system

Abstract

We provide the system. [Solution] A voice acquisition means that captures the user's voice and saves or processes it as digital data, A speech recognition means that converts acquired audio data into text data, A question analysis tool that understands the intent of a question recognized as text data and generates an appropriate answer, A response generation means that creates an appropriate answer based on question analysis, A speech generation means that synthesizes the generated text-formatted response as audio data, A means of providing a response to the user by providing the generated audio data, A system that includes this.

Description

Technical Field

[0005]

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern times, there are various risks associated with children accessing the Internet. For example, there is a risk of accessing harmful content or being deceived. There is also a current situation where busy parents cannot immediately answer their children's questions or provide English education. Furthermore, supervision to ensure the safety of children is also important. In order to solve these problems and provide a safer and more effective educational environment, this invention provides a system that provides educational support while ensuring safety without connecting to the Internet.

Means for Solving the Problems

[0005] This invention provides an educational support watch that includes means for acquiring audio, voice recognition, question analysis, answer generation, audio generation, and answer presentation. This system does not connect to the internet, eliminating the risk of children accessing harmful content. It also includes means for acquiring audio of lesson content, converting the audio to text, summarizing, saving, and presenting it, thereby facilitating post-lesson review. Furthermore, it includes means for acquiring English conversation audio, converting it to text, generating answers using an English conversation model, converting them back to audio, and playing them back, thereby supporting 24-hour English learning. This makes it possible to provide a safe and effective educational environment.

[0006] "Voice acquisition means" refers to a function for capturing the user's voice and saving or processing it as digital data.

[0007] "Speech recognition means" refers to a function that converts acquired speech data into text data.

[0008] "Question analysis means" refers to a function that understands the intent of a question recognized as text data and performs analysis to generate an appropriate answer.

[0009] "Answer generation means" refers to a function that generates appropriate answers based on question analysis.

[0010] "Voice generation means" refers to a function that synthesizes the generated text-formatted response into audio data.

[0011] "Answer presentation means" refers to a function for providing the generated audio data to the user.

[0012] An "educational support watch" refers to an electronic device worn by the user that implements the aforementioned functions for educational support.

[0013] "Method for acquiring audio of lecture content" refers to a function that captures audio spoken during lectures and saves it as digital data.

[0014] The "means for converting voice into text" refers to the function of converting the acquired voice data of the lecture content into text data.

[0015] The "means for summarizing text" refers to the function of analyzing text data, extracting important parts, and summarizing them.

[0016] The "means for saving the summary result" refers to the function of saving the summarized text data in a storage medium.

[0017] The "means for presenting the summary result" refers to the function of displaying the saved summary result to the user.

[0018] The "means for acquiring English conversation voice" refers to the function of capturing the English voice uttered by the user.

[0019] The "English conversation model" refers to an algorithm or dataset for generating appropriate English responses to the user's questions.

[0020] The "means for converting the generated answer into voice" refers to the function of converting the English answer in text form into voice data.

[0021] The "means for playing voice" refers to the function of playing the generated voice data with a speaker or the like and letting the user hear it.

Brief Description of Drawings

[0022] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0023] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0026] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0027] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0028] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0030] [First Embodiment]

[0031] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0032] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0035] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0038] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0042] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0043] This invention relates to a wristwatch-type device for supporting education while ensuring the safety of children. Because this device operates locally without an internet connection, it offers high security. Furthermore, it provides educational support and monitoring functions using a system equipped with voice recognition and natural language processing.

[0044] The present invention has the following main features:

[0045] 1. Question answering function

[0046] 2. Function to summarize lesson content

[0047] 3. English conversation support function

[0048] 4. Monitoring function

[0049] Question answering function

[0050] Terminal:

[0051] It is equipped with a microphone for users to input questions by voice. When a user asks a question, this voice data is converted into text data by the speech recognition engine in the device.

[0052] server:

[0053] The system receives text data and analyzes the questions. The analysis process uses natural language processing techniques to understand the intent of the questions and generates appropriate answers from a local database.

[0054] server:

[0055] The generated response is converted into audio data using a speech synthesis engine. The converted audio data is then sent to the terminal.

[0056] Terminal:

[0057] The system plays back the received audio data to the user and provides answers to their questions.

[0058] Specific user examples:

[0059] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[0060] Lesson content summary function

[0061] Terminal:

[0062] Audio from the classroom is captured via a micro-fan and saved as audio data.

[0063] Terminal:

[0064] At regular intervals, audio data is converted to text using a speech recognition engine as a batch process. This text data is then sent to the server.

[0065] server:

[0066] The received text data is summarized using natural language processing techniques. The summarization results are stored in a local database.

[0067] Terminal:

[0068] In response to user requests, the system retrieves a summary of the lesson and displays it on the screen.

[0069] Specific user examples:

[0070] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson is..."

[0071] English conversation support function

[0072] Terminal:

[0073] It is equipped with a microphone for users to ask questions or initiate conversations in English. It acquires voice data, and a speech recognition engine converts it to text.

[0074] server:

[0075] The system receives text data and generates appropriate responses using an English conversation model. The generated responses are then converted into speech data using a speech synthesis engine.

[0076] Terminal:

[0077] The generated audio data is played for the user.

[0078] Specific user examples:

[0079] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[0080] Monitoring function

[0081] Terminal:

[0082] It is equipped with a GPS system and an accelerometer, which continuously acquire the user's location and movement information. This data is transmitted to a server.

[0083] server:

[0084] The system analyzes the received data and detects anomalies. If an anomaly is detected, it sends an alert to the parent system or emergency services.

[0085] Terminal:

[0086] It receives alerts and notifies the user via voice or vibration as needed.

[0087] Specific user examples:

[0088] If a user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is [location]."

[0089] These features allow children to learn safely, and parents can watch over them with peace of mind. Furthermore, they provide an effective learning environment through English conversation support and lesson review.

[0090] The following describes the processing flow.

[0091] Question answering function

[0092] Step 1:

[0093] Terminal: The user speaks a voice question into the microphone.

[0094] Step 2:

[0095] Terminal: The speech recognition engine converts the user's voice into text data.

[0096] Step 3:

[0097] Terminal: Sends the converted text data to the server.

[0098] Step 4:

[0099] Server: Receives text data and analyzes the question using a natural language processing algorithm.

[0100] Step 5:

[0101] Server: Based on the analysis results, it searches for and generates the appropriate answer from the local database.

[0102] Step 6:

[0103] Server: Converts the generated response text into speech data using a speech synthesis engine.

[0104] Step 7:

[0105] Server: Sends audio data to the terminal.

[0106] Step 8:

[0107] Terminal: Plays the received audio data through the speaker and presents the answer to the user.

[0108] Lesson content summary function

[0109] Step 1:

[0110] Terminal: This device uses a micro-fan to capture audio during class and saves it as audio data.

[0111] Step 2:

[0112] Terminal: Uses a speech recognition engine to convert speech data into text at regular intervals.

[0113] Step 3:

[0114] Terminal: Sends the converted text data to the server.

[0115] Step 4:

[0116] Server: Receives text data and summarizes it using a natural language processing algorithm.

[0117] Step 5:

[0118] Server: Saves the summary results to the local database.

[0119] Step 6:

[0120] Terminal: Receives requests from users and requests summary results from the server.

[0121] Step 7:

[0122] Server: Sends the summary results to the terminal.

[0123] Step 8:

[0124] Terminal: Displays the received summary results on the screen.

[0125] English conversation support function

[0126] Step 1:

[0127] Device: Speak your voice question in English into the microphone.

[0128] Step 2:

[0129] Terminal: The speech recognition engine converts the speech data into text data.

[0130] Step 3:

[0131] Terminal: Sends the converted text data to the server.

[0132] Step 4:

[0133] Server: Receives text data and generates appropriate English responses based on an English conversation model.

[0134] Step 5:

[0135] Server: Converts the generated English text into speech data using a speech synthesis engine.

[0136] Step 6:

[0137] Server: Sends audio data to the terminal.

[0138] Step 7:

[0139] Terminal: Plays the received audio data through the speaker and provides the user with a response.

[0140] Monitoring function

[0141] Step 1:

[0142] Terminal: Periodically acquires the user's location information and movement using a GPS system and accelerometer.

[0143] Step 2:

[0144] Terminal: Sends acquired data to the server.

[0145] Step 3:

[0146] Server: Monitors location information and operational data in real time and detects anomalies.

[0147] Step 4:

[0148] Server: If an anomaly is detected, it sends an alert to the parent or emergency services.

[0149] Step 5:

[0150] Device: Notifies the user via voice alerts or vibration.

[0151] This should clarify the specific processing steps for each function.

[0152] (Example 1)

[0153] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0154] Traditional educational support devices required an internet connection, posing a high security risk. Furthermore, few devices integrated child safety monitoring and educational support functions, making it difficult for parents to constantly monitor their children with peace of mind. Additionally, educational support features such as lesson summaries and English conversation support were often insufficient.

[0155] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0156] In this invention, the server includes a voice acquisition means, a voice recognition means, a question analysis means, an answer generation means, a voice generation means, an answer presentation means, a location information acquisition means, a motion detection means, an anomaly detection means, and an anomaly notification means. This allows for secure operation even without an internet connection and enables the integrated provision of educational support and child monitoring functions. Furthermore, by providing functions such as lesson content summarization and English conversation support, the effectiveness of children's learning can be enhanced.

[0157] "Voice acquisition means" refers to a device or function for picking up the voice emitted by the user.

[0158] "Speech recognition means" refers to a technology or device that converts acquired speech into text data.

[0159] "Question analysis means" refers to a technology or device for analyzing user intent from transcribed audio data.

[0160] "Answer generation means" refers to a technology or device that generates an appropriate answer based on the analysis results.

[0161] "Speech generation means" refers to a technology or device that converts generated text-based responses into speech.

[0162] "Answer presentation means" refers to a technology or device for presenting generated audio to the user.

[0163] "Location information acquisition means" refers to technology or devices that use GPS or similar systems to acquire the user's current location.

[0164] "Motion detection means" refers to a technology or device that detects the user's movement status using an acceleration sensor or the like.

[0165] An "anomaly detection means" is a technology or device for detecting abnormal situations from user activity information or location information.

[0166] An "anomaly notification means" is a technology or device for sending an alert when an anomaly is detected.

[0167] "Means for acquiring audio of lecture content" refers to a device or function for capturing audio during lectures.

[0168] "Means for converting audio of lecture content into text" refers to technology or equipment that converts acquired audio into text data.

[0169] "Means of summarizing text" refers to techniques or devices that analyze text data, extract only the important information, and summarize it.

[0170] "Means for saving the summary results" refers to a technology or device for saving the summarized text data to a storage device.

[0171] "Means for presenting summary results" refers to a technology or device for presenting summarized text data to a user.

[0172] "Means for acquiring English conversation audio" refers to a device or function for picking up English audio spoken by a user.

[0173] "Means for converting spoken English into text" refers to a technology or device that converts acquired spoken English into text data.

[0174] "Means for generating responses using an English conversation model" refers to a technology or device that analyzes English text data to generate appropriate English responses.

[0175] "Means for converting generated responses into speech" refers to a technology or device that converts generated text-based responses into speech.

[0176] "Means for playing back audio" refers to the technology or device used to play back generated audio data for the user.

[0177] This invention relates to an arm-worn device for supporting education while ensuring the safety of children. Because this device operates locally without an internet connection, it offers high security. Furthermore, it utilizes speech recognition and natural language processing technologies to provide multiple educational support and monitoring functions.

[0178] Question answering function

[0179] The device of this invention is equipped with a microphone for the user to input questions by voice. The voice data is converted into text data by a speech recognition engine (e.g., Google Speech-to-Text API) within the device. This text data is sent to a server.

[0180] The server receives text data and analyzes the question using natural language processing techniques (e.g., the NLTK library). Based on the analysis, it generates an appropriate answer from a local database. The generated answer is converted into audio data by a speech synthesis engine (e.g., Google Text-to-Speech API) and sent to the terminal. The terminal plays this audio data and provides the answer to the user.

[0181] Specific example:

[0182] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[0183] Lesson content summary function

[0184] Audio during lessons is captured through the device's microphone and saved as audio data. This audio data is converted into text data at regular intervals by a speech recognition engine (e.g., Google Speech-to-Text API). The converted text data is then sent to a server.

[0185] The server summarizes the received text data using natural language processing techniques (e.g., the NLTK library). The summary is stored in a local database. The user can retrieve this summary upon request and display it on their terminal screen.

[0186] Specific example:

[0187] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson is..."

[0188] English conversation support function

[0189] The user uses the device's microphone to ask a question or initiate a conversation in English. The audio data is converted into text data by a speech recognition engine (e.g., Google Speech-to-Text API). This text data is then sent to the server.

[0190] The server receives text data and generates an appropriate response using an English conversation model (e.g., OpenAI's GPT-3). The generated response is converted into audio data by a speech synthesis engine (e.g., Google Text-to-Speech API) and sent to the terminal. The terminal plays this audio data and provides the response to the user.

[0191] Specific example:

[0192] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[0193] Monitoring function

[0194] The device of the present invention is equipped with a GPS system and an accelerometer, and continuously acquires the user's location information and movements. This data is transmitted to a server.

[0195] The server analyzes the received data and, if it detects an anomaly, sends an alert to the parent or emergency services. The device receives the alert and notifies the user via sound or vibration.

[0196] Specific example:

[0197] If the user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is [location]."

[0198] Examples of prompt statements

[0199] The following are specific examples of prompt statements to be input to a generative AI model:

[0200] 1. Prompts for the question function:

[0201] User: "Why is the sky blue?"

[0202] Prompt: "Explain why the sky appears blue."

[0203] 2. Prompts for English conversation support function:

[0204] User: "How is the weather today?"

[0205] Prompt: "Please describe the weather today."

[0206] Based on these concrete examples, children can learn safely, and parents can watch over them with peace of mind. Furthermore, it becomes possible to provide an effective learning environment through English conversation support and review of lessons.

[0207] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0208] Question answering function

[0209] Step 1:

[0210] The user enters the question by voice.

[0211] Input: User's voice question

[0212] Specific action: The user asks the question, "Why is the sky blue?"

[0213] Step 2:

[0214] The device acquires voice data and converts it into text data using a speech recognition engine.

[0215] Input: User's voice data

[0216] Data processing: Convert speech to text using a speech recognition engine (e.g., Google Speech-to-Text API).

[0217] Output: Text data ("Why is the sky blue?")

[0218] Step 3:

[0219] The terminal sends the converted text data to the server.

[0220] Input: Text data

[0221] Data processing: Uses HTTP requests

[0222] Output: Sending text data to the server

[0223] Step 4:

[0224] The server receives text data and analyzes the question using natural language processing technology.

[0225] Input: Text data ("Why is the sky blue?")

[0226] Data processing: Analyze the intent of the question using natural language processing techniques (e.g., NLTK library).

[0227] Output: Analysis results

[0228] Step 5:

[0229] The server generates the appropriate answer from the local database based on the analysis results.

[0230] Input: Analysis results

[0231] Data calculation: Generate answers from a local database.

[0232] Output: Answer text ("The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes.")

[0233] Step 6:

[0234] The server converts the generated response text into audio data and sends it to the terminal.

[0235] Input: Answer text

[0236] Data processing: Convert to audio data using a speech synthesis engine (e.g., Google Text-to-Speech API).

[0237] Output: Generation of audio data and transmission to the terminal.

[0238] Step 7:

[0239] The device plays the audio data it received.

[0240] Input: Audio data

[0241] Specific action: Play the answer through the device's speaker (telling the user, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes.")

[0242] Lesson content summary function

[0243] Step 1:

[0244] The device acquires audio during the lesson.

[0245] Input: Audio from class

[0246] Specific action: Audio during class is captured using a microphone.

[0247] Step 2:

[0248] The device saves the audio data.

[0249] Input: Acquired audio data

[0250] Data processing: Saving to internal storage

[0251] Output: Saved audio data

[0252] Step 3:

[0253] The device converts voice data into text data using a speech recognition engine at regular intervals.

[0254] Input: Saved audio data

[0255] Data processing: Convert speech to text using a speech recognition engine (e.g., Google Speech-to-Text API).

[0256] Output: Text data

[0257] Step 4:

[0258] The terminal sends text data to the server.

[0259] Input: Text data

[0260] Data processing: Uses HTTP requests

[0261] Output: Sending text data to the server

[0262] Step 5:

[0263] Summarize the text data received by the server.

[0264] Input: Text data

[0265] Data processing: Summarizing using natural language processing techniques (e.g., NLTK library).

[0266] Output: Summary text data

[0267] Step 6:

[0268] The server sends the summary results to the terminal.

[0269] Input: Summary text data

[0270] Data processing: Uses HTTP requests

[0271] Output: Send summary text to terminal

[0272] Step 7:

[0273] The terminal displays the summary result in response to a request from the user.

[0274] Input: User request, summary text data

[0275] Specific operation: When the user requests "Tell me the content of today's class", the summary result is displayed on the screen

[0276] Output: Summary result displayed on the screen ("What was learned in today's class is 〇〇〇〇")

[0277] English conversation support function

[0278] Step 1:

[0279] The user speaks in English.

[0280] Input: User's English voice

[0281] Specific operation: The user speaks in English, "How is the weather today?"

[0282] Step 2:

[0283] The terminal acquires the voice data and converts it into text data using a speech recognition engine.

[0284] Input: User's English voice data

[0285] [[ID=四十八]]Data calculation: Convert the voice to text using a speech recognition engine (e.g., Google Speech-to-Text API) <九〇三〇〇〇〇>

[0286] Output: Text data ("How is the weather today?")

[0287] Step 3:

[0288] The terminal sends the text data to the server.

[0289] Input: Text data

[0290] Data processing: Uses HTTP requests

[0291] Output: Sending text data to the server

[0292] Step 4:

[0293] The server generates the response using an English conversation model.

[0294] Input: Text data ("How is the weather today?")

[0295] Data processing: Generate answers using an English conversation model (e.g., OpenAI's GPT-3).

[0296] Output: Answer text ("The weather is sunny today.")

[0297] Step 5:

[0298] The server converts the generated response text into audio data and sends it to the terminal.

[0299] Input: Answer text

[0300] Data processing: Convert to audio data using a speech synthesis engine (e.g., Google Text-to-Speech API).

[0301] Output: Generation of audio data and transmission to the terminal.

[0302] Step 6:

[0303] The device plays the audio data it received.

[0304] Input: Audio data

[0305] Specific action: Play the answer through the terminal's speaker (tell the user "The weather is sunny today.")

[0306] Monitoring function

[0307] Step 1:

[0308] The terminal obtains the user's location information and movement.

[0309] Input: Data from GPS and accelerometer

[0310] Specific action: The terminal obtains location information and movement using the GPS system and accelerometer

[0311] Step 2:

[0312] The terminal sends the location information and movement to the server.

[0313] Input: Location information data, movement data

[0314] Data processing: Use HTTP request <​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​

[0323] Input: Anomaly detection result

[0324] Specific actions: Send an alert via SMS or email to parents or emergency services (e.g., "Your child has fallen. Their current location is [location].")

[0325] Step 5:

[0326] The device receives an alert and sends a notification.

[0327] Input: Alert data

[0328] Specific action: The device notifies the user of an abnormality via sound or vibration.

[0329] Through the processing steps described above, this system can effectively provide educational support while ensuring the safety of children.

[0330] (Application Example 1)

[0331] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0332] There is a need for effective methods to maximize educational effectiveness while ensuring children's safety. Furthermore, a system is required that operates locally without an internet connection, ensuring data security while efficiently handling question answering, lesson summaries, English conversation support, and monitoring. Conventional technologies lack sufficient child anomaly detection and emergency notification features, making it difficult to ensure children's safety.

[0333] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0334] In this invention, the server includes voice acquisition means, voice recognition means, question analysis means, answer generation means, voice generation means, answer presentation means, location information acquisition means, emergency notification means, and anomaly detection means. This enables question answering based on voice input, acquisition and summarization of educational content, and provision of English conversation support, as well as the ability to detect location information and abnormal operation and provide notifications in emergencies.

[0335] "Sound acquisition means" refers to a device or software that has the function of collecting ambient sounds or voices.

[0336] "Speech recognition means" refers to a device or software that analyzes acquired speech data and converts it into text data.

[0337] "Question analysis means" refers to a device or software that analyzes text data generated by speech recognition means and understands the intent of a question.

[0338] "Answer generation means" refers to a device or software that generates an appropriate answer based on the intent of a question understood by the question analysis means.

[0339] "Voice generation means" refers to a device or software that converts generated text-based responses into voice data.

[0340] "Answer presentation means" refers to a device or software that has the function of playing back audio data generated by the audio generation means to the user.

[0341] "Location information acquisition means" refers to a device or software that acquires a user's location information using GPS or other location measurement technologies.

[0342] An "emergency notification system" is a device or software that has the function of sending notifications to pre-configured contacts when an abnormality or emergency is detected.

[0343] An "anomaly detection means" is a device or software that has the function of detecting abnormal operation or condition based on data from sensors.

[0344] "Means for acquiring audio recordings of class content" refers to a device or software that has the function of recording lectures and discussions during class.

[0345] "Means of converting speech to text" refers to a device or software that analyzes speech data acquired during a lesson and converts it into text data.

[0346] "Means for summarizing text" refers to a device or software that analyzes converted text data, extracts important points, and summarizes them.

[0347] "Means for saving the summary results" refers to a device or software that has the function of saving the summarized information to a storage device.

[0348] "Means for presenting summary results" refers to a device or software that has the function of displaying or playing back saved summary results to the user.

[0349] "Means for acquiring English conversation audio" refers to a device or software that has the function of collecting audio data of English spoken by a user.

[0350] "Means for generating responses using an English conversation model" refers to a device or software that analyzes acquired English text data and uses an English conversation model to generate appropriate responses.

[0351] "Means for converting generated responses into speech" refers to a device or software that converts text-based responses generated by an English conversation model into speech data.

[0352] "Means for playing audio" refers to a device or software that has the function of playing converted audio data back to the user.

[0353] This invention provides a system to support education while ensuring the safety of children. This system has the following main functions:

[0354] 1. Voice acquisition and recognition function

[0355] Terminal:

[0356] It is equipped with a microphone for users to input questions by voice. When a user asks a question, this voice data is converted into text data by the speech recognition engine in the device. A standard microphone is used as the hardware.

[0357] server:

[0358] The system receives text data and analyzes the questions. The analysis process uses natural language processing techniques to understand the intent of the questions and generates appropriate answers from a local database. The software used includes natural language processing libraries (e.g., NLTK, spaCy).

[0359] Specific example:

[0360] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[0361] Example of a prompt:

[0362] Generate an answer to the question, "Why is the sky blue?"

[0363] 2. Function to summarize educational content

[0364] Terminal:

[0365] Audio during class is captured via a microphone and saved as audio data. A standard microphone is used as the hardware.

[0366] server:

[0367] The received text data is summarized using natural language processing techniques. The summarization results are stored in a local database. The software used includes a text summarization library (e.g., Sumy).

[0368] Specific example:

[0369] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson was about the main events of World War II."

[0370] Example of a prompt:

[0371] Generate a summary of what a user would say if they requested, "Tell me what was covered in today's lesson."

[0372] 3. English conversation support function

[0373] Terminal:

[0374] It is equipped with a microphone for users to ask questions or initiate conversations in English. It acquires voice data, and a speech recognition engine converts it to text.

[0375] server:

[0376] The system receives text data and generates appropriate responses using an English conversation model. The generated responses are then converted into speech data using a speech synthesis engine. The software used includes an English conversation model (e.g., GPT-3).

[0377] Specific example:

[0378] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[0379] Example of a prompt:

[0380] Please generate an answer for the user's question, "How is the weather today?"

[0381] 4. Child monitoring function

[0382] Terminal:

[0383] It is equipped with a GPS system and an accelerometer to continuously acquire the user's location and movement. This data is transmitted to a server. The hardware used includes a GPS module (e.g., u-blox) and an accelerometer (e.g., MEMS sensor).

[0384] server:

[0385] The system analyzes the received data and detects anomalies. If an anomaly is detected, it sends an alert to the parent system or emergency services. The software used includes an anomaly detection algorithm.

[0386] Specific example:

[0387] If the user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is 35.6895, 139.6917."

[0388] Example of a prompt:

[0389] Please explain how to send an emergency notification to parents if a child falls.

[0390] This allows children to learn safely, and parents can watch over them with peace of mind. Furthermore, it makes it possible to provide an effective learning environment through English conversation support and lesson review.

[0391] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0392] Step 1:

[0393] Voice data acquisition and recognition

[0394] The device acquires the user's voice data using the microphone. The acquired voice data is sent to the speech recognition engine within the device. The speech recognition engine converts the voice data into text data and generates the resulting text data.

[0395] Input: Audio data

[0396] Output: Text data

[0397] Step 2:

[0398] Question analysis

[0399] The server receives text data from the terminal. The server uses natural language processing technology to analyze the content of the question in the text data and performs data calculations to understand its intent. This determines the intent of the question and generates a result.

[0400] Input: Text data

[0401] Output: Question intent data

[0402] Step 3:

[0403] Generating an answer

[0404] The server generates appropriate answers based on the intent data of the questions, using a local database or a generative AI model. The generated answer text is then used as the primary output.

[0405] Input: Question intent data

[0406] Output: Response text data

[0407] Step 4:

[0408] Convert to speech

[0409] The server sends the response text data to the speech synthesis engine, which converts it into speech data. The converted speech data is then sent to the terminal.

[0410] Input: Answer text data

[0411] Output: Audio data

[0412] Step 5:

[0413] Providing an answer

[0414] The terminal plays the audio data received from the server. Based on the played audio data, the user is presented with an answer.

[0415] Input: Audio data

[0416] Output: Audio output

[0417] Step 6:

[0418] Acquisition of course content

[0419] The device uses its microphone to capture audio during class. The captured audio data is temporarily stored on the device and periodically sent to the speech recognition engine. The speech recognition engine converts the audio data into text data and generates the resulting text data.

[0420] Input: Audio data from class

[0421] Output: Text data from the lecture

[0422] Step 7:

[0423] Text summary

[0424] The server receives the lecture text data and uses natural language processing technology to summarize the text. The summary results are stored in a local database.

[0425] Input: Text data from the class

[0426] Output: Summary data

[0427] Step 8:

[0428] English conversation support

[0429] The device uses its microphone to capture the user's English conversation voice. The captured audio data is sent to the device's speech recognition engine, which converts the audio data into text data.

[0430] Input: English conversation audio data

[0431] Output: English conversation text data

[0432] Step 9:

[0433] English conversation response generation

[0434] The server receives English conversation text data and generates appropriate responses using an English conversation model. The generated response text data is used as the primary output.

[0435] Input: English conversation text data

[0436] Output: English conversation response text data

[0437] Step 10:

[0438] English conversation converted to audio

[0439] The server sends the English conversation response text data to the speech synthesis engine, which converts it into audio data. The converted audio data is then sent to the terminal.

[0440] Input: English conversation response text data

[0441] Output: English conversation audio data

[0442] Step 11:

[0443] English conversation response

[0444] The device plays English conversation audio data received from the server. Based on the played audio data, the user is presented with an English conversation response.

[0445] Input: English conversation audio data

[0446] Output: Audio output

[0447] Step 12:

[0448] Location information acquisition

[0449] The device uses the GPS system to obtain the user's location information. The obtained location data is periodically sent to the server.

[0450] Input: Location data

[0451] Output: Location data

[0452] Step 13:

[0453] Anomaly detection

[0454] The server analyzes location and acceleration data received from the terminal to detect anomalies. If an anomaly is detected, it generates anomaly data and activates an emergency notification system.

[0455] Input: Location data, acceleration data

[0456] Output: Abnormal data

[0457] Step 14:

[0458] emergency notification

[0459] The server sends notifications to parents or emergency services based on abnormal data. The notification includes the user's current location information.

[0460] Input: Abnormal data

[0461] Output: Notification message

[0462] The above outlines the specific process for implementing the present invention. This system can support effective education while ensuring the safety of children.

[0463] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0464] This invention relates to a wristwatch-type device for supporting education while ensuring the safety of children. Because this device operates locally without an internet connection, it offers high security. Furthermore, it provides educational support and monitoring functions using a system equipped with voice recognition and natural language processing. In addition, by incorporating an emotion engine that recognizes the user's emotions, it enables more personalized educational support.

[0465] The present invention has the following main features:

[0466] 1. Question answering function

[0467] 2. Function to summarize lesson content

[0468] 3. English conversation support function

[0469] 4. Monitoring function

[0470] 5. Emotion recognition function

[0471] Question answering function

[0472] Terminal:

[0473] It is equipped with a microphone for users to input questions by voice. When a user asks a question, this voice data is converted into text data by the speech recognition engine in the device.

[0474] server:

[0475] The system receives text data and analyzes the questions. The analysis process uses natural language processing techniques to understand the intent of the questions and generates appropriate answers from a local database.

[0476] server:

[0477] The generated response is converted into audio data using a speech synthesis engine. The converted audio data is then sent to the terminal.

[0478] Terminal:

[0479] The system plays back the received audio data to the user and provides answers to their questions.

[0480] Specific user examples:

[0481] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[0482] Lesson content summary function

[0483] Terminal:

[0484] Audio from the classroom is captured via a micro-fan and saved as audio data.

[0485] Terminal:

[0486] The audio data is converted to text by a speech recognition engine at regular intervals. This text data is then sent to the server.

[0487] server:

[0488] The received text data is summarized using natural language processing techniques. The summarization results are stored in a local database.

[0489] Terminal:

[0490] In response to user requests, the system retrieves a summary of the lesson and displays it on the screen.

[0491] Specific user examples:

[0492] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson is..."

[0493] English conversation support function

[0494] Terminal:

[0495] It is equipped with a microphone for users to ask questions or initiate conversations in English. It acquires voice data, and a speech recognition engine converts it to text.

[0496] server:

[0497] The system receives text data and generates appropriate English responses using an English conversation model. The generated responses are then converted into speech data using a speech synthesis engine.

[0498] Terminal:

[0499] The generated audio data is played for the user.

[0500] Specific user examples:

[0501] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[0502] Monitoring function

[0503] Terminal:

[0504] It is equipped with a GPS system and an accelerometer, which continuously acquire the user's location and movement information. This data is transmitted to a server.

[0505] server:

[0506] The system analyzes the received data and detects anomalies. If an anomaly is detected, it sends an alert to the parent system or emergency services.

[0507] Terminal:

[0508] It receives alerts and notifies the user via voice or vibration as needed.

[0509] Specific user examples:

[0510] If a user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is [location]."

[0511] Emotion recognition function

[0512] Terminal:

[0513] The system captures the user's voice and facial expressions through the camera and microphone and sends them to the emotion engine.

[0514] server:

[0515] The emotion engine analyzes the user's emotions and adjusts the response and educational content based on the results.

[0516] Terminal:

[0517] Provide users with appropriate responses and educational content.

[0518] Specific user examples:

[0519] When a user says, "I'm tired today," the device uses an emotion engine to analyze the user's tone of voice and facial expression, and the server generates a response such as, "You seem tired today, so you should take a rest," which is then played back as audio.

[0520] These features allow children to learn safely and parents to supervise with peace of mind. They also provide an effective learning environment through English conversation support and lesson review. Emotion recognition enables more personalized educational support, boosting children's motivation to learn.

[0521] The following describes the processing flow.

[0522] Question answering function

[0523] Step 1:

[0524] Terminal: The user speaks a voice question into the microphone.

[0525] Step 2:

[0526] Terminal: The speech recognition engine converts the user's voice into text data.

[0527] Step 3:

[0528] Terminal: Sends the converted text data to the server.

[0529] Step 4:

[0530] Server: Receives text data and analyzes the question using a natural language processing algorithm.

[0531] Step 5:

[0532] Server: Based on the analysis results, it searches and generates the appropriate answer from the local database.

[0533] Step 6:

[0534] Server: Converts the generated response text into speech data using a speech synthesis engine.

[0535] Step 7:

[0536] Server: Sends audio data to the terminal.

[0537] Step 8:

[0538] Terminal: Plays the received audio data through the speaker and provides the user with a response.

[0539] Lesson content summary function

[0540] Step 1:

[0541] Terminal: This device uses a micro-fan to capture audio during class and saves it as audio data.

[0542] Step 2:

[0543] Terminal: The acquired audio data is converted into text data at regular intervals using a speech recognition engine.

[0544] Step 3:

[0545] Terminal: Sends the converted text data to the server.

[0546] Step 4:

[0547] Server: Receives text data and summarizes it using a natural language processing algorithm.

[0548] Step 5:

[0549] Server: Saves the summary results to the local database.

[0550] Step 6:

[0551] Terminal: Receives a request from the user and requests the summary result from the server.

[0552] Step 7:

[0553] Server: Sends the summary results to the terminal.

[0554] Step 8:

[0555] Terminal: Displays the received summary results on the screen.

[0556] English conversation support function

[0557] Step 1:

[0558] Device: The user speaks a voice question in English into the microphone.

[0559] Step 2:

[0560] Terminal: The speech recognition engine converts the speech data into text data.

[0561] Step 3:

[0562] Terminal: Sends the converted text data to the server.

[0563] Step 4:

[0564] Server: Receives text data and generates appropriate English responses using an English conversation model.

[0565] Step 5:

[0566] Server: Converts the generated English text into speech data using a speech synthesis engine.

[0567] Step 6:

[0568] Server: Sends audio data to the terminal.

[0569] Step 7:

[0570] Terminal: Plays the received audio data through the speaker and provides the user with a response.

[0571] Monitoring function

[0572] Step 1:

[0573] Terminal: Periodically acquires the user's location information and movement using a GPS system and accelerometer.

[0574] Step 2:

[0575] Terminal: Sends acquired data to the server in real time.

[0576] Step 3:

[0577] Server: Monitors location information and operational data to detect anomalies.

[0578] Step 4:

[0579] Server: If an anomaly is detected, it sends an alert to the parent or emergency services.

[0580] Step 5:

[0581] Device: Notifies the user via sound or vibration.

[0582] Emotion recognition function

[0583] Step 1:

[0584] Device: When a user performs voice input, the device captures their voice tone and facial expressions through the camera and microphone.

[0585] Step 2:

[0586] Terminal: Sends the acquired data to the emotion engine.

[0587] Step 3:

[0588] Server: The emotion engine analyzes the data and identifies the user's emotional state.

[0589] Step 4:

[0590] Server: Generates and selects appropriate responses and educational content based on emotional state.

[0591] Step 5:

[0592] Server: Sends the generated response and educational content to the terminal as audio or text data.

[0593] Step 6:

[0594] Terminal: Presents received data to the user.

[0595] Specific user examples:

[0596] When a user says, "I'm tired today," the device sends the voice data to the emotion engine, and the server, based on the analysis results, generates a response such as, "You seem tired today, so you should rest a little," and plays it back as audio.

[0597] This should clarify the specific processing steps for each function.

[0598] (Example 2)

[0599] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0600] Traditional educational support watches offer features such as answering user questions, summarizing lesson content, and supporting English conversation, but they have been insufficient in terms of safety and emotion recognition. Therefore, there is a need for features that can track a child's location, detect abnormal situations, and analyze the user's emotions to provide appropriate responses. Furthermore, the challenge lies in integrating these functions into a single device to achieve safer and more personalized educational support.

[0601] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0602] In this invention, the server includes speech recognition means, question analysis means, answer generation means, speech generation means, anomaly detection means, and emotion analysis means. This enables the immediate provision of answers to user questions, summaries of lesson content, English conversation support, safety management through location information monitoring, and appropriate responses according to the user's emotional state.

[0603] "Voice acquisition means" refers to devices such as microphones used to collect the user's voice.

[0604] "Speech recognition means" refers to speech recognition engines or software used to convert collected speech data into text data.

[0605] "Question analysis means" refers to natural language processing technology and analysis algorithms for analyzing user questions transcribed into text by speech recognition means and understanding their intent.

[0606] "Answer generation means" refers to an engine or algorithm for generating appropriate answers based on the content analyzed by the question analysis means.

[0607] "Voice generation means" refers to a speech synthesis engine or software for converting the generated text-formatted response into speech data.

[0608] "Answer presentation means" refers to a speaker or output device for playing back audio data created by the audio generation means to the user.

[0609] "Location information acquisition means" refers to GPS systems and other location detection technologies used to obtain the user's current location.

[0610] An "anomaly detection means" is an algorithm and system for detecting abnormal behavior or conditions by analyzing data collected by a location information acquisition means.

[0611] "Emotion analysis means" refers to software and hardware that analyzes a user's voice and facial expressions to identify their emotional state.

[0612] "Means of notifying the user" refer to notification systems that inform the user of detected anomalies or analyzed information, such as vibration, voice notifications, or display notifications.

[0613] System Overview

[0614] This invention relates to a wristwatch-type device for supporting education while ensuring safety. This device is voice-operable by the user and features voice recognition, question analysis, answer generation, speech synthesis, lesson content summarization, English conversation support, monitoring, and emotion recognition. Because this device provides numerous educational support functions in an integrated manner, it creates a safe learning environment for the user.

[0615] Hardware and software to be used

[0616] This system uses the following main hardware and software:

[0617] Microphone: Collects the user's voice.

[0618] Speaker: Plays the generated audio data.

[0619] Camera: Collects user facial expressions (for emotion recognition function).

[0620] GPS system: Acquires location information.

[0621] Accelerometer: Detects user movements.

[0622] Speech recognition engine: Converts speech data into text data. Example: Google Speech-to-Text.

[0623] Text-to-speech engine: Converts text data into speech data. Example: Google Text-to-Speech.

[0624] Natural language processing models: Analyze the intent of a question and generate appropriate answers. Example: OpenAI's GPT model.

[0625] Emotion analysis engine: Analyzes the user's emotions from their voice and facial expressions.

[0626] Database: Stores data such as generated summary results.

[0627] Program processing

[0628] Question answering function

[0629] When a user inputs a question by voice, the device collects this audio and converts it into text data using a speech recognition engine. The text data is sent to a server, which uses a natural language processing model to analyze the question, generates an appropriate answer, and converts it back into audio data using a speech synthesis engine. This audio data is then sent to the device and played back to the user.

[0630] Specific example: When a user asks, "Why is the sky blue?", the device plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[0631] Lesson content summary function

[0632] Audio during lessons is collected by microphones and converted to text at regular intervals using a speech recognition engine. This text data is processed on a server and summarized using a natural language processing model. The summarization results are stored in a database and displayed on the user's terminal screen upon request.

[0633] Specific example: If a user requests "Tell me what we learned in today's lesson" after class, the device will display a summary of what we learned in today's lesson.

[0634] English conversation support function

[0635] When a user initiates a question or conversation in English, audio data is acquired and converted to text by a speech recognition engine. This text data is processed on a server, where an English conversation model is used to generate an appropriate English response, which is then converted back into audio data using a speech synthesis engine and played back to the user.

[0636] Specific example: When a user asks "How is the weather today?", the system plays the answer "The weather is sunny today." in audio.

[0637] Monitoring function

[0638] The system uses a GPS system and an accelerometer to acquire the user's location and movements, and transmits this information to a server. The server analyzes the data to detect anomalies. If an anomaly is detected, an alert is sent to the parent or emergency services, and the user is notified via the device.

[0639] Specific example: If a user falls, the device detects the abnormality and sends an SMS message to the parent saying, "Your child has fallen. Their current location is XX."

[0640] Emotion recognition function

[0641] The system captures the user's voice and facial expressions through a camera and microphone and sends them to an emotion analysis engine. The server analyzes the emotions and generates appropriate responses or educational content based on the results. The generated responses are converted into audio data using a speech synthesis engine and played back to the user.

[0642] Specific example: When a user says, "I'm tired today," a voice response is played saying, "You seem tired today, so you should rest a little."

[0643] Example of a prompt

[0644] Question answer function: A user asks, "Why is the sky blue?"

[0645] Lesson summary function: After the lesson, the user can request, "Please explain what we covered in today's lesson."

[0646] English conversation support function: The user asks "How is the weather today?".

[0647] Emotion recognition function: The user says, "I'm tired today."

[0648] These features allow children to learn safely and parents to watch over them with peace of mind. Furthermore, they provide an effective learning environment through English conversation support and lesson review, and enable more personalized educational support through emotion recognition.

[0649] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0650] Question answering function

[0651] Step 1:

[0652] The user inputs the question by voice. The device collects the user's voice using a microphone and saves it as audio data.

[0653] Input: User's voice: "Why is the sky blue?"

[0654] Output: Audio data

[0655] Step 2:

[0656] The collected audio data is converted into text data using a speech recognition engine (e.g., Google Speech-to-Text).

[0657] Input: Audio data

[0658] Output: Text data "Why is the sky blue?"

[0659] Specific operation: The speech recognition engine analyzes the audio data and generates the corresponding text.

[0660] Step 3:

[0661] Send the converted text data to the server.

[0662] Input: Text data "Why is the sky blue?"

[0663] Output: Text data sent to the server

[0664] Specific action: The terminal sends text data to the server.

[0665] Step 4:

[0666] The server receives text data and uses a natural language processing model (e.g., OpenAI's GPT model) to analyze the intent of the question.

[0667] Input: Text data "Why is the sky blue?"

[0668] Output: Analysis results (the intent of the question)

[0669] Specific operation: The natural language processing model analyzes the context of the question and extracts the information necessary to generate an appropriate answer.

[0670] Step 5:

[0671] Based on the question analysis results, the response generation means generates an appropriate response.

[0672] Input: Analysis results

[0673] Output: Answer: "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[0674] Specific operation: The response generation mechanism accesses the database and extracts or generates appropriate responses.

[0675] Step 6:

[0676] The generated responses are converted into audio data using a speech synthesis engine (e.g., Google Text-to-Speech).

[0677] Input: Response (text data)

[0678] Output: Response (audio data)

[0679] Specific operation: The speech synthesis engine converts text data into speech data.

[0680] Step 7:

[0681] Send audio data to the device.

[0682] Input: Response (audio data)

[0683] Output: Audio data sent to the terminal

[0684] Specific operation: The server sends the generated audio data to the terminal.

[0685] Step 8:

[0686] The device plays back the received audio data and provides the user with an answer.

[0687] Input: Response (audio data)

[0688] Output: Audio played to the user

[0689] Specific action: The device's speaker plays audio data, and the user listens to the response.

[0690] Lesson content summary function

[0691] Step 1:

[0692] Audio during lessons is collected using a microphone and saved as audio data.

[0693] Input: Audio from class

[0694] Output: Audio data

[0695] Specific operation: The microphone continuously collects audio during class and stores it on the device.

[0696] Step 2:

[0697] The collected audio data is converted into text data at regular intervals using a speech recognition engine.

[0698] Input: Audio data

[0699] Output: Text data

[0700] Specific operation: The speech recognition engine analyzes the collected audio data and generates corresponding text.

[0701] Step 3:

[0702] Send the converted text data to the server.

[0703] Input: Text data

[0704] Output: Text data sent to the server

[0705] Specific action: The terminal sends text data to the server.

[0706] Step 4:

[0707] The server receives text data and summarizes it using a natural language processing model.

[0708] Input: Text data

[0709] Output: Summary results

[0710] Specific operation: A natural language processing model analyzes text data, extracts key points, and generates a summary.

[0711] Step 5:

[0712] Save the summary results to the database.

[0713] Input: Summary result

[0714] Output: Saved summary data

[0715] Specific action: Record the summary results in a database and save them so they can be retrieved later.

[0716] Step 6:

[0717] The system retrieves a summary result in response to the user's request and displays it on the device screen.

[0718] Input: User Request

[0719] Output: Summary results displayed on the screen

[0720] Specific operation: Retrieve the summary results from the database and display them on the terminal screen.

[0721] English conversation support function

[0722] Step 1:

[0723] The microphone collects audio from the user to initiate questions or conversations in English.

[0724] Input: English audio

[0725] Output: Audio data

[0726] Specific action: The user asks a question in English, such as "How is the weather today?".

[0727] Step 2:

[0728] The collected audio data is converted into text data using a speech recognition engine.

[0729] Input: Audio data

[0730] Output: Text data

[0731] Specific operation: The speech recognition engine analyzes the audio data and generates the corresponding text.

[0732] Step 3:

[0733] Send the converted text data to the server.

[0734] Input: Text data

[0735] Output: Text data sent to the server

[0736] Specific action: The terminal sends text data to the server.

[0737] Step 4:

[0738] The server receives text data and generates a response using an English conversation model.

[0739] Input: Text data

[0740] Output: Response (text data)

[0741] Specific operation: The English conversation model analyzes text data and generates appropriate English responses.

[0742] Step 5:

[0743] The generated responses are converted into speech data using a speech synthesis engine.

[0744] Input: Response (text data)

[0745] Output: Response (audio data)

[0746] Specific operation: The speech synthesis engine converts text into speech data.

[0747] Step 6:

[0748] Send audio data to the device.

[0749] Input: Response (audio data)

[0750] Output: Audio data sent to the terminal

[0751] Specific operation: The server sends the generated audio data to the terminal.

[0752] Step 7:

[0753] The device plays back the received audio data and provides the user with an answer.

[0754] Input: Response (audio data)

[0755] Output: Audio played to the user

[0756] Specific action: The device's speaker plays audio data, and the user listens to the response.

[0757] Monitoring function

[0758] Step 1:

[0759] The system uses a GPS system and an accelerometer to acquire the user's location and movements.

[0760] Input: None (Sensor acquisition itself)

[0761] Output: Location information and motion data

[0762] Specific actions: Continuously acquire location information and motion data to prepare for anomaly detection.

[0763] Step 2:

[0764] The acquired data is sent to the server.

[0765] Input: Location information and motion data

[0766] Output: Data sent to the server

[0767] Specific action: The device sends the data it has acquired to the server.

[0768] Step 3:

[0769] The server analyzes the received data and detects abnormal conditions.

[0770] Input: Location information and motion data

[0771] Output: Analysis result (normal / abnormal)

[0772] Specific operation: The data analysis engine analyzes the data and detects abnormal conditions.

[0773] Step 4:

[0774] If an anomaly is detected, an alert will be sent to parents or emergency services.

[0775] Input: Analysis results (abnormal)

[0776] Output: Alert notification

[0777] Specific operation: The server sends alerts via SMS or email.

[0778] Step 5:

[0779] The device notifies the user of an anomaly.

[0780] Input: None (based on alert notifications from the server)

[0781] Output: Voice notifications, vibration, and display.

[0782] Specific actions: The device will notify the user of an anomaly using sound, vibration, or a display message.

[0783] Emotion recognition function

[0784] Step 1:

[0785] The system captures the user's voice and facial expressions through a camera and microphone.

[0786] Input: Voice and facial expressions

[0787] Output: Audio data and facial expression data

[0788] Specific operation: The microphone collects audio, and the camera collects facial expressions.

[0789] Step 2:

[0790] The acquired data is sent to the emotion analysis engine.

[0791] Input: Voice data and facial expression data

[0792] Output: Emotion analysis results

[0793] Specific operation: The emotion analysis engine analyzes voice and facial expressions to identify emotions.

[0794] Step 3:

[0795] The server generates an appropriate response based on the sentiment analysis results.

[0796] Input: Sentiment analysis results

[0797] Output: Response (text data)

[0798] Specific operation: The response generation engine generates an appropriate response based on the analyzed emotions.

[0799] Step 4:

[0800] The generated response is converted into speech data using a speech synthesis engine.

[0801] Input: Response (text data)

[0802] Output: Response (audio data)

[0803] Specific operation: The speech synthesis engine converts text data into speech data.

[0804] Step 5:

[0805] The response audio data is sent to the terminal.

[0806] Input: Response (voice data)

[0807] Output: Audio data sent to the terminal

[0808] Specific operation: The server sends the generated audio data to the terminal.

[0809] Step 6:

[0810] The device plays back the received audio data and provides a response to the user.

[0811] Input: Response (voice data)

[0812] Output: Audio played to the user

[0813] Specific action: The device's speaker plays audio data, and the user listens to the response.

[0814] The above is a detailed explanation of the program's processing flow.

[0815] (Application Example 2)

[0816] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0817] Conventional educational support devices lack features to ensure children's safety, such as monitoring functions, and personalized educational support that responds to their emotions. Furthermore, they lack the ability to track children's location in real time and detect anomalies, making rapid response in emergencies difficult. This invention aims to solve these problems and provide safer, more secure, and more effective educational support.

[0818] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0819] In this invention, the server includes voice acquisition means, voice recognition means, question analysis means, answer generation means, voice generation means, answer presentation means, location information acquisition means, location data transmission means, anomaly detection means, alert transmission means, emotion acquisition means, emotion analysis means, and system response generation means. This makes it possible to provide individually optimized educational support in real time while ensuring user safety.

[0820] A "voice acquisition means" is a device that captures the user's voice and stores it as digital data.

[0821] "Speech recognition means" refers to technology that converts acquired speech data into text data.

[0822] A "question analysis tool" is an algorithm that analyzes text data obtained by a speech recognition tool to understand the intent and content of a question.

[0823] "Answer generation means" refers to a technology that generates appropriate answers based on the results obtained from question analysis means.

[0824] "Speech generation means" refers to technology that converts generated text data into speech data.

[0825] A "response presentation means" is a device that plays back the generated audio data and presents it to the user.

[0826] "Location information acquisition means" refers to technologies that use GPS or other technologies to acquire the user's current location.

[0827] "Location data transmission means" refers to technology that transmits acquired location information to a server or other terminal.

[0828] An "anomaly detection method" is a technology that detects user behavior or states that are different from the norm.

[0829] An "alert transmission method" is a technology for quickly notifying of anomalies detected by anomaly detection methods.

[0830] "Emotion acquisition methods" refer to technologies that acquire emotions from the user's voice and facial expressions.

[0831] "Emotional analysis means" refers to technology that analyzes acquired emotional data to determine the user's emotional state.

[0832] "System response generation means" refers to a technology that generates appropriate responses or messages based on data obtained from emotion analysis means.

[0833] Explanation of program generation and processing

[0834] This invention is a system that provides individually optimized educational support in real time while ensuring user safety, and utilizes the following main hardware and software.

[0835] Hardware to use

[0836] smartphone

[0837] Smart Glasses

[0838] GPS sensor

[0839] Accelerometer

[0840] microphone

[0841] camera

[0842] Software to use

[0843] Google Speech API (speech recognition engine)

[0844] TensorFlow (natural language processing engine)

[0845] Microsoft Azure Emotion API (emotion recognition engine)

[0846] Google Maps API (location tracking)

[0847] Twilio API (Alert Notifications)

[0848] Detailed processing of the system

[0849] Location tracking

[0850] The server acquires location information from GPS sensors installed in smartphones and smart glasses. This acquired location information is transmitted to the server in real time via a location data transmission device. The server analyzes the received location data and detects abnormal movements using an anomaly detection device. If an anomaly is detected, an alert transmission device is used to notify parents or administrators via SMS or push notification.

[0851] Emotion monitoring

[0852] The server uses the microphone and camera of a smartphone or smart glasses to acquire the user's voice and facial expressions using emotion acquisition means. This data is analyzed by emotion analysis means to determine the user's emotional state. Next, system response generation means is used to generate an appropriate response based on the emotional state. This response is converted into audio data by voice generation means and presented to the user.

[0853] Learning support

[0854] The server acquires the user's voice question using a voice acquisition means and converts it into text data using a speech recognition means. The text data is analyzed by a question analysis means, and based on the results, an answer is generated by an answer generation means. The generated answer is converted into speech by a speech generation means and presented to the user through an answer presentation means.

[0855] Specific examples and prompt statements

[0856] Examples of emotion recognition

[0857] When a user says, "I'm tired today," the device uses an emotion engine to analyze the user's tone of voice and facial expression. Based on the analysis, the server generates a response such as, "You seem tired today, so you should take a rest," and plays it back as audio.

[0858] Example of a prompt:

[0859] Voice input: "Something unpleasant happened at school today."

[0860] Facial expression input: Image data (child's facial expression)

[0861] Examples of learning support

[0862] User question: "Why is the sky blue?"

[0863] Response: "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[0864] Example of a prompt:

[0865] "Why is the sky blue?"

[0866] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0867] Step 1:

[0868] The device uses a microphone to acquire the user's voice. The acquired voice data is stored as a digital signal. In this case, the voice data is the input.

[0869] Step 2:

[0870] The device converts the acquired audio data, obtained using speech recognition (Google Speech API), into text data. This conversion results in the output of the text data.

[0871] Step 3:

[0872] The terminal sends the converted text data to the server. Here, the text data is passed to the server as input.

[0873] Step 4:

[0874] The server analyzes the text data using a question analysis tool (TensorFlow) to understand the intent of the question. The intent of the question is output as the analysis result.

[0875] Step 5:

[0876] The server generates an appropriate answer using an answer generation mechanism based on the question analysis results. The generated answer is output as text data.

[0877] Step 6:

[0878] The server converts the generated text response into audio data using a speech generation tool (Google Speech API). This conversion results in the output of the audio data.

[0879] Step 7:

[0880] The server sends the generated audio data to the terminal. Here, the audio data is passed to the terminal as input.

[0881] Step 8:

[0882] The terminal presents the user with the answer to the question by playing the audio data through the answer presentation means. Here, the audio is presented to the user.

[0883] Step 9:

[0884] The device uses a GPS sensor to obtain the user's location information. This information is stored as location data. In this case, location data is the input.

[0885] Step 10:

[0886] The device sends location data to the server. The location data is passed to the server as input.

[0887] Step 11:

[0888] The server analyzes location data using anomaly detection mechanisms to detect unusual movement. If an anomaly is detected, anomaly data is output.

[0889] Step 12:

[0890] When abnormal data is detected, the server sends a notification to the parent or administrator using the alert sending mechanism (Twilio API). This notification triggers an alert.

[0891] Step 13:

[0892] The device acquires user voice and facial expression data using emotion acquisition methods. This emotion data is the input.

[0893] Step 14:

[0894] The server analyzes emotional data using an emotion analysis tool (Microsoft Azure Emotion API) to determine the user's emotional state. The emotion analysis results are then output.

[0895] Step 15:

[0896] The server generates an appropriate response using a system response generation tool (TensorFlow) based on the sentiment analysis results. This response is output as text data.

[0897] Step 16:

[0898] The server converts the generated text response into audio data using a speech generation tool (Google Speech API). This conversion results in the output of the audio data.

[0899] Step 17:

[0900] The server sends the generated audio data to the terminal. Here, the audio data is passed to the terminal as input.

[0901] Step 18:

[0902] The device plays audio data to the user, providing a response that corresponds to their emotional state. Here, the audio is presented to the user.

[0903] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0904] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0905] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0906] [Second Embodiment]

[0907] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0908] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0909] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0910] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0911] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0912] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0913] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0914] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0915] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0916] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0917] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0918] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0919] This invention relates to a wristwatch-type device for supporting education while ensuring the safety of children. Because this device operates locally without an internet connection, it offers high security. Furthermore, it provides educational support and monitoring functions using a system equipped with voice recognition and natural language processing.

[0920] The present invention has the following main features:

[0921] 1. Question answering function

[0922] 2. Function to summarize lesson content

[0923] 3. English conversation support function

[0924] 4. Monitoring function

[0925] Question answering function

[0926] Terminal:

[0927] It is equipped with a microphone for users to input questions by voice. When a user asks a question, this voice data is converted into text data by the speech recognition engine in the device.

[0928] server:

[0929] The system receives text data and analyzes the questions. The analysis process uses natural language processing techniques to understand the intent of the questions and generates appropriate answers from a local database.

[0930] server:

[0931] The generated response is converted into audio data using a speech synthesis engine. The converted audio data is then sent to the terminal.

[0932] Terminal:

[0933] The system plays back the received audio data to the user and provides answers to their questions.

[0934] Specific user examples:

[0935] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[0936] Lesson content summary function

[0937] Terminal:

[0938] Audio from the classroom is captured via a micro-fan and saved as audio data.

[0939] Terminal:

[0940] At regular intervals, audio data is converted to text using a speech recognition engine as a batch process. This text data is then sent to the server.

[0941] server:

[0942] The received text data is summarized using natural language processing techniques. The summarization results are stored in a local database.

[0943] Terminal:

[0944] In response to user requests, the system retrieves a summary of the lesson and displays it on the screen.

[0945] Specific user examples:

[0946] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson is..."

[0947] English conversation support function

[0948] Terminal:

[0949] It is equipped with a microphone for users to ask questions or initiate conversations in English. It acquires voice data, and a speech recognition engine converts it to text.

[0950] server:

[0951] The system receives text data and generates appropriate responses using an English conversation model. The generated responses are then converted into speech data using a speech synthesis engine.

[0952] Terminal:

[0953] The generated audio data is played for the user.

[0954] Specific user examples:

[0955] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[0956] Monitoring function

[0957] Terminal:

[0958] It is equipped with a GPS system and an accelerometer, which continuously acquire the user's location and movement information. This data is transmitted to a server.

[0959] server:

[0960] The system analyzes the received data and detects anomalies. If an anomaly is detected, it sends an alert to the parent system or emergency services.

[0961] Terminal:

[0962] It receives alerts and notifies the user via voice or vibration as needed.

[0963] Specific user examples:

[0964] If a user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is [location]."

[0965] These features allow children to learn safely, and parents can watch over them with peace of mind. Furthermore, they provide an effective learning environment through English conversation support and lesson review.

[0966] The following describes the processing flow.

[0967] Question answering function

[0968] Step 1:

[0969] Terminal: The user speaks a voice question into the microphone.

[0970] Step 2:

[0971] Terminal: The speech recognition engine converts the user's voice into text data.

[0972] Step 3:

[0973] Terminal: Sends the converted text data to the server.

[0974] Step 4:

[0975] Server: Receives text data and analyzes the question using a natural language processing algorithm.

[0976] Step 5:

[0977] Server: Based on the analysis results, it searches for and generates the appropriate answer from the local database.

[0978] Step 6:

[0979] Server: Converts the generated response text into speech data using a speech synthesis engine.

[0980] Step 7:

[0981] Server: Sends audio data to the terminal.

[0982] Step 8:

[0983] Terminal: Plays the received audio data through the speaker and presents the answer to the user.

[0984] Lesson content summary function

[0985] Step 1:

[0986] Terminal: This device uses a micro-fan to capture audio during class and saves it as audio data.

[0987] Step 2:

[0988] Terminal: Uses a speech recognition engine to convert speech data into text at regular intervals.

[0989] Step 3:

[0990] Terminal: Sends the converted text data to the server.

[0991] Step 4:

[0992] Server: Receives text data and summarizes it using a natural language processing algorithm.

[0993] Step 5:

[0994] Server: Saves the summary results to the local database.

[0995] Step 6:

[0996] Terminal: Receives requests from users and requests summary results from the server.

[0997] Step 7:

[0998] Server: Sends the summary results to the terminal.

[0999] Step 8:

[1000] Terminal: Displays the received summary results on the screen.

[1001] English conversation support function

[1002] Step 1:

[1003] Device: Speak your voice question in English into the microphone.

[1004] Step 2:

[1005] Terminal: The speech recognition engine converts the speech data into text data.

[1006] Step 3:

[1007] Terminal: Sends the converted text data to the server.

[1008] Step 4:

[1009] Server: Receives text data and generates appropriate English responses based on an English conversation model.

[1010] Step 5:

[1011] Server: Converts the generated English text into speech data using a speech synthesis engine.

[1012] Step 6:

[1013] Server: Sends audio data to the terminal.

[1014] Step 7:

[1015] Terminal: Plays the received audio data through the speaker and provides the user with a response.

[1016] Monitoring function

[1017] Step 1:

[1018] Terminal: Periodically acquires the user's location information and movement using a GPS system and accelerometer.

[1019] Step 2:

[1020] Terminal: Sends acquired data to the server.

[1021] Step 3:

[1022] Server: Monitors location information and operational data in real time and detects anomalies.

[1023] Step 4:

[1024] Server: If an anomaly is detected, it sends an alert to the parent or emergency services.

[1025] Step 5:

[1026] Device: Notifies the user via voice alerts or vibration.

[1027] This should clarify the specific processing steps for each function.

[1028] (Example 1)

[1029] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1030] Traditional educational support devices required an internet connection, posing a high security risk. Furthermore, few devices integrated child safety monitoring and educational support functions, making it difficult for parents to constantly monitor their children with peace of mind. Additionally, educational support features such as lesson summaries and English conversation support were often insufficient.

[1031] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1032] In this invention, the server includes a voice acquisition means, a voice recognition means, a question analysis means, an answer generation means, a voice generation means, an answer presentation means, a location information acquisition means, a motion detection means, an anomaly detection means, and an anomaly notification means. This allows for secure operation even without an internet connection and enables the integrated provision of educational support and child monitoring functions. Furthermore, by providing functions such as lesson content summarization and English conversation support, the effectiveness of children's learning can be enhanced.

[1033] "Voice acquisition means" refers to a device or function for picking up the voice emitted by the user.

[1034] "Speech recognition means" refers to a technology or device that converts acquired speech into text data.

[1035] "Question analysis means" refers to a technology or device for analyzing user intent from transcribed audio data.

[1036] "Answer generation means" refers to a technology or device that generates an appropriate answer based on the analysis results.

[1037] "Speech generation means" refers to a technology or device that converts generated text-based responses into speech.

[1038] "Answer presentation means" refers to a technology or device for presenting generated audio to the user.

[1039] "Location information acquisition means" refers to technology or devices that use GPS or similar systems to acquire the user's current location.

[1040] "Motion detection means" refers to a technology or device that detects the user's movement status using an acceleration sensor or the like.

[1041] An "anomaly detection means" is a technology or device for detecting abnormal situations from user activity information or location information.

[1042] An "anomaly notification means" is a technology or device for sending an alert when an anomaly is detected.

[1043] "Means for acquiring audio of lecture content" refers to a device or function for capturing audio during lectures.

[1044] "Means for converting audio of lecture content into text" refers to technology or equipment that converts acquired audio into text data.

[1045] "Means of summarizing text" refers to techniques or devices that analyze text data, extract only the important information, and summarize it.

[1046] "Means for saving the summary results" refers to a technology or device for saving the summarized text data to a storage device.

[1047] "Means for presenting summary results" refers to a technology or device for presenting summarized text data to a user.

[1048] "Means for acquiring English conversation audio" refers to a device or function for picking up English audio spoken by a user.

[1049] "Means for converting spoken English into text" refers to a technology or device that converts acquired spoken English into text data.

[1050] "Means for generating responses using an English conversation model" refers to a technology or device that analyzes English text data to generate appropriate English responses.

[1051] "Means for converting generated responses into speech" refers to a technology or device that converts generated text-based responses into speech.

[1052] "Means for playing back audio" refers to the technology or device used to play back generated audio data for the user.

[1053] This invention relates to an arm-worn device for supporting education while ensuring the safety of children. Because this device operates locally without an internet connection, it offers high security. Furthermore, it utilizes speech recognition and natural language processing technologies to provide multiple educational support and monitoring functions.

[1054] Question answering function

[1055] The device of this invention is equipped with a microphone for the user to input questions by voice. The voice data is converted into text data by a speech recognition engine (e.g., Google Speech-to-Text API) within the device. This text data is sent to a server.

[1056] The server receives text data and analyzes the question using natural language processing techniques (e.g., the NLTK library). Based on the analysis, it generates an appropriate answer from a local database. The generated answer is converted into audio data by a speech synthesis engine (e.g., Google Text-to-Speech API) and sent to the terminal. The terminal plays this audio data and provides the answer to the user.

[1057] Specific example:

[1058] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[1059] Lesson content summary function

[1060] Audio during lessons is captured through the device's microphone and saved as audio data. This audio data is converted into text data at regular intervals by a speech recognition engine (e.g., Google Speech-to-Text API). The converted text data is then sent to a server.

[1061] The server summarizes the received text data using natural language processing techniques (e.g., the NLTK library). The summary is stored in a local database. The user can retrieve this summary upon request and display it on their terminal screen.

[1062] Specific example:

[1063] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson is..."

[1064] English conversation support function

[1065] The user uses the device's microphone to ask a question or initiate a conversation in English. The audio data is converted into text data by a speech recognition engine (e.g., Google Speech-to-Text API). This text data is then sent to the server.

[1066] The server receives text data and generates an appropriate response using an English conversation model (e.g., OpenAI's GPT-3). The generated response is converted into audio data by a speech synthesis engine (e.g., Google Text-to-Speech API) and sent to the terminal. The terminal plays this audio data and provides the response to the user.

[1067] Specific example:

[1068] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[1069] Monitoring function

[1070] The device of the present invention is equipped with a GPS system and an accelerometer, and continuously acquires the user's location information and movements. This data is transmitted to a server.

[1071] The server analyzes the received data and, if it detects an anomaly, sends an alert to the parent or emergency services. The device receives the alert and notifies the user via sound or vibration.

[1072] Specific example:

[1073] If the user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is [location]."

[1074] Examples of prompt statements

[1075] The following are specific examples of prompt statements to be input to a generative AI model:

[1076] 1. Prompts for the question function:

[1077] User: "Why is the sky blue?"

[1078] Prompt: "Explain why the sky appears blue."

[1079] 2. Prompts for English conversation support function:

[1080] User: "How is the weather today?"

[1081] Prompt: "Please describe the weather today."

[1082] Based on these concrete examples, children can learn safely, and parents can watch over them with peace of mind. Furthermore, it becomes possible to provide an effective learning environment through English conversation support and review of lessons.

[1083] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1084] Question answering function

[1085] Step 1:

[1086] The user enters the question by voice.

[1087] Input: User's voice question

[1088] Specific action: The user asks the question, "Why is the sky blue?"

[1089] Step 2:

[1090] The device acquires voice data and converts it into text data using a speech recognition engine.

[1091] Input: User's voice data

[1092] Data processing: Convert speech to text using a speech recognition engine (e.g., Google Speech-to-Text API).

[1093] Output: Text data ("Why is the sky blue?")

[1094] Step 3:

[1095] The terminal sends the converted text data to the server.

[1096] Input: Text data

[1097] Data processing: Uses HTTP requests

[1098] Output: Sending text data to the server

[1099] Step 4:

[1100] The server receives text data and analyzes the question using natural language processing technology.

[1101] Input: Text data ("Why is the sky blue?")

[1102] Data processing: Analyze the intent of the question using natural language processing techniques (e.g., NLTK library).

[1103] Output: Analysis results

[1104] Step 5:

[1105] The server generates the appropriate answer from the local database based on the analysis results.

[1106] Input: Analysis results

[1107] Data calculation: Generate answers from a local database.

[1108] Output: Answer text ("The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes.")

[1109] Step 6:

[1110] The server converts the generated response text into audio data and sends it to the terminal.

[1111] Input: Answer text

[1112] Data processing: Convert to audio data using a speech synthesis engine (e.g., Google Text-to-Speech API).

[1113] Output: Generation of audio data and transmission to the terminal.

[1114] Step 7:

[1115] The device plays the audio data it received.

[1116] Input: Audio data

[1117] Specific action: Play the answer through the device's speaker (telling the user, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes.")

[1118] Lesson content summary function

[1119] Step 1:

[1120] The device acquires audio during the lesson.

[1121] Input: Audio from class

[1122] Specific action: Audio during class is captured using a microphone.

[1123] Step 2:

[1124] The device saves the audio data.

[1125] Input: Acquired audio data

[1126] Data processing: Saving to internal storage

[1127] Output: Saved audio data

[1128] Step 3:

[1129] The device converts voice data into text data using a speech recognition engine at regular intervals.

[1130] Input: Saved audio data

[1131] Data processing: Convert speech to text using a speech recognition engine (e.g., Google Speech-to-Text API).

[1132] Output: Text data

[1133] Step 4:

[1134] The terminal sends text data to the server.

[1135] Input: Text data

[1136] Data processing: Uses HTTP requests

[1137] Output: Sending text data to the server

[1138] Step 5:

[1139] Summarize the text data received by the server.

[1140] Input: Text data

[1141] Data processing: Summarizing using natural language processing techniques (e.g., NLTK library).

[1142] Output: Summary text data

[1143] Step 6:

[1144] The server sends the summary results to the terminal.

[1145] Input: Summary text data

[1146] Data processing: Uses HTTP requests

[1147] Output: Send summary text to terminal

[1148] Step 7:

[1149] The device displays a summary result in response to a user request.

[1150] Input: User request, summary text data

[1151] Specific operation: When the user requests "Tell me what we covered in today's lesson," a summary is displayed on the screen.

[1152] Output: Summary displayed on the screen ("What we learned in today's lesson is ______")

[1153] English conversation support function

[1154] Step 1:

[1155] The user speaks in English.

[1156] Input: User's English voice

[1157] Specific action: The user speaks in English, saying "How is the weather today?"

[1158] Step 2:

[1159] The device acquires voice data and converts it into text data using a speech recognition engine.

[1160] Input: User's English voice data

[1161] Data processing: Convert speech to text using a speech recognition engine (e.g., Google Speech-to-Text API).

[1162] Output: Text data ("How is the weather today?")

[1163] Step 3:

[1164] The terminal sends text data to the server.

[1165] Input: Text data

[1166] Data processing: Uses HTTP requests

[1167] Output: Sending text data to the server

[1168] Step 4:

[1169] The server generates the response using an English conversation model.

[1170] Input: Text data ("How is the weather today?")

[1171] Data processing: Generate answers using an English conversation model (e.g., OpenAI's GPT-3).

[1172] Output: Answer text ("The weather is sunny today.")

[1173] Step 5:

[1174] The server converts the generated response text into audio data and sends it to the terminal.

[1175] Input: Answer text

[1176] Data processing: Convert to audio data using a speech synthesis engine (e.g., Google Text-to-Speech API).

[1177] Output: Generation of audio data and transmission to the terminal.

[1178] Step 6:

[1179] The device plays the audio data it received.

[1180] Input: Audio data

[1181] Specific action: Play the answer through the device's speaker (telling the user "The weather is sunny today.")

[1182] Monitoring function

[1183] Step 1:

[1184] The device acquires the user's location information and movements.

[1185] Input: GPS and accelerometer data

[1186] Specific operation: The device uses the GPS system and accelerometer to obtain location information and movement.

[1187] Step 2:

[1188] The device sends location information and movement data to the server.

[1189] Input: Location data, motion data

[1190] Data processing: Uses HTTP requests

[1191] Output: Sending data to the server

[1192] Step 3:

[1193] The server analyzes the data and detects anomalies.

[1194] Input: Location data, motion data

[1195] Data processing: Anomaly detection using algorithms

[1196] Output: Anomaly detection result

[1197] Step 4:

[1198] If the server detects an anomaly, it will send an alert.

[1199] Input: Anomaly detection result

[1200] Specific actions: Send an alert via SMS or email to parents or emergency services (e.g., "Your child has fallen. Their current location is [location].")

[1201] Step 5:

[1202] The device receives an alert and sends a notification.

[1203] Input: Alert data

[1204] Specific action: The device notifies the user of an abnormality via sound or vibration.

[1205] Through the processing steps described above, this system can effectively provide educational support while ensuring the safety of children.

[1206] (Application Example 1)

[1207] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1208] There is a need for effective methods to maximize educational effectiveness while ensuring children's safety. Furthermore, a system is required that operates locally without an internet connection, ensuring data security while efficiently handling question answering, lesson summaries, English conversation support, and monitoring. Conventional technologies lack sufficient child anomaly detection and emergency notification features, making it difficult to ensure children's safety.

[1209] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1210] In this invention, the server includes voice acquisition means, voice recognition means, question analysis means, answer generation means, voice generation means, answer presentation means, location information acquisition means, emergency notification means, and anomaly detection means. This enables question answering based on voice input, acquisition and summarization of educational content, and provision of English conversation support, as well as the ability to detect location information and abnormal operation and provide notifications in emergencies.

[1211] "Sound acquisition means" refers to a device or software that has the function of collecting ambient sounds or voices.

[1212] "Speech recognition means" refers to a device or software that analyzes acquired speech data and converts it into text data.

[1213] "Question analysis means" refers to a device or software that analyzes text data generated by speech recognition means and understands the intent of a question.

[1214] "Answer generation means" refers to a device or software that generates an appropriate answer based on the intent of a question understood by the question analysis means.

[1215] "Voice generation means" refers to a device or software that converts generated text-based responses into voice data.

[1216] "Answer presentation means" refers to a device or software that has the function of playing back audio data generated by the audio generation means to the user.

[1217] "Location information acquisition means" refers to a device or software that acquires a user's location information using GPS or other location measurement technologies.

[1218] An "emergency notification system" is a device or software that has the function of sending notifications to pre-configured contacts when an abnormality or emergency is detected.

[1219] An "anomaly detection means" is a device or software that has the function of detecting abnormal operation or condition based on data from sensors.

[1220] "Means for acquiring audio recordings of class content" refers to a device or software that has the function of recording lectures and discussions during class.

[1221] "Means of converting speech to text" refers to a device or software that analyzes speech data acquired during a lesson and converts it into text data.

[1222] "Means for summarizing text" refers to a device or software that analyzes converted text data, extracts important points, and summarizes them.

[1223] "Means for saving the summary results" refers to a device or software that has the function of saving the summarized information to a storage device.

[1224] "Means for presenting summary results" refers to a device or software that has the function of displaying or playing back saved summary results to the user.

[1225] "Means for acquiring English conversation audio" refers to a device or software that has the function of collecting audio data of English spoken by a user.

[1226] "Means for generating responses using an English conversation model" refers to a device or software that analyzes acquired English text data and uses an English conversation model to generate appropriate responses.

[1227] "Means for converting generated responses into speech" refers to a device or software that converts text-based responses generated by an English conversation model into speech data.

[1228] "Means for playing audio" refers to a device or software that has the function of playing converted audio data back to the user.

[1229] This invention provides a system to support education while ensuring the safety of children. This system has the following main functions:

[1230] 1. Voice acquisition and recognition function

[1231] Terminal:

[1232] It is equipped with a microphone for users to input questions by voice. When a user asks a question, this voice data is converted into text data by the speech recognition engine in the device. A standard microphone is used as the hardware.

[1233] server:

[1234] The system receives text data and analyzes the questions. The analysis process uses natural language processing techniques to understand the intent of the questions and generates appropriate answers from a local database. The software used includes natural language processing libraries (e.g., NLTK, spaCy).

[1235] Specific example:

[1236] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[1237] Example of a prompt:

[1238] Generate an answer to the question, "Why is the sky blue?"

[1239] 2. Function to summarize educational content

[1240] Terminal:

[1241] Audio during class is captured via a microphone and saved as audio data. A standard microphone is used as the hardware.

[1242] server:

[1243] The received text data is summarized using natural language processing techniques. The summarization results are stored in a local database. The software used includes a text summarization library (e.g., Sumy).

[1244] Specific example:

[1245] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson was about the main events of World War II."

[1246] Example of a prompt:

[1247] Generate a summary of what a user would say if they requested, "Tell me what was covered in today's lesson."

[1248] 3. English conversation support function

[1249] Terminal:

[1250] It is equipped with a microphone for users to ask questions or initiate conversations in English. It acquires voice data, and a speech recognition engine converts it to text.

[1251] server:

[1252] The system receives text data and generates appropriate responses using an English conversation model. The generated responses are then converted into speech data using a speech synthesis engine. The software used includes an English conversation model (e.g., GPT-3).

[1253] Specific example:

[1254] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[1255] Example of a prompt:

[1256] Please generate an answer for the user's question, "How is the weather today?"

[1257] 4. Child monitoring function

[1258] Terminal:

[1259] It is equipped with a GPS system and an accelerometer to continuously acquire the user's location and movement. This data is transmitted to a server. The hardware used includes a GPS module (e.g., u-blox) and an accelerometer (e.g., MEMS sensor).

[1260] server:

[1261] The system analyzes the received data and detects anomalies. If an anomaly is detected, it sends an alert to the parent system or emergency services. The software used includes an anomaly detection algorithm.

[1262] Specific example:

[1263] If the user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is 35.6895, 139.6917."

[1264] Example of a prompt:

[1265] Please explain how to send an emergency notification to parents if a child falls.

[1266] This allows children to learn safely, and parents can watch over them with peace of mind. Furthermore, it makes it possible to provide an effective learning environment through English conversation support and lesson review.

[1267] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1268] Step 1:

[1269] Voice data acquisition and recognition

[1270] The device acquires the user's voice data using the microphone. The acquired voice data is sent to the speech recognition engine within the device. The speech recognition engine converts the voice data into text data and generates the resulting text data.

[1271] Input: Audio data

[1272] Output: Text data

[1273] Step 2:

[1274] Question analysis

[1275] The server receives text data from the terminal. The server uses natural language processing technology to analyze the content of the question in the text data and performs data calculations to understand its intent. This determines the intent of the question and generates a result.

[1276] Input: Text data

[1277] Output: Question intent data

[1278] Step 3:

[1279] Generating an answer

[1280] The server generates appropriate answers based on the intent data of the questions, using a local database or a generative AI model. The generated answer text is then used as the primary output.

[1281] Input: Question intent data

[1282] Output: Response text data

[1283] Step 4:

[1284] Convert to speech

[1285] The server sends the response text data to the speech synthesis engine, which converts it into speech data. The converted speech data is then sent to the terminal.

[1286] Input: Answer text data

[1287] Output: Audio data

[1288] Step 5:

[1289] Providing an answer

[1290] The terminal plays the audio data received from the server. Based on the played audio data, the user is presented with an answer.

[1291] Input: Audio data

[1292] Output: Audio output

[1293] Step 6:

[1294] Acquisition of course content

[1295] The device uses its microphone to capture audio during class. The captured audio data is temporarily stored on the device and periodically sent to the speech recognition engine. The speech recognition engine converts the audio data into text data and generates the resulting text data.

[1296] Input: Audio data from class

[1297] Output: Text data from the lecture

[1298] Step 7:

[1299] Text summary

[1300] The server receives the lecture text data and uses natural language processing technology to summarize the text. The summary results are stored in a local database.

[1301] Input: Text data from the class

[1302] Output: Summary data

[1303] Step 8:

[1304] English conversation support

[1305] The device uses its microphone to capture the user's English conversation voice. The captured audio data is sent to the device's speech recognition engine, which converts the audio data into text data.

[1306] Input: English conversation audio data

[1307] Output: English conversation text data

[1308] Step 9:

[1309] English conversation response generation

[1310] The server receives English conversation text data and generates appropriate responses using an English conversation model. The generated response text data is used as the primary output.

[1311] Input: English conversation text data

[1312] Output: English conversation response text data

[1313] Step 10:

[1314] English conversation converted to audio

[1315] The server sends the English conversation response text data to the speech synthesis engine, which converts it into audio data. The converted audio data is then sent to the terminal.

[1316] Input: English conversation response text data

[1317] Output: English conversation audio data

[1318] Step 11:

[1319] English conversation response

[1320] The device plays English conversation audio data received from the server. Based on the played audio data, the user is presented with an English conversation response.

[1321] Input: English conversation audio data

[1322] Output: Audio output

[1323] Step 12:

[1324] Location information acquisition

[1325] The device uses the GPS system to obtain the user's location information. The obtained location data is periodically sent to the server.

[1326] Input: Location data

[1327] Output: Location data

[1328] Step 13:

[1329] Anomaly detection

[1330] The server analyzes location and acceleration data received from the terminal to detect anomalies. If an anomaly is detected, it generates anomaly data and activates an emergency notification system.

[1331] Input: Location data, acceleration data

[1332] Output: Abnormal data

[1333] Step 14:

[1334] emergency notification

[1335] The server sends notifications to parents or emergency services based on abnormal data. The notification includes the user's current location information.

[1336] Input: Abnormal data

[1337] Output: Notification message

[1338] The above outlines the specific process for implementing the present invention. This system can support effective education while ensuring the safety of children.

[1339] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1340] This invention relates to a wristwatch-type device for supporting education while ensuring the safety of children. Because this device operates locally without an internet connection, it offers high security. Furthermore, it provides educational support and monitoring functions using a system equipped with voice recognition and natural language processing. In addition, by incorporating an emotion engine that recognizes the user's emotions, it enables more personalized educational support.

[1341] The present invention has the following main features:

[1342] 1. Question answering function

[1343] 2. Function to summarize lesson content

[1344] 3. English conversation support function

[1345] 4. Monitoring function

[1346] 5. Emotion recognition function

[1347] Question answering function

[1348] Terminal:

[1349] It is equipped with a microphone for users to input questions by voice. When a user asks a question, this voice data is converted into text data by the speech recognition engine in the device.

[1350] server:

[1351] The system receives text data and analyzes the questions. The analysis process uses natural language processing techniques to understand the intent of the questions and generates appropriate answers from a local database.

[1352] server:

[1353] The generated response is converted into audio data using a speech synthesis engine. The converted audio data is then sent to the terminal.

[1354] Terminal:

[1355] The system plays back the received audio data to the user and provides answers to their questions.

[1356] Specific user examples:

[1357] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[1358] Lesson content summary function

[1359] Terminal:

[1360] Audio from the classroom is captured via a micro-fan and saved as audio data.

[1361] Terminal:

[1362] The audio data is converted to text by a speech recognition engine at regular intervals. This text data is then sent to the server.

[1363] server:

[1364] The received text data is summarized using natural language processing techniques. The summarization results are stored in a local database.

[1365] Terminal:

[1366] In response to user requests, the system retrieves a summary of the lesson and displays it on the screen.

[1367] Specific user examples:

[1368] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson is..."

[1369] English conversation support function

[1370] Terminal:

[1371] It is equipped with a microphone for users to ask questions or initiate conversations in English. It acquires voice data, and a speech recognition engine converts it to text.

[1372] server:

[1373] The system receives text data and generates appropriate English responses using an English conversation model. The generated responses are then converted into speech data using a speech synthesis engine.

[1374] Terminal:

[1375] The generated audio data is played for the user.

[1376] Specific user examples:

[1377] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[1378] Monitoring function

[1379] Terminal:

[1380] It is equipped with a GPS system and an accelerometer, which continuously acquire the user's location and movement information. This data is transmitted to a server.

[1381] server:

[1382] The system analyzes the received data and detects anomalies. If an anomaly is detected, it sends an alert to the parent system or emergency services.

[1383] Terminal:

[1384] It receives alerts and notifies the user via voice or vibration as needed.

[1385] Specific user examples:

[1386] If a user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is [location]."

[1387] Emotion recognition function

[1388] Terminal:

[1389] The system captures the user's voice and facial expressions through the camera and microphone and sends them to the emotion engine.

[1390] server:

[1391] The emotion engine analyzes the user's emotions and adjusts the response and educational content based on the results.

[1392] Terminal:

[1393] Provide users with appropriate responses and educational content.

[1394] Specific user examples:

[1395] When a user says, "I'm tired today," the device uses an emotion engine to analyze the user's tone of voice and facial expression, and the server generates a response such as, "You seem tired today, so you should take a rest," which is then played back as audio.

[1396] These features allow children to learn safely and parents to supervise with peace of mind. They also provide an effective learning environment through English conversation support and lesson review. Emotion recognition enables more personalized educational support, boosting children's motivation to learn.

[1397] The following describes the processing flow.

[1398] Question answering function

[1399] Step 1:

[1400] Terminal: The user speaks a voice question into the microphone.

[1401] Step 2:

[1402] Terminal: The speech recognition engine converts the user's voice into text data.

[1403] Step 3:

[1404] Terminal: Sends the converted text data to the server.

[1405] Step 4:

[1406] Server: Receives text data and analyzes the question using a natural language processing algorithm.

[1407] Step 5:

[1408] Server: Based on the analysis results, it searches and generates the appropriate answer from the local database.

[1409] Step 6:

[1410] Server: Converts the generated response text into speech data using a speech synthesis engine.

[1411] Step 7:

[1412] Server: Sends audio data to the terminal.

[1413] Step 8:

[1414] Terminal: Plays the received audio data through the speaker and provides the user with a response.

[1415] Lesson content summary function

[1416] Step 1:

[1417] Terminal: This device uses a micro-fan to capture audio during class and saves it as audio data.

[1418] Step 2:

[1419] Terminal: The acquired audio data is converted into text data at regular intervals using a speech recognition engine.

[1420] Step 3:

[1421] Terminal: Sends the converted text data to the server.

[1422] Step 4:

[1423] Server: Receives text data and summarizes it using a natural language processing algorithm.

[1424] Step 5:

[1425] Server: Saves the summary results to the local database.

[1426] Step 6:

[1427] Terminal: Receives a request from the user and requests the summary result from the server.

[1428] Step 7:

[1429] Server: Sends the summary results to the terminal.

[1430] Step 8:

[1431] Terminal: Displays the received summary results on the screen.

[1432] English conversation support function

[1433] Step 1:

[1434] Device: The user speaks a voice question in English into the microphone.

[1435] Step 2:

[1436] Terminal: The speech recognition engine converts the speech data into text data.

[1437] Step 3:

[1438] Terminal: Sends the converted text data to the server.

[1439] Step 4:

[1440] Server: Receives text data and generates appropriate English responses using an English conversation model.

[1441] Step 5:

[1442] Server: Converts the generated English text into speech data using a speech synthesis engine.

[1443] Step 6:

[1444] Server: Sends audio data to the terminal.

[1445] Step 7:

[1446] Terminal: Plays the received audio data through the speaker and provides the user with a response.

[1447] Monitoring function

[1448] Step 1:

[1449] Terminal: Periodically acquires the user's location information and movement using a GPS system and accelerometer.

[1450] Step 2:

[1451] Terminal: Sends acquired data to the server in real time.

[1452] Step 3:

[1453] Server: Monitors location information and operational data to detect anomalies.

[1454] Step 4:

[1455] Server: If an anomaly is detected, it sends an alert to the parent or emergency services.

[1456] Step 5:

[1457] Device: Notifies the user via sound or vibration.

[1458] Emotion recognition function

[1459] Step 1:

[1460] Device: When a user performs voice input, the device captures their voice tone and facial expressions through the camera and microphone.

[1461] Step 2:

[1462] Terminal: Sends the acquired data to the emotion engine.

[1463] Step 3:

[1464] Server: The emotion engine analyzes the data and identifies the user's emotional state.

[1465] Step 4:

[1466] Server: Generates and selects appropriate responses and educational content based on emotional state.

[1467] Step 5:

[1468] Server: Sends the generated response and educational content to the terminal as audio or text data.

[1469] Step 6:

[1470] Terminal: Presents received data to the user.

[1471] Specific user examples:

[1472] When a user says, "I'm tired today," the device sends the voice data to the emotion engine, and the server, based on the analysis results, generates a response such as, "You seem tired today, so you should rest a little," and plays it back as audio.

[1473] This should clarify the specific processing steps for each function.

[1474] (Example 2)

[1475] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1476] Traditional educational support watches offer features such as answering user questions, summarizing lesson content, and supporting English conversation, but they have been insufficient in terms of safety and emotion recognition. Therefore, there is a need for features that can track a child's location, detect abnormal situations, and analyze the user's emotions to provide appropriate responses. Furthermore, the challenge lies in integrating these functions into a single device to achieve safer and more personalized educational support.

[1477] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1478] In this invention, the server includes speech recognition means, question analysis means, answer generation means, speech generation means, anomaly detection means, and emotion analysis means. This enables the immediate provision of answers to user questions, summaries of lesson content, English conversation support, safety management through location information monitoring, and appropriate responses according to the user's emotional state.

[1479] "Voice acquisition means" refers to devices such as microphones used to collect the user's voice.

[1480] "Speech recognition means" refers to speech recognition engines or software used to convert collected speech data into text data.

[1481] "Question analysis means" refers to natural language processing technology and analysis algorithms for analyzing user questions transcribed into text by speech recognition means and understanding their intent.

[1482] "Answer generation means" refers to an engine or algorithm for generating appropriate answers based on the content analyzed by the question analysis means.

[1483] "Voice generation means" refers to a speech synthesis engine or software for converting the generated text-formatted response into speech data.

[1484] "Answer presentation means" refers to a speaker or output device for playing back audio data created by the audio generation means to the user.

[1485] "Location information acquisition means" refers to GPS systems and other location detection technologies used to obtain the user's current location.

[1486] An "anomaly detection means" is an algorithm and system for detecting abnormal behavior or conditions by analyzing data collected by a location information acquisition means.

[1487] "Emotion analysis means" refers to software and hardware that analyzes a user's voice and facial expressions to identify their emotional state.

[1488] "Means of notifying the user" refer to notification systems that inform the user of detected anomalies or analyzed information, such as vibration, voice notifications, or display notifications.

[1489] System Overview

[1490] This invention relates to a wristwatch-type device for supporting education while ensuring safety. This device is voice-operable by the user and features voice recognition, question analysis, answer generation, speech synthesis, lesson content summarization, English conversation support, monitoring, and emotion recognition. Because this device provides numerous educational support functions in an integrated manner, it creates a safe learning environment for the user.

[1491] Hardware and software to be used

[1492] This system uses the following main hardware and software:

[1493] Microphone: Collects the user's voice.

[1494] Speaker: Plays the generated audio data.

[1495] Camera: Collects user facial expressions (for emotion recognition function).

[1496] GPS system: Acquires location information.

[1497] Accelerometer: Detects user movements.

[1498] Speech recognition engine: Converts speech data into text data. Example: Google Speech-to-Text.

[1499] Text-to-speech engine: Converts text data into speech data. Example: Google Text-to-Speech.

[1500] Natural language processing models: Analyze the intent of a question and generate appropriate answers. Example: OpenAI's GPT model.

[1501] Emotion analysis engine: Analyzes the user's emotions from their voice and facial expressions.

[1502] Database: Stores data such as generated summary results.

[1503] Program processing

[1504] Question answering function

[1505] When a user inputs a question by voice, the device collects this audio and converts it into text data using a speech recognition engine. The text data is sent to a server, which uses a natural language processing model to analyze the question, generates an appropriate answer, and converts it back into audio data using a speech synthesis engine. This audio data is then sent to the device and played back to the user.

[1506] Specific example: When a user asks, "Why is the sky blue?", the device plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[1507] Lesson content summary function

[1508] Audio during lessons is collected by microphones and converted to text at regular intervals using a speech recognition engine. This text data is processed on a server and summarized using a natural language processing model. The summarization results are stored in a database and displayed on the user's terminal screen upon request.

[1509] Specific example: If a user requests "Tell me what we learned in today's lesson" after class, the device will display a summary of what we learned in today's lesson.

[1510] English conversation support function

[1511] When a user initiates a question or conversation in English, audio data is acquired and converted to text by a speech recognition engine. This text data is processed on a server, where an English conversation model is used to generate an appropriate English response, which is then converted back into audio data using a speech synthesis engine and played back to the user.

[1512] Specific example: When a user asks "How is the weather today?", the system plays the answer "The weather is sunny today." in audio.

[1513] Monitoring function

[1514] The system uses a GPS system and an accelerometer to acquire the user's location and movements, and transmits this information to a server. The server analyzes the data to detect anomalies. If an anomaly is detected, an alert is sent to the parent or emergency services, and the user is notified via the device.

[1515] Specific example: If a user falls, the device detects the abnormality and sends an SMS message to the parent saying, "Your child has fallen. Their current location is XX."

[1516] Emotion recognition function

[1517] The system captures the user's voice and facial expressions through a camera and microphone and sends them to an emotion analysis engine. The server analyzes the emotions and generates appropriate responses or educational content based on the results. The generated responses are converted into audio data using a speech synthesis engine and played back to the user.

[1518] Specific example: When a user says, "I'm tired today," a voice response is played saying, "You seem tired today, so you should rest a little."

[1519] Example of a prompt

[1520] Question answer function: A user asks, "Why is the sky blue?"

[1521] Lesson summary function: After the lesson, the user can request, "Please explain what we covered in today's lesson."

[1522] English conversation support function: The user asks "How is the weather today?".

[1523] Emotion recognition function: The user says, "I'm tired today."

[1524] These features allow children to learn safely and parents to watch over them with peace of mind. Furthermore, they provide an effective learning environment through English conversation support and lesson review, and enable more personalized educational support through emotion recognition.

[1525] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1526] Question answering function

[1527] Step 1:

[1528] The user inputs the question by voice. The device collects the user's voice using a microphone and saves it as audio data.

[1529] Input: User's voice: "Why is the sky blue?"

[1530] Output: Audio data

[1531] Step 2:

[1532] The collected audio data is converted into text data using a speech recognition engine (e.g., Google Speech-to-Text).

[1533] Input: Audio data

[1534] Output: Text data "Why is the sky blue?"

[1535] Specific operation: The speech recognition engine analyzes the audio data and generates the corresponding text.

[1536] Step 3:

[1537] Send the converted text data to the server.

[1538] Input: Text data "Why is the sky blue?"

[1539] Output: Text data sent to the server

[1540] Specific action: The terminal sends text data to the server.

[1541] Step 4:

[1542] The server receives text data and uses a natural language processing model (e.g., OpenAI's GPT model) to analyze the intent of the question.

[1543] Input: Text data "Why is the sky blue?"

[1544] Output: Analysis results (the intent of the question)

[1545] Specific operation: The natural language processing model analyzes the context of the question and extracts the information necessary to generate an appropriate answer.

[1546] Step 5:

[1547] Based on the question analysis results, the response generation means generates an appropriate response.

[1548] Input: Analysis results

[1549] Output: Answer: "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[1550] Specific operation: The response generation mechanism accesses the database and extracts or generates appropriate responses.

[1551] Step 6:

[1552] The generated responses are converted into audio data using a speech synthesis engine (e.g., Google Text-to-Speech).

[1553] Input: Response (text data)

[1554] Output: Response (audio data)

[1555] Specific operation: The speech synthesis engine converts text data into speech data.

[1556] Step 7:

[1557] Send audio data to the device.

[1558] Input: Response (audio data)

[1559] Output: Audio data sent to the terminal

[1560] Specific operation: The server sends the generated audio data to the terminal.

[1561] Step 8:

[1562] The device plays back the received audio data and provides the user with an answer.

[1563] Input: Response (audio data)

[1564] Output: Audio played to the user

[1565] Specific action: The device's speaker plays audio data, and the user listens to the response.

[1566] Lesson content summary function

[1567] Step 1:

[1568] Audio during lessons is collected using a microphone and saved as audio data.

[1569] Input: Audio from class

[1570] Output: Audio data

[1571] Specific operation: The microphone continuously collects audio during class and stores it on the device.

[1572] Step 2:

[1573] The collected audio data is converted into text data at regular intervals using a speech recognition engine.

[1574] Input: Audio data

[1575] Output: Text data

[1576] Specific operation: The speech recognition engine analyzes the collected audio data and generates corresponding text.

[1577] Step 3:

[1578] Send the converted text data to the server.

[1579] Input: Text data

[1580] Output: Text data sent to the server

[1581] Specific action: The terminal sends text data to the server.

[1582] Step 4:

[1583] The server receives text data and summarizes it using a natural language processing model.

[1584] Input: Text data

[1585] Output: Summary results

[1586] Specific operation: A natural language processing model analyzes text data, extracts key points, and generates a summary.

[1587] Step 5:

[1588] Save the summary results to the database.

[1589] Input: Summary result

[1590] Output: Saved summary data

[1591] Specific action: Record the summary results in a database and save them so they can be retrieved later.

[1592] Step 6:

[1593] The system retrieves a summary result in response to the user's request and displays it on the device screen.

[1594] Input: User Request

[1595] Output: Summary results displayed on the screen

[1596] Specific operation: Retrieve the summary results from the database and display them on the terminal screen.

[1597] English conversation support function

[1598] Step 1:

[1599] The microphone collects audio from the user to initiate questions or conversations in English.

[1600] Input: English audio

[1601] Output: Audio data

[1602] Specific action: The user asks a question in English, such as "How is the weather today?".

[1603] Step 2:

[1604] The collected audio data is converted into text data using a speech recognition engine.

[1605] Input: Audio data

[1606] Output: Text data

[1607] Specific operation: The speech recognition engine analyzes the audio data and generates the corresponding text.

[1608] Step 3:

[1609] Send the converted text data to the server.

[1610] Input: Text data

[1611] Output: Text data sent to the server

[1612] Specific action: The terminal sends text data to the server.

[1613] Step 4:

[1614] The server receives text data and generates a response using an English conversation model.

[1615] Input: Text data

[1616] Output: Response (text data)

[1617] Specific operation: The English conversation model analyzes text data and generates appropriate English responses.

[1618] Step 5:

[1619] The generated responses are converted into speech data using a speech synthesis engine.

[1620] Input: Response (text data)

[1621] Output: Response (audio data)

[1622] Specific operation: The speech synthesis engine converts text into speech data.

[1623] Step 6:

[1624] Send audio data to the device.

[1625] Input: Response (audio data)

[1626] Output: Audio data sent to the terminal

[1627] Specific operation: The server sends the generated audio data to the terminal.

[1628] Step 7:

[1629] The device plays back the received audio data and provides the user with an answer.

[1630] Input: Response (audio data)

[1631] Output: Audio played to the user

[1632] Specific action: The device's speaker plays audio data, and the user listens to the response.

[1633] Monitoring function

[1634] Step 1:

[1635] The system uses a GPS system and an accelerometer to acquire the user's location and movements.

[1636] Input: None (Sensor acquisition itself)

[1637] Output: Location information and motion data

[1638] Specific actions: Continuously acquire location information and motion data to prepare for anomaly detection.

[1639] Step 2:

[1640] The acquired data is sent to the server.

[1641] Input: Location information and motion data

[1642] Output: Data sent to the server

[1643] Specific action: The device sends the data it has acquired to the server.

[1644] Step 3:

[1645] The server analyzes the received data and detects abnormal conditions.

[1646] Input: Location information and motion data

[1647] Output: Analysis result (normal / abnormal)

[1648] Specific operation: The data analysis engine analyzes the data and detects abnormal conditions.

[1649] Step 4:

[1650] If an anomaly is detected, an alert will be sent to parents or emergency services.

[1651] Input: Analysis results (abnormal)

[1652] Output: Alert notification

[1653] Specific operation: The server sends alerts via SMS or email.

[1654] Step 5:

[1655] The device notifies the user of an anomaly.

[1656] Input: None (based on alert notifications from the server)

[1657] Output: Voice notifications, vibration, and display.

[1658] Specific actions: The device will notify the user of an anomaly using sound, vibration, or a display message.

[1659] Emotion recognition function

[1660] Step 1:

[1661] The system captures the user's voice and facial expressions through a camera and microphone.

[1662] Input: Voice and facial expressions

[1663] Output: Audio data and facial expression data

[1664] Specific operation: The microphone collects audio, and the camera collects facial expressions.

[1665] Step 2:

[1666] The acquired data is sent to the emotion analysis engine.

[1667] Input: Voice data and facial expression data

[1668] Output: Emotion analysis results

[1669] Specific operation: The emotion analysis engine analyzes voice and facial expressions to identify emotions.

[1670] Step 3:

[1671] The server generates an appropriate response based on the sentiment analysis results.

[1672] Input: Sentiment analysis results

[1673] Output: Response (text data)

[1674] Specific operation: The response generation engine generates an appropriate response based on the analyzed emotions.

[1675] Step 4:

[1676] The generated response is converted into speech data using a speech synthesis engine.

[1677] Input: Response (text data)

[1678] Output: Response (audio data)

[1679] Specific operation: The speech synthesis engine converts text data into speech data.

[1680] Step 5:

[1681] The response audio data is sent to the terminal.

[1682] Input: Response (voice data)

[1683] Output: Audio data sent to the terminal

[1684] Specific operation: The server sends the generated audio data to the terminal.

[1685] Step 6:

[1686] The device plays back the received audio data and provides a response to the user.

[1687] Input: Response (voice data)

[1688] Output: Audio played to the user

[1689] Specific action: The device's speaker plays audio data, and the user listens to the response.

[1690] The above is a detailed explanation of the program's processing flow.

[1691] (Application Example 2)

[1692] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1693] Conventional educational support devices lack features to ensure children's safety, such as monitoring functions, and personalized educational support that responds to their emotions. Furthermore, they lack the ability to track children's location in real time and detect anomalies, making rapid response in emergencies difficult. This invention aims to solve these problems and provide safer, more secure, and more effective educational support.

[1694] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1695] In this invention, the server includes voice acquisition means, voice recognition means, question analysis means, answer generation means, voice generation means, answer presentation means, location information acquisition means, location data transmission means, anomaly detection means, alert transmission means, emotion acquisition means, emotion analysis means, and system response generation means. This makes it possible to provide individually optimized educational support in real time while ensuring user safety.

[1696] A "voice acquisition means" is a device that captures the user's voice and stores it as digital data.

[1697] "Speech recognition means" refers to technology that converts acquired speech data into text data.

[1698] A "question analysis tool" is an algorithm that analyzes text data obtained by a speech recognition tool to understand the intent and content of a question.

[1699] "Answer generation means" refers to a technology that generates appropriate answers based on the results obtained from question analysis means.

[1700] "Speech generation means" refers to technology that converts generated text data into speech data.

[1701] A "response presentation means" is a device that plays back the generated audio data and presents it to the user.

[1702] "Location information acquisition means" refers to technologies that use GPS or other technologies to acquire the user's current location.

[1703] "Location data transmission means" refers to technology that transmits acquired location information to a server or other terminal.

[1704] An "anomaly detection method" is a technology that detects user behavior or states that are different from the norm.

[1705] An "alert transmission method" is a technology for quickly notifying of anomalies detected by anomaly detection methods.

[1706] "Emotion acquisition methods" refer to technologies that acquire emotions from the user's voice and facial expressions.

[1707] "Emotional analysis means" refers to technology that analyzes acquired emotional data to determine the user's emotional state.

[1708] "System response generation means" refers to a technology that generates appropriate responses or messages based on data obtained from emotion analysis means.

[1709] Explanation of program generation and processing

[1710] This invention is a system that provides individually optimized educational support in real time while ensuring user safety, and utilizes the following main hardware and software.

[1711] Hardware to use

[1712] smartphone

[1713] Smart Glasses

[1714] GPS sensor

[1715] Accelerometer

[1716] microphone

[1717] camera

[1718] Software to use

[1719] Google Speech API (speech recognition engine)

[1720] TensorFlow (natural language processing engine)

[1721] Microsoft Azure Emotion API (emotion recognition engine)

[1722] Google Maps API (location tracking)

[1723] Twilio API (Alert Notifications)

[1724] Detailed processing of the system

[1725] Location tracking

[1726] The server acquires location information from GPS sensors installed in smartphones and smart glasses. This acquired location information is transmitted to the server in real time via a location data transmission device. The server analyzes the received location data and detects abnormal movements using an anomaly detection device. If an anomaly is detected, an alert transmission device is used to notify parents or administrators via SMS or push notification.

[1727] Emotion monitoring

[1728] The server uses the microphone and camera of a smartphone or smart glasses to acquire the user's voice and facial expressions using emotion acquisition means. This data is analyzed by emotion analysis means to determine the user's emotional state. Next, system response generation means is used to generate an appropriate response based on the emotional state. This response is converted into audio data by voice generation means and presented to the user.

[1729] Learning support

[1730] The server acquires the user's voice question using a voice acquisition means and converts it into text data using a speech recognition means. The text data is analyzed by a question analysis means, and based on the results, an answer is generated by an answer generation means. The generated answer is converted into speech by a speech generation means and presented to the user through an answer presentation means.

[1731] Specific examples and prompt statements

[1732] Examples of emotion recognition

[1733] When a user says, "I'm tired today," the device uses an emotion engine to analyze the user's tone of voice and facial expression. Based on the analysis, the server generates a response such as, "You seem tired today, so you should take a rest," and plays it back as audio.

[1734] Example of a prompt:

[1735] Voice input: "Something unpleasant happened at school today."

[1736] Facial expression input: Image data (child's facial expression)

[1737] Examples of learning support

[1738] User question: "Why is the sky blue?"

[1739] Response: "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[1740] Example of a prompt:

[1741] "Why is the sky blue?"

[1742] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1743] Step 1:

[1744] The device uses a microphone to acquire the user's voice. The acquired voice data is stored as a digital signal. In this case, the voice data is the input.

[1745] Step 2:

[1746] The device converts the acquired audio data, obtained using speech recognition (Google Speech API), into text data. This conversion results in the output of the text data.

[1747] Step 3:

[1748] The terminal sends the converted text data to the server. Here, the text data is passed to the server as input.

[1749] Step 4:

[1750] The server analyzes the text data using a question analysis tool (TensorFlow) to understand the intent of the question. The intent of the question is output as the analysis result.

[1751] Step 5:

[1752] The server generates an appropriate answer using an answer generation mechanism based on the question analysis results. The generated answer is output as text data.

[1753] Step 6:

[1754] The server converts the generated text response into audio data using a speech generation tool (Google Speech API). This conversion results in the output of the audio data.

[1755] Step 7:

[1756] The server sends the generated audio data to the terminal. Here, the audio data is passed to the terminal as input.

[1757] Step 8:

[1758] The terminal presents the user with the answer to the question by playing the audio data through the answer presentation means. Here, the audio is presented to the user.

[1759] Step 9:

[1760] The device uses a GPS sensor to obtain the user's location information. This information is stored as location data. In this case, location data is the input.

[1761] Step 10:

[1762] The device sends location data to the server. The location data is passed to the server as input.

[1763] Step 11:

[1764] The server analyzes location data using anomaly detection mechanisms to detect unusual movement. If an anomaly is detected, anomaly data is output.

[1765] Step 12:

[1766] When abnormal data is detected, the server sends a notification to the parent or administrator using the alert sending mechanism (Twilio API). This notification triggers an alert.

[1767] Step 13:

[1768] The device acquires user voice and facial expression data using emotion acquisition methods. This emotion data is the input.

[1769] Step 14:

[1770] The server analyzes emotional data using an emotion analysis tool (Microsoft Azure Emotion API) to determine the user's emotional state. The emotion analysis results are then output.

[1771] Step 15:

[1772] The server generates an appropriate response using a system response generation tool (TensorFlow) based on the sentiment analysis results. This response is output as text data.

[1773] Step 16:

[1774] The server converts the generated text response into audio data using a speech generation tool (Google Speech API). This conversion results in the output of the audio data.

[1775] Step 17:

[1776] The server sends the generated audio data to the terminal. Here, the audio data is passed to the terminal as input.

[1777] Step 18:

[1778] The device plays audio data to the user, providing a response that corresponds to their emotional state. Here, the audio is presented to the user.

[1779] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1780] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1781] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1782] [Third Embodiment]

[1783] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1784] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1785] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1786] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1787] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1788] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1789] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1790] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1791] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1792] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1793] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1794] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1795] This invention relates to a wristwatch-type device for supporting education while ensuring the safety of children. Because this device operates locally without an internet connection, it offers high security. Furthermore, it provides educational support and monitoring functions using a system equipped with voice recognition and natural language processing.

[1796] The present invention has the following main features:

[1797] 1. Question answering function

[1798] 2. Function to summarize lesson content

[1799] 3. English conversation support function

[1800] 4. Monitoring function

[1801] Question answering function

[1802] Terminal:

[1803] It is equipped with a microphone for users to input questions by voice. When a user asks a question, this voice data is converted into text data by the speech recognition engine in the device.

[1804] server:

[1805] The system receives text data and analyzes the questions. The analysis process uses natural language processing techniques to understand the intent of the questions and generates appropriate answers from a local database.

[1806] server:

[1807] The generated response is converted into audio data using a speech synthesis engine. The converted audio data is then sent to the terminal.

[1808] Terminal:

[1809] The system plays back the received audio data to the user and provides answers to their questions.

[1810] Specific user examples:

[1811] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[1812] Lesson content summary function

[1813] Terminal:

[1814] Audio from the classroom is captured via a micro-fan and saved as audio data.

[1815] Terminal:

[1816] At regular intervals, audio data is converted to text using a speech recognition engine as a batch process. This text data is then sent to the server.

[1817] server:

[1818] The received text data is summarized using natural language processing techniques. The summarization results are stored in a local database.

[1819] Terminal:

[1820] In response to user requests, the system retrieves a summary of the lesson and displays it on the screen.

[1821] Specific user examples:

[1822] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson is..."

[1823] English conversation support function

[1824] Terminal:

[1825] It is equipped with a microphone for users to ask questions or initiate conversations in English. It acquires voice data, and a speech recognition engine converts it to text.

[1826] server:

[1827] The system receives text data and generates appropriate responses using an English conversation model. The generated responses are then converted into speech data using a speech synthesis engine.

[1828] Terminal:

[1829] The generated audio data is played for the user.

[1830] Specific user examples:

[1831] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[1832] Monitoring function

[1833] Terminal:

[1834] It is equipped with a GPS system and an accelerometer, which continuously acquire the user's location and movement information. This data is transmitted to a server.

[1835] server:

[1836] The system analyzes the received data and detects anomalies. If an anomaly is detected, it sends an alert to the parent system or emergency services.

[1837] Terminal:

[1838] It receives alerts and notifies the user via voice or vibration as needed.

[1839] Specific user examples:

[1840] If a user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is [location]."

[1841] These features allow children to learn safely, and parents can watch over them with peace of mind. Furthermore, they provide an effective learning environment through English conversation support and lesson review.

[1842] The following describes the processing flow.

[1843] Question answering function

[1844] Step 1:

[1845] Terminal: The user speaks a voice question into the microphone.

[1846] Step 2:

[1847] Terminal: The speech recognition engine converts the user's voice into text data.

[1848] Step 3:

[1849] Terminal: Sends the converted text data to the server.

[1850] Step 4:

[1851] Server: Receives text data and analyzes the question using a natural language processing algorithm.

[1852] Step 5:

[1853] Server: Based on the analysis results, it searches for and generates the appropriate answer from the local database.

[1854] Step 6:

[1855] Server: Converts the generated response text into speech data using a speech synthesis engine.

[1856] Step 7:

[1857] Server: Sends audio data to the terminal.

[1858] Step 8:

[1859] Terminal: Plays the received audio data through the speaker and presents the answer to the user.

[1860] Lesson content summary function

[1861] Step 1:

[1862] Terminal: This device uses a micro-fan to capture audio during class and saves it as audio data.

[1863] Step 2:

[1864] Terminal: Uses a speech recognition engine to convert speech data into text at regular intervals.

[1865] Step 3:

[1866] Terminal: Sends the converted text data to the server.

[1867] Step 4:

[1868] Server: Receives text data and summarizes it using a natural language processing algorithm.

[1869] Step 5:

[1870] Server: Saves the summary results to the local database.

[1871] Step 6:

[1872] Terminal: Receives requests from users and requests summary results from the server.

[1873] Step 7:

[1874] Server: Sends the summary results to the terminal.

[1875] Step 8:

[1876] Terminal: Displays the received summary results on the screen.

[1877] English conversation support function

[1878] Step 1:

[1879] Device: Speak your voice question in English into the microphone.

[1880] Step 2:

[1881] Terminal: The speech recognition engine converts the speech data into text data.

[1882] Step 3:

[1883] Terminal: Sends the converted text data to the server.

[1884] Step 4:

[1885] Server: Receives text data and generates appropriate English responses based on an English conversation model.

[1886] Step 5:

[1887] Server: Converts the generated English text into speech data using a speech synthesis engine.

[1888] Step 6:

[1889] Server: Sends audio data to the terminal.

[1890] Step 7:

[1891] Terminal: Plays the received audio data through the speaker and provides the user with a response.

[1892] Monitoring function

[1893] Step 1:

[1894] Terminal: Periodically acquires the user's location information and movement using a GPS system and accelerometer.

[1895] Step 2:

[1896] Terminal: Sends acquired data to the server.

[1897] Step 3:

[1898] Server: Monitors location information and operational data in real time and detects anomalies.

[1899] Step 4:

[1900] Server: If an anomaly is detected, it sends an alert to the parent or emergency services.

[1901] Step 5:

[1902] Device: Notifies the user via voice alerts or vibration.

[1903] This should clarify the specific processing steps for each function.

[1904] (Example 1)

[1905] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1906] Traditional educational support devices required an internet connection, posing a high security risk. Furthermore, few devices integrated child safety monitoring and educational support functions, making it difficult for parents to constantly monitor their children with peace of mind. Additionally, educational support features such as lesson summaries and English conversation support were often insufficient.

[1907] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1908] In this invention, the server includes a voice acquisition means, a voice recognition means, a question analysis means, an answer generation means, a voice generation means, an answer presentation means, a location information acquisition means, a motion detection means, an anomaly detection means, and an anomaly notification means. This allows for secure operation even without an internet connection and enables the integrated provision of educational support and child monitoring functions. Furthermore, by providing functions such as lesson content summarization and English conversation support, the effectiveness of children's learning can be enhanced.

[1909] "Voice acquisition means" refers to a device or function for picking up the voice emitted by the user.

[1910] "Speech recognition means" refers to a technology or device that converts acquired speech into text data.

[1911] "Question analysis means" refers to a technology or device for analyzing user intent from transcribed audio data.

[1912] "Answer generation means" refers to a technology or device that generates an appropriate answer based on the analysis results.

[1913] "Speech generation means" refers to a technology or device that converts generated text-based responses into speech.

[1914] "Answer presentation means" refers to a technology or device for presenting generated audio to the user.

[1915] "Location information acquisition means" refers to technology or devices that use GPS or similar systems to acquire the user's current location.

[1916] "Motion detection means" refers to a technology or device that detects the user's movement status using an acceleration sensor or the like.

[1917] An "anomaly detection means" is a technology or device for detecting abnormal situations from user activity information or location information.

[1918] An "anomaly notification means" is a technology or device for sending an alert when an anomaly is detected.

[1919] "Means for acquiring audio of lecture content" refers to a device or function for capturing audio during lectures.

[1920] "Means for converting audio of lecture content into text" refers to technology or equipment that converts acquired audio into text data.

[1921] "Means of summarizing text" refers to techniques or devices that analyze text data, extract only the important information, and summarize it.

[1922] "Means for saving the summary results" refers to a technology or device for saving the summarized text data to a storage device.

[1923] "Means for presenting summary results" refers to a technology or device for presenting summarized text data to a user.

[1924] "Means for acquiring English conversation audio" refers to a device or function for picking up English audio spoken by a user.

[1925] "Means for converting spoken English into text" refers to a technology or device that converts acquired spoken English into text data.

[1926] "Means for generating responses using an English conversation model" refers to a technology or device that analyzes English text data to generate appropriate English responses.

[1927] "Means for converting generated responses into speech" refers to a technology or device that converts generated text-based responses into speech.

[1928] "Means for playing back audio" refers to the technology or device used to play back generated audio data for the user.

[1929] This invention relates to an arm-worn device for supporting education while ensuring the safety of children. Because this device operates locally without an internet connection, it offers high security. Furthermore, it utilizes speech recognition and natural language processing technologies to provide multiple educational support and monitoring functions.

[1930] Question answering function

[1931] The device of this invention is equipped with a microphone for the user to input questions by voice. The voice data is converted into text data by a speech recognition engine (e.g., Google Speech-to-Text API) within the device. This text data is sent to a server.

[1932] The server receives text data and analyzes the question using natural language processing techniques (e.g., the NLTK library). Based on the analysis, it generates an appropriate answer from a local database. The generated answer is converted into audio data by a speech synthesis engine (e.g., Google Text-to-Speech API) and sent to the terminal. The terminal plays this audio data and provides the answer to the user.

[1933] Specific example:

[1934] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[1935] Lesson content summary function

[1936] Audio during lessons is captured through the device's microphone and saved as audio data. This audio data is converted into text data at regular intervals by a speech recognition engine (e.g., Google Speech-to-Text API). The converted text data is then sent to a server.

[1937] The server summarizes the received text data using natural language processing techniques (e.g., the NLTK library). The summary is stored in a local database. The user can retrieve this summary upon request and display it on their terminal screen.

[1938] Specific example:

[1939] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson is..."

[1940] English conversation support function

[1941] The user uses the device's microphone to ask a question or initiate a conversation in English. The audio data is converted into text data by a speech recognition engine (e.g., Google Speech-to-Text API). This text data is then sent to the server.

[1942] The server receives text data and generates an appropriate response using an English conversation model (e.g., OpenAI's GPT-3). The generated response is converted into audio data by a speech synthesis engine (e.g., Google Text-to-Speech API) and sent to the terminal. The terminal plays this audio data and provides the response to the user.

[1943] Specific example:

[1944] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[1945] Monitoring function

[1946] The device of the present invention is equipped with a GPS system and an accelerometer, and continuously acquires the user's location information and movements. This data is transmitted to a server.

[1947] The server analyzes the received data and, if it detects an anomaly, sends an alert to the parent or emergency services. The device receives the alert and notifies the user via sound or vibration.

[1948] Specific example:

[1949] If the user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is [location]."

[1950] Examples of prompt statements

[1951] The following are specific examples of prompt statements to be input to a generative AI model:

[1952] 1. Prompts for the question function:

[1953] User: "Why is the sky blue?"

[1954] Prompt: "Explain why the sky appears blue."

[1955] 2. Prompts for English conversation support function:

[1956] User: "How is the weather today?"

[1957] Prompt: "Please describe the weather today."

[1958] Based on these concrete examples, children can learn safely, and parents can watch over them with peace of mind. Furthermore, it becomes possible to provide an effective learning environment through English conversation support and review of lessons.

[1959] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1960] Question answering function

[1961] Step 1:

[1962] The user enters the question by voice.

[1963] Input: User's voice question

[1964] Specific action: The user asks the question, "Why is the sky blue?"

[1965] Step 2:

[1966] The device acquires voice data and converts it into text data using a speech recognition engine.

[1967] Input: User's voice data

[1968] Data processing: Convert speech to text using a speech recognition engine (e.g., Google Speech-to-Text API).

[1969] Output: Text data ("Why is the sky blue?")

[1970] Step 3:

[1971] The terminal sends the converted text data to the server.

[1972] Input: Text data

[1973] Data processing: Uses HTTP requests

[1974] Output: Sending text data to the server

[1975] Step 4:

[1976] The server receives text data and analyzes the question using natural language processing technology.

[1977] Input: Text data ("Why is the sky blue?")

[1978] Data processing: Analyze the intent of the question using natural language processing techniques (e.g., NLTK library).

[1979] Output: Analysis results

[1980] Step 5:

[1981] The server generates the appropriate answer from the local database based on the analysis results.

[1982] Input: Analysis results

[1983] Data calculation: Generate answers from a local database.

[1984] Output: Answer text ("The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes.")

[1985] Step 6:

[1986] The server converts the generated response text into audio data and sends it to the terminal.

[1987] Input: Answer text

[1988] Data processing: Convert to audio data using a speech synthesis engine (e.g., Google Text-to-Speech API).

[1989] Output: Generation of audio data and transmission to the terminal.

[1990] Step 7:

[1991] The device plays the audio data it received.

[1992] Input: Audio data

[1993] Specific action: Play the answer through the device's speaker (telling the user, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes.")

[1994] Lesson content summary function

[1995] Step 1:

[1996] The device acquires audio during the lesson.

[1997] Input: Audio from class

[1998] Specific action: Audio during class is captured using a microphone.

[1999] Step 2:

[2000] The device saves the audio data.

[2001] Input: Acquired audio data

[2002] Data processing: Saving to internal storage

[2003] Output: Saved audio data

[2004] Step 3:

[2005] The device converts voice data into text data using a speech recognition engine at regular intervals.

[2006] Input: Saved audio data

[2007] Data processing: Convert speech to text using a speech recognition engine (e.g., Google Speech-to-Text API).

[2008] Output: Text data

[2009] Step 4:

[2010] The terminal sends text data to the server.

[2011] Input: Text data

[2012] Data processing: Uses HTTP requests

[2013] Output: Sending text data to the server

[2014] Step 5:

[2015] Summarize the text data received by the server.

[2016] Input: Text data

[2017] Data processing: Summarizing using natural language processing techniques (e.g., NLTK library).

[2018] Output: Summary text data

[2019] Step 6:

[2020] The server sends the summary results to the terminal.

[2021] Input: Summary text data

[2022] Data processing: Uses HTTP requests

[2023] Output: Send summary text to terminal

[2024] Step 7:

[2025] The device displays a summary result in response to a user request.

[2026] Input: User request, summary text data

[2027] Specific operation: When the user requests "Tell me what we covered in today's lesson," a summary is displayed on the screen.

[2028] Output: Summary displayed on the screen ("What we learned in today's lesson is ______")

[2029] English conversation support function

[2030] Step 1:

[2031] The user speaks in English.

[2032] Input: User's English voice

[2033] Specific action: The user speaks in English, saying "How is the weather today?"

[2034] Step 2:

[2035] The device acquires voice data and converts it into text data using a speech recognition engine.

[2036] Input: User's English voice data

[2037] Data processing: Convert speech to text using a speech recognition engine (e.g., Google Speech-to-Text API).

[2038] Output: Text data ("How is the weather today?")

[2039] Step 3:

[2040] The terminal sends text data to the server.

[2041] Input: Text data

[2042] Data processing: Uses HTTP requests

[2043] Output: Sending text data to the server

[2044] Step 4:

[2045] The server generates the response using an English conversation model.

[2046] Input: Text data ("How is the weather today?")

[2047] Data processing: Generate answers using an English conversation model (e.g., OpenAI's GPT-3).

[2048] Output: Answer text ("The weather is sunny today.")

[2049] Step 5:

[2050] The server converts the generated response text into audio data and sends it to the terminal.

[2051] Input: Answer text

[2052] Data processing: Convert to audio data using a speech synthesis engine (e.g., Google Text-to-Speech API).

[2053] Output: Generation of audio data and transmission to the terminal.

[2054] Step 6:

[2055] The device plays the audio data it received.

[2056] Input: Audio data

[2057] Specific action: Play the answer through the device's speaker (telling the user "The weather is sunny today.")

[2058] Monitoring function

[2059] Step 1:

[2060] The device acquires the user's location information and movements.

[2061] Input: GPS and accelerometer data

[2062] Specific operation: The device uses the GPS system and accelerometer to obtain location information and movement.

[2063] Step 2:

[2064] The device sends location information and movement data to the server.

[2065] Input: Location data, motion data

[2066] Data processing: Uses HTTP requests

[2067] Output: Sending data to the server

[2068] Step 3:

[2069] The server analyzes the data and detects anomalies.

[2070] Input: Location data, motion data

[2071] Data processing: Anomaly detection using algorithms

[2072] Output: Anomaly detection result

[2073] Step 4:

[2074] If the server detects an anomaly, it will send an alert.

[2075] Input: Anomaly detection result

[2076] Specific actions: Send an alert via SMS or email to parents or emergency services (e.g., "Your child has fallen. Their current location is [location].")

[2077] Step 5:

[2078] The device receives an alert and sends a notification.

[2079] Input: Alert data

[2080] Specific action: The device notifies the user of an abnormality via sound or vibration.

[2081] Through the processing steps described above, this system can effectively provide educational support while ensuring the safety of children.

[2082] (Application Example 1)

[2083] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[2084] There is a need for effective methods to maximize educational effectiveness while ensuring children's safety. Furthermore, a system is required that operates locally without an internet connection, ensuring data security while efficiently handling question answering, lesson summaries, English conversation support, and monitoring. Conventional technologies lack sufficient child anomaly detection and emergency notification features, making it difficult to ensure children's safety.

[2085] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[2086] In this invention, the server includes voice acquisition means, voice recognition means, question analysis means, answer generation means, voice generation means, answer presentation means, location information acquisition means, emergency notification means, and anomaly detection means. This enables question answering based on voice input, acquisition and summarization of educational content, and provision of English conversation support, as well as the ability to detect location information and abnormal operation and provide notifications in emergencies.

[2087] "Sound acquisition means" refers to a device or software that has the function of collecting ambient sounds or voices.

[2088] "Speech recognition means" refers to a device or software that analyzes acquired speech data and converts it into text data.

[2089] "Question analysis means" refers to a device or software that analyzes text data generated by speech recognition means and understands the intent of a question.

[2090] "Answer generation means" refers to a device or software that generates an appropriate answer based on the intent of a question understood by the question analysis means.

[2091] "Voice generation means" refers to a device or software that converts generated text-based responses into voice data.

[2092] "Answer presentation means" refers to a device or software that has the function of playing back audio data generated by the audio generation means to the user.

[2093] "Location information acquisition means" refers to a device or software that acquires a user's location information using GPS or other location measurement technologies.

[2094] An "emergency notification system" is a device or software that has the function of sending notifications to pre-configured contacts when an abnormality or emergency is detected.

[2095] An "anomaly detection means" is a device or software that has the function of detecting abnormal operation or condition based on data from sensors.

[2096] "Means for acquiring audio recordings of class content" refers to a device or software that has the function of recording lectures and discussions during class.

[2097] "Means of converting speech to text" refers to a device or software that analyzes speech data acquired during a lesson and converts it into text data.

[2098] "Means for summarizing text" refers to a device or software that analyzes converted text data, extracts important points, and summarizes them.

[2099] "Means for saving the summary results" refers to a device or software that has the function of saving the summarized information to a storage device.

[2100] "Means for presenting summary results" refers to a device or software that has the function of displaying or playing back saved summary results to the user.

[2101] "Means for acquiring English conversation audio" refers to a device or software that has the function of collecting audio data of English spoken by a user.

[2102] "Means for generating responses using an English conversation model" refers to a device or software that analyzes acquired English text data and uses an English conversation model to generate appropriate responses.

[2103] "Means for converting generated responses into speech" refers to a device or software that converts text-based responses generated by an English conversation model into speech data.

[2104] "Means for playing audio" refers to a device or software that has the function of playing converted audio data back to the user.

[2105] This invention provides a system to support education while ensuring the safety of children. This system has the following main functions:

[2106] 1. Voice acquisition and recognition function

[2107] Terminal:

[2108] It is equipped with a microphone for users to input questions by voice. When a user asks a question, this voice data is converted into text data by the speech recognition engine in the device. A standard microphone is used as the hardware.

[2109] server:

[2110] The system receives text data and analyzes the questions. The analysis process uses natural language processing techniques to understand the intent of the questions and generates appropriate answers from a local database. The software used includes natural language processing libraries (e.g., NLTK, spaCy).

[2111] Specific example:

[2112] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[2113] Example of a prompt:

[2114] Generate an answer to the question, "Why is the sky blue?"

[2115] 2. Function to summarize educational content

[2116] Terminal:

[2117] Audio during class is captured via a microphone and saved as audio data. A standard microphone is used as the hardware.

[2118] server:

[2119] The received text data is summarized using natural language processing techniques. The summarization results are stored in a local database. The software used includes a text summarization library (e.g., Sumy).

[2120] Specific example:

[2121] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson was about the main events of World War II."

[2122] Example of a prompt:

[2123] Generate a summary of what a user would say if they requested, "Tell me what was covered in today's lesson."

[2124] 3. English conversation support function

[2125] Terminal:

[2126] It is equipped with a microphone for users to ask questions or initiate conversations in English. It acquires voice data, and a speech recognition engine converts it to text.

[2127] server:

[2128] The system receives text data and generates appropriate responses using an English conversation model. The generated responses are then converted into speech data using a speech synthesis engine. The software used includes an English conversation model (e.g., GPT-3).

[2129] Specific example:

[2130] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[2131] Example of a prompt:

[2132] Please generate an answer for the user's question, "How is the weather today?"

[2133] 4. Child monitoring function

[2134] Terminal:

[2135] It is equipped with a GPS system and an accelerometer to continuously acquire the user's location and movement. This data is transmitted to a server. The hardware used includes a GPS module (e.g., u-blox) and an accelerometer (e.g., MEMS sensor).

[2136] server:

[2137] The system analyzes the received data and detects anomalies. If an anomaly is detected, it sends an alert to the parent system or emergency services. The software used includes an anomaly detection algorithm.

[2138] Specific example:

[2139] If the user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is 35.6895, 139.6917."

[2140] Example of a prompt:

[2141] Please explain how to send an emergency notification to parents if a child falls.

[2142] This allows children to learn safely, and parents can watch over them with peace of mind. Furthermore, it makes it possible to provide an effective learning environment through English conversation support and lesson review.

[2143] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[2144] Step 1:

[2145] Voice data acquisition and recognition

[2146] The device acquires the user's voice data using the microphone. The acquired voice data is sent to the speech recognition engine within the device. The speech recognition engine converts the voice data into text data and generates the resulting text data.

[2147] Input: Audio data

[2148] Output: Text data

[2149] Step 2:

[2150] Question analysis

[2151] The server receives text data from the terminal. The server uses natural language processing technology to analyze the content of the question in the text data and performs data calculations to understand its intent. This determines the intent of the question and generates a result.

[2152] Input: Text data

[2153] Output: Question intent data

[2154] Step 3:

[2155] Generating an answer

[2156] The server generates appropriate answers based on the intent data of the questions, using a local database or a generative AI model. The generated answer text is then used as the primary output.

[2157] Input: Question intent data

[2158] Output: Response text data

[2159] Step 4:

[2160] Convert to speech

[2161] The server sends the response text data to the speech synthesis engine, which converts it into speech data. The converted speech data is then sent to the terminal.

[2162] Input: Answer text data

[2163] Output: Audio data

[2164] Step 5:

[2165] Providing an answer

[2166] The terminal plays the audio data received from the server. Based on the played audio data, the user is presented with an answer.

[2167] Input: Audio data

[2168] Output: Audio output

[2169] Step 6:

[2170] Acquisition of course content

[2171] The device uses its microphone to capture audio during class. The captured audio data is temporarily stored on the device and periodically sent to the speech recognition engine. The speech recognition engine converts the audio data into text data and generates the resulting text data.

[2172] Input: Audio data from class

[2173] Output: Text data from the lecture

[2174] Step 7:

[2175] Text summary

[2176] The server receives the lecture text data and uses natural language processing technology to summarize the text. The summary results are stored in a local database.

[2177] Input: Text data from the class

[2178] Output: Summary data

[2179] Step 8:

[2180] English conversation support

[2181] The device uses its microphone to capture the user's English conversation voice. The captured audio data is sent to the device's speech recognition engine, which converts the audio data into text data.

[2182] Input: English conversation audio data

[2183] Output: English conversation text data

[2184] Step 9:

[2185] English conversation response generation

[2186] The server receives English conversation text data and generates appropriate responses using an English conversation model. The generated response text data is used as the primary output.

[2187] Input: English conversation text data

[2188] Output: English conversation response text data

[2189] Step 10:

[2190] English conversation converted to audio

[2191] The server sends the English conversation response text data to the speech synthesis engine, which converts it into audio data. The converted audio data is then sent to the terminal.

[2192] Input: English conversation response text data

[2193] Output: English conversation audio data

[2194] Step 11:

[2195] English conversation response

[2196] The device plays English conversation audio data received from the server. Based on the played audio data, the user is presented with an English conversation response.

[2197] Input: English conversation audio data

[2198] Output: Audio output

[2199] Step 12:

[2200] Location information acquisition

[2201] The device uses the GPS system to obtain the user's location information. The obtained location data is periodically sent to the server.

[2202] Input: Location data

[2203] Output: Location data

[2204] Step 13:

[2205] Anomaly detection

[2206] The server analyzes location and acceleration data received from the terminal to detect anomalies. If an anomaly is detected, it generates anomaly data and activates an emergency notification system.

[2207] Input: Location data, acceleration data

[2208] Output: Abnormal data

[2209] Step 14:

[2210] emergency notification

[2211] The server sends notifications to parents or emergency services based on abnormal data. The notification includes the user's current location information.

[2212] Input: Abnormal data

[2213] Output: Notification message

[2214] The above outlines the specific process for implementing the present invention. This system can support effective education while ensuring the safety of children.

[2215] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[2216] This invention relates to a wristwatch-type device for supporting education while ensuring the safety of children. Because this device operates locally without an internet connection, it offers high security. Furthermore, it provides educational support and monitoring functions using a system equipped with voice recognition and natural language processing. In addition, by incorporating an emotion engine that recognizes the user's emotions, it enables more personalized educational support.

[2217] The present invention has the following main features:

[2218] 1. Question answering function

[2219] 2. Function to summarize lesson content

[2220] 3. English conversation support function

[2221] 4. Monitoring function

[2222] 5. Emotion recognition function

[2223] Question answering function

[2224] Terminal:

[2225] It is equipped with a microphone for users to input questions by voice. When a user asks a question, this voice data is converted into text data by the speech recognition engine in the device.

[2226] server:

[2227] The system receives text data and analyzes the questions. The analysis process uses natural language processing techniques to understand the intent of the questions and generates appropriate answers from a local database.

[2228] server:

[2229] The generated response is converted into audio data using a speech synthesis engine. The converted audio data is then sent to the terminal.

[2230] Terminal:

[2231] The system plays back the received audio data to the user and provides answers to their questions.

[2232] Specific user examples:

[2233] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[2234] Lesson content summary function

[2235] Terminal:

[2236] Audio from the classroom is captured via a micro-fan and saved as audio data.

[2237] Terminal:

[2238] The audio data is converted to text by a speech recognition engine at regular intervals. This text data is then sent to the server.

[2239] server:

[2240] The received text data is summarized using natural language processing techniques. The summarization results are stored in a local database.

[2241] Terminal:

[2242] In response to user requests, the system retrieves a summary of the lesson and displays it on the screen.

[2243] Specific user examples:

[2244] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson is..."

[2245] English conversation support function

[2246] Terminal:

[2247] It is equipped with a microphone for users to ask questions or initiate conversations in English. It acquires voice data, and a speech recognition engine converts it to text.

[2248] server:

[2249] The system receives text data and generates appropriate English responses using an English conversation model. The generated responses are then converted into speech data using a speech synthesis engine.

[2250] Terminal:

[2251] The generated audio data is played for the user.

[2252] Specific user examples:

[2253] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[2254] Monitoring function

[2255] Terminal:

[2256] It is equipped with a GPS system and an accelerometer, which continuously acquire the user's location and movement information. This data is transmitted to a server.

[2257] server:

[2258] The system analyzes the received data and detects anomalies. If an anomaly is detected, it sends an alert to the parent system or emergency services.

[2259] Terminal:

[2260] It receives alerts and notifies the user via voice or vibration as needed.

[2261] Specific user examples:

[2262] If a user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is [location]."

[2263] Emotion recognition function

[2264] Terminal:

[2265] The system captures the user's voice and facial expressions through the camera and microphone and sends them to the emotion engine.

[2266] server:

[2267] The emotion engine analyzes the user's emotions and adjusts the response and educational content based on the results.

[2268] Terminal:

[2269] Provide users with appropriate responses and educational content.

[2270] Specific user examples:

[2271] When a user says, "I'm tired today," the device uses an emotion engine to analyze the user's tone of voice and facial expression, and the server generates a response such as, "You seem tired today, so you should take a rest," which is then played back as audio.

[2272] These features allow children to learn safely and parents to supervise with peace of mind. They also provide an effective learning environment through English conversation support and lesson review. Emotion recognition enables more personalized educational support, boosting children's motivation to learn.

[2273] The following describes the processing flow.

[2274] Question answering function

[2275] Step 1:

[2276] Terminal: The user speaks a voice question into the microphone.

[2277] Step 2:

[2278] Terminal: The speech recognition engine converts the user's voice into text data.

[2279] Step 3:

[2280] Terminal: Sends the converted text data to the server.

[2281] Step 4:

[2282] Server: Receives text data and analyzes the question using a natural language processing algorithm.

[2283] Step 5:

[2284] Server: Based on the analysis results, it searches and generates the appropriate answer from the local database.

[2285] Step 6:

[2286] Server: Converts the generated response text into speech data using a speech synthesis engine.

[2287] Step 7:

[2288] Server: Sends audio data to the terminal.

[2289] Step 8:

[2290] Terminal: Plays the received audio data through the speaker and provides the user with a response.

[2291] Lesson content summary function

[2292] Step 1:

[2293] Terminal: This device uses a micro-fan to capture audio during class and saves it as audio data.

[2294] Step 2:

[2295] Terminal: The acquired audio data is converted into text data at regular intervals using a speech recognition engine.

[2296] Step 3:

[2297] Terminal: Sends the converted text data to the server.

[2298] Step 4:

[2299] Server: Receives text data and summarizes it using a natural language processing algorithm.

[2300] Step 5:

[2301] Server: Saves the summary results to the local database.

[2302] Step 6:

[2303] Terminal: Receives a request from the user and requests the summary result from the server.

[2304] Step 7:

[2305] Server: Sends the summary results to the terminal.

[2306] Step 8:

[2307] Terminal: Displays the received summary results on the screen.

[2308] English conversation support function

[2309] Step 1:

[2310] Device: The user speaks a voice question in English into the microphone.

[2311] Step 2:

[2312] Terminal: The speech recognition engine converts the speech data into text data.

[2313] Step 3:

[2314] Terminal: Sends the converted text data to the server.

[2315] Step 4:

[2316] Server: Receives text data and generates appropriate English responses using an English conversation model.

[2317] Step 5:

[2318] Server: Converts the generated English text into speech data using a speech synthesis engine.

[2319] Step 6:

[2320] Server: Sends audio data to the terminal.

[2321] Step 7:

[2322] Terminal: Plays the received audio data through the speaker and provides the user with a response.

[2323] Monitoring function

[2324] Step 1:

[2325] Terminal: Periodically acquires the user's location information and movement using a GPS system and accelerometer.

[2326] Step 2:

[2327] Terminal: Sends acquired data to the server in real time.

[2328] Step 3:

[2329] Server: Monitors location information and operational data to detect anomalies.

[2330] Step 4:

[2331] Server: If an anomaly is detected, it sends an alert to the parent or emergency services.

[2332] Step 5:

[2333] Device: Notifies the user via sound or vibration.

[2334] Emotion recognition function

[2335] Step 1:

[2336] Device: When a user performs voice input, the device captures their voice tone and facial expressions through the camera and microphone.

[2337] Step 2:

[2338] Terminal: Sends the acquired data to the emotion engine.

[2339] Step 3:

[2340] Server: The emotion engine analyzes the data and identifies the user's emotional state.

[2341] Step 4:

[2342] Server: Generates and selects appropriate responses and educational content based on emotional state.

[2343] Step 5:

[2344] Server: Sends the generated response and educational content to the terminal as audio or text data.

[2345] Step 6:

[2346] Terminal: Presents received data to the user.

[2347] Specific user examples:

[2348] When a user says, "I'm tired today," the device sends the voice data to the emotion engine, and the server, based on the analysis results, generates a response such as, "You seem tired today, so you should rest a little," and plays it back as audio.

[2349] This should clarify the specific processing steps for each function.

[2350] (Example 2)

[2351] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[2352] Traditional educational support watches offer features such as answering user questions, summarizing lesson content, and supporting English conversation, but they have been insufficient in terms of safety and emotion recognition. Therefore, there is a need for features that can track a child's location, detect abnormal situations, and analyze the user's emotions to provide appropriate responses. Furthermore, the challenge lies in integrating these functions into a single device to achieve safer and more personalized educational support.

[2353] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[2354] In this invention, the server includes speech recognition means, question analysis means, answer generation means, speech generation means, anomaly detection means, and emotion analysis means. This enables the immediate provision of answers to user questions, summaries of lesson content, English conversation support, safety management through location information monitoring, and appropriate responses according to the user's emotional state.

[2355] "Voice acquisition means" refers to devices such as microphones used to collect the user's voice.

[2356] "Speech recognition means" refers to speech recognition engines or software used to convert collected speech data into text data.

[2357] "Question analysis means" refers to natural language processing technology and analysis algorithms for analyzing user questions transcribed into text by speech recognition means and understanding their intent.

[2358] "Answer generation means" refers to an engine or algorithm for generating appropriate answers based on the content analyzed by the question analysis means.

[2359] "Voice generation means" refers to a speech synthesis engine or software for converting the generated text-formatted response into speech data.

[2360] "Answer presentation means" refers to a speaker or output device for playing back audio data created by the audio generation means to the user.

[2361] "Location information acquisition means" refers to GPS systems and other location detection technologies used to obtain the user's current location.

[2362] An "anomaly detection means" is an algorithm and system for detecting abnormal behavior or conditions by analyzing data collected by a location information acquisition means.

[2363] "Emotion analysis means" refers to software and hardware that analyzes a user's voice and facial expressions to identify their emotional state.

[2364] "Means of notifying the user" refer to notification systems that inform the user of detected anomalies or analyzed information, such as vibration, voice notifications, or display notifications.

[2365] System Overview

[2366] This invention relates to a wristwatch-type device for supporting education while ensuring safety. This device is voice-operable by the user and features voice recognition, question analysis, answer generation, speech synthesis, lesson content summarization, English conversation support, monitoring, and emotion recognition. Because this device provides numerous educational support functions in an integrated manner, it creates a safe learning environment for the user.

[2367] Hardware and software to be used

[2368] This system uses the following main hardware and software:

[2369] Microphone: Collects the user's voice.

[2370] Speaker: Plays the generated audio data.

[2371] Camera: Collects user facial expressions (for emotion recognition function).

[2372] GPS system: Acquires location information.

[2373] Accelerometer: Detects user movements.

[2374] Speech recognition engine: Converts speech data into text data. Example: Google Speech-to-Text.

[2375] Text-to-speech engine: Converts text data into speech data. Example: Google Text-to-Speech.

[2376] Natural language processing models: Analyze the intent of a question and generate appropriate answers. Example: OpenAI's GPT model.

[2377] Emotion analysis engine: Analyzes the user's emotions from their voice and facial expressions.

[2378] Database: Stores data such as generated summary results.

[2379] Program processing

[2380] Question answering function

[2381] When a user inputs a question by voice, the device collects this audio and converts it into text data using a speech recognition engine. The text data is sent to a server, which uses a natural language processing model to analyze the question, generates an appropriate answer, and converts it back into audio data using a speech synthesis engine. This audio data is then sent to the device and played back to the user.

[2382] Specific example: When a user asks, "Why is the sky blue?", the device plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[2383] Lesson content summary function

[2384] Audio during lessons is collected by microphones and converted to text at regular intervals using a speech recognition engine. This text data is processed on a server and summarized using a natural language processing model. The summarization results are stored in a database and displayed on the user's terminal screen upon request.

[2385] Specific example: If a user requests "Tell me what we learned in today's lesson" after class, the device will display a summary of what we learned in today's lesson.

[2386] English conversation support function

[2387] When a user initiates a question or conversation in English, audio data is acquired and converted to text by a speech recognition engine. This text data is processed on a server, where an English conversation model is used to generate an appropriate English response, which is then converted back into audio data using a speech synthesis engine and played back to the user.

[2388] Specific example: When a user asks "How is the weather today?", the system plays the answer "The weather is sunny today." in audio.

[2389] Monitoring function

[2390] The system uses a GPS system and an accelerometer to acquire the user's location and movements, and transmits this information to a server. The server analyzes the data to detect anomalies. If an anomaly is detected, an alert is sent to the parent or emergency services, and the user is notified via the device.

[2391] Specific example: If a user falls, the device detects the abnormality and sends an SMS message to the parent saying, "Your child has fallen. Their current location is XX."

[2392] Emotion recognition function

[2393] The system captures the user's voice and facial expressions through a camera and microphone and sends them to an emotion analysis engine. The server analyzes the emotions and generates appropriate responses or educational content based on the results. The generated responses are converted into audio data using a speech synthesis engine and played back to the user.

[2394] Specific example: When a user says, "I'm tired today," a voice response is played saying, "You seem tired today, so you should rest a little."

[2395] Example of a prompt

[2396] Question answer function: A user asks, "Why is the sky blue?"

[2397] Lesson summary function: After the lesson, the user can request, "Please explain what we covered in today's lesson."

[2398] English conversation support function: The user asks "How is the weather today?".

[2399] Emotion recognition function: The user says, "I'm tired today."

[2400] These features allow children to learn safely and parents to watch over them with peace of mind. Furthermore, they provide an effective learning environment through English conversation support and lesson review, and enable more personalized educational support through emotion recognition.

[2401] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2402] Question answering function

[2403] Step 1:

[2404] The user inputs the question by voice. The device collects the user's voice using a microphone and saves it as audio data.

[2405] Input: User's voice: "Why is the sky blue?"

[2406] Output: Audio data

[2407] Step 2:

[2408] The collected audio data is converted into text data using a speech recognition engine (e.g., Google Speech-to-Text).

[2409] Input: Audio data

[2410] Output: Text data "Why is the sky blue?"

[2411] Specific operation: The speech recognition engine analyzes the audio data and generates the corresponding text.

[2412] Step 3:

[2413] Send the converted text data to the server.

[2414] Input: Text data "Why is the sky blue?"

[2415] Output: Text data sent to the server

[2416] Specific action: The terminal sends text data to the server.

[2417] Step 4:

[2418] The server receives text data and uses a natural language processing model (e.g., OpenAI's GPT model) to analyze the intent of the question.

[2419] Input: Text data "Why is the sky blue?"

[2420] Output: Analysis results (the intent of the question)

[2421] Specific operation: The natural language processing model analyzes the context of the question and extracts the information necessary to generate an appropriate answer.

[2422] Step 5:

[2423] Based on the question analysis results, the response generation means generates an appropriate response.

[2424] Input: Analysis results

[2425] Output: Answer: "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[2426] Specific operation: The response generation mechanism accesses the database and extracts or generates appropriate responses.

[2427] Step 6:

[2428] The generated responses are converted into audio data using a speech synthesis engine (e.g., Google Text-to-Speech).

[2429] Input: Response (text data)

[2430] Output: Response (audio data)

[2431] Specific operation: The speech synthesis engine converts text data into speech data.

[2432] Step 7:

[2433] Send audio data to the device.

[2434] Input: Response (audio data)

[2435] Output: Audio data sent to the terminal

[2436] Specific operation: The server sends the generated audio data to the terminal.

[2437] Step 8:

[2438] The device plays back the received audio data and provides the user with an answer.

[2439] Input: Response (audio data)

[2440] Output: Audio played to the user

[2441] Specific action: The device's speaker plays audio data, and the user listens to the response.

[2442] Lesson content summary function

[2443] Step 1:

[2444] Audio during lessons is collected using a microphone and saved as audio data.

[2445] Input: Audio from class

[2446] Output: Audio data

[2447] Specific operation: The microphone continuously collects audio during class and stores it on the device.

[2448] Step 2:

[2449] The collected audio data is converted into text data at regular intervals using a speech recognition engine.

[2450] Input: Audio data

[2451] Output: Text data

[2452] Specific operation: The speech recognition engine analyzes the collected audio data and generates corresponding text.

[2453] Step 3:

[2454] Send the converted text data to the server.

[2455] Input: Text data

[2456] Output: Text data sent to the server

[2457] Specific action: The terminal sends text data to the server.

[2458] Step 4:

[2459] The server receives text data and summarizes it using a natural language processing model.

[2460] Input: Text data

[2461] Output: Summary results

[2462] Specific operation: A natural language processing model analyzes text data, extracts key points, and generates a summary.

[2463] Step 5:

[2464] Save the summary results to the database.

[2465] Input: Summary result

[2466] Output: Saved summary data

[2467] Specific action: Record the summary results in a database and save them so they can be retrieved later.

[2468] Step 6:

[2469] The system retrieves a summary result in response to the user's request and displays it on the device screen.

[2470] Input: User Request

[2471] Output: Summary results displayed on the screen

[2472] Specific operation: Retrieve the summary results from the database and display them on the terminal screen.

[2473] English conversation support function

[2474] Step 1:

[2475] The microphone collects audio from the user to initiate questions or conversations in English.

[2476] Input: English audio

[2477] Output: Audio data

[2478] Specific action: The user asks a question in English, such as "How is the weather today?".

[2479] Step 2:

[2480] The collected audio data is converted into text data using a speech recognition engine.

[2481] Input: Audio data

[2482] Output: Text data

[2483] Specific operation: The speech recognition engine analyzes the audio data and generates the corresponding text.

[2484] Step 3:

[2485] Send the converted text data to the server.

[2486] Input: Text data

[2487] Output: Text data sent to the server

[2488] Specific action: The terminal sends text data to the server.

[2489] Step 4:

[2490] The server receives text data and generates a response using an English conversation model.

[2491] Input: Text data

[2492] Output: Response (text data)

[2493] Specific operation: The English conversation model analyzes text data and generates appropriate English responses.

[2494] Step 5:

[2495] The generated responses are converted into speech data using a speech synthesis engine.

[2496] Input: Response (text data)

[2497] Output: Response (audio data)

[2498] Specific operation: The speech synthesis engine converts text into speech data.

[2499] Step 6:

[2500] Send audio data to the device.

[2501] Input: Response (audio data)

[2502] Output: Audio data sent to the terminal

[2503] Specific operation: The server sends the generated audio data to the terminal.

[2504] Step 7:

[2505] The device plays back the received audio data and provides the user with an answer.

[2506] Input: Response (audio data)

[2507] Output: Audio played to the user

[2508] Specific action: The device's speaker plays audio data, and the user listens to the response.

[2509] Monitoring function

[2510] Step 1:

[2511] The system uses a GPS system and an accelerometer to acquire the user's location and movements.

[2512] Input: None (Sensor acquisition itself)

[2513] Output: Location information and motion data

[2514] Specific actions: Continuously acquire location information and motion data to prepare for anomaly detection.

[2515] Step 2:

[2516] The acquired data is sent to the server.

[2517] Input: Location information and motion data

[2518] Output: Data sent to the server

[2519] Specific action: The device sends the data it has acquired to the server.

[2520] Step 3:

[2521] The server analyzes the received data and detects abnormal conditions.

[2522] Input: Location information and motion data

[2523] Output: Analysis result (normal / abnormal)

[2524] Specific operation: The data analysis engine analyzes the data and detects abnormal conditions.

[2525] Step 4:

[2526] If an anomaly is detected, an alert will be sent to parents or emergency services.

[2527] Input: Analysis results (abnormal)

[2528] Output: Alert notification

[2529] Specific operation: The server sends alerts via SMS or email.

[2530] Step 5:

[2531] The device notifies the user of an anomaly.

[2532] Input: None (based on alert notifications from the server)

[2533] Output: Voice notifications, vibration, and display.

[2534] Specific actions: The device will notify the user of an anomaly using sound, vibration, or a display message.

[2535] Emotion recognition function

[2536] Step 1:

[2537] The system captures the user's voice and facial expressions through a camera and microphone.

[2538] Input: Voice and facial expressions

[2539] Output: Audio data and facial expression data

[2540] Specific operation: The microphone collects audio, and the camera collects facial expressions.

[2541] Step 2:

[2542] The acquired data is sent to the emotion analysis engine.

[2543] Input: Voice data and facial expression data

[2544] Output: Emotion analysis results

[2545] Specific operation: The emotion analysis engine analyzes voice and facial expressions to identify emotions.

[2546] Step 3:

[2547] The server generates an appropriate response based on the sentiment analysis results.

[2548] Input: Sentiment analysis results

[2549] Output: Response (text data)

[2550] Specific operation: The response generation engine generates an appropriate response based on the analyzed emotions.

[2551] Step 4:

[2552] The generated response is converted into speech data using a speech synthesis engine.

[2553] Input: Response (text data)

[2554] Output: Response (audio data)

[2555] Specific operation: The speech synthesis engine converts text data into speech data.

[2556] Step 5:

[2557] The response audio data is sent to the terminal.

[2558] Input: Response (voice data)

[2559] Output: Audio data sent to the terminal

[2560] Specific operation: The server sends the generated audio data to the terminal.

[2561] Step 6:

[2562] The device plays back the received audio data and provides a response to the user.

[2563] Input: Response (voice data)

[2564] Output: Audio played to the user

[2565] Specific action: The device's speaker plays audio data, and the user listens to the response.

[2566] The above is a detailed explanation of the program's processing flow.

[2567] (Application Example 2)

[2568] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[2569] Conventional educational support devices lack features to ensure children's safety, such as monitoring functions, and personalized educational support that responds to their emotions. Furthermore, they lack the ability to track children's location in real time and detect anomalies, making rapid response in emergencies difficult. This invention aims to solve these problems and provide safer, more secure, and more effective educational support.

[2570] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[2571] In this invention, the server includes voice acquisition means, voice recognition means, question analysis means, answer generation means, voice generation means, answer presentation means, location information acquisition means, location data transmission means, anomaly detection means, alert transmission means, emotion acquisition means, emotion analysis means, and system response generation means. This makes it possible to provide individually optimized educational support in real time while ensuring user safety.

[2572] A "voice acquisition means" is a device that captures the user's voice and stores it as digital data.

[2573] "Speech recognition means" refers to technology that converts acquired speech data into text data.

[2574] A "question analysis tool" is an algorithm that analyzes text data obtained by a speech recognition tool to understand the intent and content of a question.

[2575] "Answer generation means" refers to a technology that generates appropriate answers based on the results obtained from question analysis means.

[2576] "Speech generation means" refers to technology that converts generated text data into speech data.

[2577] A "response presentation means" is a device that plays back the generated audio data and presents it to the user.

[2578] "Location information acquisition means" refers to technologies that use GPS or other technologies to acquire the user's current location.

[2579] "Location data transmission means" refers to technology that transmits acquired location information to a server or other terminal.

[2580] An "anomaly detection method" is a technology that detects user behavior or states that are different from the norm.

[2581] An "alert transmission method" is a technology for quickly notifying of anomalies detected by anomaly detection methods.

[2582] "Emotion acquisition methods" refer to technologies that acquire emotions from the user's voice and facial expressions.

[2583] "Emotional analysis means" refers to technology that analyzes acquired emotional data to determine the user's emotional state.

[2584] "System response generation means" refers to a technology that generates appropriate responses or messages based on data obtained from emotion analysis means.

[2585] Explanation of program generation and processing

[2586] This invention is a system that provides individually optimized educational support in real time while ensuring user safety, and utilizes the following main hardware and software.

[2587] Hardware to use

[2588] smartphone

[2589] Smart Glasses

[2590] GPS sensor

[2591] Accelerometer

[2592] microphone

[2593] camera

[2594] Software to use

[2595] Google Speech API (speech recognition engine)

[2596] TensorFlow (natural language processing engine)

[2597] Microsoft Azure Emotion API (emotion recognition engine)

[2598] Google Maps API (location tracking)

[2599] Twilio API (Alert Notifications)

[2600] Detailed processing of the system

[2601] Location tracking

[2602] The server acquires location information from GPS sensors installed in smartphones and smart glasses. This acquired location information is transmitted to the server in real time via a location data transmission device. The server analyzes the received location data and detects abnormal movements using an anomaly detection device. If an anomaly is detected, an alert transmission device is used to notify parents or administrators via SMS or push notification.

[2603] Emotion monitoring

[2604] The server uses the microphone and camera of a smartphone or smart glasses to acquire the user's voice and facial expressions using emotion acquisition means. This data is analyzed by emotion analysis means to determine the user's emotional state. Next, system response generation means is used to generate an appropriate response based on the emotional state. This response is converted into audio data by voice generation means and presented to the user.

[2605] Learning support

[2606] The server acquires the user's voice question using a voice acquisition means and converts it into text data using a speech recognition means. The text data is analyzed by a question analysis means, and based on the results, an answer is generated by an answer generation means. The generated answer is converted into speech by a speech generation means and presented to the user through an answer presentation means.

[2607] Specific examples and prompt statements

[2608] Examples of emotion recognition

[2609] When a user says, "I'm tired today," the device uses an emotion engine to analyze the user's tone of voice and facial expression. Based on the analysis, the server generates a response such as, "You seem tired today, so you should take a rest," and plays it back as audio.

[2610] Example of a prompt:

[2611] Voice input: "Something unpleasant happened at school today."

[2612] Facial expression input: Image data (child's facial expression)

[2613] Examples of learning support

[2614] User question: "Why is the sky blue?"

[2615] Response: "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[2616] Example of a prompt:

[2617] "Why is the sky blue?"

[2618] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2619] Step 1:

[2620] The device uses a microphone to acquire the user's voice. The acquired voice data is stored as a digital signal. In this case, the voice data is the input.

[2621] Step 2:

[2622] The device converts the acquired audio data, obtained using speech recognition (Google Speech API), into text data. This conversion results in the output of the text data.

[2623] Step 3:

[2624] The terminal sends the converted text data to the server. Here, the text data is passed to the server as input.

[2625] Step 4:

[2626] The server analyzes the text data using a question analysis tool (TensorFlow) to understand the intent of the question. The intent of the question is output as the analysis result.

[2627] Step 5:

[2628] The server generates an appropriate answer using an answer generation mechanism based on the question analysis results. The generated answer is output as text data.

[2629] Step 6:

[2630] The server converts the generated text response into audio data using a speech generation tool (Google Speech API). This conversion results in the output of the audio data.

[2631] Step 7:

[2632] The server sends the generated audio data to the terminal. Here, the audio data is passed to the terminal as input.

[2633] Step 8:

[2634] The terminal presents the user with the answer to the question by playing the audio data through the answer presentation means. Here, the audio is presented to the user.

[2635] Step 9:

[2636] The device uses a GPS sensor to obtain the user's location information. This information is stored as location data. In this case, location data is the input.

[2637] Step 10:

[2638] The device sends location data to the server. The location data is passed to the server as input.

[2639] Step 11:

[2640] The server analyzes location data using anomaly detection mechanisms to detect unusual movement. If an anomaly is detected, anomaly data is output.

[2641] Step 12:

[2642] When abnormal data is detected, the server sends a notification to the parent or administrator using the alert sending mechanism (Twilio API). This notification triggers an alert.

[2643] Step 13:

[2644] The device acquires user voice and facial expression data using emotion acquisition methods. This emotion data is the input.

[2645] Step 14:

[2646] The server analyzes emotional data using an emotion analysis tool (Microsoft Azure Emotion API) to determine the user's emotional state. The emotion analysis results are then output.

[2647] Step 15:

[2648] The server generates an appropriate response using a system response generation tool (TensorFlow) based on the sentiment analysis results. This response is output as text data.

[2649] Step 16:

[2650] The server converts the generated text response into audio data using a speech generation tool (Google Speech API). This conversion results in the output of the audio data.

[2651] Step 17:

[2652] The server sends the generated audio data to the terminal. Here, the audio data is passed to the terminal as input.

[2653] Step 18:

[2654] The device plays audio data to the user, providing a response that corresponds to their emotional state. Here, the audio is presented to the user.

[2655] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[2656] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2657] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[2658] [Fourth Embodiment]

[2659] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[2660] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2661] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2662] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[2663] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[2664] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[2665] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[2666] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[2667] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[2668] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2669] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2670] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[2671] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2672] This invention relates to a wristwatch-type device for supporting education while ensuring the safety of children. Because this device operates locally without an internet connection, it offers high security. Furthermore, it provides educational support and monitoring functions using a system equipped with voice recognition and natural language processing.

[2673] The present invention has the following main features:

[2674] 1. Question answering function

[2675] 2. Function to summarize lesson content

[2676] 3. English conversation support function

[2677] 4. Monitoring function

[2678] Question answering function

[2679] Terminal:

[2680] It is equipped with a microphone for users to input questions by voice. When a user asks a question, this voice data is converted into text data by the speech recognition engine in the device.

[2681] server:

[2682] The system receives text data and analyzes the questions. The analysis process uses natural language processing techniques to understand the intent of the questions and generates appropriate answers from a local database.

[2683] server:

[2684] The generated response is converted into audio data using a speech synthesis engine. The converted audio data is then sent to the terminal.

[2685] Terminal:

[2686] The system plays back the received audio data to the user and provides answers to their questions.

[2687] Specific user examples:

[2688] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[2689] Lesson content summary function

[2690] Terminal:

[2691] Audio from the classroom is captured via a micro-fan and saved as audio data.

[2692] Terminal:

[2693] At regular intervals, audio data is converted to text using a speech recognition engine as a batch process. This text data is then sent to the server.

[2694] server:

[2695] The received text data is summarized using natural language processing techniques. The summarization results are stored in a local database.

[2696] Terminal:

[2697] In response to user requests, the system retrieves a summary of the lesson and displays it on the screen.

[2698] Specific user examples:

[2699] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson is..."

[2700] English conversation support function

[2701] Terminal:

[2702] It is equipped with a microphone for users to ask questions or initiate conversations in English. It acquires voice data, and a speech recognition engine converts it to text.

[2703] server:

[2704] The system receives text data and generates appropriate responses using an English conversation model. The generated responses are then converted into speech data using a speech synthesis engine.

[2705] Terminal:

[2706] The generated audio data is played for the user.

[2707] Specific user examples:

[2708] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[2709] Monitoring function

[2710] Terminal:

[2711] It is equipped with a GPS system and an accelerometer, which continuously acquire the user's location and movement information. This data is transmitted to a server.

[2712] server:

[2713] The system analyzes the received data and detects anomalies. If an anomaly is detected, it sends an alert to the parent system or emergency services.

[2714] Terminal:

[2715] It receives alerts and notifies the user via voice or vibration as needed.

[2716] Specific user examples:

[2717] If a user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is [location]."

[2718] These features allow children to learn safely, and parents can watch over them with peace of mind. Furthermore, they provide an effective learning environment through English conversation support and lesson review.

[2719] The following describes the processing flow.

[2720] Question answering function

[2721] Step 1:

[2722] Terminal: The user speaks a voice question into the microphone.

[2723] Step 2:

[2724] Terminal: The speech recognition engine converts the user's voice into text data.

[2725] Step 3:

[2726] Terminal: Sends the converted text data to the server.

[2727] Step 4:

[2728] Server: Receives text data and analyzes the question using a natural language processing algorithm.

[2729] Step 5:

[2730] Server: Based on the analysis results, it searches for and generates the appropriate answer from the local database.

[2731] Step 6:

[2732] Server: Converts the generated response text into speech data using a speech synthesis engine.

[2733] Step 7:

[2734] Server: Sends audio data to the terminal.

[2735] Step 8:

[2736] Terminal: Plays the received audio data through the speaker and presents the answer to the user.

[2737] Lesson content summary function

[2738] Step 1:

[2739] Terminal: This device uses a micro-fan to capture audio during class and saves it as audio data.

[2740] Step 2:

[2741] Terminal: Uses a speech recognition engine to convert speech data into text at regular intervals.

[2742] Step 3:

[2743] Terminal: Sends the converted text data to the server.

[2744] Step 4:

[2745] Server: Receives text data and summarizes it using a natural language processing algorithm.

[2746] Step 5:

[2747] Server: Saves the summary results to the local database.

[2748] Step 6:

[2749] Terminal: Receives requests from users and requests summary results from the server.

[2750] Step 7:

[2751] Server: Sends the summary results to the terminal.

[2752] Step 8:

[2753] Terminal: Displays the received summary results on the screen.

[2754] English conversation support function

[2755] Step 1:

[2756] Device: Speak your voice question in English into the microphone.

[2757] Step 2:

[2758] Terminal: The speech recognition engine converts the speech data into text data.

[2759] Step 3:

[2760] Terminal: Sends the converted text data to the server.

[2761] Step 4:

[2762] Server: Receives text data and generates appropriate English responses based on an English conversation model.

[2763] Step 5:

[2764] Server: Converts the generated English text into speech data using a speech synthesis engine.

[2765] Step 6:

[2766] Server: Sends audio data to the terminal.

[2767] Step 7:

[2768] Terminal: Plays the received audio data through the speaker and provides the user with a response.

[2769] Monitoring function

[2770] Step 1:

[2771] Terminal: Periodically acquires the user's location information and movement using a GPS system and accelerometer.

[2772] Step 2:

[2773] Terminal: Sends acquired data to the server.

[2774] Step 3:

[2775] Server: Monitors location information and operational data in real time and detects anomalies.

[2776] Step 4:

[2777] Server: If an anomaly is detected, it sends an alert to the parent or emergency services.

[2778] Step 5:

[2779] Device: Notifies the user via voice alerts or vibration.

[2780] This should clarify the specific processing steps for each function.

[2781] (Example 1)

[2782] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2783] Traditional educational support devices required an internet connection, posing a high security risk. Furthermore, few devices integrated child safety monitoring and educational support functions, making it difficult for parents to constantly monitor their children with peace of mind. Additionally, educational support features such as lesson summaries and English conversation support were often insufficient.

[2784] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[2785] In this invention, the server includes a voice acquisition means, a voice recognition means, a question analysis means, an answer generation means, a voice generation means, an answer presentation means, a location information acquisition means, a motion detection means, an anomaly detection means, and an anomaly notification means. This allows for secure operation even without an internet connection and enables the integrated provision of educational support and child monitoring functions. Furthermore, by providing functions such as lesson content summarization and English conversation support, the effectiveness of children's learning can be enhanced.

[2786] "Voice acquisition means" refers to a device or function for picking up the voice emitted by the user.

[2787] "Speech recognition means" refers to a technology or device that converts acquired speech into text data.

[2788] "Question analysis means" refers to a technology or device for analyzing user intent from transcribed audio data.

[2789] "Answer generation means" refers to a technology or device that generates an appropriate answer based on the analysis results.

[2790] "Speech generation means" refers to a technology or device that converts generated text-based responses into speech.

[2791] "Answer presentation means" refers to a technology or device for presenting generated audio to the user.

[2792] "Location information acquisition means" refers to technology or devices that use GPS or similar systems to acquire the user's current location.

[2793] "Motion detection means" refers to a technology or device that detects the user's movement status using an acceleration sensor or the like.

[2794] An "anomaly detection means" is a technology or device for detecting abnormal situations from user activity information or location information.

[2795] An "anomaly notification means" is a technology or device for sending an alert when an anomaly is detected.

[2796] "Means for acquiring audio of lecture content" refers to a device or function for capturing audio during lectures.

[2797] "Means for converting audio of lecture content into text" refers to technology or equipment that converts acquired audio into text data.

[2798] "Means of summarizing text" refers to techniques or devices that analyze text data, extract only the important information, and summarize it.

[2799] "Means for saving the summary results" refers to a technology or device for saving the summarized text data to a storage device.

[2800] "Means for presenting summary results" refers to a technology or device for presenting summarized text data to a user.

[2801] "Means for acquiring English conversation audio" refers to a device or function for picking up English audio spoken by a user.

[2802] "Means for converting spoken English into text" refers to a technology or device that converts acquired spoken English into text data.

[2803] "Means for generating responses using an English conversation model" refers to a technology or device that analyzes English text data to generate appropriate English responses.

[2804] "Means for converting generated responses into speech" refers to a technology or device that converts generated text-based responses into speech.

[2805] "Means for playing back audio" refers to the technology or device used to play back generated audio data for the user.

[2806] This invention relates to an arm-worn device for supporting education while ensuring the safety of children. Because this device operates locally without an internet connection, it offers high security. Furthermore, it utilizes speech recognition and natural language processing technologies to provide multiple educational support and monitoring functions.

[2807] Question answering function

[2808] The device of this invention is equipped with a microphone for the user to input questions by voice. The voice data is converted into text data by a speech recognition engine (e.g., Google Speech-to-Text API) within the device. This text data is sent to a server.

[2809] The server receives text data and analyzes the question using natural language processing techniques (e.g., the NLTK library). Based on the analysis, it generates an appropriate answer from a local database. The generated answer is converted into audio data by a speech synthesis engine (e.g., Google Text-to-Speech API) and sent to the terminal. The terminal plays this audio data and provides the answer to the user.

[2810] Specific example:

[2811] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[2812] Lesson content summary function

[2813] Audio during lessons is captured through the device's microphone and saved as audio data. This audio data is converted into text data at regular intervals by a speech recognition engine (e.g., Google Speech-to-Text API). The converted text data is then sent to a server.

[2814] The server summarizes the received text data using natural language processing techniques (e.g., the NLTK library). The summary is stored in a local database. The user can retrieve this summary upon request and display it on their terminal screen.

[2815] Specific example:

[2816] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson is..."

[2817] English conversation support function

[2818] The user uses the device's microphone to ask a question or initiate a conversation in English. The audio data is converted into text data by a speech recognition engine (e.g., Google Speech-to-Text API). This text data is then sent to the server.

[2819] The server receives text data and generates an appropriate response using an English conversation model (e.g., OpenAI's GPT-3). The generated response is converted into audio data by a speech synthesis engine (e.g., Google Text-to-Speech API) and sent to the terminal. The terminal plays this audio data and provides the response to the user.

[2820] Specific example:

[2821] The user asks, "How is the weather today?" The device recognizes the voice, and the server responds, "The weather is sunny today," and plays it back aloud.

[2822] Monitoring function

[2823] The device of the present invention is equipped with a GPS system and an accelerometer, and continuously acquires the user's location information and movements. This data is transmitted to a server.

[2824] The server analyzes the received data and, if it detects an anomaly, sends an alert to the parent or emergency services. The device receives the alert and notifies the user via sound or vibration.

[2825] Specific example:

[2826] If the user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is [location]."

[2827] Examples of prompt statements

[2828] The following are specific examples of prompt statements to be input to a generative AI model:

[2829] 1. Prompts for the question function:

[2830] User: "Why is the sky blue?"

[2831] Prompt: "Explain why the sky appears blue."

[2832] 2. Prompts for English conversation support function:

[2833] User: "How is the weather today?"

[2834] Prompt: "Please describe the weather today."

[2835] Based on these concrete examples, children can learn safely, and parents can watch over them with peace of mind. Furthermore, it becomes possible to provide an effective learning environment through English conversation support and review of lessons.

[2836] The flow of the specific processing in Example 1 will be explained using Figure 11.

[2837] Question answering function

[2838] Step 1:

[2839] The user enters the question by voice.

[2840] Input: User's voice question

[2841] Specific action: The user asks the question, "Why is the sky blue?"

[2842] Step 2:

[2843] The device acquires voice data and converts it into text data using a speech recognition engine.

[2844] Input: User's voice data

[2845] Data processing: Convert speech to text using a speech recognition engine (e.g., Google Speech-to-Text API).

[2846] Output: Text data ("Why is the sky blue?")

[2847] Step 3:

[2848] The terminal sends the converted text data to the server.

[2849] Input: Text data

[2850] Data processing: Uses HTTP requests

[2851] Output: Sending text data to the server

[2852] Step 4:

[2853] The server receives text data and analyzes the question using natural language processing technology.

[2854] Input: Text data ("Why is the sky blue?")

[2855] Data processing: Analyze the intent of the question using natural language processing techniques (e.g., NLTK library).

[2856] Output: Analysis results

[2857] Step 5:

[2858] The server generates the appropriate answer from the local database based on the analysis results.

[2859] Input: Analysis results

[2860] Data calculation: Generate answers from a local database.

[2861] Output: Answer text ("The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes.")

[2862] Step 6:

[2863] The server converts the generated response text into audio data and sends it to the terminal.

[2864] Input: Answer text

[2865] Data processing: Convert to audio data using a speech synthesis engine (e.g., Google Text-to-Speech API).

[2866] Output: Generation of audio data and transmission to the terminal.

[2867] Step 7:

[2868] The device plays the audio data it received.

[2869] Input: Audio data

[2870] Specific action: Play the answer through the device's speaker (telling the user, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes.")

[2871] Lesson content summary function

[2872] Step 1:

[2873] The device acquires audio during the lesson.

[2874] Input: Audio from class

[2875] Specific action: Audio during class is captured using a microphone.

[2876] Step 2:

[2877] The device saves the audio data.

[2878] Input: Acquired audio data

[2879] Data processing: Saving to internal storage

[2880] Output: Saved audio data

[2881] Step 3:

[2882] The device converts voice data into text data using a speech recognition engine at regular intervals.

[2883] Input: Saved audio data

[2884] Data processing: Convert speech to text using a speech recognition engine (e.g., Google Speech-to-Text API).

[2885] Output: Text data

[2886] Step 4:

[2887] The terminal sends text data to the server.

[2888] Input: Text data

[2889] Data processing: Uses HTTP requests

[2890] Output: Sending text data to the server

[2891] Step 5:

[2892] Summarize the text data received by the server.

[2893] Input: Text data

[2894] Data processing: Summarizing using natural language processing techniques (e.g., NLTK library).

[2895] Output: Summary text data

[2896] Step 6:

[2897] The server sends the summary results to the terminal.

[2898] Input: Summary text data

[2899] Data processing: Uses HTTP requests

[2900] Output: Send summary text to terminal

[2901] Step 7:

[2902] The device displays a summary result in response to a user request.

[2903] Input: User request, summary text data

[2904] Specific operation: When the user requests "Tell me what we covered in today's lesson," a summary is displayed on the screen.

[2905] Output: Summary displayed on the screen ("What we learned in today's lesson is ______")

[2906] English conversation support function

[2907] Step 1:

[2908] The user speaks in English.

[2909] Input: User's English voice

[2910] Specific action: The user speaks in English, saying "How is the weather today?"

[2911] Step 2:

[2912] The device acquires voice data and converts it into text data using a speech recognition engine.

[2913] Input: User's English voice data

[2914] Data processing: Convert speech to text using a speech recognition engine (e.g., Google Speech-to-Text API).

[2915] Output: Text data ("How is the weather today?")

[2916] Step 3:

[2917] The terminal sends text data to the server.

[2918] Input: Text data

[2919] Data processing: Uses HTTP requests

[2920] Output: Sending text data to the server

[2921] Step 4:

[2922] The server generates the response using an English conversation model.

[2923] Input: Text data ("How is the weather today?")

[2924] Data processing: Generate answers using an English conversation model (e.g., OpenAI's GPT-3).

[2925] Output: Answer text ("The weather is sunny today.")

[2926] Step 5:

[2927] The server converts the generated response text into audio data and sends it to the terminal.

[2928] Input: Answer text

[2929] Data processing: Convert to audio data using a speech synthesis engine (e.g., Google Text-to-Speech API).

[2930] Output: Generation of audio data and transmission to the terminal.

[2931] Step 6:

[2932] The device plays the audio data it received.

[2933] Input: Audio data

[2934] Specific action: Play the answer through the device's speaker (telling the user "The weather is sunny today.")

[2935] Monitoring function

[2936] Step 1:

[2937] The device acquires the user's location information and movements.

[2938] Input: GPS and accelerometer data

[2939] Specific operation: The device uses the GPS system and accelerometer to obtain location information and movement.

[2940] Step 2:

[2941] The device sends location information and movement data to the server.

[2942] Input: Location data, motion data

[2943] Data processing: Uses HTTP requests

[2944] Output: Sending data to the server

[2945] Step 3:

[2946] The server analyzes the data and detects anomalies.

[2947] Input: Location data, motion data

[2948] Data processing: Anomaly detection using algorithms

[2949] Output: Anomaly detection result

[2950] Step 4:

[2951] If the server detects an anomaly, it will send an alert.

[2952] Input: Anomaly detection result

[2953] Specific actions: Send an alert via SMS or email to parents or emergency services (e.g., "Your child has fallen. Their current location is [location].")

[2954] Step 5:

[2955] The device receives an alert and sends a notification.

[2956] Input: Alert data

[2957] Specific action: The device notifies the user of an abnormality via sound or vibration.

[2958] Through the processing steps described above, this system can effectively provide educational support while ensuring the safety of children.

[2959] (Application Example 1)

[2960] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2961] There is a need for effective methods to maximize educational effectiveness while ensuring children's safety. Furthermore, a system is required that operates locally without an internet connection, ensuring data security while efficiently handling question answering, lesson summaries, English conversation support, and monitoring. Conventional technologies lack sufficient child anomaly detection and emergency notification features, making it difficult to ensure children's safety.

[2962] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[2963] In this invention, the server includes voice acquisition means, voice recognition means, question analysis means, answer generation means, voice generation means, answer presentation means, location information acquisition means, emergency notification means, and anomaly detection means. This enables question answering based on voice input, acquisition and summarization of educational content, and provision of English conversation support, as well as the ability to detect location information and abnormal operation and provide notifications in emergencies.

[2964] "Sound acquisition means" refers to a device or software that has the function of collecting ambient sounds or voices.

[2965] "Speech recognition means" refers to a device or software that analyzes acquired speech data and converts it into text data.

[2966] "Question analysis means" refers to a device or software that analyzes text data generated by speech recognition means and understands the intent of a question.

[2967] "Answer generation means" refers to a device or software that generates an appropriate answer based on the intent of a question understood by the question analysis means.

[2968] "Voice generation means" refers to a device or software that converts generated text-based responses into voice data.

[2969] "Answer presentation means" refers to a device or software that has the function of playing back audio data generated by the audio generation means to the user.

[2970] "Location information acquisition means" refers to a device or software that acquires a user's location information using GPS or other location measurement technologies.

[2971] An "emergency notification system" is a device or software that has the function of sending notifications to pre-configured contacts when an abnormality or emergency is detected.

[2972] An "anomaly detection means" is a device or software that has the function of detecting abnormal operation or condition based on data from sensors.

[2973] "Means for acquiring audio recordings of class content" refers to a device or software that has the function of recording lectures and discussions during class.

[2974] "Means of converting speech to text" refers to a device or software that analyzes speech data acquired during a lesson and converts it into text data.

[2975] "Means for summarizing text" refers to a device or software that analyzes converted text data, extracts important points, and summarizes them.

[2976] "Means for saving the summary results" refers to a device or software that has the function of saving the summarized information to a storage device.

[2977] "Means for presenting summary results" refers to a device or software that has the function of displaying or playing back saved summary results to the user.

[2978] "Means for acquiring English conversation audio" refers to a device or software that has the function of collecting audio data of English spoken by a user.

[2979] The "means for generating an answer using an English conversation model" is a device or software that analyzes the acquired English text data and uses an English conversation model to generate an appropriate response.

[2980] The "means for converting the generated answer into speech" is a device or software that converts the text-based answer generated by the English conversation model into speech data.

[2981] The "means for playing the speech" is a device or software having a function of playing the converted speech data to the user.

[2982] This invention provides a system for supporting education while ensuring the safety of children. This system has the following main functions.

[2983] 1. Voice acquisition and recognition function

[2984] Terminal:

[2985] It is equipped with a microphone for the user to input questions in voice. When the user asks a question, this voice data is converted into text data by the voice recognition engine in the terminal. As hardware, a general microphone is used.

[2986] Server:

[2987] Receives the text data and analyzes the question. In the analysis process, natural language processing technology is used to understand the intention of the question and generate an appropriate answer from the local database. The software used includes natural language processing libraries (e.g., NLTK, spaCy).

[2988] Specific example:

[2989] The user asks, "Why is the sky blue?" The device recognizes the voice, and the server generates and plays the answer, "The sky appears blue because sunlight is scattered in the atmosphere, and the blue light reaches our eyes."

[2990] Example of a prompt:

[2991] Generate an answer to the question, "Why is the sky blue?"

[2992] 2. Function to summarize educational content

[2993] Terminal:

[2994] Audio during class is captured via a microphone and saved as audio data. A standard microphone is used as the hardware.

[2995] server:

[2996] The received text data is summarized using natural language processing techniques. The summarization results are stored in a local database. The software used includes a text summarization library (e.g., Sumy).

[2997] Specific example:

[2998] After class, the user requests, "Tell me what we learned in today's lesson." The device displays a summary, stating, "What we learned in today's lesson was about the main events of World War II."

[2999] Example of a prompt:

[3000] Generate a summary of what a user would say if they requested, "Tell me what was covered in today's lesson."

[3001] 3. English conversation support function

[3002] Terminal:

[3003] It has a microphone for the user to start questions and conversations in English. It acquires voice data and performs text conversion using a speech recognition engine.

[3004] Server:

[3005] It receives text data and generates an appropriate answer using an English conversation model. It converts the generated answer into voice data using a speech synthesis engine. The software used includes an English conversation model (e.g., GPT-3).

[3006] Specific example:

[3007] The user asks, "How is the weather today?" The terminal recognizes the voice, and the server answers, "The weather is sunny today." and plays it back in voice.

[3008] Examples of prompt sentences:

[3009] Please generate an answer when the user asks, "How is the weather today?"

[3010] 4. Child monitoring function

[3011] Terminal:

[3012] It is equipped with a GPS system and an acceleration sensor, and continuously acquires the user's location information and movements. These data are sent to the server. As hardware, a GPS module (e.g., u-blox) and an acceleration sensor (e.g., MEMS sensor) are used.

[3013] Server:

[3014] It analyzes the received data and detects abnormalities. If an abnormality is detected, it sends an alert to the parent or emergency services. The software used includes an abnormality detection algorithm.

[3015] Specific example:

[3016] If the user falls, the device detects the abnormality and automatically sends an SMS message to the parent saying, "Your child has fallen. Their current location is 35.6895, 139.6917."

[3017] Example of a prompt:

[3018] Please explain how to send an emergency notification to parents if a child falls.

[3019] This allows children to learn safely, and parents can watch over them with peace of mind. Furthermore, it makes it possible to provide an effective learning environment through English conversation support and lesson review.

[3020] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[3021] Step 1:

[3022] Voice data acquisition and recognition

[3023] The device acquires the user's voice data using the microphone. The acquired voice data is sent to the speech recognition engine within the device. The speech recognition engine converts the voice data into text data and generates the resulting text data.

[3024] Input: Audio data

[3025] Output: Text data

[3026] Step 2:

[3027] Question analysis

[3028] The server receives text data from the terminal. The server uses natural language processing technology to analyze the content of the question in the text data and performs data calculations to understand its intent. This determines the intent of the question and generates a result.

[3029] Input: Text data

[3030] Output: Question intent data

[3031] Step 3:

[3032] Generating an answer

[3033] The server generates appropriate answers based on the intent data of the questions, using a local database or a generative AI model. The generated answer text is then used as the primary output.

[3034] Input: Question intent data

[3035] Output: Response text data

[3036] Step 4:

[3037] Convert to speech

[3038] The server sends the response text data to the speech synthesis engine, which converts it into speech data. The converted speech data is then sent to the terminal.

[3039] Input: Answer text data

[3040] Output: Audio data

[3041] Step 5:

[3042] Providing an answer

[3043] The terminal plays the audio data received from the server. Based on the played audio data, the user is presented with an answer.

[3044] Input: Audio data

[3045] Output: Audio output

[3046] Step 6:

[3047] Acquisition of course content

[3048] The device uses its microphone to capture audio during class. The captured audio data is temporarily stored on the device and periodically sent to the speech recognition engine. The speech recognition engine converts the audio data into text data and generates the resulting text data.

[3049] Input: Audio data from class

[3050] Output: Text data from the lecture

[3051] Step 7:

[3052] Text summary

[3053] The server receives the lecture text data and uses natural language processing technology to summarize the text. The summary results are stored in a local database.

[3054] Input: Text data from the class

[3055] Output: Summary data

[3056] Step 8:

[3057] English conversation support

[3058] The device uses its microphone to capture the user's English conversation voice. The captured audio data is sent to the device's speech recognition engine, which converts the audio data into text data.

[3059] Input: English conversation audio data

[3060] Output: English conversation text data

[3061] Step 9:

[3062] English conversation response generation

[3063] The server receives English conversation text data and generates appropriate responses using a...

Claims

1. A voice acquisition means that captures the user's voice and stores or processes it as digital data, A speech recognition means that converts acquired audio data into text data, A question analysis tool that understands the intent of a question recognized as text data and generates an appropriate answer, A response generation means that creates an appropriate answer based on question analysis, A speech generation means that synthesizes the generated text-formatted response as audio data, A means of providing a response to the user by providing the generated audio data, A system that includes this.

2. Methods for obtaining audio of the lesson content, A means of converting speech to text, Means of summarizing text, Means for saving the summary results, It is equipped with means for presenting the summary results. The system according to claim 1.

3. Methods for acquiring English conversation audio, A means of converting speech to text, A means of generating answers using an English conversation model, A means of converting the generated response into speech, Equipped with means for playing sound, The system according to claim 1.

Images