System

Abstract

A system is provided.SOLUTION: The system includes a user authentication means, a voice recognition means, a voice data transmission means, a natural language processing means, a voice synthesis means, a profile data recording means, a learning activity monitoring means, and a data sharing means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's education system, it is difficult to provide educational resources that are appropriate for each individual child, and they often lack motivation to learn. There are also limited means for parents and educators to effectively track children's learning progress and provide appropriate feedback. Another challenge is that companies need to develop new educational services at high development costs. [Means for solving the problem]

[0005] In order to solve these problems, the present invention provides the following means.

[0006] The user authentication means authenticates each user individually and loads a corresponding profile. The voice recognition means and voice data transmission means transmit the user's speech to the server in real time, and the natural language processing means generates an appropriate response. The generated response is output as voice data by the voice synthesis means. The learning activity monitoring means monitors learning activities in real time and records them by the profile data recording means. This data is shared with parents and educators using the data sharing means.

[0007] This will provide an effective, individually optimized educational environment, enable information sharing with parents and educators, and enable companies to offer new educational services at low cost.

[0008] A "user authentication means" is a part of a system that has the functionality to identify a particular user and load profile data corresponding to the user.

[0009] "Speech recognition means" is a part of the system that has the function of receiving a user's voice and converting it into text data.

[0010] The "audio data transmission means" is a part of the system that has the function of transmitting audio data to the server in real time.

[0011] The "natural language processing means" is a part of the system that has the functionality to analyze the text converted by the speech recognition means and generate an appropriate response.

[0012] The "speech synthesis means" is a part of the system that has the function of converting the text response generated by the natural language processing means into voice data and playing it back to the user.

[0013] The "profile data recording means" is a part of the system that has the function of recording and saving the user's learning activities and conversation contents.

[0014] The "learning activity monitoring means" is a part of the system that has the function of monitoring the user's learning activities in real time and recording their progress and content.

[0015] "Data sharing means" is a part of the system that has the functionality to share and notify parents and educators of recorded learning data. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing the system of the present invention will be described below.

[0038] System configuration

[0039] This system mainly consists of a terminal, a server, and a network interface. The terminal is a device that the child directly operates, and the server processes data in the backend. The network interface is responsible for connecting the terminal and the server via communication.

[0040] Program processing

[0041] 1. User Authentication

[0042] Terminal: When the power is turned on, the initialization process begins. First, ambient sound is received through the microphone. The received sound data is sent to the server in real time.

[0043] Server: Analyzes the received voice data and compares it with the voice of pre-registered users. If authentication is successful, it sends the corresponding user profile data to the device.

[0044] Terminal: The received profile data is loaded into memory and a message indicating authentication completion is given to the user.

[0045] 2. Conversations with children

[0046] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[0047] Terminal: Receives audio via microphone and transmits audio data to the server in real time.

[0048] Server: The received voice data is passed through a speech recognition engine and converted into text. The converted text data is sent to a natural language processing engine, which generates an appropriate response (e.g., "Today is math class. Let's practice addition together!").

[0049] Server: Passes the generated response text to the speech synthesis engine to generate voice data, which is then sent to the terminal.

[0050] Terminal: Plays back the received voice data and conveys the response to the child.

[0051] 3. Recording learning activities

[0052] Device: Monitors conversations and learning activities with children in real time. This data is sent to the server.

[0053] Server: Records the transmitted data in a database. Analyzes the recorded data, extracts the user's learning progress and areas of interest, and generates the next learning plan.

[0054] 4. Information sharing

[0055] Server: Aggregates learning data for specific periods and creates reports. Generated notifications are sent to parents and educators via email or a dedicated app (e.g., "Today your child learned the basics of math. Next, we'll introduce multiplication.").

[0056] Specific examples

[0057] Example 1: A child asks Tomo, "What are we going to study today?"

[0058] Terminal: Receives audio and sends it to the server.

[0059] Server: Converts the speech data into text and generates a response ("Today is math time. Let's practice addition together!").

[0060] Device: The generated response is spoken to the child.

[0061] Example 2: Parents keep track of their children's learning progress.

[0062] Server: Collects learning data and creates reports.

[0063] Server: Send the report to the parent's email address.

[0064] In this way, the system of the present invention provides an individually optimized educational environment and enables information sharing with parents and educators.

[0065] The processing flow will be explained below.

[0066] User authentication process

[0067] Step 1:

[0068] Device: When powered on, it begins an initialization process, preparing to receive ambient audio through the microphone.

[0069] Step 2:

[0070] Device: Collects voice data through a microphone and transmits the collected voice data to a server in real time.

[0071] Step 3:

[0072] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If the comparison is successful, authentication of the user is completed.

[0073] Step 4:

[0074] Server: If the authentication is successful, it sends the corresponding user profile data to the terminal.

[0075] Step 5:

[0076] Terminal: Loads the received profile data into memory.

[0077] Step 6:

[0078] Terminal: A message indicating authentication completion is given to the user via voice.

[0079] Handling conversations with children

[0080] Step 1:

[0081] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[0082] Step 2:

[0083] Terminal: Receives audio via microphone and transmits it to the server as audio data in real time.

[0084] Step 3:

[0085] Server: The received voice data is passed through a voice recognition engine and converted into text.

[0086] Step 4:

[0087] Server: Sends the converted text data to a natural language processing engine to generate an appropriate response.

[0088] Step 5:

[0089] Server: Passes the generated response text to the speech synthesis engine to generate voice data.

[0090] Step 6:

[0091] Server: Sends the generated voice data to the terminal.

[0092] Step 7:

[0093] Terminal: Plays back the received voice data and conveys the response to the child.

[0094] Processing learning activity records

[0095] Step 1:

[0096] Device: Monitor your child's conversations and learning activities in real time.

[0097] Step 2:

[0098] Terminal: Sends monitored data to the server.

[0099] Step 3:

[0100] Server: Records the transmitted data in a database.

[0101] Step 4:

[0102] Server: Analyzes the recorded data and extracts the user's learning progress and areas of interest.

[0103] Step 5:

[0104] Server: Generates the next learning plan based on the analysis results.

[0105] Information Sharing Process

[0106] Step 1:

[0107] Server: Aggregates learning data for specific periods and creates reports.

[0108] Step 2:

[0109] Server: Create notifications to share generated reports with parents and educators.

[0110] Step 3:

[0111] Server: Sends reports to parents and educators via email or a dedicated app.

[0112] Processing of specific examples

[0113] Example 1: The process of a child asking Tomo, "What are we going to study today?"

[0114] Terminal: The child's questions are received by a microphone and sent to the server as voice data.

[0115] Server: Analyzes the voice data and generates an appropriate response.

[0116] Server: Converts the generated response into audio data and sends it to the terminal.

[0117] Device: Plays the response as audio and relays it to the child.

[0118] Example 2: The process by which parents keep track of their children's learning progress.

[0119] Server: Aggregates learning data and creates reports.

[0120] Server: Generates and emails the report to the parent.

[0121] Example 1

[0122] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0123] In today's educational environment, there is a demand for systems that provide individually optimized educational support. Conventional systems lack the ability to analyze voice data in real time and generate appropriate feedback, limiting their educational effectiveness. Furthermore, they often do not effectively record and analyze learning progress or share information with parents and educators. This raises concerns that children's learning efficiency may decline.

[0124] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0125] In this invention, the server includes a user authentication means, a voice recognition means, a natural language processing means, a voice synthesis means, a learning activity monitoring means, a data sharing means, a learning activity recording means, a means for generating responses using a generative AI model, and a learning progress analysis means, which enable real-time analysis of voice data, generation of appropriate feedback, recording and analysis of learning progress, and effective information sharing with parents and educators.

[0126] "User authentication means" is a means for verifying whether a user is a registered user by using voice or other biometric information.

[0127] "Speech recognition means" refers to means having the function of converting voice data into text data.

[0128] The "audio data transmission means" is a means having a function of transmitting audio data received by the terminal to the server.

[0129] "Natural language processing means" refers to means that has the function of analyzing text data and generating an appropriate response.

[0130] The "voice synthesis means" is a means having a function of converting text data into voice data.

[0131] "Profile data recording means" means a means having the function of recording and storing profile data of authenticated users.

[0132] The "learning activity monitoring means" is a means having a function of monitoring the user's learning activity in real time and collecting the data.

[0133] "Data sharing means" refers to a means that has the function of sharing collected and analyzed learning data with parents and educators at regular intervals.

[0134] A "means for recording learning activities" is a means that has the function of recording data during learning activities.

[0135] A "means for generating a response using a generative artificial intelligence model" is a means that has the function of generating an appropriate text response to a voice input using a generative AI model.

[0136] The "learning progress analysis means" is a means having a function of analyzing collected learning data and generating learning progress and the next learning plan.

[0137] The present invention is implemented by a system constructed by combining specified hardware and software. Specific embodiments will be described below.

[0138] System configuration

[0139] This system consists of a terminal, a server, and a network interface. The terminal is a device that the child directly operates, and the server processes data in the backend. The network interface connects the terminal and the server via communication.

[0140] Hardware and software used

[0141] Device:

[0142] Microphone: Used to receive audio data.

[0143] Speaker: Used to play audio data.

[0144] Network interface: Responsible for communication with the server.

[0145] server:

[0146] Speech analysis engine: Uses Google Cloud Speech-to-Text to convert voice data into text.

[0147] Natural language processing engine: Uses OpenAI GPT-3 to generate appropriate responses from received text data.

[0148] Speech synthesis engine: Uses Amazon Polly to convert the generated text into speech data.

[0149] Database: MongoDB is used to store profile data and records of learning activities.

[0150] Data sharing system: Use Firebase to share learning progress reports with parents and educators.

[0151] System operating principle

[0152] User authentication

[0153] Device: When powered on, it starts an initialization process and receives ambient audio through its microphone, which is then sent to the server in real time.

[0154] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If authentication is successful, it sends the corresponding user profile data to the device.

[0155] Terminal: The received profile data is loaded into memory and a message indicating authentication completion is given to the user.

[0156] Conversations with children

[0157] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[0158] Terminal: Receives audio via microphone and transmits audio data to the server in real time.

[0159] Server: The received voice data is passed through a speech recognition engine and converted into text. The converted text data is sent to a natural language processing engine, which generates an appropriate response (e.g., "Today is math class. Let's practice addition together!").

[0160] Server: Passes the generated response text to the speech synthesis engine to generate voice data, which is then sent to the terminal.

[0161] Terminal: Plays back the received voice data and conveys the response to the child.

[0162] Recording learning activities

[0163] Device: Monitors conversations and learning activities with children in real time. This data is sent to the server.

[0164] Server: Records the transmitted data in a database. Analyzes the recorded data, extracts the user's learning progress and areas of interest, and generates the next learning plan.

[0165] Information sharing

[0166] Server: Collects learning data at regular intervals and creates reports. Generated notifications are sent to parents and educators via email or a dedicated app. For example, "Today, your child learned the basics of arithmetic. Next, we'll introduce multiplication."

[0167] Specific examples

[0168] Example 1: A child asks Tomo, "What are we going to study today?"

[0169] Terminal: Receives audio and sends it to the server.

[0170] Server: Converts the speech data into text and generates a response ("Today is math time. Let's practice addition together!").

[0171] Device: The generated response is spoken to the child.

[0172] Example 2: Parents keep track of their children's learning progress.

[0173] Server: Collects learning data and creates reports.

[0174] Server: Send the report to the parent's email address.

[0175] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0176] Specific explanation of processing steps

[0177] Step 1: Initialize the system

[0178] Device: When the device is powered on, it begins an initialization process, which checks system memory, verifies network connectivity, and verifies the status of devices such as the microphone and speaker. If successful, the device proceeds to the next step: user authentication.

[0179] Input: Power on

[0180] Output: Initialization complete signal

[0181] Step 2: User authentication

[0182] Device: Once initialization is complete, it receives surrounding sounds through the microphone. For example, when a child says "Hello, Tomo," the voice data is captured. The captured voice data is sent to the server in real time.

[0183] Server: Analyzes the received voice data using a voice analysis engine (e.g., Google Cloud Speech-to-Text). The analysis results are compared with pre-registered user voices. If authentication is successful, the server sends the corresponding user profile data to the device.

[0184] Terminal: Loads the received profile data into memory and generates a message indicating authentication is complete. The message is then output to the speaker, saying "Authentication complete. Hello, [username]!"

[0185] Input: Audio data

[0186] Output: User profile data, voice messages

[0187] Step 3: Talk to your child

[0188] User (child): Once authenticated, the child can talk to Tomo, for example, asking questions like, "What are we going to study today?"

[0189] Terminal: This audio is received by a microphone and sent to the server in real time as audio data.

[0190] Server: The received voice data is converted into text data through a speech recognition engine (e.g., Google Cloud Speech-to-Text). The converted text is sent to a natural language processing engine (e.g., OpenAI GPT-3), which generates an appropriate response (e.g., "Today is math class. Let's practice addition together!").

[0191] Server: Pass the generated response text to a speech synthesis engine (e.g., Amazon Polly) to generate voice data.

[0192] Server: Sends the generated voice data to the terminal.

[0193] Terminal: The received voice data is played back on the speaker and the response is conveyed to the child.

[0194] Input: Child's voice data

[0195] Output: Response voice data

[0196] Step 4: Record your learning activities

[0197] Device: Monitors your child's conversations and learning activities with Tomo in real time (e.g., collects data when answering questions like "What is 1 + 1?").

[0198] Terminal: These data are sent to the server in real time.

[0199] Server: Records the submitted data in a database (e.g., MongoDB). The recorded data is analyzed based on each data point, such as what was learned, the accuracy of the answers, and the learning time.

[0200] Server: From the analysis results, the user's learning progress and areas of interest are extracted and the next learning plan is generated.

[0201] Input: Learning activity data

[0202] Output: Analysis results, learning plan

[0203] Step 5: Information sharing

[0204] Server: Compiles learning data at regular intervals and creates reports. For example, a "weekly learning progress report" might include information such as what was learned, the percentage of correct answers, and the time spent.

[0205] Server: Generated reports are sent to parents and educators via email or a dedicated app (e.g., Firebase). For example, a report might say, "Today, your child learned the basics of math. Next, we'll introduce multiplication."

[0206] Input: Training data

[0207] Output: Learning progress report

[0208] This will improve the efficiency of learning activities and ensure effective collaboration with parents and educators.

[0209] (Application example 1)

[0210] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0211] There is a need for efficient collaboration between workers and robots in factories. However, there is no system in place that allows workers to give voice instructions to robots and have them execute them accurately and quickly. There is also a lack of appropriate support for monitoring the progress and efficiency of work in real time and planning the next work. This is reducing work efficiency in the factory and hindering overall productivity improvements.

[0212] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0213] In this invention, the server includes a user authentication means, a voice recognition means, a voice data transmission means, a natural language processing means, a voice synthesis means, a profile data recording means, a work activity monitoring means, a data sharing means, and a work instruction generation means. This allows an employee to give work instructions to a robot by voice, and the robot to appropriately process and execute those instructions. Furthermore, it is possible to monitor the progress of work in real time and optimize the next work plan, thereby improving overall work efficiency and productivity.

[0214] The "user authentication means" is a means for analyzing the employee's voice data and authenticating them by comparing it with pre-stored profile data.

[0215] "Speech recognition means" is a means for analyzing voice data and converting it into text data.

[0216] The "audio data transmission means" is a means for transmitting audio data to a server in real time.

[0217] The "natural language processing means" is a means for analyzing text data that has been speech-recognized and generating an appropriate response or work instruction.

[0218] The "voice synthesis means" is a means for generating voice data from text data.

[0219] "Profile data recording means" refers to a means for recording and managing profile data of authenticated employees.

[0220] The "work activity monitoring means" is a means for monitoring the work status of the robot in real time and recording the progress.

[0221] "Data sharing means" refers to a means for sharing work progress and analysis results with relevant parties.

[0222] The "work instruction generation means" is a means for analyzing the voice instructions of an employee and generating specific work instructions.

[0223] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing this system will be described below.

[0224] System configuration

[0225] This system mainly consists of a terminal (the control unit of the factory robot), a server, and a network interface. The terminal is a robot that operates in the factory, and the server processes data on the back end. The network interface is responsible for connecting the terminal and server via communication.

[0226] Program processing

[0227] The server implements the system using the following means:

[0228] 1. User authentication method:

[0229] When the terminal (the robot's control unit) is turned on, it receives audio through a microphone and sends the audio data to the server.

[0230] The server analyzes the received voice data using a voice recognition engine and compares it with pre-registered profile data. If authentication is successful, the corresponding employee profile data is sent to the terminal.

[0231] 2. Voice recognition means:

[0232] The server analyzes the voice data sent from the device in real time and converts it into text data, specifically using the SpeechRecognition library.

[0233] 3. Work instruction generation means:

[0234] The server sends the text data to a natural language processing engine to generate appropriate work instructions, which are then sent to the terminal, where the robot begins working based on the instructions.

[0235] 4. Work activity monitoring means:

[0236] The terminal monitors the status of the work being performed in real time and sends the data to the server.

[0237] The server records the received data in a database and analyzes the progress. The analysis results are used to generate the next work plan.

[0238] 5. Data sharing methods:

[0239] The server has a means to share the progress and efficiency of work with the relevant parties. For example, it generates a work report for each specific period and notifies the supervisor.

[0240] Hardware and software used

[0241] Hardware:

[0242] Microphone: A device for receiving employee voice.

[0243] Robot control unit: CPU and memory for processing voice data and receiving and executing work instructions.

[0244] software:

[0245] SpeechRecognition Library: A library for real-time speech recognition in Python.

[0246] Requests library: An HTTP client library for communicating with the server.

[0247] Specific examples

[0248] Employee dialogue: "Hey robot, sort the ingredients."

[0249] Server: Converts voice data into text and generates work instructions such as "Start sorting materials" based on that text data.

[0250] Terminal: Based on the work instructions sent from the server, the robot begins sorting materials.

[0251] Example of prompt input:

[0252] User: "Robot, sort the ingredients."

[0253] Response: "I'll start sorting the ingredients."

[0254] In this way, the system of the present invention supports efficient collaboration between employees and robots within a factory, improving work efficiency and productivity.

[0255] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0256] Step 1:

[0257] The user (employee) gives the robot instructions by voice, for example, "sort the materials." This voice data becomes the input.

[0258] Step 2:

[0259] The terminal (the robot's control unit) receives the user's voice through a microphone and prepares to send this voice data to the server. Here, the voice data is the input, and the output is to convert it into the format required to send it to the server. Specifically, the recorded voice data is converted into a digital format so that it can be sent over the network.

[0260] Step 3:

[0261] The server receives the voice data sent from the terminal. Here, the voice data is the input. The server converts this voice data into text data using a voice recognition system (SpeechRecognition library). The text data is the output.

[0262] Step 4:

[0263] The server sends the converted text data to a natural language processing engine, which analyzes the employee's intent. This text data is the input, and the analyzed work instructions are the output. Specifically, the instruction "sort the materials" is converted into the specific work instruction "sort the materials into the designated locations."

[0264] Step 5:

[0265] The server converts the generated work instruction data into voice data using a voice synthesis engine and prepares to send it to the terminal. Here, the work instruction data is the input and the voice data is the output.

[0266] Step 6:

[0267] The terminal receives the voice data sent from the server and issues appropriate work instructions to the robot. Here, the voice data is the input, and the robot starts a specific task (e.g., sorting materials) based on that voice data.

[0268] Step 7:

[0269] The terminal monitors the robot's work progress in real time and sends the data to the server. Here, the progress data is the input, and converting it into a format that can be sent to the server is the output.

[0270] Step 8:

[0271] The server records the received work progress data in a database and analyzes the progress. The progress data is the input and the analysis results are the output. The next work plan is generated based on the analysis results.

[0272] Step 9:

[0273] The server generates work reports for each specific period and sends them to the relevant parties. Analysis results and progress data are the input, and the work reports are the output. An example of a report might be, "Today, the robot sorted 100 items of material."

[0274] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0275] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing the system of the present invention will be described below.

[0276] System configuration

[0277] This system mainly consists of a terminal, a server, a network interface, and an emotion engine. The terminal is a device that the child directly operates, and the server processes data at the backend. The network interface is responsible for connecting the terminal and the server via communication. The emotion engine provides the functionality to recognize the user's emotions and generate responses based on them.

[0278] Program processing

[0279] 1. User Authentication

[0280] Device: When powered on, it begins an initialization process, preparing to receive ambient audio through the microphone.

[0281] Device: Collects voice data through a microphone and transmits the collected voice data to a server in real time.

[0282] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If the comparison is successful, authentication of the user is completed.

[0283] Server: If the authentication is successful, it sends the corresponding user profile data to the terminal.

[0284] Terminal: Loads the received profile data into memory.

[0285] Terminal: A message indicating authentication completion is given to the user via voice.

[0286] 2. Conversations with children

[0287] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[0288] Terminal: Receives audio via microphone and transmits it to the server as audio data in real time.

[0289] Server: The received voice data is passed through a voice recognition engine and converted into text.

[0290] Server: The converted text data is sent to a natural language processing engine to generate an appropriate response, including emotion recognition using an emotion engine.

[0291] Emotion engine: Analyzes the user's emotions and provides the results to the natural language processing engine.

[0292] Server: Generates response text including emotion recognition results and passes it to the speech synthesis engine.

[0293] Server: Sends the generated voice data to the terminal.

[0294] Terminal: Plays back the received voice data and conveys the response to the child.

[0295] 3. Recording learning activities

[0296] Device: Monitor your child's conversations and learning activities in real time.

[0297] Terminal: Sends monitored data to the server.

[0298] Server: Records the transmitted data in a database, including emotional data.

[0299] Server: Analyzes the recorded data and extracts the user's learning progress and areas of interest. The analysis also takes into account the user's emotional data.

[0300] Server: Generates the next learning plan based on the analysis results.

[0301] 4. Information sharing

[0302] Server: Aggregates learning data for specific periods and creates reports, including emotional data.

[0303] Server: Prepares generated notifications to be shared with parents and educators.

[0304] Server: Sends reports to parents and educators via email or a dedicated app.

[0305] Specific examples

[0306] Example 1: The process of a child asking Tomo, "What are we going to study today?"

[0307] Terminal: The child's questions are received by a microphone and sent to the server as voice data.

[0308] Server: Analyzes the voice data, converts it into text, and recognizes emotions using an emotion engine. Based on the response and emotion data, it generates an optimal response (e.g., "Today is math class. Let's practice addition together!").

[0309] Server: Converts the generated response into audio data and sends it to the terminal.

[0310] Device: Plays the response aloud and relays it to the child.

[0311] Example 2: The process by which parents keep track of their children's learning progress.

[0312] Server: Aggregates learning data and emotion data and creates reports.

[0313] Server: Generates and emails the report to the parent, containing information about the learning content and the child's emotional state.

[0314] In this way, the system of the present invention provides an individually optimized educational environment, and by combining it with emotion recognition, it enables more effective learning support and information sharing with parents and educators.

[0315] The processing flow will be explained below.

[0316] User authentication process

[0317] Step 1:

[0318] Device: When powered on, it begins an initialization process, preparing to receive ambient audio through the microphone.

[0319] Step 2:

[0320] Device: Collects voice data through a microphone and transmits the collected voice data to a server in real time.

[0321] Step 3:

[0322] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If the comparison is successful, authentication of the user is completed.

[0323] Step 4:

[0324] Server: If the authentication is successful, it sends the corresponding user profile data to the terminal.

[0325] Step 5:

[0326] Terminal: Loads the received profile data into memory.

[0327] Step 6:

[0328] Terminal: A message indicating authentication completion is given to the user via voice.

[0329] Handling conversations with children

[0330] Step 1:

[0331] User (child): Talk to the device (e.g., "What are we going to study today?").

[0332] Step 2:

[0333] Terminal: Receives audio via microphone and transmits it to the server as audio data in real time.

[0334] Step 3:

[0335] Server: The received voice data is passed through a voice recognition engine and converted into text.

[0336] Step 4:

[0337] Server: The converted text data is sent to a natural language processing engine to generate an appropriate response, including emotion recognition using an emotion engine.

[0338] Step 5:

[0339] Emotion engine: Analyzes emotions from the user's voice and provides the results to the natural language processing engine.

[0340] Step 6:

[0341] Server: Generates a response text that reflects the emotion recognition results and passes it to the speech synthesis engine (e.g., "Today is math class. Let's practice addition together!").

[0342] Step 7:

[0343] Server: Sends the generated voice data to the terminal.

[0344] Step 8:

[0345] Terminal: Plays back the received voice data and conveys the response to the child.

[0346] Processing learning activity records

[0347] Step 1:

[0348] Device: Monitor your child's conversations and learning activities in real time.

[0349] Step 2:

[0350] Terminal: Sends monitored data to the server.

[0351] Step 3:

[0352] Server: Records the transmitted data in a database, including emotional data.

[0353] Step 4:

[0354] Server: Analyzes the recorded data and extracts the user's learning progress and areas of interest. The analysis also takes into account the user's emotional data.

[0355] Step 5:

[0356] Server: Generates the next learning plan based on the analysis results.

[0357] Information Sharing Process

[0358] Step 1:

[0359] Server: Aggregates learning data for specific periods and creates reports, including emotional data.

[0360] Step 2:

[0361] Server: Prepares generated notifications to be shared with parents and educators.

[0362] Step 3:

[0363] Server: Sends reports to parents and educators via email or a dedicated app.

[0364] Processing of specific examples

[0365] Example 1: The process of a child asking their device, "What are we going to study today?"

[0366] Terminal: The child's questions are received by a microphone and sent to the server as voice data.

[0367] Server: Analyzes the voice data, converts it into text, and recognizes emotions using an emotion engine. Based on the response and emotion data, it generates an optimal response (e.g., "Today is math class. Let's practice addition together!").

[0368] Server: Converts the generated response into audio data and sends it to the terminal.

[0369] Device: Plays the response aloud and relays it to the child.

[0370] Example 2: The process by which parents keep track of their children's learning progress.

[0371] Server: Aggregates learning data and emotion data and creates reports.

[0372] Server: Generates and emails the report to the parent, containing information about the learning content and the child's emotional state.

[0373] Example 2

[0374] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0375] Conventional voice recognition systems are inadequate for complex processing, including user authentication and emotion recognition, making it difficult to optimize the educational environment and provide effective learning support. Furthermore, monitoring and sharing of users' emotion data and learning progress is not smooth, resulting in low quality information provided to parents and educators. There was a need to resolve these issues and provide a more advanced educational support system.

[0376] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0377] In this invention, the server includes a user authentication means, a voice data transmission means, a data analysis means, an emotion recognition means, and a means for generating a response using a generative AI model. This enables accurate authentication based on the user's voice, real-time voice data processing, and the provision of natural responses and learning plans based on emotion analysis. It also makes it possible to realize a system that efficiently collects and analyzes data related to learning progress and emotional state, providing useful information to parents and educators.

[0378] The "user authentication means" is a function that collects the user's voice via a microphone when the device is turned on, sends it to a server for analysis, and compares it with the voice of a previously registered user for authentication.

[0379] The "voice recognition means" is a function that transmits voice data collected by the terminal to a server in real time, analyzes the voice data, and converts it into text.

[0380] The "audio data transmission means" is a function that encrypts the audio data collected by the microphone and transmits it to the server in real time using a communication protocol.

[0381] "Natural language processing means" is a function that uses a natural language processing engine to analyze the converted text data and generate an appropriate response.

[0382] The "voice synthesis means" is a function that uses a voice synthesis engine to convert the generated response text into voice data.

[0383] The "profile data recording means" is a function that loads the profile data of a user into memory when the user is successfully authenticated, and records and manages that information.

[0384] The "learning activity monitoring means" is a function that monitors conversations with children and their learning activities in real time and sends the collected data to a server.

[0385] "Data sharing means" is a function that periodically compiles learning data and emotional data and provides them as reports to parents and educators.

[0386] The "emotion recognition means" is a function that analyzes the user's voice and behavior, determines their emotional state, and provides the results to the natural language processing engine.

[0387] "Means for generating a response using a generative AI model" refers to a function that generates an optimal response using a generative AI model based on received text data and emotion data.

[0388] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing the system of the present invention will be described below.

[0389] System configuration

[0390] This system mainly consists of a terminal, a server, a network interface, and an emotion engine. The terminal is a device that the user directly operates, and the server processes data at the backend. The network interface is responsible for connecting the terminal and the server via communication. The emotion engine provides the functionality to recognize the user's emotions and generate responses based on them.

[0391] Program processing

[0392] This system uses the following hardware and software to process and calculate data.

[0393] Terminal: Contains a microphone to receive the user's voice, a speaker to play the voice, and internal memory. The terminal is responsible for collecting voice data, authenticating the user, and managing profile data.

[0394] Server: Equipped with high-speed computing power for analyzing voice data, natural language processing, emotion recognition, and database management. The server uses a speech recognition engine (e.g., Google Speech-to-Text API), a natural language processing engine (e.g., OpenAI GPT-3), a speech synthesis engine (e.g., IBM Watson Text-to-Speech), and an emotion recognition engine (e.g., IBM Watson Tone Analyzer).

[0395] Network interface: A communication protocol (e.g., UDP or HTTP) is used to allow the terminal and server to send and receive data in real time.

[0396] Specific examples

[0397] User authentication

[0398] The server receives the voice data sent from the device and analyzes it using a voice recognition engine (e.g., Google Speech-to-Text API). The analyzed voice is compared with the voice of the user registered in advance, and if authentication is successful, the user's profile data is sent to the device. The device loads the profile data into memory and audibly announces authentication completion to the user (e.g., "Hello, I'm Tomo.").

[0399] Conversations with children

[0400] When a user (child) speaks to the device, the microphone collects voice data and sends it to the server. The server then uses a speech recognition engine to convert it into text and an emotion recognition engine to analyze emotions. It then uses a natural language processing engine (e.g., OpenAI GPT-3) to generate the optimal response, and then uses a speech synthesis engine to convert the response into voice data. The generated voice data is sent to the device, which then plays it back and communicates it to the user.

[0401] Recording learning activities

[0402] The device monitors the child's conversations and learning activities and sends the collected data to a server. The server records the data in a database and uses an analysis algorithm to extract learning progress and areas of interest. The next lesson plan is generated based on the analysis results.

[0403] Information sharing

[0404] The server aggregates learning and emotional data over a specific period of time and creates reports for parents and educators, which are then sent to them via email or a dedicated app.

[0405] Examples of prompt statements

[0406] Example 1: The process of a child asking Tomo, "What are we going to study today?"

[0407] Terminal: "What are we studying today?"

[0408] Server: Best response: "It's math time today. Let's practice addition together!"

[0409] Example 2: The process by which parents keep track of their children's learning progress.

[0410] Server: Report based on learning data and sentiment data

[0411] Report contents: "Information about learning content and the child's emotional state"

[0412] In this way, the system of the present invention provides an individually optimized educational environment, and by combining emotion recognition, it realizes more effective learning support and information sharing with parents and educators. The above system configuration and program processing can provide new value.

[0413] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0414] Step 1:

[0415] Device: When powered on, an initialization process begins. The device checks its internal components and enables the microphone to collect audio data. It is then ready to receive audio data.

[0416] Input: Power on, system initialization sequence

[0417] Output: Microphone ready message, interface ready state

[0418] Step 2:

[0419] Device: Collects surrounding audio data through a microphone, which is encrypted in real time and prepared for transmission to a server.

[0420] Input: Audio data

[0421] Output: Encrypted audio data packets

[0422] Step 3:

[0423] Terminal: The terminal transmits collected audio data to the server in real time using communication protocols such as UDP and HTTP.

[0424] Input: Encrypted voice data packets

[0425] Output: Request to send audio data to the server

[0426] Step 4:

[0427] Server: Decodes the received voice data and passes it to a speech recognition engine (e.g., Google Speech-to-Text API) to begin analysis. Converts the voice data into text data.

[0428] Input: Encrypted audio data

[0429] Output: Text data

[0430] Step 5:

[0431] Server: The voice data is analyzed and compared with the voice of the user registered in advance. User authentication is performed based on this comparison.

[0432] Input: Text data

[0433] Output: Authentication result (success / failure)

[0434] Step 6:

[0435] Server: If authentication is successful, retrieves the user's profile data from the database and sends it to the terminal.

[0436] Input: Authentication result (success)

[0437] Output: Request to send profile data

[0438] Step 7:

[0439] Terminal: Loads the received profile data into memory, initializes the user's settings and preferences, and prepares to provide a user-specific interface based on the profile data.

[0440] Input: Profile data

[0441] Output: User interface initialization completion message

[0442] Step 8:

[0443] On the device: Play a message to the user to let them know that authentication is complete (e.g., "Hello, this is Tomo.").

[0444] Input: Authentication successful, profile data initialization completed

[0445] Output: Playback of voice message

[0446] Step 9:

[0447] User (child): Talk to the device. For example, say, "What are we going to study today?"

[0448] Input: User question

[0449] Output: Start collecting audio data through the microphone

[0450] Step 10:

[0451] Terminal: The microphone receives the user's question and collects the voice data. This voice data is then encrypted again and sent to the server.

[0452] Input: User's voice data

[0453] Output: Encrypted audio data packets

[0454] Step 11:

[0455] Server: Passes the received voice data to the voice recognition engine and converts it into text data. The voice data is then parsed into a string format.

[0456] Input: Encrypted audio data

[0457] Output: Text data

[0458] Step 12:

[0459] Server: The converted text data is sent to an emotion recognition engine (e.g., IBM Watson Tone Analyzer) for sentiment analysis. At the same time, the text is sent to a natural language processing engine (e.g., OpenAI GPT-3) to generate an appropriate response.

[0460] Input: Text data

[0461] Output: Emotion data, primary response text

[0462] Step 13:

[0463] Server: Based on the emotion recognition results, the server uses a generative AI model to construct an optimal response. It generates a natural-sounding response by integrating the response text and emotion data.

[0464] Input: Emotion data, primary response text

[0465] Output: Best response text

[0466] Step 14:

[0467] Server: The generated response text is sent to a speech synthesis engine (e.g., IBM Watson Text-to-Speech) and converted into voice data.

[0468] Input: Best response text

[0469] Output: Audio data

[0470] Step 15:

[0471] Server: Sends the generated voice data to the terminal.

[0472] Input: Audio data

[0473] Output: Request to send audio data to the device

[0474] Step 16:

[0475] Terminal: Plays back the received voice data and communicates the response to the user (e.g., "It's math time today. Let's practice addition together!").

[0476] Input: Audio data

[0477] Output: Playback of voice message

[0478] Step 17:

[0479] Device: Continuously monitors conversations and learning activities with your child. Monitored data includes conversation content, usage time, and facial expressions (if camera is used).

[0480] Input: Learning activity data

[0481] Output: Monitoring report

[0482] Step 18:

[0483] Terminal: Collected data is sent to the server in stages. The data is encrypted and sent securely.

[0484] Input: Monitoring report

[0485] Output: A request to send data to the server

[0486] Step 19:

[0487] Server: Records the submitted data in a database and applies analytical algorithms to extract learning progress and areas of interest, using machine learning models if necessary.

[0488] Input: Learning activity data

[0489] Output: Analysis result report

[0490] Step 20:

[0491] Server: Based on the analysis results, the next learning plan is generated. This plan is optimized for each individual user.

[0492] Input: Analysis result report

[0493] Output: Next lesson plan

[0494] Step 21:

[0495] Server: Aggregates learning and sentiment data for specific time periods and creates reports for parents and educators.

[0496] Input: training data, emotion data

[0497] Output: Report

[0498] Step 22:

[0499] Server: Generated reports are sent to parents and educators via email or a dedicated app.

[0500] Input: Report

[0501] Output: Send completion notification

[0502] This allows the processing flow of the entire system to be executed in a concrete manner, providing an individually optimized educational environment.

[0503] (Application example 2)

[0504] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0505] In order to improve passenger comfort and safety in autonomous vehicles, it is necessary to analyze passengers' emotional states and requests in real time and respond appropriately. However, existing technologies lack the means to accurately recognize passengers' emotional states and optimize the in-car environment in real time. Therefore, there is a need for a system that analyzes passengers' emotional states and automatically adjusts the in-car environment.

[0506] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion recognition means, an in-vehicle environment adjustment means, and a data analysis means. This makes it possible to analyze the emotional state of passengers in real time and optimize the in-vehicle environment based on the analysis.

[0507] "User authentication means" is a function that identifies individuals through voice or facial recognition and allows them access to the system.

[0508] "Speech recognition means" is a function that converts the user's voice into text data.

[0509] The "voice data transmission means" is a function that transmits collected voice data to a server in real time.

[0510] "Natural language processing means" is a function that analyzes text data and generates an appropriate response.

[0511] The "voice synthesis means" is a function that converts the generated text data into voice data.

[0512] The "profile data recording means" is a function for recording personal information and history data of users.

[0513] The "learning activity monitoring means" is a function that monitors the user's learning activities and collects and records the data.

[0514] "Data sharing means" is a function that allows collected data to be shared with other users or systems.

[0515] "Emotion recognition means" is a function that analyzes the user's emotional state from their facial expressions and voice.

[0516] The "in-car environment adjustment means" is a function that automatically adjusts the in-car environment, such as temperature, lighting, and music, based on the analyzed emotional state.

[0517] "Data analysis means" is a function that analyzes collected data and generates the next action or response.

[0518] The present invention provides an emotion recognition system for an autonomous vehicle that analyzes passenger emotions in real time and adjusts the in-vehicle environment appropriately to maximize passenger comfort and safety. Specific embodiments for implementing this system are described below.

[0519] System configuration

[0520] This system mainly consists of the following elements:

[0521] Terminal: A device installed inside an autonomous vehicle that is equipped with a camera and microphone.

[0522] Server: A back-end system that analyzes and processes data.

[0523] Emotion recognition engine: Software that analyzes passenger emotions.

[0524] Speech recognition engine: Software that converts speech into text data.

[0525] Natural language processing engine: Software that generates appropriate responses.

[0526] IoT control system: A system that automatically adjusts the in-car environment.

[0527] Program processing procedure

[0528] 1. User Authentication

[0529] Terminal: Using a camera and microphone, the device recognizes the passenger's face and voice to identify the individual. This data is then sent to the server via a voice data transmission means.

[0530] Server: Analyzes the received voice and facial data and compares it with profile data to authenticate the user. If authentication is successful, it sends the corresponding profile data to the device.

[0531] 2. Passenger sentiment analysis

[0532] Emotion recognition engine: Analyzes passengers' facial expressions captured by the camera and the voice data sent from the voice recognition means to identify their emotional state. The emotion recognition results are sent to the data analysis means.

[0533] 3. Automatic adjustment of the in-car environment

[0534] IoT Control System: Automatically adjusts environmental settings such as in-car temperature, lighting, seat reclining angle, and music based on emotion recognition results and profile data.

[0535] Specific use cases

[0536] For example, if a passenger looks tired, the emotion recognition engine will detect this and send a command to the IoT control system, which will then recline the seat, dim the interior lights, and play soft music.

[0537] Prompt Sentence Examples

[0538] Emotion analysis prompt: "Based on the facial image of the passenger, guess their current emotional state."

[0539] Speech recognition prompt: "Please convert the passenger's request and sentiment from the recorded conversation into text and return it to us."

[0540] In this way, the system of the present invention analyzes passenger emotions in real time and adjusts the in-car environment based on that, providing a comfortable and safe travel experience.

[0541] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0542] Step 1:

[0543] Terminal: Activates the camera and microphone to capture the passenger's face and voice data. The input is the passenger's real-time face image and voice. An audio data transmission means is used to send this to the server. The output is the captured data.

[0544] Step 2:

[0545] Server: Analyzes the received voice data and facial image data. It uses a voice recognition means to convert the voice data into text, and then uses an emotion recognition means to analyze the emotional state from the facial image and voice. The inputs are the transmitted voice data and facial image data. The output is the analyzed text data and emotional state data.

[0546] Step 3:

[0547] Server: The text data generated by the speech recognition means is compared with the profile data to authenticate the user. The emotional state data is also compared with the profile data to confirm the passenger's emotional tendency. The inputs are text data, emotional state data, and profile data. The output is the authentication result and emotional tendency data.

[0548] Step 4:

[0549] Server: Using emotion recognition means and data analysis means, generates appropriate in-car environment settings based on the analyzed emotion data. The inputs are emotional state data, profile data, and past emotion tendency data. The output is in-car environment setting data.

[0550] Step 5:

[0551] Terminal: Using the in-car environment adjustment means, the temperature, lighting, seat position, music, etc. in the car are automatically adjusted based on the generated in-car environment setting data. The in-car environment setting data is used as input, and the adjusted in-car environment is obtained as output.

[0552] Step 6:

[0553] Terminal: After the adjustment is completed, the passenger is notified by voice that the environment has been adjusted. A predefined notification message is generated using a speech synthesis means and played through the speaker. The input is the text data of the notification message. The output is a voice notification.

[0554] In this way, each step works together to create a system that provides passenger comfort and safety.

[0555] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0556] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0557] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0558] [Second embodiment]

[0559] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0560] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0561] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0562] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0563] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0564] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0565] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0566] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0567] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0568] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0569] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0570] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0571] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing the system of the present invention will be described below.

[0572] System configuration

[0573] This system mainly consists of a terminal, a server, and a network interface. The terminal is a device that the child directly operates, and the server processes data in the backend. The network interface is responsible for connecting the terminal and the server via communication.

[0574] Program processing

[0575] 1. User Authentication

[0576] Terminal: When the power is turned on, the initialization process begins. First, ambient sound is received through the microphone. The received sound data is sent to the server in real time.

[0577] Server: Analyzes the received voice data and compares it with the voice of pre-registered users. If authentication is successful, it sends the corresponding user profile data to the device.

[0578] Terminal: The received profile data is loaded into memory and a message indicating authentication completion is given to the user.

[0579] 2. Conversations with children

[0580] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[0581] Terminal: Receives audio via microphone and transmits audio data to the server in real time.

[0582] Server: The received voice data is passed through a speech recognition engine and converted into text. The converted text data is sent to a natural language processing engine, which generates an appropriate response (e.g., "Today is math class. Let's practice addition together!").

[0583] Server: Passes the generated response text to the speech synthesis engine to generate voice data, which is then sent to the terminal.

[0584] Terminal: Plays back the received voice data and conveys the response to the child.

[0585] 3. Recording learning activities

[0586] Device: Monitors conversations and learning activities with children in real time. This data is sent to the server.

[0587] Server: Records the transmitted data in a database. Analyzes the recorded data, extracts the user's learning progress and areas of interest, and generates the next learning plan.

[0588] 4. Information sharing

[0589] Server: Aggregates learning data for specific periods and creates reports. Generated notifications are sent to parents and educators via email or a dedicated app (e.g., "Today your child learned the basics of math. Next, we'll introduce multiplication.").

[0590] Specific examples

[0591] Example 1: A child asks Tomo, "What are we going to study today?"

[0592] Terminal: Receives audio and sends it to the server.

[0593] Server: Converts the speech data into text and generates a response ("Today is math time. Let's practice addition together!").

[0594] Device: The generated response is spoken to the child.

[0595] Example 2: Parents keep track of their children's learning progress.

[0596] Server: Collects learning data and creates reports.

[0597] Server: Send the report to the parent's email address.

[0598] In this way, the system of the present invention provides an individually optimized educational environment and enables information sharing with parents and educators.

[0599] The processing flow will be explained below.

[0600] User authentication process

[0601] Step 1:

[0602] Device: When powered on, it begins an initialization process, preparing to receive ambient audio through the microphone.

[0603] Step 2:

[0604] Device: Collects voice data through a microphone and transmits the collected voice data to a server in real time.

[0605] Step 3:

[0606] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If the comparison is successful, authentication of the user is completed.

[0607] Step 4:

[0608] Server: If the authentication is successful, it sends the corresponding user profile data to the terminal.

[0609] Step 5:

[0610] Terminal: Loads the received profile data into memory.

[0611] Step 6:

[0612] Terminal: A message indicating authentication completion is given to the user via voice.

[0613] Handling conversations with children

[0614] Step 1:

[0615] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[0616] Step 2:

[0617] Terminal: Receives audio via microphone and transmits it to the server as audio data in real time.

[0618] Step 3:

[0619] Server: The received voice data is passed through a voice recognition engine and converted into text.

[0620] Step 4:

[0621] Server: Sends the converted text data to a natural language processing engine to generate an appropriate response.

[0622] Step 5:

[0623] Server: Passes the generated response text to the speech synthesis engine to generate voice data.

[0624] Step 6:

[0625] Server: Sends the generated voice data to the terminal.

[0626] Step 7:

[0627] Terminal: Plays back the received voice data and conveys the response to the child.

[0628] Processing learning activity records

[0629] Step 1:

[0630] Device: Monitor your child's conversations and learning activities in real time.

[0631] Step 2:

[0632] Terminal: Sends monitored data to the server.

[0633] Step 3:

[0634] Server: Records the transmitted data in a database.

[0635] Step 4:

[0636] Server: Analyzes the recorded data and extracts the user's learning progress and areas of interest.

[0637] Step 5:

[0638] Server: Generates the next learning plan based on the analysis results.

[0639] Information Sharing Process

[0640] Step 1:

[0641] Server: Aggregates learning data for specific periods and creates reports.

[0642] Step 2:

[0643] Server: Create notifications to share generated reports with parents and educators.

[0644] Step 3:

[0645] Server: Sends reports to parents and educators via email or a dedicated app.

[0646] Processing of specific examples

[0647] Example 1: The process of a child asking Tomo, "What are we going to study today?"

[0648] Terminal: The child's questions are received by a microphone and sent to the server as voice data.

[0649] Server: Analyzes the voice data and generates an appropriate response.

[0650] Server: Converts the generated response into audio data and sends it to the terminal.

[0651] Device: Plays the response as audio and relays it to the child.

[0652] Example 2: The process by which parents keep track of their children's learning progress.

[0653] Server: Aggregates learning data and creates reports.

[0654] Server: Generates and emails the report to the parent.

[0655] Example 1

[0656] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0657] In today's educational environment, there is a demand for systems that provide individually optimized educational support. Conventional systems lack the ability to analyze voice data in real time and generate appropriate feedback, limiting their educational effectiveness. Furthermore, they often do not effectively record and analyze learning progress or share information with parents and educators. This raises concerns that children's learning efficiency may decline.

[0658] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0659] In this invention, the server includes a user authentication means, a voice recognition means, a natural language processing means, a voice synthesis means, a learning activity monitoring means, a data sharing means, a learning activity recording means, a means for generating responses using a generative AI model, and a learning progress analysis means, which enable real-time analysis of voice data, generation of appropriate feedback, recording and analysis of learning progress, and effective information sharing with parents and educators.

[0660] "User authentication means" is a means for verifying whether a user is a registered user by using voice or other biometric information.

[0661] "Speech recognition means" refers to means having the function of converting voice data into text data.

[0662] The "audio data transmission means" is a means having a function of transmitting audio data received by the terminal to the server.

[0663] "Natural language processing means" refers to means that has the function of analyzing text data and generating an appropriate response.

[0664] The "voice synthesis means" is a means having a function of converting text data into voice data.

[0665] "Profile data recording means" means a means having the function of recording and storing profile data of authenticated users.

[0666] The "learning activity monitoring means" is a means having a function of monitoring the user's learning activity in real time and collecting the data.

[0667] "Data sharing means" refers to a means that has the function of sharing collected and analyzed learning data with parents and educators at regular intervals.

[0668] A "means for recording learning activities" is a means that has the function of recording data during learning activities.

[0669] A "means for generating a response using a generative artificial intelligence model" is a means that has the function of generating an appropriate text response to a voice input using a generative AI model.

[0670] The "learning progress analysis means" is a means having a function of analyzing collected learning data and generating learning progress and the next learning plan.

[0671] The present invention is implemented by a system constructed by combining specified hardware and software. Specific embodiments will be described below.

[0672] System configuration

[0673] This system consists of a terminal, a server, and a network interface. The terminal is a device that the child directly operates, and the server processes data in the backend. The network interface connects the terminal and the server via communication.

[0674] Hardware and software used

[0675] Device:

[0676] Microphone: Used to receive audio data.

[0677] Speaker: Used to play audio data.

[0678] Network interface: Responsible for communication with the server.

[0679] server:

[0680] Speech analysis engine: Uses Google Cloud Speech-to-Text to convert voice data into text.

[0681] Natural language processing engine: Uses OpenAI GPT-3 to generate appropriate responses from received text data.

[0682] Speech synthesis engine: Uses Amazon Polly to convert the generated text into speech data.

[0683] Database: MongoDB is used to store profile data and records of learning activities.

[0684] Data sharing system: Use Firebase to share learning progress reports with parents and educators.

[0685] System operating principle

[0686] User authentication

[0687] Device: When powered on, it starts an initialization process and receives ambient audio through its microphone, which is then sent to the server in real time.

[0688] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If authentication is successful, it sends the corresponding user profile data to the device.

[0689] Terminal: The received profile data is loaded into memory and a message indicating authentication completion is given to the user.

[0690] Conversations with children

[0691] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[0692] Terminal: Receives audio via microphone and transmits audio data to the server in real time.

[0693] Server: The received voice data is passed through a speech recognition engine and converted into text. The converted text data is sent to a natural language processing engine, which generates an appropriate response (e.g., "Today is math class. Let's practice addition together!").

[0694] Server: Passes the generated response text to the speech synthesis engine to generate voice data, which is then sent to the terminal.

[0695] Terminal: Plays back the received voice data and conveys the response to the child.

[0696] Recording learning activities

[0697] Device: Monitors conversations and learning activities with children in real time. This data is sent to the server.

[0698] Server: Records the transmitted data in a database. Analyzes the recorded data, extracts the user's learning progress and areas of interest, and generates the next learning plan.

[0699] Information sharing

[0700] Server: Collects learning data at regular intervals and creates reports. Generated notifications are sent to parents and educators via email or a dedicated app. For example, "Today, your child learned the basics of arithmetic. Next, we'll introduce multiplication."

[0701] Specific examples

[0702] Example 1: A child asks Tomo, "What are we going to study today?"

[0703] Terminal: Receives audio and sends it to the server.

[0704] Server: Converts the speech data into text and generates a response ("Today is math time. Let's practice addition together!").

[0705] Device: The generated response is spoken to the child.

[0706] Example 2: Parents keep track of their children's learning progress.

[0707] Server: Collects learning data and creates reports.

[0708] Server: Send the report to the parent's email address.

[0709] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0710] Specific explanation of processing steps

[0711] Step 1: Initialize the system

[0712] Device: When the device is powered on, it begins an initialization process, which checks system memory, verifies network connectivity, and verifies the status of devices such as the microphone and speaker. If successful, the device proceeds to the next step: user authentication.

[0713] Input: Power on

[0714] Output: Initialization complete signal

[0715] Step 2: User authentication

[0716] Device: Once initialization is complete, it receives surrounding sounds through the microphone. For example, when a child says "Hello, Tomo," the voice data is captured. The captured voice data is sent to the server in real time.

[0717] Server: Analyzes the received voice data using a voice analysis engine (e.g., Google Cloud Speech-to-Text). The analysis results are compared with pre-registered user voices. If authentication is successful, the server sends the corresponding user profile data to the device.

[0718] Terminal: Loads the received profile data into memory and generates a message indicating authentication is complete. The message is then output to the speaker, saying "Authentication complete. Hello, [username]!"

[0719] Input: Audio data

[0720] Output: User profile data, voice messages

[0721] Step 3: Talk to your child

[0722] User (child): Once authenticated, the child can talk to Tomo, for example, asking questions like, "What are we going to study today?"

[0723] Terminal: This audio is received by a microphone and sent to the server in real time as audio data.

[0724] Server: The received voice data is converted into text data through a speech recognition engine (e.g., Google Cloud Speech-to-Text). The converted text is sent to a natural language processing engine (e.g., OpenAI GPT-3), which generates an appropriate response (e.g., "Today is math class. Let's practice addition together!").

[0725] Server: Pass the generated response text to a speech synthesis engine (e.g., Amazon Polly) to generate voice data.

[0726] Server: Sends the generated voice data to the terminal.

[0727] Terminal: The received voice data is played back on the speaker and the response is conveyed to the child.

[0728] Input: Child's voice data

[0729] Output: Response voice data

[0730] Step 4: Record your learning activities

[0731] Device: Monitors your child's conversations and learning activities with Tomo in real time (e.g., collects data when answering questions like "What is 1 + 1?").

[0732] Terminal: These data are sent to the server in real time.

[0733] Server: Records the submitted data in a database (e.g., MongoDB). The recorded data is analyzed based on each data point, such as what was learned, the accuracy of the answers, and the learning time.

[0734] Server: From the analysis results, the user's learning progress and areas of interest are extracted and the next learning plan is generated.

[0735] Input: Learning activity data

[0736] Output: Analysis results, learning plan

[0737] Step 5: Information sharing

[0738] Server: Compiles learning data at regular intervals and creates reports. For example, a "weekly learning progress report" might include information such as what was learned, the percentage of correct answers, and the time spent.

[0739] Server: Generated reports are sent to parents and educators via email or a dedicated app (e.g., Firebase). For example, a report might say, "Today, your child learned the basics of math. Next, we'll introduce multiplication."

[0740] Input: Training data

[0741] Output: Learning progress report

[0742] This will improve the efficiency of learning activities and ensure effective collaboration with parents and educators.

[0743] (Application example 1)

[0744] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0745] There is a need for efficient collaboration between workers and robots in factories. However, there is no system in place that allows workers to give voice instructions to robots and have them execute them accurately and quickly. There is also a lack of appropriate support for monitoring the progress and efficiency of work in real time and planning the next work. This is reducing work efficiency in the factory and hindering overall productivity improvements.

[0746] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0747] In this invention, the server includes a user authentication means, a voice recognition means, a voice data transmission means, a natural language processing means, a voice synthesis means, a profile data recording means, a work activity monitoring means, a data sharing means, and a work instruction generation means. This allows an employee to give work instructions to a robot by voice, and the robot to appropriately process and execute those instructions. Furthermore, it is possible to monitor the progress of work in real time and optimize the next work plan, thereby improving overall work efficiency and productivity.

[0748] The "user authentication means" is a means for analyzing the employee's voice data and authenticating them by comparing it with pre-stored profile data.

[0749] "Speech recognition means" is a means for analyzing voice data and converting it into text data.

[0750] The "audio data transmission means" is a means for transmitting audio data to a server in real time.

[0751] The "natural language processing means" is a means for analyzing text data that has been speech-recognized and generating an appropriate response or work instruction.

[0752] The "voice synthesis means" is a means for generating voice data from text data.

[0753] "Profile data recording means" refers to a means for recording and managing profile data of authenticated employees.

[0754] The "work activity monitoring means" is a means for monitoring the work status of the robot in real time and recording the progress.

[0755] "Data sharing means" refers to a means for sharing work progress and analysis results with relevant parties.

[0756] The "work instruction generation means" is a means for analyzing the voice instructions of an employee and generating specific work instructions.

[0757] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing this system will be described below.

[0758] System configuration

[0759] This system mainly consists of a terminal (the control unit of the factory robot), a server, and a network interface. The terminal is a robot that operates in the factory, and the server processes data on the back end. The network interface is responsible for connecting the terminal and server via communication.

[0760] Program processing

[0761] The server implements the system using the following means:

[0762] 1. User authentication method:

[0763] When the terminal (the robot's control unit) is turned on, it receives audio through a microphone and sends the audio data to the server.

[0764] The server analyzes the received voice data using a voice recognition engine and compares it with pre-registered profile data. If authentication is successful, the corresponding employee profile data is sent to the terminal.

[0765] 2. Voice recognition means:

[0766] The server analyzes the voice data sent from the device in real time and converts it into text data, specifically using the SpeechRecognition library.

[0767] 3. Work instruction generation means:

[0768] The server sends the text data to a natural language processing engine to generate appropriate work instructions, which are then sent to the terminal, where the robot begins working based on the instructions.

[0769] 4. Work activity monitoring means:

[0770] The terminal monitors the status of the work being performed in real time and sends the data to the server.

[0771] The server records the received data in a database and analyzes the progress. The analysis results are used to generate the next work plan.

[0772] 5. Data sharing methods:

[0773] The server has a means to share the progress and efficiency of work with the relevant parties. For example, it generates a work report for each specific period and notifies the supervisor.

[0774] Hardware and software used

[0775] Hardware:

[0776] Microphone: A device for receiving employee voice.

[0777] Robot control unit: CPU and memory for processing voice data and receiving and executing work instructions.

[0778] software:

[0779] SpeechRecognition Library: A library for real-time speech recognition in Python.

[0780] Requests library: An HTTP client library for communicating with the server.

[0781] Specific examples

[0782] Employee dialogue: "Hey robot, sort the ingredients."

[0783] Server: Converts voice data into text and generates work instructions such as "Start sorting materials" based on that text data.

[0784] Terminal: Based on the work instructions sent from the server, the robot begins sorting materials.

[0785] Example of prompt input:

[0786] User: "Robot, sort the ingredients."

[0787] Response: "I'll start sorting the ingredients."

[0788] In this way, the system of the present invention supports efficient collaboration between employees and robots within a factory, improving work efficiency and productivity.

[0789] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0790] Step 1:

[0791] The user (employee) gives the robot instructions by voice, for example, "sort the materials." This voice data becomes the input.

[0792] Step 2:

[0793] The terminal (the robot's control unit) receives the user's voice through a microphone and prepares to send this voice data to the server. Here, the voice data is the input, and the output is to convert it into the format required to send it to the server. Specifically, the recorded voice data is converted into a digital format so that it can be sent over the network.

[0794] Step 3:

[0795] The server receives the voice data sent from the terminal. Here, the voice data is the input. The server converts this voice data into text data using a voice recognition system (SpeechRecognition library). The text data is the output.

[0796] Step 4:

[0797] The server sends the converted text data to a natural language processing engine, which analyzes the employee's intent. This text data is the input, and the analyzed work instructions are the output. Specifically, the instruction "sort the materials" is converted into the specific work instruction "sort the materials into the designated locations."

[0798] Step 5:

[0799] The server converts the generated work instruction data into voice data using a voice synthesis engine and prepares to send it to the terminal. Here, the work instruction data is the input and the voice data is the output.

[0800] Step 6:

[0801] The terminal receives the voice data sent from the server and issues appropriate work instructions to the robot. Here, the voice data is the input, and the robot starts a specific task (e.g., sorting materials) based on that voice data.

[0802] Step 7:

[0803] The terminal monitors the robot's work progress in real time and sends the data to the server. Here, the progress data is the input, and converting it into a format that can be sent to the server is the output.

[0804] Step 8:

[0805] The server records the received work progress data in a database and analyzes the progress. The progress data is the input and the analysis results are the output. The next work plan is generated based on the analysis results.

[0806] Step 9:

[0807] The server generates work reports for each specific period and sends them to the relevant parties. Analysis results and progress data are the input, and the work reports are the output. An example of a report might be, "Today, the robot sorted 100 items of material."

[0808] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0809] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing the system of the present invention will be described below.

[0810] System configuration

[0811] This system mainly consists of a terminal, a server, a network interface, and an emotion engine. The terminal is a device that the child directly operates, and the server processes data at the backend. The network interface is responsible for connecting the terminal and the server via communication. The emotion engine provides the functionality to recognize the user's emotions and generate responses based on them.

[0812] Program processing

[0813] 1. User Authentication

[0814] Device: When powered on, it begins an initialization process, preparing to receive ambient audio through the microphone.

[0815] Device: Collects voice data through a microphone and transmits the collected voice data to a server in real time.

[0816] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If the comparison is successful, authentication of the user is completed.

[0817] Server: If the authentication is successful, it sends the corresponding user profile data to the terminal.

[0818] Terminal: Loads the received profile data into memory.

[0819] Terminal: A message indicating authentication completion is given to the user via voice.

[0820] 2. Conversations with children

[0821] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[0822] Terminal: Receives audio via microphone and transmits it to the server as audio data in real time.

[0823] Server: The received voice data is passed through a voice recognition engine and converted into text.

[0824] Server: The converted text data is sent to a natural language processing engine to generate an appropriate response, including emotion recognition using an emotion engine.

[0825] Emotion engine: Analyzes the user's emotions and provides the results to the natural language processing engine.

[0826] Server: Generates response text including emotion recognition results and passes it to the speech synthesis engine.

[0827] Server: Sends the generated voice data to the terminal.

[0828] Terminal: Plays back the received voice data and conveys the response to the child.

[0829] 3. Recording learning activities

[0830] Device: Monitor your child's conversations and learning activities in real time.

[0831] Terminal: Sends monitored data to the server.

[0832] Server: Records the transmitted data in a database, including emotional data.

[0833] Server: Analyzes the recorded data and extracts the user's learning progress and areas of interest. The analysis also takes into account the user's emotional data.

[0834] Server: Generates the next learning plan based on the analysis results.

[0835] 4. Information sharing

[0836] Server: Aggregates learning data for specific periods and creates reports, including emotional data.

[0837] Server: Prepares generated notifications to be shared with parents and educators.

[0838] Server: Sends reports to parents and educators via email or a dedicated app.

[0839] Specific examples

[0840] Example 1: The process of a child asking Tomo, "What are we going to study today?"

[0841] Terminal: The child's questions are received by a microphone and sent to the server as voice data.

[0842] Server: Analyzes the voice data, converts it into text, and recognizes emotions using an emotion engine. Based on the response and emotion data, it generates an optimal response (e.g., "Today is math class. Let's practice addition together!").

[0843] Server: Converts the generated response into audio data and sends it to the terminal.

[0844] Device: Plays the response aloud and relays it to the child.

[0845] Example 2: The process by which parents keep track of their children's learning progress.

[0846] Server: Aggregates learning data and emotion data and creates reports.

[0847] Server: Generates and emails the report to the parent, containing information about the learning content and the child's emotional state.

[0848] In this way, the system of the present invention provides an individually optimized educational environment, and by combining it with emotion recognition, it enables more effective learning support and information sharing with parents and educators.

[0849] The processing flow will be explained below.

[0850] User authentication process

[0851] Step 1:

[0852] Device: When powered on, it begins an initialization process, preparing to receive ambient audio through the microphone.

[0853] Step 2:

[0854] Device: Collects voice data through a microphone and transmits the collected voice data to a server in real time.

[0855] Step 3:

[0856] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If the comparison is successful, authentication of the user is completed.

[0857] Step 4:

[0858] Server: If the authentication is successful, it sends the corresponding user profile data to the terminal.

[0859] Step 5:

[0860] Terminal: Loads the received profile data into memory.

[0861] Step 6:

[0862] Terminal: A message indicating authentication completion is given to the user via voice.

[0863] Handling conversations with children

[0864] Step 1:

[0865] User (child): Talk to the device (e.g., "What are we going to study today?").

[0866] Step 2:

[0867] Terminal: Receives audio via microphone and transmits it to the server as audio data in real time.

[0868] Step 3:

[0869] Server: The received voice data is passed through a voice recognition engine and converted into text.

[0870] Step 4:

[0871] Server: The converted text data is sent to a natural language processing engine to generate an appropriate response, including emotion recognition using an emotion engine.

[0872] Step 5:

[0873] Emotion engine: Analyzes emotions from the user's voice and provides the results to the natural language processing engine.

[0874] Step 6:

[0875] Server: Generates a response text that reflects the emotion recognition results and passes it to the speech synthesis engine (e.g., "Today is math class. Let's practice addition together!").

[0876] Step 7:

[0877] Server: Sends the generated voice data to the terminal.

[0878] Step 8:

[0879] Terminal: Plays back the received voice data and conveys the response to the child.

[0880] Processing learning activity records

[0881] Step 1:

[0882] Device: Monitor your child's conversations and learning activities in real time.

[0883] Step 2:

[0884] Terminal: Sends monitored data to the server.

[0885] Step 3:

[0886] Server: Records the transmitted data in a database, including emotional data.

[0887] Step 4:

[0888] Server: Analyzes the recorded data and extracts the user's learning progress and areas of interest. The analysis also takes into account the user's emotional data.

[0889] Step 5:

[0890] Server: Generates the next learning plan based on the analysis results.

[0891] Information Sharing Process

[0892] Step 1:

[0893] Server: Aggregates learning data for specific periods and creates reports, including emotional data.

[0894] Step 2:

[0895] Server: Prepares generated notifications to be shared with parents and educators.

[0896] Step 3:

[0897] Server: Sends reports to parents and educators via email or a dedicated app.

[0898] Processing of specific examples

[0899] Example 1: The process of a child asking their device, "What are we going to study today?"

[0900] Terminal: The child's questions are received by a microphone and sent to the server as voice data.

[0901] Server: Analyzes the voice data, converts it into text, and recognizes emotions using an emotion engine. Based on the response and emotion data, it generates an optimal response (e.g., "Today is math class. Let's practice addition together!").

[0902] Server: Converts the generated response into audio data and sends it to the terminal.

[0903] Device: Plays the response aloud and relays it to the child.

[0904] Example 2: The process by which parents keep track of their children's learning progress.

[0905] Server: Aggregates learning data and emotion data and creates reports.

[0906] Server: Generates and emails the report to the parent, containing information about the learning content and the child's emotional state.

[0907] Example 2

[0908] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0909] Conventional voice recognition systems are inadequate for complex processing, including user authentication and emotion recognition, making it difficult to optimize the educational environment and provide effective learning support. Furthermore, monitoring and sharing of users' emotion data and learning progress is not smooth, resulting in low quality information provided to parents and educators. There was a need to resolve these issues and provide a more advanced educational support system.

[0910] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0911] In this invention, the server includes a user authentication means, a voice data transmission means, a data analysis means, an emotion recognition means, and a means for generating a response using a generative AI model. This enables accurate authentication based on the user's voice, real-time voice data processing, and the provision of natural responses and learning plans based on emotion analysis. It also makes it possible to realize a system that efficiently collects and analyzes data related to learning progress and emotional state, providing useful information to parents and educators.

[0912] The "user authentication means" is a function that collects the user's voice via a microphone when the device is turned on, sends it to a server for analysis, and compares it with the voice of a previously registered user for authentication.

[0913] The "voice recognition means" is a function that transmits voice data collected by the terminal to a server in real time, analyzes the voice data, and converts it into text.

[0914] The "audio data transmission means" is a function that encrypts the audio data collected by the microphone and transmits it to the server in real time using a communication protocol.

[0915] "Natural language processing means" is a function that uses a natural language processing engine to analyze the converted text data and generate an appropriate response.

[0916] The "voice synthesis means" is a function that uses a voice synthesis engine to convert the generated response text into voice data.

[0917] The "profile data recording means" is a function that loads the profile data of a user into memory when the user is successfully authenticated, and records and manages that information.

[0918] The "learning activity monitoring means" is a function that monitors conversations with children and their learning activities in real time and sends the collected data to a server.

[0919] "Data sharing means" is a function that periodically compiles learning data and emotional data and provides them as reports to parents and educators.

[0920] The "emotion recognition means" is a function that analyzes the user's voice and behavior, determines their emotional state, and provides the results to the natural language processing engine.

[0921] "Means for generating a response using a generative AI model" refers to a function that generates an optimal response using a generative AI model based on received text data and emotion data.

[0922] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing the system of the present invention will be described below.

[0923] System configuration

[0924] This system mainly consists of a terminal, a server, a network interface, and an emotion engine. The terminal is a device that the user directly operates, and the server processes data at the backend. The network interface is responsible for connecting the terminal and the server via communication. The emotion engine provides the functionality to recognize the user's emotions and generate responses based on them.

[0925] Program processing

[0926] This system uses the following hardware and software to process and calculate data.

[0927] Terminal: Contains a microphone to receive the user's voice, a speaker to play the voice, and internal memory. The terminal is responsible for collecting voice data, authenticating the user, and managing profile data.

[0928] Server: Equipped with high-speed computing power for analyzing voice data, natural language processing, emotion recognition, and database management. The server uses a speech recognition engine (e.g., Google Speech-to-Text API), a natural language processing engine (e.g., OpenAI GPT-3), a speech synthesis engine (e.g., IBM Watson Text-to-Speech), and an emotion recognition engine (e.g., IBM Watson Tone Analyzer).

[0929] Network interface: A communication protocol (e.g., UDP or HTTP) is used to allow the terminal and server to send and receive data in real time.

[0930] Specific examples

[0931] User authentication

[0932] The server receives the voice data sent from the device and analyzes it using a voice recognition engine (e.g., Google Speech-to-Text API). The analyzed voice is compared with the voice of the user registered in advance, and if authentication is successful, the user's profile data is sent to the device. The device loads the profile data into memory and audibly announces authentication completion to the user (e.g., "Hello, I'm Tomo.").

[0933] Conversations with children

[0934] When a user (child) speaks to the device, the microphone collects voice data and sends it to the server. The server then uses a speech recognition engine to convert it into text and an emotion recognition engine to analyze emotions. It then uses a natural language processing engine (e.g., OpenAI GPT-3) to generate the optimal response, and then uses a speech synthesis engine to convert the response into voice data. The generated voice data is sent to the device, which then plays it back and communicates it to the user.

[0935] Recording learning activities

[0936] The device monitors the child's conversations and learning activities and sends the collected data to a server. The server records the data in a database and uses an analysis algorithm to extract learning progress and areas of interest. The next lesson plan is generated based on the analysis results.

[0937] Information sharing

[0938] The server aggregates learning and emotional data over a specific period of time and creates reports for parents and educators, which are then sent to them via email or a dedicated app.

[0939] Examples of prompt statements

[0940] Example 1: The process of a child asking Tomo, "What are we going to study today?"

[0941] Terminal: "What are we studying today?"

[0942] Server: Best response: "It's math time today. Let's practice addition together!"

[0943] Example 2: The process by which parents keep track of their children's learning progress.

[0944] Server: Report based on learning data and sentiment data

[0945] Report contents: "Information about learning content and the child's emotional state"

[0946] In this way, the system of the present invention provides an individually optimized educational environment, and by combining emotion recognition, it realizes more effective learning support and information sharing with parents and educators. The above system configuration and program processing can provide new value.

[0947] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0948] Step 1:

[0949] Device: When powered on, an initialization process begins. The device checks its internal components and enables the microphone to collect audio data. It is then ready to receive audio data.

[0950] Input: Power on, system initialization sequence

[0951] Output: Microphone ready message, interface ready state

[0952] Step 2:

[0953] Device: Collects surrounding audio data through a microphone, which is encrypted in real time and prepared for transmission to a server.

[0954] Input: Audio data

[0955] Output: Encrypted audio data packets

[0956] Step 3:

[0957] Terminal: The terminal transmits collected audio data to the server in real time using communication protocols such as UDP and HTTP.

[0958] Input: Encrypted voice data packets

[0959] Output: Request to send audio data to the server

[0960] Step 4:

[0961] Server: Decodes the received voice data and passes it to a speech recognition engine (e.g., Google Speech-to-Text API) to begin analysis. Converts the voice data into text data.

[0962] Input: Encrypted audio data

[0963] Output: Text data

[0964] Step 5:

[0965] Server: The voice data is analyzed and compared with the voice of the user registered in advance. User authentication is performed based on this comparison.

[0966] Input: Text data

[0967] Output: Authentication result (success / failure)

[0968] Step 6:

[0969] Server: If authentication is successful, retrieves the user's profile data from the database and sends it to the terminal.

[0970] Input: Authentication result (success)

[0971] Output: Request to send profile data

[0972] Step 7:

[0973] Terminal: Loads the received profile data into memory, initializes the user's settings and preferences, and prepares to provide a user-specific interface based on the profile data.

[0974] Input: Profile data

[0975] Output: User interface initialization completion message

[0976] Step 8:

[0977] On the device: Play a message to the user to let them know that authentication is complete (e.g., "Hello, this is Tomo.").

[0978] Input: Authentication successful, profile data initialization completed

[0979] Output: Playback of voice message

[0980] Step 9:

[0981] User (child): Talk to the device. For example, say, "What are we going to study today?"

[0982] Input: User question

[0983] Output: Start collecting audio data through the microphone

[0984] Step 10:

[0985] Terminal: The microphone receives the user's question and collects the voice data. This voice data is then encrypted again and sent to the server.

[0986] Input: User's voice data

[0987] Output: Encrypted audio data packets

[0988] Step 11:

[0989] Server: Passes the received voice data to the voice recognition engine and converts it into text data. The voice data is then parsed into a string format.

[0990] Input: Encrypted audio data

[0991] Output: Text data

[0992] Step 12:

[0993] Server: The converted text data is sent to an emotion recognition engine (e.g., IBM Watson Tone Analyzer) for sentiment analysis. At the same time, the text is sent to a natural language processing engine (e.g., OpenAI GPT-3) to generate an appropriate response.

[0994] Input: Text data

[0995] Output: Emotion data, primary response text

[0996] Step 13:

[0997] Server: Based on the emotion recognition results, the server uses a generative AI model to construct an optimal response. It generates a natural-sounding response by integrating the response text and emotion data.

[0998] Input: Emotion data, primary response text

[0999] Output: Best response text

[1000] Step 14:

[1001] Server: The generated response text is sent to a speech synthesis engine (e.g., IBM Watson Text-to-Speech) and converted into voice data.

[1002] Input: Best response text

[1003] Output: Audio data

[1004] Step 15:

[1005] Server: Sends the generated voice data to the terminal.

[1006] Input: Audio data

[1007] Output: Request to send audio data to the device

[1008] Step 16:

[1009] Terminal: Plays back the received voice data and communicates the response to the user (e.g., "It's math time today. Let's practice addition together!").

[1010] Input: Audio data

[1011] Output: Playback of voice message

[1012] Step 17:

[1013] Device: Continuously monitors conversations and learning activities with your child. Monitored data includes conversation content, usage time, and facial expressions (if camera is used).

[1014] Input: Learning activity data

[1015] Output: Monitoring report

[1016] Step 18:

[1017] Terminal: Collected data is sent to the server in stages. The data is encrypted and sent securely.

[1018] Input: Monitoring report

[1019] Output: A request to send data to the server

[1020] Step 19:

[1021] Server: Records the submitted data in a database and applies analytical algorithms to extract learning progress and areas of interest, using machine learning models if necessary.

[1022] Input: Learning activity data

[1023] Output: Analysis result report

[1024] Step 20:

[1025] Server: Based on the analysis results, the next learning plan is generated. This plan is optimized for each individual user.

[1026] Input: Analysis result report

[1027] Output: Next lesson plan

[1028] Step 21:

[1029] Server: Aggregates learning and sentiment data for specific time periods and creates reports for parents and educators.

[1030] Input: training data, emotion data

[1031] Output: Report

[1032] Step 22:

[1033] Server: Generated reports are sent to parents and educators via email or a dedicated app.

[1034] Input: Report

[1035] Output: Send completion notification

[1036] This allows the processing flow of the entire system to be executed in a concrete manner, providing an individually optimized educational environment.

[1037] (Application example 2)

[1038] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1039] In order to improve passenger comfort and safety in autonomous vehicles, it is necessary to analyze passengers' emotional states and requests in real time and respond appropriately. However, existing technologies lack the means to accurately recognize passengers' emotional states and optimize the in-car environment in real time. Therefore, there is a need for a system that analyzes passengers' emotional states and automatically adjusts the in-car environment.

[1040] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion recognition means, an in-vehicle environment adjustment means, and a data analysis means. This makes it possible to analyze the emotional state of passengers in real time and optimize the in-vehicle environment based on the analysis.

[1041] "User authentication means" is a function that identifies individuals through voice or facial recognition and allows them access to the system.

[1042] "Speech recognition means" is a function that converts the user's voice into text data.

[1043] The "voice data transmission means" is a function that transmits collected voice data to a server in real time.

[1044] "Natural language processing means" is a function that analyzes text data and generates an appropriate response.

[1045] The "voice synthesis means" is a function that converts the generated text data into voice data.

[1046] The "profile data recording means" is a function for recording personal information and history data of users.

[1047] The "learning activity monitoring means" is a function that monitors the user's learning activities and collects and records the data.

[1048] "Data sharing means" is a function that allows collected data to be shared with other users or systems.

[1049] "Emotion recognition means" is a function that analyzes the user's emotional state from their facial expressions and voice.

[1050] The "in-car environment adjustment means" is a function that automatically adjusts the in-car environment, such as temperature, lighting, and music, based on the analyzed emotional state.

[1051] "Data analysis means" is a function that analyzes collected data and generates the next action or response.

[1052] The present invention provides an emotion recognition system for an autonomous vehicle that analyzes passenger emotions in real time and adjusts the in-vehicle environment appropriately to maximize passenger comfort and safety. Specific embodiments for implementing this system are described below.

[1053] System configuration

[1054] This system mainly consists of the following elements:

[1055] Terminal: A device installed inside an autonomous vehicle that is equipped with a camera and microphone.

[1056] Server: A back-end system that analyzes and processes data.

[1057] Emotion recognition engine: Software that analyzes passenger emotions.

[1058] Speech recognition engine: Software that converts speech into text data.

[1059] Natural language processing engine: Software that generates appropriate responses.

[1060] IoT control system: A system that automatically adjusts the in-car environment.

[1061] Program processing procedure

[1062] 1. User Authentication

[1063] Terminal: Using a camera and microphone, the device recognizes the passenger's face and voice to identify the individual. This data is then sent to the server via a voice data transmission means.

[1064] Server: Analyzes the received voice and facial data and compares it with profile data to authenticate the user. If authentication is successful, it sends the corresponding profile data to the device.

[1065] 2. Passenger sentiment analysis

[1066] Emotion recognition engine: Analyzes passengers' facial expressions captured by the camera and the voice data sent from the voice recognition means to identify their emotional state. The emotion recognition results are sent to the data analysis means.

[1067] 3. Automatic adjustment of the in-car environment

[1068] IoT Control System: Automatically adjusts environmental settings such as in-car temperature, lighting, seat reclining angle, and music based on emotion recognition results and profile data.

[1069] Specific use cases

[1070] For example, if a passenger looks tired, the emotion recognition engine will detect this and send a command to the IoT control system, which will then recline the seat, dim the interior lights, and play soft music.

[1071] Prompt Sentence Examples

[1072] Emotion analysis prompt: "Based on the facial image of the passenger, guess their current emotional state."

[1073] Speech recognition prompt: "Please convert the passenger's request and sentiment from the recorded conversation into text and return it to us."

[1074] In this way, the system of the present invention analyzes passenger emotions in real time and adjusts the in-car environment based on that, providing a comfortable and safe travel experience.

[1075] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1076] Step 1:

[1077] Terminal: Activates the camera and microphone to capture the passenger's face and voice data. The input is the passenger's real-time face image and voice. An audio data transmission means is used to send this to the server. The output is the captured data.

[1078] Step 2:

[1079] Server: Analyzes the received voice data and facial image data. It uses a voice recognition means to convert the voice data into text, and then uses an emotion recognition means to analyze the emotional state from the facial image and voice. The inputs are the transmitted voice data and facial image data. The output is the analyzed text data and emotional state data.

[1080] Step 3:

[1081] Server: The text data generated by the speech recognition means is compared with the profile data to authenticate the user. The emotional state data is also compared with the profile data to confirm the passenger's emotional tendency. The inputs are text data, emotional state data, and profile data. The output is the authentication result and emotional tendency data.

[1082] Step 4:

[1083] Server: Using emotion recognition means and data analysis means, generates appropriate in-car environment settings based on the analyzed emotion data. The inputs are emotional state data, profile data, and past emotion tendency data. The output is in-car environment setting data.

[1084] Step 5:

[1085] Terminal: Using the in-car environment adjustment means, the temperature, lighting, seat position, music, etc. in the car are automatically adjusted based on the generated in-car environment setting data. The in-car environment setting data is used as input, and the adjusted in-car environment is obtained as output.

[1086] Step 6:

[1087] Terminal: After the adjustment is completed, the passenger is notified by voice that the environment has been adjusted. A predefined notification message is generated using a speech synthesis means and played through the speaker. The input is the text data of the notification message. The output is a voice notification.

[1088] In this way, each step works together to create a system that provides passenger comfort and safety.

[1089] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1090] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1091] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1092] [Third embodiment]

[1093] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1094] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1095] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1096] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1097] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1098] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1099] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1100] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1101] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1102] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1103] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1104] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1105] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing the system of the present invention will be described below.

[1106] System configuration

[1107] This system mainly consists of a terminal, a server, and a network interface. The terminal is a device that the child directly operates, and the server processes data in the backend. The network interface is responsible for connecting the terminal and the server via communication.

[1108] Program processing

[1109] 1. User Authentication

[1110] Terminal: When the power is turned on, the initialization process begins. First, ambient sound is received through the microphone. The received sound data is sent to the server in real time.

[1111] Server: Analyzes the received voice data and compares it with the voice of pre-registered users. If authentication is successful, it sends the corresponding user profile data to the device.

[1112] Terminal: The received profile data is loaded into memory and a message indicating authentication completion is given to the user.

[1113] 2. Conversations with children

[1114] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[1115] Terminal: Receives audio via microphone and transmits audio data to the server in real time.

[1116] Server: The received voice data is passed through a speech recognition engine and converted into text. The converted text data is sent to a natural language processing engine, which generates an appropriate response (e.g., "Today is math class. Let's practice addition together!").

[1117] Server: Passes the generated response text to the speech synthesis engine to generate voice data, which is then sent to the terminal.

[1118] Terminal: Plays back the received voice data and conveys the response to the child.

[1119] 3. Recording learning activities

[1120] Device: Monitors conversations and learning activities with children in real time. This data is sent to the server.

[1121] Server: Records the transmitted data in a database. Analyzes the recorded data, extracts the user's learning progress and areas of interest, and generates the next learning plan.

[1122] 4. Information sharing

[1123] Server: Aggregates learning data for specific periods and creates reports. Generated notifications are sent to parents and educators via email or a dedicated app (e.g., "Today your child learned the basics of math. Next, we'll introduce multiplication.").

[1124] Specific examples

[1125] Example 1: A child asks Tomo, "What are we going to study today?"

[1126] Terminal: Receives audio and sends it to the server.

[1127] Server: Converts the speech data into text and generates a response ("Today is math time. Let's practice addition together!").

[1128] Device: The generated response is spoken to the child.

[1129] Example 2: Parents keep track of their children's learning progress.

[1130] Server: Collects learning data and creates reports.

[1131] Server: Send the report to the parent's email address.

[1132] In this way, the system of the present invention provides an individually optimized educational environment and enables information sharing with parents and educators.

[1133] The processing flow will be explained below.

[1134] User authentication process

[1135] Step 1:

[1136] Device: When powered on, it begins an initialization process, preparing to receive ambient audio through the microphone.

[1137] Step 2:

[1138] Device: Collects voice data through a microphone and transmits the collected voice data to a server in real time.

[1139] Step 3:

[1140] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If the comparison is successful, authentication of the user is completed.

[1141] Step 4:

[1142] Server: If the authentication is successful, it sends the corresponding user profile data to the terminal.

[1143] Step 5:

[1144] Terminal: Loads the received profile data into memory.

[1145] Step 6:

[1146] Terminal: A message indicating authentication completion is given to the user via voice.

[1147] Handling conversations with children

[1148] Step 1:

[1149] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[1150] Step 2:

[1151] Terminal: Receives audio via microphone and transmits it to the server as audio data in real time.

[1152] Step 3:

[1153] Server: The received voice data is passed through a voice recognition engine and converted into text.

[1154] Step 4:

[1155] Server: Sends the converted text data to a natural language processing engine to generate an appropriate response.

[1156] Step 5:

[1157] Server: Passes the generated response text to the speech synthesis engine to generate voice data.

[1158] Step 6:

[1159] Server: Sends the generated voice data to the terminal.

[1160] Step 7:

[1161] Terminal: Plays back the received voice data and conveys the response to the child.

[1162] Processing learning activity records

[1163] Step 1:

[1164] Device: Monitor your child's conversations and learning activities in real time.

[1165] Step 2:

[1166] Terminal: Sends monitored data to the server.

[1167] Step 3:

[1168] Server: Records the transmitted data in a database.

[1169] Step 4:

[1170] Server: Analyzes the recorded data and extracts the user's learning progress and areas of interest.

[1171] Step 5:

[1172] Server: Generates the next learning plan based on the analysis results.

[1173] Information Sharing Process

[1174] Step 1:

[1175] Server: Aggregates learning data for specific periods and creates reports.

[1176] Step 2:

[1177] Server: Create notifications to share generated reports with parents and educators.

[1178] Step 3:

[1179] Server: Sends reports to parents and educators via email or a dedicated app.

[1180] Processing of specific examples

[1181] Example 1: The process of a child asking Tomo, "What are we going to study today?"

[1182] Terminal: The child's questions are received by a microphone and sent to the server as voice data.

[1183] Server: Analyzes the voice data and generates an appropriate response.

[1184] Server: Converts the generated response into audio data and sends it to the terminal.

[1185] Device: Plays the response as audio and relays it to the child.

[1186] Example 2: The process by which parents keep track of their children's learning progress.

[1187] Server: Aggregates learning data and creates reports.

[1188] Server: Generates and emails the report to the parent.

[1189] Example 1

[1190] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1191] In today's educational environment, there is a demand for systems that provide individually optimized educational support. Conventional systems lack the ability to analyze voice data in real time and generate appropriate feedback, limiting their educational effectiveness. Furthermore, they often do not effectively record and analyze learning progress or share information with parents and educators. This raises concerns that children's learning efficiency may decline.

[1192] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1193] In this invention, the server includes a user authentication means, a voice recognition means, a natural language processing means, a voice synthesis means, a learning activity monitoring means, a data sharing means, a learning activity recording means, a means for generating responses using a generative AI model, and a learning progress analysis means, which enable real-time analysis of voice data, generation of appropriate feedback, recording and analysis of learning progress, and effective information sharing with parents and educators.

[1194] "User authentication means" is a means for verifying whether a user is a registered user by using voice or other biometric information.

[1195] "Speech recognition means" refers to means having the function of converting voice data into text data.

[1196] The "audio data transmission means" is a means having a function of transmitting audio data received by the terminal to the server.

[1197] "Natural language processing means" refers to means that has the function of analyzing text data and generating an appropriate response.

[1198] The "voice synthesis means" is a means having a function of converting text data into voice data.

[1199] "Profile data recording means" means a means having the function of recording and storing profile data of authenticated users.

[1200] The "learning activity monitoring means" is a means having a function of monitoring the user's learning activity in real time and collecting the data.

[1201] "Data sharing means" refers to a means that has the function of sharing collected and analyzed learning data with parents and educators at regular intervals.

[1202] A "means for recording learning activities" is a means that has the function of recording data during learning activities.

[1203] A "means for generating a response using a generative artificial intelligence model" is a means that has the function of generating an appropriate text response to a voice input using a generative AI model.

[1204] The "learning progress analysis means" is a means having a function of analyzing collected learning data and generating learning progress and the next learning plan.

[1205] The present invention is implemented by a system constructed by combining specified hardware and software. Specific embodiments will be described below.

[1206] System configuration

[1207] This system consists of a terminal, a server, and a network interface. The terminal is a device that the child directly operates, and the server processes data in the backend. The network interface connects the terminal and the server via communication.

[1208] Hardware and software used

[1209] Device:

[1210] Microphone: Used to receive audio data.

[1211] Speaker: Used to play audio data.

[1212] Network interface: Responsible for communication with the server.

[1213] server:

[1214] Speech analysis engine: Uses Google Cloud Speech-to-Text to convert voice data into text.

[1215] Natural language processing engine: Uses OpenAI GPT-3 to generate appropriate responses from received text data.

[1216] Speech synthesis engine: Uses Amazon Polly to convert the generated text into speech data.

[1217] Database: MongoDB is used to store profile data and records of learning activities.

[1218] Data sharing system: Use Firebase to share learning progress reports with parents and educators.

[1219] System operating principle

[1220] User authentication

[1221] Device: When powered on, it starts an initialization process and receives ambient audio through its microphone, which is then sent to the server in real time.

[1222] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If authentication is successful, it sends the corresponding user profile data to the device.

[1223] Terminal: The received profile data is loaded into memory and a message indicating authentication completion is given to the user.

[1224] Conversations with children

[1225] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[1226] Terminal: Receives audio via microphone and transmits audio data to the server in real time.

[1227] Server: The received voice data is passed through a speech recognition engine and converted into text. The converted text data is sent to a natural language processing engine, which generates an appropriate response (e.g., "Today is math class. Let's practice addition together!").

[1228] Server: Passes the generated response text to the speech synthesis engine to generate voice data, which is then sent to the terminal.

[1229] Terminal: Plays back the received voice data and conveys the response to the child.

[1230] Recording learning activities

[1231] Device: Monitors conversations and learning activities with children in real time. This data is sent to the server.

[1232] Server: Records the transmitted data in a database. Analyzes the recorded data, extracts the user's learning progress and areas of interest, and generates the next learning plan.

[1233] Information sharing

[1234] Server: Collects learning data at regular intervals and creates reports. Generated notifications are sent to parents and educators via email or a dedicated app. For example, "Today, your child learned the basics of arithmetic. Next, we'll introduce multiplication."

[1235] Specific examples

[1236] Example 1: A child asks Tomo, "What are we going to study today?"

[1237] Terminal: Receives audio and sends it to the server.

[1238] Server: Converts the speech data into text and generates a response ("Today is math time. Let's practice addition together!").

[1239] Device: The generated response is spoken to the child.

[1240] Example 2: Parents keep track of their children's learning progress.

[1241] Server: Collects learning data and creates reports.

[1242] Server: Send the report to the parent's email address.

[1243] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1244] Specific explanation of processing steps

[1245] Step 1: Initialize the system

[1246] Device: When the device is powered on, it begins an initialization process, which checks system memory, verifies network connectivity, and verifies the status of devices such as the microphone and speaker. If successful, the device proceeds to the next step: user authentication.

[1247] Input: Power on

[1248] Output: Initialization complete signal

[1249] Step 2: User authentication

[1250] Device: Once initialization is complete, it receives surrounding sounds through the microphone. For example, when a child says "Hello, Tomo," the voice data is captured. The captured voice data is sent to the server in real time.

[1251] Server: Analyzes the received voice data using a voice analysis engine (e.g., Google Cloud Speech-to-Text). The analysis results are compared with pre-registered user voices. If authentication is successful, the server sends the corresponding user profile data to the device.

[1252] Terminal: Loads the received profile data into memory and generates a message indicating authentication is complete. The message is then output to the speaker, saying "Authentication complete. Hello, [username]!"

[1253] Input: Audio data

[1254] Output: User profile data, voice messages

[1255] Step 3: Talk to your child

[1256] User (child): Once authenticated, the child can talk to Tomo, for example, asking questions like, "What are we going to study today?"

[1257] Terminal: This audio is received by a microphone and sent to the server in real time as audio data.

[1258] Server: The received voice data is converted into text data through a speech recognition engine (e.g., Google Cloud Speech-to-Text). The converted text is sent to a natural language processing engine (e.g., OpenAI GPT-3), which generates an appropriate response (e.g., "Today is math class. Let's practice addition together!").

[1259] Server: Pass the generated response text to a speech synthesis engine (e.g., Amazon Polly) to generate voice data.

[1260] Server: Sends the generated voice data to the terminal.

[1261] Terminal: The received voice data is played back on the speaker and the response is conveyed to the child.

[1262] Input: Child's voice data

[1263] Output: Response voice data

[1264] Step 4: Record your learning activities

[1265] Device: Monitors your child's conversations and learning activities with Tomo in real time (e.g., collects data when answering questions like "What is 1 + 1?").

[1266] Terminal: These data are sent to the server in real time.

[1267] Server: Records the submitted data in a database (e.g., MongoDB). The recorded data is analyzed based on each data point, such as what was learned, the accuracy of the answers, and the learning time.

[1268] Server: From the analysis results, the user's learning progress and areas of interest are extracted and the next learning plan is generated.

[1269] Input: Learning activity data

[1270] Output: Analysis results, learning plan

[1271] Step 5: Information sharing

[1272] Server: Compiles learning data at regular intervals and creates reports. For example, a "weekly learning progress report" might include information such as what was learned, the percentage of correct answers, and the time spent.

[1273] Server: Generated reports are sent to parents and educators via email or a dedicated app (e.g., Firebase). For example, a report might say, "Today, your child learned the basics of math. Next, we'll introduce multiplication."

[1274] Input: Training data

[1275] Output: Learning progress report

[1276] This will improve the efficiency of learning activities and ensure effective collaboration with parents and educators.

[1277] (Application example 1)

[1278] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1279] There is a need for efficient collaboration between workers and robots in factories. However, there is no system in place that allows workers to give voice instructions to robots and have them execute them accurately and quickly. There is also a lack of appropriate support for monitoring the progress and efficiency of work in real time and planning the next work. This is reducing work efficiency in the factory and hindering overall productivity improvements.

[1280] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1281] In this invention, the server includes a user authentication means, a voice recognition means, a voice data transmission means, a natural language processing means, a voice synthesis means, a profile data recording means, a work activity monitoring means, a data sharing means, and a work instruction generation means. This allows an employee to give work instructions to a robot by voice, and the robot to appropriately process and execute those instructions. Furthermore, it is possible to monitor the progress of work in real time and optimize the next work plan, thereby improving overall work efficiency and productivity.

[1282] The "user authentication means" is a means for analyzing the employee's voice data and authenticating them by comparing it with pre-stored profile data.

[1283] "Speech recognition means" is a means for analyzing voice data and converting it into text data.

[1284] The "audio data transmission means" is a means for transmitting audio data to a server in real time.

[1285] The "natural language processing means" is a means for analyzing text data that has been speech-recognized and generating an appropriate response or work instruction.

[1286] The "voice synthesis means" is a means for generating voice data from text data.

[1287] "Profile data recording means" refers to a means for recording and managing profile data of authenticated employees.

[1288] The "work activity monitoring means" is a means for monitoring the work status of the robot in real time and recording the progress.

[1289] "Data sharing means" refers to a means for sharing work progress and analysis results with relevant parties.

[1290] The "work instruction generation means" is a means for analyzing the voice instructions of an employee and generating specific work instructions.

[1291] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing this system will be described below.

[1292] System configuration

[1293] This system mainly consists of a terminal (the control unit of the factory robot), a server, and a network interface. The terminal is a robot that operates in the factory, and the server processes data on the back end. The network interface is responsible for connecting the terminal and server via communication.

[1294] Program processing

[1295] The server implements the system using the following means:

[1296] 1. User authentication method:

[1297] When the terminal (the robot's control unit) is turned on, it receives audio through a microphone and sends the audio data to the server.

[1298] The server analyzes the received voice data using a voice recognition engine and compares it with pre-registered profile data. If authentication is successful, the corresponding employee profile data is sent to the terminal.

[1299] 2. Voice recognition means:

[1300] The server analyzes the voice data sent from the device in real time and converts it into text data, specifically using the SpeechRecognition library.

[1301] 3. Work instruction generation means:

[1302] The server sends the text data to a natural language processing engine to generate appropriate work instructions, which are then sent to the terminal, where the robot begins working based on the instructions.

[1303] 4. Work activity monitoring means:

[1304] The terminal monitors the status of the work being performed in real time and sends the data to the server.

[1305] The server records the received data in a database and analyzes the progress. The analysis results are used to generate the next work plan.

[1306] 5. Data sharing methods:

[1307] The server has a means to share the progress and efficiency of work with the relevant parties. For example, it generates a work report for each specific period and notifies the supervisor.

[1308] Hardware and software used

[1309] Hardware:

[1310] Microphone: A device for receiving employee voice.

[1311] Robot control unit: CPU and memory for processing voice data and receiving and executing work instructions.

[1312] software:

[1313] SpeechRecognition Library: A library for real-time speech recognition in Python.

[1314] Requests library: An HTTP client library for communicating with the server.

[1315] Specific examples

[1316] Employee dialogue: "Hey robot, sort the ingredients."

[1317] Server: Converts voice data into text and generates work instructions such as "Start sorting materials" based on that text data.

[1318] Terminal: Based on the work instructions sent from the server, the robot begins sorting materials.

[1319] Example of prompt input:

[1320] User: "Robot, sort the ingredients."

[1321] Response: "I'll start sorting the ingredients."

[1322] In this way, the system of the present invention supports efficient collaboration between employees and robots within a factory, improving work efficiency and productivity.

[1323] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1324] Step 1:

[1325] The user (employee) gives the robot instructions by voice, for example, "sort the materials." This voice data becomes the input.

[1326] Step 2:

[1327] The terminal (the robot's control unit) receives the user's voice through a microphone and prepares to send this voice data to the server. Here, the voice data is the input, and the output is to convert it into the format required to send it to the server. Specifically, the recorded voice data is converted into a digital format so that it can be sent over the network.

[1328] Step 3:

[1329] The server receives the voice data sent from the terminal. Here, the voice data is the input. The server converts this voice data into text data using a voice recognition system (SpeechRecognition library). The text data is the output.

[1330] Step 4:

[1331] The server sends the converted text data to a natural language processing engine, which analyzes the employee's intent. This text data is the input, and the analyzed work instructions are the output. Specifically, the instruction "sort the materials" is converted into the specific work instruction "sort the materials into the designated locations."

[1332] Step 5:

[1333] The server converts the generated work instruction data into voice data using a voice synthesis engine and prepares to send it to the terminal. Here, the work instruction data is the input and the voice data is the output.

[1334] Step 6:

[1335] The terminal receives the voice data sent from the server and issues appropriate work instructions to the robot. Here, the voice data is the input, and the robot starts a specific task (e.g., sorting materials) based on that voice data.

[1336] Step 7:

[1337] The terminal monitors the robot's work progress in real time and sends the data to the server. Here, the progress data is the input, and converting it into a format that can be sent to the server is the output.

[1338] Step 8:

[1339] The server records the received work progress data in a database and analyzes the progress. The progress data is the input and the analysis results are the output. The next work plan is generated based on the analysis results.

[1340] Step 9:

[1341] The server generates work reports for each specific period and sends them to the relevant parties. Analysis results and progress data are the input, and the work reports are the output. An example of a report might be, "Today, the robot sorted 100 items of material."

[1342] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1343] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing the system of the present invention will be described below.

[1344] System configuration

[1345] This system mainly consists of a terminal, a server, a network interface, and an emotion engine. The terminal is a device that the child directly operates, and the server processes data at the backend. The network interface is responsible for connecting the terminal and the server via communication. The emotion engine provides the functionality to recognize the user's emotions and generate responses based on them.

[1346] Program processing

[1347] 1. User Authentication

[1348] Device: When powered on, it begins an initialization process, preparing to receive ambient audio through the microphone.

[1349] Device: Collects voice data through a microphone and transmits the collected voice data to a server in real time.

[1350] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If the comparison is successful, authentication of the user is completed.

[1351] Server: If the authentication is successful, it sends the corresponding user profile data to the terminal.

[1352] Terminal: Loads the received profile data into memory.

[1353] Terminal: A message indicating authentication completion is given to the user via voice.

[1354] 2. Conversations with children

[1355] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[1356] Terminal: Receives audio via microphone and transmits it to the server as audio data in real time.

[1357] Server: The received voice data is passed through a voice recognition engine and converted into text.

[1358] Server: The converted text data is sent to a natural language processing engine to generate an appropriate response, including emotion recognition using an emotion engine.

[1359] Emotion engine: Analyzes the user's emotions and provides the results to the natural language processing engine.

[1360] Server: Generates response text including emotion recognition results and passes it to the speech synthesis engine.

[1361] Server: Sends the generated voice data to the terminal.

[1362] Terminal: Plays back the received voice data and conveys the response to the child.

[1363] 3. Recording learning activities

[1364] Device: Monitor your child's conversations and learning activities in real time.

[1365] Terminal: Sends monitored data to the server.

[1366] Server: Records the transmitted data in a database, including emotional data.

[1367] Server: Analyzes the recorded data and extracts the user's learning progress and areas of interest. The analysis also takes into account the user's emotional data.

[1368] Server: Generates the next learning plan based on the analysis results.

[1369] 4. Information sharing

[1370] Server: Aggregates learning data for specific periods and creates reports, including emotional data.

[1371] Server: Prepares generated notifications to be shared with parents and educators.

[1372] Server: Sends reports to parents and educators via email or a dedicated app.

[1373] Specific examples

[1374] Example 1: The process of a child asking Tomo, "What are we going to study today?"

[1375] Terminal: The child's questions are received by a microphone and sent to the server as voice data.

[1376] Server: Analyzes the voice data, converts it into text, and recognizes emotions using an emotion engine. Based on the response and emotion data, it generates an optimal response (e.g., "Today is math class. Let's practice addition together!").

[1377] Server: Converts the generated response into audio data and sends it to the terminal.

[1378] Device: Plays the response aloud and relays it to the child.

[1379] Example 2: The process by which parents keep track of their children's learning progress.

[1380] Server: Aggregates learning data and emotion data and creates reports.

[1381] Server: Generates and emails the report to the parent, containing information about the learning content and the child's emotional state.

[1382] In this way, the system of the present invention provides an individually optimized educational environment, and by combining it with emotion recognition, it enables more effective learning support and information sharing with parents and educators.

[1383] The processing flow will be explained below.

[1384] User authentication process

[1385] Step 1:

[1386] Device: When powered on, it begins an initialization process, preparing to receive ambient audio through the microphone.

[1387] Step 2:

[1388] Device: Collects voice data through a microphone and transmits the collected voice data to a server in real time.

[1389] Step 3:

[1390] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If the comparison is successful, authentication of the user is completed.

[1391] Step 4:

[1392] Server: If the authentication is successful, it sends the corresponding user profile data to the terminal.

[1393] Step 5:

[1394] Terminal: Loads the received profile data into memory.

[1395] Step 6:

[1396] Terminal: A message indicating authentication completion is given to the user via voice.

[1397] Handling conversations with children

[1398] Step 1:

[1399] User (child): Talk to the device (e.g., "What are we going to study today?").

[1400] Step 2:

[1401] Terminal: Receives audio via microphone and transmits it to the server as audio data in real time.

[1402] Step 3:

[1403] Server: The received voice data is passed through a voice recognition engine and converted into text.

[1404] Step 4:

[1405] Server: The converted text data is sent to a natural language processing engine to generate an appropriate response, including emotion recognition using an emotion engine.

[1406] Step 5:

[1407] Emotion engine: Analyzes emotions from the user's voice and provides the results to the natural language processing engine.

[1408] Step 6:

[1409] Server: Generates a response text that reflects the emotion recognition results and passes it to the speech synthesis engine (e.g., "Today is math class. Let's practice addition together!").

[1410] Step 7:

[1411] Server: Sends the generated voice data to the terminal.

[1412] Step 8:

[1413] Terminal: Plays back the received voice data and conveys the response to the child.

[1414] Processing learning activity records

[1415] Step 1:

[1416] Device: Monitor your child's conversations and learning activities in real time.

[1417] Step 2:

[1418] Terminal: Sends monitored data to the server.

[1419] Step 3:

[1420] Server: Records the transmitted data in a database, including emotional data.

[1421] Step 4:

[1422] Server: Analyzes the recorded data and extracts the user's learning progress and areas of interest. The analysis also takes into account the user's emotional data.

[1423] Step 5:

[1424] Server: Generates the next learning plan based on the analysis results.

[1425] Information Sharing Process

[1426] Step 1:

[1427] Server: Aggregates learning data for specific periods and creates reports, including emotional data.

[1428] Step 2:

[1429] Server: Prepares generated notifications to be shared with parents and educators.

[1430] Step 3:

[1431] Server: Sends reports to parents and educators via email or a dedicated app.

[1432] Processing of specific examples

[1433] Example 1: The process of a child asking their device, "What are we going to study today?"

[1434] Terminal: The child's questions are received by a microphone and sent to the server as voice data.

[1435] Server: Analyzes the voice data, converts it into text, and recognizes emotions using an emotion engine. Based on the response and emotion data, it generates an optimal response (e.g., "Today is math class. Let's practice addition together!").

[1436] Server: Converts the generated response into audio data and sends it to the terminal.

[1437] Device: Plays the response aloud and relays it to the child.

[1438] Example 2: The process by which parents keep track of their children's learning progress.

[1439] Server: Aggregates learning data and emotion data and creates reports.

[1440] Server: Generates and emails the report to the parent, containing information about the learning content and the child's emotional state.

[1441] Example 2

[1442] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1443] Conventional voice recognition systems are inadequate for complex processing, including user authentication and emotion recognition, making it difficult to optimize the educational environment and provide effective learning support. Furthermore, monitoring and sharing of users' emotion data and learning progress is not smooth, resulting in low quality information provided to parents and educators. There was a need to resolve these issues and provide a more advanced educational support system.

[1444] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1445] In this invention, the server includes a user authentication means, a voice data transmission means, a data analysis means, an emotion recognition means, and a means for generating a response using a generative AI model. This enables accurate authentication based on the user's voice, real-time voice data processing, and the provision of natural responses and learning plans based on emotion analysis. It also makes it possible to realize a system that efficiently collects and analyzes data related to learning progress and emotional state, providing useful information to parents and educators.

[1446] The "user authentication means" is a function that collects the user's voice via a microphone when the device is turned on, sends it to a server for analysis, and compares it with the voice of a previously registered user for authentication.

[1447] The "voice recognition means" is a function that transmits voice data collected by the terminal to a server in real time, analyzes the voice data, and converts it into text.

[1448] The "audio data transmission means" is a function that encrypts the audio data collected by the microphone and transmits it to the server in real time using a communication protocol.

[1449] "Natural language processing means" is a function that uses a natural language processing engine to analyze the converted text data and generate an appropriate response.

[1450] The "voice synthesis means" is a function that uses a voice synthesis engine to convert the generated response text into voice data.

[1451] The "profile data recording means" is a function that loads the profile data of a user into memory when the user is successfully authenticated, and records and manages that information.

[1452] The "learning activity monitoring means" is a function that monitors conversations with children and their learning activities in real time and sends the collected data to a server.

[1453] "Data sharing means" is a function that periodically compiles learning data and emotional data and provides them as reports to parents and educators.

[1454] The "emotion recognition means" is a function that analyzes the user's voice and behavior, determines their emotional state, and provides the results to the natural language processing engine.

[1455] "Means for generating a response using a generative AI model" refers to a function that generates an optimal response using a generative AI model based on received text data and emotion data.

[1456] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing the system of the present invention will be described below.

[1457] System configuration

[1458] This system mainly consists of a terminal, a server, a network interface, and an emotion engine. The terminal is a device that the user directly operates, and the server processes data at the backend. The network interface is responsible for connecting the terminal and the server via communication. The emotion engine provides the functionality to recognize the user's emotions and generate responses based on them.

[1459] Program processing

[1460] This system uses the following hardware and software to process and calculate data.

[1461] Terminal: Contains a microphone to receive the user's voice, a speaker to play the voice, and internal memory. The terminal is responsible for collecting voice data, authenticating the user, and managing profile data.

[1462] Server: Equipped with high-speed computing power for analyzing voice data, natural language processing, emotion recognition, and database management. The server uses a speech recognition engine (e.g., Google Speech-to-Text API), a natural language processing engine (e.g., OpenAI GPT-3), a speech synthesis engine (e.g., IBM Watson Text-to-Speech), and an emotion recognition engine (e.g., IBM Watson Tone Analyzer).

[1463] Network interface: A communication protocol (e.g., UDP or HTTP) is used to allow the terminal and server to send and receive data in real time.

[1464] Specific examples

[1465] User authentication

[1466] The server receives the voice data sent from the device and analyzes it using a voice recognition engine (e.g., Google Speech-to-Text API). The analyzed voice is compared with the voice of the user registered in advance, and if authentication is successful, the user's profile data is sent to the device. The device loads the profile data into memory and audibly announces authentication completion to the user (e.g., "Hello, I'm Tomo.").

[1467] Conversations with children

[1468] When a user (child) speaks to the device, the microphone collects voice data and sends it to the server. The server then uses a speech recognition engine to convert it into text and an emotion recognition engine to analyze emotions. It then uses a natural language processing engine (e.g., OpenAI GPT-3) to generate the optimal response, and then uses a speech synthesis engine to convert the response into voice data. The generated voice data is sent to the device, which then plays it back and communicates it to the user.

[1469] Recording learning activities

[1470] The device monitors the child's conversations and learning activities and sends the collected data to a server. The server records the data in a database and uses an analysis algorithm to extract learning progress and areas of interest. The next lesson plan is generated based on the analysis results.

[1471] Information sharing

[1472] The server aggregates learning and emotional data over a specific period of time and creates reports for parents and educators, which are then sent to them via email or a dedicated app.

[1473] Examples of prompt statements

[1474] Example 1: The process of a child asking Tomo, "What are we going to study today?"

[1475] Terminal: "What are we studying today?"

[1476] Server: Best response: "It's math time today. Let's practice addition together!"

[1477] Example 2: The process by which parents keep track of their children's learning progress.

[1478] Server: Report based on learning data and sentiment data

[1479] Report contents: "Information about learning content and the child's emotional state"

[1480] In this way, the system of the present invention provides an individually optimized educational environment, and by combining emotion recognition, it realizes more effective learning support and information sharing with parents and educators. The above system configuration and program processing can provide new value.

[1481] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1482] Step 1:

[1483] Device: When powered on, an initialization process begins. The device checks its internal components and enables the microphone to collect audio data. It is then ready to receive audio data.

[1484] Input: Power on, system initialization sequence

[1485] Output: Microphone ready message, interface ready state

[1486] Step 2:

[1487] Device: Collects surrounding audio data through a microphone, which is encrypted in real time and prepared for transmission to a server.

[1488] Input: Audio data

[1489] Output: Encrypted audio data packets

[1490] Step 3:

[1491] Terminal: The terminal transmits collected audio data to the server in real time using communication protocols such as UDP and HTTP.

[1492] Input: Encrypted voice data packets

[1493] Output: Request to send audio data to the server

[1494] Step 4:

[1495] Server: Decodes the received voice data and passes it to a speech recognition engine (e.g., Google Speech-to-Text API) to begin analysis. Converts the voice data into text data.

[1496] Input: Encrypted audio data

[1497] Output: Text data

[1498] Step 5:

[1499] Server: The voice data is analyzed and compared with the voice of the user registered in advance. User authentication is performed based on this comparison.

[1500] Input: Text data

[1501] Output: Authentication result (success / failure)

[1502] Step 6:

[1503] Server: If authentication is successful, retrieves the user's profile data from the database and sends it to the terminal.

[1504] Input: Authentication result (success)

[1505] Output: Request to send profile data

[1506] Step 7:

[1507] Terminal: Loads the received profile data into memory, initializes the user's settings and preferences, and prepares to provide a user-specific interface based on the profile data.

[1508] Input: Profile data

[1509] Output: User interface initialization completion message

[1510] Step 8:

[1511] On the device: Play a message to the user to let them know that authentication is complete (e.g., "Hello, this is Tomo.").

[1512] Input: Authentication successful, profile data initialization completed

[1513] Output: Playback of voice message

[1514] Step 9:

[1515] User (child): Talk to the device. For example, say, "What are we going to study today?"

[1516] Input: User question

[1517] Output: Start collecting audio data through the microphone

[1518] Step 10:

[1519] Terminal: The microphone receives the user's question and collects the voice data. This voice data is then encrypted again and sent to the server.

[1520] Input: User's voice data

[1521] Output: Encrypted audio data packets

[1522] Step 11:

[1523] Server: Passes the received voice data to the voice recognition engine and converts it into text data. The voice data is then parsed into a string format.

[1524] Input: Encrypted audio data

[1525] Output: Text data

[1526] Step 12:

[1527] Server: The converted text data is sent to an emotion recognition engine (e.g., IBM Watson Tone Analyzer) for sentiment analysis. At the same time, the text is sent to a natural language processing engine (e.g., OpenAI GPT-3) to generate an appropriate response.

[1528] Input: Text data

[1529] Output: Emotion data, primary response text

[1530] Step 13:

[1531] Server: Based on the emotion recognition results, the server uses a generative AI model to construct an optimal response. It generates a natural-sounding response by integrating the response text and emotion data.

[1532] Input: Emotion data, primary response text

[1533] Output: Best response text

[1534] Step 14:

[1535] Server: The generated response text is sent to a speech synthesis engine (e.g., IBM Watson Text-to-Speech) and converted into voice data.

[1536] Input: Best response text

[1537] Output: Audio data

[1538] Step 15:

[1539] Server: Sends the generated voice data to the terminal.

[1540] Input: Audio data

[1541] Output: Request to send audio data to the device

[1542] Step 16:

[1543] Terminal: Plays back the received voice data and communicates the response to the user (e.g., "It's math time today. Let's practice addition together!").

[1544] Input: Audio data

[1545] Output: Playback of voice message

[1546] Step 17:

[1547] Device: Continuously monitors conversations and learning activities with your child. Monitored data includes conversation content, usage time, and facial expressions (if camera is used).

[1548] Input: Learning activity data

[1549] Output: Monitoring report

[1550] Step 18:

[1551] Terminal: Collected data is sent to the server in stages. The data is encrypted and sent securely.

[1552] Input: Monitoring report

[1553] Output: A request to send data to the server

[1554] Step 19:

[1555] Server: Records the submitted data in a database and applies analytical algorithms to extract learning progress and areas of interest, using machine learning models if necessary.

[1556] Input: Learning activity data

[1557] Output: Analysis result report

[1558] Step 20:

[1559] Server: Based on the analysis results, the next learning plan is generated. This plan is optimized for each individual user.

[1560] Input: Analysis result report

[1561] Output: Next lesson plan

[1562] Step 21:

[1563] Server: Aggregates learning and sentiment data for specific time periods and creates reports for parents and educators.

[1564] Input: training data, emotion data

[1565] Output: Report

[1566] Step 22:

[1567] Server: Generated reports are sent to parents and educators via email or a dedicated app.

[1568] Input: Report

[1569] Output: Send completion notification

[1570] This allows the processing flow of the entire system to be executed in a concrete manner, providing an individually optimized educational environment.

[1571] (Application example 2)

[1572] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1573] In order to improve passenger comfort and safety in autonomous vehicles, it is necessary to analyze passengers' emotional states and requests in real time and respond appropriately. However, existing technologies lack the means to accurately recognize passengers' emotional states and optimize the in-car environment in real time. Therefore, there is a need for a system that analyzes passengers' emotional states and automatically adjusts the in-car environment.

[1574] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion recognition means, an in-vehicle environment adjustment means, and a data analysis means. This makes it possible to analyze the emotional state of passengers in real time and optimize the in-vehicle environment based on the analysis.

[1575] "User authentication means" is a function that identifies individuals through voice or facial recognition and allows them access to the system.

[1576] "Speech recognition means" is a function that converts the user's voice into text data.

[1577] The "voice data transmission means" is a function that transmits collected voice data to a server in real time.

[1578] "Natural language processing means" is a function that analyzes text data and generates an appropriate response.

[1579] The "voice synthesis means" is a function that converts the generated text data into voice data.

[1580] The "profile data recording means" is a function for recording personal information and history data of users.

[1581] The "learning activity monitoring means" is a function that monitors the user's learning activities and collects and records the data.

[1582] "Data sharing means" is a function that allows collected data to be shared with other users or systems.

[1583] "Emotion recognition means" is a function that analyzes the user's emotional state from their facial expressions and voice.

[1584] The "in-car environment adjustment means" is a function that automatically adjusts the in-car environment, such as temperature, lighting, and music, based on the analyzed emotional state.

[1585] "Data analysis means" is a function that analyzes collected data and generates the next action or response.

[1586] The present invention provides an emotion recognition system for an autonomous vehicle that analyzes passenger emotions in real time and adjusts the in-vehicle environment appropriately to maximize passenger comfort and safety. Specific embodiments for implementing this system are described below.

[1587] System configuration

[1588] This system mainly consists of the following elements:

[1589] Terminal: A device installed inside an autonomous vehicle that is equipped with a camera and microphone.

[1590] Server: A back-end system that analyzes and processes data.

[1591] Emotion recognition engine: Software that analyzes passenger emotions.

[1592] Speech recognition engine: Software that converts speech into text data.

[1593] Natural language processing engine: Software that generates appropriate responses.

[1594] IoT control system: A system that automatically adjusts the in-car environment.

[1595] Program processing procedure

[1596] 1. User Authentication

[1597] Terminal: Using a camera and microphone, the device recognizes the passenger's face and voice to identify the individual. This data is then sent to the server via a voice data transmission means.

[1598] Server: Analyzes the received voice and facial data and compares it with profile data to authenticate the user. If authentication is successful, it sends the corresponding profile data to the device.

[1599] 2. Passenger sentiment analysis

[1600] Emotion recognition engine: Analyzes passengers' facial expressions captured by the camera and the voice data sent from the voice recognition means to identify their emotional state. The emotion recognition results are sent to the data analysis means.

[1601] 3. Automatic adjustment of the in-car environment

[1602] IoT Control System: Automatically adjusts environmental settings such as in-car temperature, lighting, seat reclining angle, and music based on emotion recognition results and profile data.

[1603] Specific use cases

[1604] For example, if a passenger looks tired, the emotion recognition engine will detect this and send a command to the IoT control system, which will then recline the seat, dim the interior lights, and play soft music.

[1605] Prompt Sentence Examples

[1606] Emotion analysis prompt: "Based on the facial image of the passenger, guess their current emotional state."

[1607] Speech recognition prompt: "Please convert the passenger's request and sentiment from the recorded conversation into text and return it to us."

[1608] In this way, the system of the present invention analyzes passenger emotions in real time and adjusts the in-car environment based on that, providing a comfortable and safe travel experience.

[1609] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1610] Step 1:

[1611] Terminal: Activates the camera and microphone to capture the passenger's face and voice data. The input is the passenger's real-time face image and voice. An audio data transmission means is used to send this to the server. The output is the captured data.

[1612] Step 2:

[1613] Server: Analyzes the received voice data and facial image data. It uses a voice recognition means to convert the voice data into text, and then uses an emotion recognition means to analyze the emotional state from the facial image and voice. The inputs are the transmitted voice data and facial image data. The output is the analyzed text data and emotional state data.

[1614] Step 3:

[1615] Server: The text data generated by the speech recognition means is compared with the profile data to authenticate the user. The emotional state data is also compared with the profile data to confirm the passenger's emotional tendency. The inputs are text data, emotional state data, and profile data. The output is the authentication result and emotional tendency data.

[1616] Step 4:

[1617] Server: Using emotion recognition means and data analysis means, generates appropriate in-car environment settings based on the analyzed emotion data. The inputs are emotional state data, profile data, and past emotion tendency data. The output is in-car environment setting data.

[1618] Step 5:

[1619] Terminal: Using the in-car environment adjustment means, the temperature, lighting, seat position, music, etc. in the car are automatically adjusted based on the generated in-car environment setting data. The in-car environment setting data is used as input, and the adjusted in-car environment is obtained as output.

[1620] Step 6:

[1621] Terminal: After the adjustment is completed, the passenger is notified by voice that the environment has been adjusted. A predefined notification message is generated using a speech synthesis means and played through the speaker. The input is the text data of the notification message. The output is a voice notification.

[1622] In this way, each step works together to create a system that provides passenger comfort and safety.

[1623] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1624] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1625] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1626] [Fourth embodiment]

[1627] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1628] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1629] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1630] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1631] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1632] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1633] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1634] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1635] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1636] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1637] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1638] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1639] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1640] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing the system of the present invention will be described below.

[1641] System configuration

[1642] This system mainly consists of a terminal, a server, and a network interface. The terminal is a device that the child directly operates, and the server processes data in the backend. The network interface is responsible for connecting the terminal and the server via communication.

[1643] Program processing

[1644] 1. User Authentication

[1645] Terminal: When the power is turned on, the initialization process begins. First, ambient sound is received through the microphone. The received sound data is sent to the server in real time.

[1646] Server: Analyzes the received voice data and compares it with the voice of pre-registered users. If authentication is successful, it sends the corresponding user profile data to the device.

[1647] Terminal: The received profile data is loaded into memory and a message indicating authentication completion is given to the user.

[1648] 2. Conversations with children

[1649] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[1650] Terminal: Receives audio via microphone and transmits audio data to the server in real time.

[1651] Server: The received voice data is passed through a speech recognition engine and converted into text. The converted text data is sent to a natural language processing engine, which generates an appropriate response (e.g., "Today is math class. Let's practice addition together!").

[1652] Server: Passes the generated response text to the speech synthesis engine to generate voice data, which is then sent to the terminal.

[1653] Terminal: Plays back the received voice data and conveys the response to the child.

[1654] 3. Recording learning activities

[1655] Device: Monitors conversations and learning activities with children in real time. This data is sent to the server.

[1656] Server: Records the transmitted data in a database. Analyzes the recorded data, extracts the user's learning progress and areas of interest, and generates the next learning plan.

[1657] 4. Information sharing

[1658] Server: Aggregates learning data for specific periods and creates reports. Generated notifications are sent to parents and educators via email or a dedicated app (e.g., "Today your child learned the basics of math. Next, we'll introduce multiplication.").

[1659] Specific examples

[1660] Example 1: A child asks Tomo, "What are we going to study today?"

[1661] Terminal: Receives audio and sends it to the server.

[1662] Server: Converts the speech data into text and generates a response ("Today is math time. Let's practice addition together!").

[1663] Device: The generated response is spoken to the child.

[1664] Example 2: Parents keep track of their children's learning progress.

[1665] Server: Collects learning data and creates reports.

[1666] Server: Send the report to the parent's email address.

[1667] In this way, the system of the present invention provides an individually optimized educational environment and enables information sharing with parents and educators.

[1668] The processing flow will be explained below.

[1669] User authentication process

[1670] Step 1:

[1671] Device: When powered on, it begins an initialization process, preparing to receive ambient audio through the microphone.

[1672] Step 2:

[1673] Device: Collects voice data through a microphone and transmits the collected voice data to a server in real time.

[1674] Step 3:

[1675] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If the comparison is successful, authentication of the user is completed.

[1676] Step 4:

[1677] Server: If the authentication is successful, it sends the corresponding user profile data to the terminal.

[1678] Step 5:

[1679] Terminal: Loads the received profile data into memory.

[1680] Step 6:

[1681] Terminal: A message indicating authentication completion is given to the user via voice.

[1682] Handling conversations with children

[1683] Step 1:

[1684] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[1685] Step 2:

[1686] Terminal: Receives audio via microphone and transmits it to the server as audio data in real time.

[1687] Step 3:

[1688] Server: The received voice data is passed through a voice recognition engine and converted into text.

[1689] Step 4:

[1690] Server: Sends the converted text data to a natural language processing engine to generate an appropriate response.

[1691] Step 5:

[1692] Server: Passes the generated response text to the speech synthesis engine to generate voice data.

[1693] Step 6:

[1694] Server: Sends the generated voice data to the terminal.

[1695] Step 7:

[1696] Terminal: Plays back the received voice data and conveys the response to the child.

[1697] Processing learning activity records

[1698] Step 1:

[1699] Device: Monitor your child's conversations and learning activities in real time.

[1700] Step 2:

[1701] Terminal: Sends monitored data to the server.

[1702] Step 3:

[1703] Server: Records the transmitted data in a database.

[1704] Step 4:

[1705] Server: Analyzes the recorded data and extracts the user's learning progress and areas of interest.

[1706] Step 5:

[1707] Server: Generates the next learning plan based on the analysis results.

[1708] Information Sharing Process

[1709] Step 1:

[1710] Server: Aggregates learning data for specific periods and creates reports.

[1711] Step 2:

[1712] Server: Create notifications to share generated reports with parents and educators.

[1713] Step 3:

[1714] Server: Sends reports to parents and educators via email or a dedicated app.

[1715] Processing of specific examples

[1716] Example 1: The process of a child asking Tomo, "What are we going to study today?"

[1717] Terminal: The child's questions are received by a microphone and sent to the server as voice data.

[1718] Server: Analyzes the voice data and generates an appropriate response.

[1719] Server: Converts the generated response into audio data and sends it to the terminal.

[1720] Device: Plays the response as audio and relays it to the child.

[1721] Example 2: The process by which parents keep track of their children's learning progress.

[1722] Server: Aggregates learning data and creates reports.

[1723] Server: Generates and emails the report to the parent.

[1724] Example 1

[1725] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1726] In today's educational environment, there is a demand for systems that provide individually optimized educational support. Conventional systems lack the ability to analyze voice data in real time and generate appropriate feedback, limiting their educational effectiveness. Furthermore, they often do not effectively record and analyze learning progress or share information with parents and educators. This raises concerns that children's learning efficiency may decline.

[1727] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1728] In this invention, the server includes a user authentication means, a voice recognition means, a natural language processing means, a voice synthesis means, a learning activity monitoring means, a data sharing means, a learning activity recording means, a means for generating responses using a generative AI model, and a learning progress analysis means, which enable real-time analysis of voice data, generation of appropriate feedback, recording and analysis of learning progress, and effective information sharing with parents and educators.

[1729] "User authentication means" is a means for verifying whether a user is a registered user by using voice or other biometric information.

[1730] "Speech recognition means" refers to means having the function of converting voice data into text data.

[1731] The "audio data transmission means" is a means having a function of transmitting audio data received by the terminal to the server.

[1732] "Natural language processing means" refers to means that has the function of analyzing text data and generating an appropriate response.

[1733] The "voice synthesis means" is a means having a function of converting text data into voice data.

[1734] "Profile data recording means" means a means having the function of recording and storing profile data of authenticated users.

[1735] The "learning activity monitoring means" is a means having a function of monitoring the user's learning activity in real time and collecting the data.

[1736] "Data sharing means" refers to a means that has the function of sharing collected and analyzed learning data with parents and educators at regular intervals.

[1737] A "means for recording learning activities" is a means that has the function of recording data during learning activities.

[1738] A "means for generating a response using a generative artificial intelligence model" is a means that has the function of generating an appropriate text response to a voice input using a generative AI model.

[1739] The "learning progress analysis means" is a means having a function of analyzing collected learning data and generating learning progress and the next learning plan.

[1740] The present invention is implemented by a system constructed by combining specified hardware and software. Specific embodiments will be described below.

[1741] System configuration

[1742] This system consists of a terminal, a server, and a network interface. The terminal is a device that the child directly operates, and the server processes data in the backend. The network interface connects the terminal and the server via communication.

[1743] Hardware and software used

[1744] Device:

[1745] Microphone: Used to receive audio data.

[1746] Speaker: Used to play audio data.

[1747] Network interface: Responsible for communication with the server.

[1748] server:

[1749] Speech analysis engine: Uses Google Cloud Speech-to-Text to convert voice data into text.

[1750] Natural language processing engine: Uses OpenAI GPT-3 to generate appropriate responses from received text data.

[1751] Speech synthesis engine: Uses Amazon Polly to convert the generated text into speech data.

[1752] Database: MongoDB is used to store profile data and records of learning activities.

[1753] Data sharing system: Use Firebase to share learning progress reports with parents and educators.

[1754] System operating principle

[1755] User authentication

[1756] Device: When powered on, it starts an initialization process and receives ambient audio through its microphone, which is then sent to the server in real time.

[1757] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If authentication is successful, it sends the corresponding user profile data to the device.

[1758] Terminal: The received profile data is loaded into memory and a message indicating authentication completion is given to the user.

[1759] Conversations with children

[1760] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[1761] Terminal: Receives audio via microphone and transmits audio data to the server in real time.

[1762] Server: The received voice data is passed through a speech recognition engine and converted into text. The converted text data is sent to a natural language processing engine, which generates an appropriate response (e.g., "Today is math class. Let's practice addition together!").

[1763] Server: Passes the generated response text to the speech synthesis engine to generate voice data, which is then sent to the terminal.

[1764] Terminal: Plays back the received voice data and conveys the response to the child.

[1765] Recording learning activities

[1766] Device: Monitors conversations and learning activities with children in real time. This data is sent to the server.

[1767] Server: Records the transmitted data in a database. Analyzes the recorded data, extracts the user's learning progress and areas of interest, and generates the next learning plan.

[1768] Information sharing

[1769] Server: Collects learning data at regular intervals and creates reports. Generated notifications are sent to parents and educators via email or a dedicated app. For example, "Today, your child learned the basics of arithmetic. Next, we'll introduce multiplication."

[1770] Specific examples

[1771] Example 1: A child asks Tomo, "What are we going to study today?"

[1772] Terminal: Receives audio and sends it to the server.

[1773] Server: Converts the speech data into text and generates a response ("Today is math time. Let's practice addition together!").

[1774] Device: The generated response is spoken to the child.

[1775] Example 2: Parents keep track of their children's learning progress.

[1776] Server: Collects learning data and creates reports.

[1777] Server: Send the report to the parent's email address.

[1778] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1779] Specific explanation of processing steps

[1780] Step 1: Initialize the system

[1781] Device: When the device is powered on, it begins an initialization process, which checks system memory, verifies network connectivity, and verifies the status of devices such as the microphone and speaker. If successful, the device proceeds to the next step: user authentication.

[1782] Input: Power on

[1783] Output: Initialization complete signal

[1784] Step 2: User authentication

[1785] Device: Once initialization is complete, it receives surrounding sounds through the microphone. For example, when a child says "Hello, Tomo," the voice data is captured. The captured voice data is sent to the server in real time.

[1786] Server: Analyzes the received voice data using a voice analysis engine (e.g., Google Cloud Speech-to-Text). The analysis results are compared with pre-registered user voices. If authentication is successful, the server sends the corresponding user profile data to the device.

[1787] Terminal: Loads the received profile data into memory and generates a message indicating authentication is complete. The message is then output to the speaker, saying "Authentication complete. Hello, [username]!"

[1788] Input: Audio data

[1789] Output: User profile data, voice messages

[1790] Step 3: Talk to your child

[1791] User (child): Once authenticated, the child can talk to Tomo, for example, asking questions like, "What are we going to study today?"

[1792] Terminal: This audio is received by a microphone and sent to the server in real time as audio data.

[1793] Server: The received voice data is converted into text data through a speech recognition engine (e.g., Google Cloud Speech-to-Text). The converted text is sent to a natural language processing engine (e.g., OpenAI GPT-3), which generates an appropriate response (e.g., "Today is math class. Let's practice addition together!").

[1794] Server: Pass the generated response text to a speech synthesis engine (e.g., Amazon Polly) to generate voice data.

[1795] Server: Sends the generated voice data to the terminal.

[1796] Terminal: The received voice data is played back on the speaker and the response is conveyed to the child.

[1797] Input: Child's voice data

[1798] Output: Response voice data

[1799] Step 4: Record your learning activities

[1800] Device: Monitors your child's conversations and learning activities with Tomo in real time (e.g., collects data when answering questions like "What is 1 + 1?").

[1801] Terminal: These data are sent to the server in real time.

[1802] Server: Records the submitted data in a database (e.g., MongoDB). The recorded data is analyzed based on each data point, such as what was learned, the accuracy of the answers, and the learning time.

[1803] Server: From the analysis results, the user's learning progress and areas of interest are extracted and the next learning plan is generated.

[1804] Input: Learning activity data

[1805] Output: Analysis results, learning plan

[1806] Step 5: Information sharing

[1807] Server: Compiles learning data at regular intervals and creates reports. For example, a "weekly learning progress report" might include information such as what was learned, the percentage of correct answers, and the time spent.

[1808] Server: Generated reports are sent to parents and educators via email or a dedicated app (e.g., Firebase). For example, a report might say, "Today, your child learned the basics of math. Next, we'll introduce multiplication."

[1809] Input: Training data

[1810] Output: Learning progress report

[1811] This will improve the efficiency of learning activities and ensure effective collaboration with parents and educators.

[1812] (Application example 1)

[1813] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1814] There is a need for efficient collaboration between workers and robots in factories. However, there is no system in place that allows workers to give voice instructions to robots and have them execute them accurately and quickly. There is also a lack of appropriate support for monitoring the progress and efficiency of work in real time and planning the next work. This is reducing work efficiency in the factory and hindering overall productivity improvements.

[1815] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1816] In this invention, the server includes a user authentication means, a voice recognition means, a voice data transmission means, a natural language processing means, a voice synthesis means, a profile data recording means, a work activity monitoring means, a data sharing means, and a work instruction generation means. This allows an employee to give work instructions to a robot by voice, and the robot to appropriately process and execute those instructions. Furthermore, it is possible to monitor the progress of work in real time and optimize the next work plan, thereby improving overall work efficiency and productivity.

[1817] The "user authentication means" is a means for analyzing the employee's voice data and authenticating them by comparing it with pre-stored profile data.

[1818] "Speech recognition means" is a means for analyzing voice data and converting it into text data.

[1819] The "audio data transmission means" is a means for transmitting audio data to a server in real time.

[1820] The "natural language processing means" is a means for analyzing text data that has been speech-recognized and generating an appropriate response or work instruction.

[1821] The "voice synthesis means" is a means for generating voice data from text data.

[1822] "Profile data recording means" refers to a means for recording and managing profile data of authenticated employees.

[1823] The "work activity monitoring means" is a means for monitoring the work status of the robot in real time and recording the progress.

[1824] "Data sharing means" refers to a means for sharing work progress and analysis results with relevant parties.

[1825] The "work instruction generation means" is a means for analyzing the voice instructions of an employee and generating specific work instructions.

[1826] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing this system will be described below.

[1827] System configuration

[1828] This system mainly consists of a terminal (the control unit of the factory robot), a server, and a network interface. The terminal is a robot that operates in the factory, and the server processes data on the back end. The network interface is responsible for connecting the terminal and server via communication.

[1829] Program processing

[1830] The server implements the system using the following means:

[1831] 1. User authentication method:

[1832] When the terminal (the robot's control unit) is turned on, it receives audio through a microphone and sends the audio data to the server.

[1833] The server analyzes the received voice data using a voice recognition engine and compares it with pre-registered profile data. If authentication is successful, the corresponding employee profile data is sent to the terminal.

[1834] 2. Voice recognition means:

[1835] The server analyzes the voice data sent from the device in real time and converts it into text data, specifically using the SpeechRecognition library.

[1836] 3. Work instruction generation means:

[1837] The server sends the text data to a natural language processing engine to generate appropriate work instructions, which are then sent to the terminal, where the robot begins working based on the instructions.

[1838] 4. Work activity monitoring means:

[1839] The terminal monitors the status of the work being performed in real time and sends the data to the server.

[1840] The server records the received data in a database and analyzes the progress. The analysis results are used to generate the next work plan.

[1841] 5. Data sharing methods:

[1842] The server has a means to share the progress and efficiency of work with the relevant parties. For example, it generates a work report for each specific period and notifies the supervisor.

[1843] Hardware and software used

[1844] Hardware:

[1845] Microphone: A device for receiving employee voice.

[1846] Robot control unit: CPU and memory for processing voice data and receiving and executing work instructions.

[1847] software:

[1848] SpeechRecognition Library: A library for real-time speech recognition in Python.

[1849] Requests library: An HTTP client library for communicating with the server.

[1850] Specific examples

[1851] Employee dialogue: "Hey robot, sort the ingredients."

[1852] Server: Converts voice data into text and generates work instructions such as "Start sorting materials" based on that text data.

[1853] Terminal: Based on the work instructions sent from the server, the robot begins sorting materials.

[1854] Example of prompt input:

[1855] User: "Robot, sort the ingredients."

[1856] Response: "I'll start sorting the ingredients."

[1857] In this way, the system of the present invention supports efficient collaboration between employees and robots within a factory, improving work efficiency and productivity.

[1858] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1859] Step 1:

[1860] The user (employee) gives the robot instructions by voice, for example, "sort the materials." This voice data becomes the input.

[1861] Step 2:

[1862] The terminal (the robot's control unit) receives the user's voice through a microphone and prepares to send this voice data to the server. Here, the voice data is the input, and the output is to convert it into the format required to send it to the server. Specifically, the recorded voice data is converted into a digital format so that it can be sent over the network.

[1863] Step 3:

[1864] The server receives the voice data sent from the terminal. Here, the voice data is the input. The server converts this voice data into text data using a voice recognition system (SpeechRecognition library). The text data is the output.

[1865] Step 4:

[1866] The server sends the converted text data to a natural language processing engine, which analyzes the employee's intent. This text data is the input, and the analyzed work instructions are the output. Specifically, the instruction "sort the materials" is converted into the specific work instruction "sort the materials into the designated locations."

[1867] Step 5:

[1868] The server converts the generated work instruction data into voice data using a voice synthesis engine and prepares to send it to the terminal. Here, the work instruction data is the input and the voice data is the output.

[1869] Step 6:

[1870] The terminal receives the voice data sent from the server and issues appropriate work instructions to the robot. Here, the voice data is the input, and the robot starts a specific task (e.g., sorting materials) based on that voice data.

[1871] Step 7:

[1872] The terminal monitors the robot's work progress in real time and sends the data to the server. Here, the progress data is the input, and converting it into a format that can be sent to the server is the output.

[1873] Step 8:

[1874] The server records the received work progress data in a database and analyzes the progress. The progress data is the input and the analysis results are the output. The next work plan is generated based on the analysis results.

[1875] Step 9:

[1876] The server generates work reports for each specific period and sends them to the relevant parties. Analysis results and progress data are the input, and the work reports are the output. An example of a report might be, "Today, the robot sorted 100 items of material."

[1877] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1878] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing the system of the present invention will be described below.

[1879] System configuration

[1880] This system mainly consists of a terminal, a server, a network interface, and an emotion engine. The terminal is a device that the child directly operates, and the server processes data at the backend. The network interface is responsible for connecting the terminal and the server via communication. The emotion engine provides the functionality to recognize the user's emotions and generate responses based on them.

[1881] Program processing

[1882] 1. User Authentication

[1883] Device: When powered on, it begins an initialization process, preparing to receive ambient audio through the microphone.

[1884] Device: Collects voice data through a microphone and transmits the collected voice data to a server in real time.

[1885] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If the comparison is successful, authentication of the user is completed.

[1886] Server: If the authentication is successful, it sends the corresponding user profile data to the terminal.

[1887] Terminal: Loads the received profile data into memory.

[1888] Terminal: A message indicating authentication completion is given to the user via voice.

[1889] 2. Conversations with children

[1890] User (child): Talk to Tomo (e.g., "What are we going to study today?").

[1891] Terminal: Receives audio via microphone and transmits it to the server as audio data in real time.

[1892] Server: The received voice data is passed through a voice recognition engine and converted into text.

[1893] Server: The converted text data is sent to a natural language processing engine to generate an appropriate response, including emotion recognition using an emotion engine.

[1894] Emotion engine: Analyzes the user's emotions and provides the results to the natural language processing engine.

[1895] Server: Generates response text including emotion recognition results and passes it to the speech synthesis engine.

[1896] Server: Sends the generated voice data to the terminal.

[1897] Terminal: Plays back the received voice data and conveys the response to the child.

[1898] 3. Recording learning activities

[1899] Device: Monitor your child's conversations and learning activities in real time.

[1900] Terminal: Sends monitored data to the server.

[1901] Server: Records the transmitted data in a database, including emotional data.

[1902] Server: Analyzes the recorded data and extracts the user's learning progress and areas of interest. The analysis also takes into account the user's emotional data.

[1903] Server: Generates the next learning plan based on the analysis results.

[1904] 4. Information sharing

[1905] Server: Aggregates learning data for specific periods and creates reports, including emotional data.

[1906] Server: Prepares generated notifications to be shared with parents and educators.

[1907] Server: Sends reports to parents and educators via email or a dedicated app.

[1908] Specific examples

[1909] Example 1: The process of a child asking Tomo, "What are we going to study today?"

[1910] Terminal: The child's questions are received by a microphone and sent to the server as voice data.

[1911] Server: Analyzes the voice data, converts it into text, and recognizes emotions using an emotion engine. Based on the response and emotion data, it generates an optimal response (e.g., "Today is math class. Let's practice addition together!").

[1912] Server: Converts the generated response into audio data and sends it to the terminal.

[1913] Device: Plays the response aloud and relays it to the child.

[1914] Example 2: The process by which parents keep track of their children's learning progress.

[1915] Server: Aggregates learning data and emotion data and creates reports.

[1916] Server: Generates and emails the report to the parent, containing information about the learning content and the child's emotional state.

[1917] In this way, the system of the present invention provides an individually optimized educational environment, and by combining it with emotion recognition, it enables more effective learning support and information sharing with parents and educators.

[1918] The processing flow will be explained below.

[1919] User authentication process

[1920] Step 1:

[1921] Device: When powered on, it begins an initialization process, preparing to receive ambient audio through the microphone.

[1922] Step 2:

[1923] Device: Collects voice data through a microphone and transmits the collected voice data to a server in real time.

[1924] Step 3:

[1925] Server: Analyzes the received voice data and compares it with the voice of the user registered in advance. If the comparison is successful, authentication of the user is completed.

[1926] Step 4:

[1927] Server: If the authentication is successful, it sends the corresponding user profile data to the terminal.

[1928] Step 5:

[1929] Terminal: Loads the received profile data into memory.

[1930] Step 6:

[1931] Terminal: A message indicating authentication completion is given to the user via voice.

[1932] Handling conversations with children

[1933] Step 1:

[1934] User (child): Talk to the device (e.g., "What are we going to study today?").

[1935] Step 2:

[1936] Terminal: Receives audio via microphone and transmits it to the server as audio data in real time.

[1937] Step 3:

[1938] Server: The received voice data is passed through a voice recognition engine and converted into text.

[1939] Step 4:

[1940] Server: The converted text data is sent to a natural language processing engine to generate an appropriate response, including emotion recognition using an emotion engine.

[1941] Step 5:

[1942] Emotion engine: Analyzes emotions from the user's voice and provides the results to the natural language processing engine.

[1943] Step 6:

[1944] Server: Generates a response text that reflects the emotion recognition results and passes it to the speech synthesis engine (e.g., "Today is math class. Let's practice addition together!").

[1945] Step 7:

[1946] Server: Sends the generated voice data to the terminal.

[1947] Step 8:

[1948] Terminal: Plays back the received voice data and conveys the response to the child.

[1949] Processing learning activity records

[1950] Step 1:

[1951] Device: Monitor your child's conversations and learning activities in real time.

[1952] Step 2:

[1953] Terminal: Sends monitored data to the server.

[1954] Step 3:

[1955] Server: Records the transmitted data in a database, including emotional data.

[1956] Step 4:

[1957] Server: Analyzes the recorded data and extracts the user's learning progress and areas of interest. The analysis also takes into account the user's emotional data.

[1958] Step 5:

[1959] Server: Generates the next learning plan based on the analysis results.

[1960] Information Sharing Process

[1961] Step 1:

[1962] Server: Aggregates learning data for specific periods and creates reports, including emotional data.

[1963] Step 2:

[1964] Server: Prepares generated notifications to be shared with parents and educators.

[1965] Step 3:

[1966] Server: Sends reports to parents and educators via email or a dedicated app.

[1967] Processing of specific examples

[1968] Example 1: The process of a child asking their device, "What are we going to study today?"

[1969] Terminal: The child's questions are received by a microphone and sent to the server as voice data.

[1970] Server: Analyzes the voice data, converts it into text, and recognizes emotions using an emotion engine. Based on the response and emotion data, it generates an optimal response (e.g., "Today is math class. Let's practice addition together!").

[1971] Server: Converts the generated response into audio data and sends it to the terminal.

[1972] Device: Plays the response aloud and relays it to the child.

[1973] Example 2: The process by which parents keep track of their children's learning progress.

[1974] Server: Aggregates learning data and emotion data and creates reports.

[1975] Server: Generates and emails the report to the parent, containing information about the learning content and the child's emotional state.

[1976] Example 2

[1977] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1978] Conventional voice recognition systems are inadequate for complex processing, including user authentication and emotion recognition, making it difficult to optimize the educational environment and provide effective learning support. Furthermore, monitoring and sharing of users' emotion data and learning progress is not smooth, resulting in low quality information provided to parents and educators. There was a need to resolve these issues and provide a more advanced educational support system.

[1979] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1980] In this invention, the server includes a user authentication means, a voice data transmission means, a data analysis means, an emotion recognition means, and a means for generating a response using a generative AI model. This enables accurate authentication based on the user's voice, real-time voice data processing, and the provision of natural responses and learning plans based on emotion analysis. It also makes it possible to realize a system that efficiently collects and analyzes data related to learning progress and emotional state, providing useful information to parents and educators.

[1981] The "user authentication means" is a function that collects the user's voice via a microphone when the device is turned on, sends it to a server for analysis, and compares it with the voice of a previously registered user for authentication.

[1982] The "voice recognition means" is a function that transmits voice data collected by the terminal to a server in real time, analyzes the voice data, and converts it into text.

[1983] The "audio data transmission means" is a function that encrypts the audio data collected by the microphone and transmits it to the server in real time using a communication protocol.

[1984] "Natural language processing means" is a function that uses a natural language processing engine to analyze the converted text data and generate an appropriate response.

[1985] The "voice synthesis means" is a function that uses a voice synthesis engine to convert the generated response text into voice data.

[1986] The "profile data recording means" is a function that loads the profile data of a user into memory when the user is successfully authenticated, and records and manages that information.

[1987] The "learning activity monitoring means" is a function that monitors conversations with children and their learning activities in real time and sends the collected data to a server.

[1988] "Data sharing means" is a function that periodically compiles learning data and emotional data and provides them as reports to parents and educators.

[1989] The "emotion recognition means" is a function that analyzes the user's voice and behavior, determines their emotional state, and provides the results to the natural language processing engine.

[1990] "Means for generating a response using a generative AI model" refers to a function that generates an optimal response using a generative AI model based on received text data and emotion data.

[1991] In order to realize the present invention, a specific combination of hardware and software is required. A specific embodiment for implementing the system of the present invention will be described below.

[1992] System configuration

[1993] This system mainly consists of a terminal, a server, a network interface, and an emotion engine. The terminal is a device that the user directly operates, and the server processes data at the backend. The network interface is responsible for connecting the terminal and the server via communication. The emotion engine provides the functionality to recognize the user's emotions and generate responses based on them.

[1994] Program processing

[1995] This system uses the following hardware and software to process and calculate data.

[1996] Terminal: Contains a microphone to receive the user's voice, a speaker to play the voice, and internal memory. The terminal is responsible for collecting voice data, authenticating the user, and managing profile data.

[1997] Server: Equipped with high-speed computing power for analyzing voice data, natural language processing, emotion recognition, and database management. The server uses a speech recognition engine (e.g., Google Speech-to-Text API), a natural language processing engine (e.g., OpenAI GPT-3), a speech synthesis engine (e.g., IBM Watson Text-to-Speech), and an emotion recognition engine (e.g., IBM Watson Tone Analyzer).

[1998] Network interface: A communication protocol (e.g., UDP or HTTP) is used to allow the terminal and server to send and receive data in real time.

[1999] Specific examples

[2000] User authentication

[2001] The server receives the voice data sent from the device and analyzes it using a voice recognition engine (e.g., Google Speech-to-Text API). The analyzed voice is compared with the voice of the user registered in advance, and if authentication is successful, the user's profile data is sent to the device. The device loads the profile data into memory and audibly announces authentication completion to the user (e.g., "Hello, I'm Tomo.").

[2002] Conversations with children

[2003] When a user (child) speaks to the device, the microphone collects voice data and sends it to the server. The server then uses a speech recognition engine to convert it into text and an emotion recognition engine to analyze emotions. It then uses a natural language processing engine (e.g., OpenAI GPT-3) to generate the optimal response, and then uses a speech synthesis engine to convert the response into voice data. The generated voice data is sent to the device, which then plays it back and communicates it to the user.

[2004] Recording learning activities

[2005] The device monitors the child's conversations and learning activities and sends the collected data to a server. The server records the data in a database and uses an analysis algorithm to extract learning progress and areas of interest. The next lesson plan is generated based on the analysis results.

[2006] Information sharing

[2007] The server aggregates learning and emotional data over a specific period of time and creates reports for parents and educators, which are then sent to them via email or a dedicated app.

[2008] Examples of prompt statements

[2009] Example 1: The process of a child asking Tomo, "What are we going to study today?"

[2010] Terminal: "What are we studying today?"

[2011] Server: Best response: "It's math time today. Let's practice addition together!"

[2012] Example 2: The process by which parents keep track of their children's learning progress.

[2013] Server: Report based on learning data and sentiment data

[2014] Report contents: "Information about learning content and the child's emotional state"

[2015] In this way, the system of the present invention provides an individually optimized educational environment, and by combining emotion recognition, it realizes more effective learning support and information sharing with parents and educators. The above system configuration and program processing can provide new value.

[2016] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2017] Step 1:

[2018] Device: When powered on, an initialization process begins. The device checks its internal components and enables the microphone to collect audio data. It is then ready to receive audio data.

[2019] Input: Power on, system initialization sequence

[2020] Output: Microphone ready message, interface ready state

[2021] Step 2:

[2022] Device: Collects surrounding audio data through a microphone, which is encrypted in real time and prepared for transmission to a server.

[2023] Input: Audio data

[2024] Output: Encrypted audio data packets

[2025] Step 3:

[2026] Terminal: The terminal transmits collected audio data to the server in real time using communication protocols such as UDP and HTTP.

[2027] Input: Encrypted voice data packets

[2028] Output: Request to send audio data to the server

[2029] Step 4:

[2030] Server: Decodes the received voice data and passes it to a speech recognition engine (e.g., Google Speech-to-Text API) to begin analysis. Converts the voice data into text data.

[2031] Input: Encrypted audio data

[2032] Output: Text data

[2033] Step 5:

[2034] Server: The voice data is analyzed and compared with the voice of the user registered in advance. User authentication is performed based on this comparison.

[2035] Input: Text data

[2036] Output: Authentication result (success / failure)

[2037] Step 6:

[2038] Server: If authentication is successful, retrieves the user's profile data from the database and sends it to the terminal.

[2039] Input: Authentication result (success)

[2040] Output: Request to send profile data

[2041] Step 7:

[2042] Terminal: Loads the received profile data into memory, initializes the user's settings and preferences, and prepares to provide a user-specific interface based on the profile data.

[2043] Input: Profile data

[2044] Output: User interface initialization completion message

[2045] Step 8:

[2046] On the device: Play a message to the user to let them know that authentication is complete (e.g., "Hello, this is Tomo.").

[2047] Input: Authentication successful, profile data initialization completed

[2048] Output: Playback of voice message

[2049] Step 9:

[2050] User (child): Talk to the device. For example, say, "What are we going to study today?"

[2051] Input: User question

[2052] Output: Start collecting audio data through the microphone

[2053] Step 10:

[2054] Terminal: The microphone receives the user's question and collects the voice data. This voice data is then encrypted again and sent to the server.

[2055] Input: User's voice data

[2056] Output: Encrypted audio data packets

[2057] Step 11:

[2058] Server: Passes the received voice data to the voice recognition engine and converts it into text data. The voice data is then parsed into a string format.

[2059] Input: Encrypted audio data

[2060] Output: Text data

[2061] Step 12:

[2062] Server: The converted text data is sent to an emotion recognition engine (e.g., IBM Watson Tone Analyzer) for sentiment analysis. At the same time, the text is sent to a natural language processing engine (e.g., OpenAI GPT-3) to generate an appropriate response.

[2063] Input: Text data

[2064] Output: Emotion data, primary response text

[2065] Step 13:

[2066] Server: Based on the emotion recognition results, the server uses a generative AI model to construct an optimal response. It generates a natural-sounding response by integrating the response text and emotion data.

[2067] Input: Emotion data, primary response text

[2068] Output: Best response text

[2069] Step 14:

[2070] Server: The generated response text is sent to a speech synthesis engine (e.g., IBM Watson Text-to-Speech) and converted into voice data.

[2071] Input: Best response text

[2072] Output: Audio data

[2073] Step 15:

[2074] Server: Sends the generated voice data to the terminal.

[2075] Input: Audio data

[2076] Output: Request to send audio data to the device

[2077] Step 16:

[2078] Terminal: Plays back the received voice data and communicates the response to the user (e.g., "It's math time today. Let's practice addition together!").

[2079] Input: Audio data

[2080] Output: Playback of voice message

[2081] Step 17:

[2082] Device: Continuously monitors conversations and learning activities with your child. Monitored data includes conversation content, usage time, and facial expressions (if camera is used).

[2083] Input: Learning activity data

[2084] Output: Monitoring report

[2085] Step 18:

[2086] Terminal: Collected data is sent to the server in stages. The data is encrypted and sent securely.

[2087] Input: Monitoring report

[2088] Output: A request to send data to the server

[2089] Step 19:

[2090] Server: Records the submitted data in a database and applies analytical algorithms to extract learning progress and areas of interest, using machine learning models if necessary.

[2091] Input: Learning activity data

[2092] Output: Analysis result report

[2093] Step 20:

[2094] Server: Based on the analysis results, the next learning plan is generated. This plan is optimized for each individual user.

[2095] Input: Analysis result report

[2096] Output: Next lesson plan

[2097] Step 21:

[2098] Server: Aggregates learning and sentiment data for specific time periods and creates reports for parents and educators.

[2099] Input: training data, emotion data

[2100] Output: Report

[2101] Step 22:

[2102] Server: Generated reports are sent to parents and educators via email or a dedicated app.

[2103] Input: Report

[2104] Output: Send completion notification

[2105] This allows the processing flow of the entire system to be executed in a concrete manner, providing an individually optimized educational environment.

[2106] (Application example 2)

[2107] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2108] In order to improve passenger comfort and safety in autonomous vehicles, it is necessary to analyze passengers' emotional states and requests in real time and respond appropriately. However, existing technologies lack the means to accurately recognize passengers' emotional states and optimize the in-car environment in real time. Therefore, there is a need for a system that analyzes passengers' emotional states and automatically adjusts the in-car environment.

[2109] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion recognition means, an in-vehicle environment adjustment means, and a data analysis means. This makes it possible to analyze the emotional state of passengers in real time and optimize the in-vehicle environment based on the analysis.

[2110] "User authentication means" is a function that identifies individuals through voice or facial recognition and allows them access to the system.

[2111] "Speech recognition means" is a function that converts the user's voice into text data.

[2112] The "voice data transmission means" is a function that transmits collected voice data to a server in real time.

[2113] "Natural language processing means" is a function that analyzes text data and generates an appropriate response.

[2114] The "voice synthesis means" is a function that converts the generated text data into voice data.

[2115] The "profile data recording means" is a function for recording personal information and history data of users.

[2116] The "learning activity monitoring means" is a function that monitors the user's learning activities and collects and records the data.

[2117] "Data sharing means" is a function that allows collected data to be shared with other users or systems.

[2118] "Emotion recognition means" is a function that analyzes the user's emotional state from their facial expressions and voice.

[2119] The "in-car environment adjustment means" is a function that automatically adjusts the in-car environment, such as temperature, lighting, and music, based on the analyzed emotional state.

[2120] "Data analysis means" is a function that analyzes collected data and generates the next action or response.

[2121] The present invention provides an emotion recognition system for an autonomous vehicle that analyzes passenger emotions in real time and adjusts the in-vehicle environment appropriately to maximize passenger comfort and safety. Specific embodiments for implementing this system are described below.

[2122] System configuration

[2123] This system mainly consists of the following elements:

[2124] Terminal: A device installed inside an autonomous vehicle that is equipped with a camera and microphone.

[2125] Server: A back-end system that analyzes and processes data.

[2126] Emotion recognition engine: Software that analyzes passenger emotions.

[2127] Speech recognition engine: Software that converts speech into text data.

[2128] Natural language processing engine: Software that generates appropriate responses.

[2129] IoT control system: A system that automatically adjusts the in-car environment.

[2130] Program processing procedure

[2131] 1. User Authentication

[2132] Terminal: Using a camera and microphone, the device recognizes the passenger's face and voice to identify the individual. This data is then sent to the server via a voice data transmission means.

[2133] Server: Analyzes the received voice and facial data and compares it with profile data to authenticate the user. If authentication is successful, it sends the corresponding profile data to the device.

[2134] 2. Passenger sentiment analysis

[2135] Emotion recognition engine: Analyzes passengers' facial expressions captured by the camera and the voice data sent from the voice recognition means to identify their emotional state. The emotion recognition results are sent to the data analysis means.

[2136] 3. Automatic adjustment of the in-car environment

[2137] IoT Control System: Automatically adjusts environmental settings such as in-car temperature, lighting, seat reclining angle, and music based on emotion recognition results and profile data.

[2138] Specific use cases

[2139] For example, if a passenger looks tired, the emotion recognition engine will detect this and send a command to the IoT control system, which will then recline the seat, dim the interior lights, and play soft music.

[2140] Prompt Sentence Examples

[2141] Emotion analysis prompt: "Based on the facial image of the passenger, guess their current emotional state."

[2142] Speech recognition prompt: "Please convert the passenger's request and sentiment from the recorded conversation into text and return it to us."

[2143] In this way, the system of the present invention analyzes passenger emotions in real time and adjusts the in-car environment based on that, providing a comfortable and safe travel experience.

[2144] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2145] Step 1:

[2146] Terminal: Activates the camera and microphone to capture the passenger's face and voice data. The input is the passenger's real-time face image and voice. An audio data transmission means is used to send this to the server. The output is the captured data.

[2147] Step 2:

[2148] Server: Analyzes the received voice data and facial image data. It uses a voice recognition means to convert the voice data into text, and then uses an emotion recognition means to analyze the emotional state from the facial image and voice. The inputs are the transmitted voice data and facial image data. The output is the analyzed text data and emotional state data.

[2149] Step 3:

[2150] Server: The text data generated by the speech recognition means is compared with the profile data to authenticate the user. The emotional state data is also compared with the profile data to confirm the passenger's emotional tendency. The inputs are text data, emotional state data, and profile data. The output is the authentication result and emotional tendency data.

[2151] Step 4:

[2152] Server: Using emotion recognition means and data analysis means, generates appropriate in-car environment settings based on the analyzed emotion data. The inputs are emotional state data, profile data, and past emotion tendency data. The output is in-car environment setting data.

[2153] Step 5:

[2154] Terminal: Using the in-car environment adjustment means, the temperature, lighting, seat position, music, etc. in the car are automatically adjusted based on the generated in-car environment setting data. The in-car environment setting data is used as input, and the adjusted in-car environment is obtained as output.

[2155] Step 6:

[2156] Terminal: After the adjustment is completed, the passenger is notified by voice that the environment has been adjusted. A predefined notification message is generated using a speech synthesis means and played through the speaker. The input is the text data of the notification message. The output is a voice notification.

[2157] In this way, each step works together to create a system that provides passenger comfort and safety.

[2158] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2159] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2160] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2161] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2162] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2163] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2164] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2165] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2166] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2167] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2168] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2169] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2170] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2171] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2172] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2173] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2174] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2175] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2176] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2177] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2178] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2179] The following is further disclosed regarding the above embodiment.

[2180] (Claim 1)

[2181] A user authentication means;

[2182] a voice recognition means;

[2183] audio data transmitting means;

[2184] natural language processing means;

[2185] a speech synthesis means;

[2186] profile data recording means;

[2187] A learning activity monitoring means;

[2188] Data sharing means;

[2189] A system including:

[2190] (Claim 2)

[2191] 2. The system according to claim 1, further comprising an audio data transmitting means for transmitting the audio data to the server in real time.

[2192] (Claim 3)

[2193] 2. The system according to claim 1, further comprising a data analysis means for analyzing the progress of the learning activity and generating a next learning plan.

[2194] "Example 1"

[2195] (Claim 1)

[2196] A user authentication means;

[2197] a voice recognition means;

[2198] audio data transmitting means;

[2199] natural language processing means;

[2200] a speech synthesis means;

[2201] profile data recording means;

[2202] A learning activity monitoring means;

[2203] Data sharing means;

[2204] A means of recording learning activities;

[2205] means for generating a response using a generative artificial intelligence model;

[2206] A learning progress analysis means;

[2207] A system including:

[2208] (Claim 2)

[2209] 2. The system according to claim 1, further comprising an audio data transmitting means for transmitting the audio data to the server in real time.

[2210] (Claim 3)

[2211] 2. The system according to claim 1, further comprising a data analysis means for analyzing the progress of the learning activity and generating a next learning plan.

[2212] "Application Example 1"

[2213] (Claim 1)

[2214] A user authentication means;

[2215] a voice recognition means;

[2216] audio data transmitting means;

[2217] natural language processing means;

[2218] a speech synthesis means;

[2219] profile data recording means;

[2220] a work activity monitoring means;

[2221] Data sharing means;

[2222] A work instruction generation means;

[2223] A system including:

[2224] (Claim 2)

[2225] 2. The system according to claim 1, further comprising an audio data transmitting means for transmitting the audio data to the server in real time.

[2226] (Claim 3)

[2227] 10. The system according to claim 1, further comprising a data analysis means for analyzing the progress of work activities and generating a next work plan.

[2228] "Example 2: Combining Emotion Engines"

[2229] (Claim 1)

[2230] A user authentication means;

[2231] a voice recognition means;

[2232] audio data transmitting means;

[2233] natural language processing means;

[2234] a speech synthesis means;

[2235] profile data recording means;

[2236] A learning activity monitoring means;

[2237] Data sharing means;

[2238] An emotion recognition means;

[2239] a means for generating a response using a generative AI model;

[2240] A system including:

[2241] (Claim 2)

[2242] 2. The system according to claim 1, further comprising an audio data transmitting means for transmitting the audio data to the server in real time.

[2243] (Claim 3)

[2244] 2. The system according to claim 1, further comprising a data analysis means for analyzing the progress of the learning activity and generating a next learning plan.

[2245] "Application example 2 when combining emotion engines"

[2246] (Claim 1)

[2247] A user authentication means;

[2248] a voice recognition means;

[2249] audio data transmitting means;

[2250] natural language processing means;

[2251] a speech synthesis means;

[2252] profile data recording means;

[2253] A learning activity monitoring means;

[2254] Data sharing means;

[2255] An emotion recognition means;

[2256] A vehicle interior environment adjustment means;

[2257] A system including:

[2258] (Claim 2)

[2259] 2. The system according to claim 1, further comprising an audio data transmitting means for transmitting the audio data to the server in real time.

[2260] (Claim 3)

[2261] 2. The system according to claim 1, further comprising a data analysis means for analyzing the progress of the learning activity and generating a next learning plan. [Explanation of symbols]

[2262] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A user authentication means; a voice recognition means; audio data transmitting means; natural language processing means; a speech synthesis means; profile data recording means; A learning activity monitoring means; Data sharing means; A system including:

2. 2. The system according to claim 1, further comprising an audio data transmitting means for transmitting the audio data to the server in real time.

3. 2. The system according to claim 1, further comprising a data analysis means for analyzing the progress of the learning activity and generating a next learning plan.

Images