System

Abstract

A system is provided.SOLUTION: A system including means for receiving sensor data, means for analyzing the received sensor data using a generative AI model, and means for feeding back an analysis result to an interactive agent and an augmented reality system.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's educational environment, it is important to provide personalized education tailored to each student's learning needs and to increase opportunities for remote learning participation. However, traditional classroom and online education systems lack the ability to adequately monitor students' real-time situations or provide individualized learning support. This can result in reduced learning effectiveness and difficulty maintaining student motivation. This invention aims to utilize IoT technology and generative AI models to realize a personalized educational environment tailored to each student's individual situation. [Means for solving the problem]

[0005] The present invention provides an educational system that includes a means for receiving sensor data, a means for analyzing the received sensor data using a generative AI model, and a means for feeding back the analysis results to an interactive agent and an augmented reality system. This system enables real-time understanding of students' situations, and the interactive agent provides individually customized learning support, thereby improving learning effectiveness. Furthermore, the augmented reality system can provide students in remote locations with a learning experience similar to that of being in person. In this way, a personalized educational environment tailored to the needs of each student is realized.

[0006] "Sensor data" refers to various data measured using sensors, including physiological and behavioral data such as heart rate and activity level.

[0007] A "generative AI model" is an artificial intelligence model that is trained based on large amounts of data and is capable of performing tasks such as natural language processing and image recognition with high accuracy.

[0008] An "interactive agent" is a program or system that provides assistance and information through dialogue with the user, and uses speech recognition and natural language processing technologies.

[0009] An "augmented reality system (xR system)" refers to technology that overlays digital information on the physical world, and includes virtual reality (VR) and augmented reality (AR).

[0010] "Personalized learning support" refers to educational support that is customized according to each student's learning situation and needs, providing teaching materials and advice that are optimized for each student's characteristics.

[0011] A "distance student" is a learner who is physically away from a classroom or learning environment and who participates in education using the internet or other communication tools. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] This invention relates to an educational system that provides personalized learning support to each student by analyzing student data collected using sensors with a generative AI model and feeding the results back to an interactive agent and an augmented reality system.

[0034] Program processing flow

[0035] Receiving sensor data (server)

[0036] The server receives sensor data sent from the student's device. Specifically, it receives data such as heart rate and activity level in JSON format. For example, if a student's heart rate is 85 and their activity level is 6, it stores the data and moves on to the next analysis step.

[0037] Data analysis (server)

[0038] The server analyzes the received sensor data using a generative AI model, which is pre-trained on a large amount of data and evaluates the student's current learning status based on sensor data such as heart rate and activity level.

[0039] For example, if the heart rate is over 80, the attention level is determined to be high, and if the activity level is 6 or higher, advanced learning materials are recommended. In this way, the analyzed data is generated in the form of "Attention level: High" and "Recommended learning materials: Advanced."

[0040] Feedback to agents and xR systems (server)

[0041] The server then feeds the generated analysis results back to the interactive agent and the augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an interactive learning environment tailored to each student's individual situation.

[0042] For example, if the student's attention level is high, the agent will provide advice such as "Take a break to relax." The xR system will also display new exercises as advanced learning materials.

[0043] Specific examples

[0044] Even when students are participating in classes remotely, they can use the system of this invention to achieve the same learning benefits as if they were in a classroom on-site. For example, when a student participates in an online class from home, sensor data is sent to a server in real time, and the generative AI model analyzes it and provides appropriate feedback.

[0045] In this way, the server can provide learning support that is optimized for each student's situation, thereby improving learning effectiveness and maintaining motivation. It is expected that the present invention will greatly improve the diversity and efficiency of education.

[0046] The processing flow will be explained below.

[0047] Step 1:

[0048] The server receives sensor data from the student's device. Specifically, it obtains heart rate and activity level data sent from the student's device in JSON format. This data represents the student's current physiological and behavioral status. For example, the sensor data is received in the format {"heart_rate": 85, "activity_level": 6}.

[0049] Step 2:

[0050] The server passes the received data to a generative AI model. This generative AI model has been trained in advance using a large amount of training data and has the ability to analyze sensor data. The server calls the generative AI model and inputs the sensor data.

[0051] Step 3:

[0052] The server analyzes the sensor data using a generative AI model. Specifically, if the heart rate is over 80, the attention level is determined to be "high," and if the activity level is 6 or higher, the recommended educational material is determined to be "advanced." For example, the analyzed data is generated in the form {"attention_level": "high", "recommended_material": "advanced"}.

[0053] Step 4:

[0054] The server sends the analysis results to the interactive agent system. The server sends JSON data containing the analysis results as a POST request to the specified endpoint of the agent system. The agent system receives this data and generates appropriate advice for the student.

[0055] Step 5:

[0056] The server sends the analysis results to the augmented reality system (xR system). The server sends json data containing the analysis results as a POST request to the specified augmented reality system endpoint. The xR system receives this data and customizes the student's learning environment appropriately.

[0057] Step 6:

[0058] The user (student) receives feedback from the conversational agent and the augmented reality system. For example, the agent may advise, "Your heart rate is high, so take a short break," and the xR system may display advanced learning materials or new exercises. This personalizes the student's learning experience and enables more effective learning.

[0059] Example 1

[0060] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0061] The lack of systems to provide students in remote locations with the same learning outcomes as those in-person and to provide learning support optimized for each individual is particularly problematic, as it is difficult to grasp students' learning status and health status in real time in a remote environment, which can lead to a decline in students' concentration and learning effectiveness.

[0062] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0063] In this invention, the server includes means for collecting and transmitting sensor data from the terminal, means for receiving the sensor data at the server and storing it in a local database, means for analyzing the stored sensor data using a generative AI model, means for feeding back the analyzed results to the interactive agent and the augmented reality system, means for the interactive agent to provide appropriate advice to the user, and means for the augmented reality system to generate an interactive learning environment, thereby making it possible to provide individually customized and effective learning support to students in remote locations.

[0064] A "terminal" is a device used by a student to collect and transmit sensor data, such as a wearable device or smartphone.

[0065] "Sensor data" refers to physiological and behavioral data, such as a student's heart rate and activity level, collected from the device.

[0066] A "server" is a computer system that receives and analyzes sensor data and feeds the results back to other systems.

[0067] A "local database" is a database system for managing sensor data stored on a server.

[0068] A "generative AI model" is an artificial intelligence model that is trained on large amounts of data and analyzes sensor data to evaluate students' learning status.

[0069] An "interactive agent" is software or a system that provides appropriate advice to users and is capable of natural dialogue with humans.

[0070] An "augmented reality system" is a system that provides students with an interactive learning environment and uses technology to overlay digital information onto the real world.

[0071] "Feedback" is the process of providing the analysis results to interactive agents and augmented reality systems to present appropriate advice and educational materials to students.

[0072] This invention is an educational system that analyzes sensor data to provide personalized learning support to each student. Specifically, it uses data analysis with a generative AI model to evaluate students' learning progress in real time and provides appropriate feedback through a conversational agent and an augmented reality system.

[0073] Hardware and Software Configuration

[0074] Device:

[0075] The devices are wearable devices or smartphones used by students. These devices are equipped with heart rate sensors and accelerometers to collect data such as students' heart rates and activity levels.

[0076] server:

[0077] The server is a computer system that receives and analyzes sensor data and provides feedback to the interactive agent and augmented reality system. The server has a database management system (e.g., MySQL®) and a generative AI model (e.g., GPT-4®) installed.

[0078] Conversational Agents:

[0079] A conversational agent is software that provides appropriate advice to users. It provides advice and instructions according to the student's learning situation through natural dialogue with the user.

[0080] Augmented Reality Systems (xR Systems):

[0081] Augmented reality systems use technology to overlay digital information onto the real world, providing an interactive learning environment that helps students learn more effectively.

[0082] Data processing and calculation

[0083] Sensor data collection and transmission (terminal)

[0084] The device uses sensors to collect data such as students' heart rate and activity level, which is converted into JSON format in real time and sent to the server.

[0085] Receiving sensor data (server)

[0086] The server receives sensor data in JSON format sent from the student's device and stores it in a local database.

[0087] Data analysis (server)

[0088] The server analyzes the received sensor data using a pre-trained generative AI model, which applies an algorithm to evaluate the student's learning progress.

[0089] Analysis result feedback (server)

[0090] The server feeds the analyzed data back to the interactive agent and augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an appropriate learning environment.

[0091] Specific examples

[0092] For example, if a student is attending a class online from home, the sensor data can be sent to the server in real time, and the generative AI model can analyze it and provide appropriate feedback. You can use the following prompt to have the generative AI model analyze the sensor data:

[0093] Prompt Sentence Examples

[0094] Analyze the data for a student with a heart rate of 85 and activity level of 6. Evaluate the student's current learning situation and recommend appropriate advice and learning materials.

[0095] In this way, the server provides learning support optimized for each student's situation, improving learning effectiveness and maintaining motivation. This makes it possible to provide effective, individually customized learning support even to students in remote locations.

[0096] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0097] Step 1:

[0098] Sensor data collection and transmission (terminal)

[0099] The device collects data such as the student's heart rate and activity level. Specifically, sensors installed in wearable devices and smartphones measure data every second. For example, data such as a heart rate of 85 and an activity level of 6 may be collected. The collected data is converted into JSON format, which is then sent to a server via the Internet.

[0100] Input: Sensor-measured data on student heart rate and activity level

[0101] Output: Sensor data converted to json format

[0102] Step 2:

[0103] Receiving sensor data (server)

[0104] The server receives the sensor data in JSON format sent from the device and stores it in a local database for subsequent analysis.

[0105] Input: JSON formatted sensor data sent from the device

[0106] Output: Sensor data stored in a local database

[0107] Specific behavior:

[0108] The server checks at regular intervals for new data.

[0109] When new data is received, extract the data in json format.

[0110] The extracted data is stored in a database.

[0111] Step 3:

[0112] Data analysis (server)

[0113] The server analyzes the received sensor data using a generative AI model. Specifically, the stored data is input into a generative AI model (e.g., GPT-4) and the data is analyzed based on a prompt. For example, the prompt might be, "Please analyze the data of a student whose heart rate is 85 and whose activity level is 6." The result of this analysis is an evaluation of the student's current learning status.

[0114] Input: Sensor data stored in the local database, prompt text

[0115] Output: Evaluation results of the learning situation analyzed by the generative AI model

[0116] Specific behavior:

[0117] The server retrieves the sensor data from the database.

[0118] Input data into the generative AI model based on the prompt sentence.

[0119] The generative AI model analyzes the input data and generates an evaluation result of the learning situation.

[0120] Step 4:

[0121] Analysis result feedback (server)

[0122] The server feeds back the data analyzed by the generative AI model to the conversational agent and augmented reality system, allowing the agent to provide advice to students through dialogue and the augmented reality system to generate an interactive learning environment.

[0123] Input: Evaluation results of learning status analyzed by the generative AI model

[0124] Output: Feedback data sent to conversational agents and augmented reality systems

[0125] Specific behavior:

[0126] The server sends the analysis results to the interactive agent.

[0127] The server sends the analysis results to the augmented reality system.

[0128] Step 5:

[0129] Providing advice and educational materials to users (users)

[0130] The user receives advice from the interactive agent and learning materials from the augmented reality system. Specifically, the interactive agent provides advice such as "Take a break to relax," and the augmented reality system displays new exercises.

[0131] Input: Feedback data from conversational agents and augmented reality systems

[0132] Output: Advice and educational materials provided to the user

[0133] Specific behavior:

[0134] The user follows the instructions of the interactive agent.

[0135] The user works on exercises displayed by the augmented reality system.

[0136] (Application example 1)

[0137] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0138] In the past, customer feedback in physical stores was mainly provided non-real-time, making it difficult to provide personalized guidance and offers that matched the customer's situation.In addition, to increase customer engagement and satisfaction, it is necessary to provide appropriate information according to each customer's individual situation, but there was a lack of effective ways to achieve this.

[0139] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0140] In this invention, the server includes a means for receiving sensor data, a means for analyzing the received sensor data using a generative AI model, a means for feeding back the analysis results to the application agent and augmented reality system, a means for personalizing guidance for the application based on the fed-back analysis results, and a means for providing application information to a display device based on the fed-back analysis results. This enables brick-and-mortar stores to utilize real-time feedback from customers to provide personalized guidance and offers tailored to each customer's individual status. As a result, customer engagement and satisfaction can be significantly improved.

[0141] "Sensor data" refers to data used to collect biometric and behavioral information about customers.

[0142] A "generative AI model" is an artificial intelligence that is trained on large amounts of data, analyzes input information, and generates useful results.

[0143] "Analysis results" are information obtained from sensor data analyzed by a generative AI model, and may indicate, for example, the psychological state and activity level of customers.

[0144] The "application agent" is an interactive system that provides guidance and offers to customers based on the analysis results.

[0145] An "augmented reality system" is a system that displays digital information overlaid on the real environment.

[0146] "Personalization" refers to providing guidance and offers that are optimized to each individual customer's situation and preferences.

[0147] A "display device" is a device for visually presenting information based on analysis results, including, for example, AR glasses and smartphones.

[0148] To put the present invention into practice, it is specifically realized by the following system configuration.

[0149] First, when a customer enters a store, sensors installed in the store or a smartphone collect the customer's biometric information (such as heart rate and activity level) in real time. The collected sensor data is sent to a server in the store.

[0150] The server uses a pre-trained generative AI model to analyze the received sensor data. This generative AI model is trained on a large amount of data and evaluates the customer's current state from sensor data such as heart rate and activity level. For example, a high heart rate is determined to be a state of tension, and a high activity level is determined to be active behavior.

[0151] The analyzed data is fed back to the agent and the augmented reality system. Based on the analysis results, the agent provides customers with individually customized guidance and offers. For example, it may recommend a relaxing rest area to a nervous customer. The augmented reality system also displays personalized information to customers through AR glasses or their smartphones based on the analysis results. For example, it may display a promotion for a new product collection to an active customer.

[0152] As a specific example, the following prompt sentence is input into the generative AI model for a customer with a heart rate of 90 and an activity level of 7.

[0153] I have a customer with a heart rate of 90 and an activity level of 7. What is their condition?

[0154] Once the generative AI model returns its analysis results, the agent advises, "Try our relaxing rest space," and the AR device displays information such as, "Check out our new collection."

[0155] The server implements a means of receiving sensor data using the Flask framework. This Flask framework processes HTTP requests and receives sensor data in JSON format. The generative AI model operates using the OpenAI (registered trademark) API, and obtains analysis results using prompt statements as input. Furthermore, since the agent and augmented reality system require analysis results to be provided in real time, AR glasses or smartphones are used as display devices.

[0156] This will enable physical stores to utilize real-time feedback from customers and provide personalized guidance and offers, significantly improving customer engagement and satisfaction.

[0157] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0158] Step 1:

[0159] The server receives sensor data from customers. Specifically, biometric information such as heart rate and activity level sent from sensors in the store or smartphones is received as input data in JSON format. The received sensor data is stored in an internal database.

[0160] Step 2:

[0161] The server uses the generative AI model to analyze the received sensor data. In this step, the sensor data (heart rate, activity level, etc.) received as input is sent to the generative AI model as a prompt. For example, a prompt sentence such as "A customer with a heart rate of 90 and an activity level of 7" is created. The generative AI model analyzes this prompt sentence, evaluates the customer's condition as an analysis result, and outputs the evaluation result.

[0162] Step 3:

[0163] The server feeds back the results of the analysis by the generative AI model to the agent and the augmented reality system. The input data for this step are the analysis results from the generative AI model. Based on the analysis results, the server instructs the conversational agent to provide personalized guidance and offers to the store visitor, and at the same time outputs data for displaying information on the augmented reality system.

[0164] Step 4:

[0165] The agent provides individually customized guidance and offers to the customer based on the analysis results. In this step, the agent system receives the analysis result data from the server and generates appropriate advice and offers based on that data. The generated guidance and offers are provided to the customer as voice or text messages.

[0166] Step 5:

[0167] The augmented reality system provides visual information to store visitors based on the analysis results. In this step, the display device (AR glasses or smartphone) receives feedback data from the server and uses that data to display personalized information to store visitors. For example, promotional information for new products or directions to rest areas are visually provided as AR.

[0168] This will allow customers to receive personalized guidance and offers in real time based on their biometric information, improving their in-store experience.

[0169] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0170] The present invention relates to an educational system that provides personalized learning support by analyzing students' situations using sensor data and an emotion engine and providing feedback to an interactive agent and an augmented reality system.

[0171] Program processing flow

[0172] Receiving sensor data and emotion data (server)

[0173] The server receives sensor data and emotion data sent from the student's device. Specifically, in addition to data such as heart rate and activity level, it also obtains the student's emotional state (e.g., joy, sadness, surprise, etc.) in JSON format. For example, the sensor data is received in the following format: {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}.

[0174] Data analysis (server)

[0175] The server passes the received sensor data and emotion data to a generative AI model. This generative AI model has been trained in advance using a large amount of training data and is capable of comprehensively analyzing the student's physiological and emotional state. The server calls the generative AI model and inputs the sensor data and emotion data.

[0176] Generation of analysis results (server)

[0177] The server analyzes the data using a generative AI model. Specifically, if the heart rate is over 80, the attention level is determined to be "high," and if the activity level is 6 or higher, the recommended learning material is "advanced." If the emotion is "happy," positive feedback is recommended. For example, the analysis results are generated in the following format: {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"}.

[0178] Feedback to agents and xR systems (server)

[0179] The server then feeds the generated analysis results back to the interactive agent and the augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an interactive learning environment tailored to each student's individual situation.

[0180] Specific examples

[0181] For example, when a user (student) is participating in an online class, emotional data such as "happy" is sent along with sensor data. This data is received by the server, and the generative AI model generates analysis results such as "Attention level: high," "Recommended learning materials: advanced," and "Positive feedback." The conversational agent provides advice such as "Your heart rate is high, so take a short break," as well as positive feedback such as "Your learning attitude is excellent!" The augmented reality system displays new exercises as advanced learning materials, optimizing the learning environment to the student's situation.

[0182] In this way, the server can provide learning support optimized for each student's physiological and emotional state, thereby improving learning effectiveness and maintaining motivation. The present invention is expected to greatly improve the diversity and efficiency of education.

[0183] The processing flow will be explained below.

[0184] Step 1:

[0185] The server receives sensor data and emotion data from the student's device. The heart rate, activity level, and emotional state are sent in JSON format from the wearable device and emotion recognition camera worn by the user (student). For example, the received data is {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}.

[0186] Step 2:

[0187] The server passes the received sensor data and emotion data to the generative AI model. The server then calls the generative AI model and inputs the acquired data for analysis. This generative AI model has been trained in advance with a large amount of training data, and performs analysis by combining the sensor data and emotion data.

[0188] Step 3:

[0189] The server analyzes sensor data and emotion data using a generative AI model. Specifically, if the heart rate is over 80, the attention level is determined to be "high," and if the activity level is 6 or higher, the recommended learning material is "advanced." If the emotion is "happy," positive feedback is generated. The analysis results are {"attention_level": "high," "recommended_material": "advanced," "emotional_feedback": "positive"}.

[0190] Step 4:

[0191] The server sends the analysis results to the interactive agent system. The analysis results are sent as JSON data to the specified agent system endpoint as a POST request. For example, if the agent system is running at the URL http: / / agent-system / feedback, the analysis results are sent to this URL.

[0192] Step 5:

[0193] The server sends the analysis results to the augmented reality system (xR system). The analysis results are sent as JSON data to the specified xR system endpoint as a POST request. For example, if the xR system is running at the URL http: / / xr-system / feedback, the analysis results are sent to this URL.

[0194] Step 6:

[0195] The user (student) receives feedback from the conversational agent and the xR system. For example, the conversational agent provides advice such as "Your heart rate is high, so take a short break," and also provides positive feedback such as "Your attitude toward studying is excellent!". The augmented reality system also displays recommended "advanced learning materials" to optimize the student's learning environment.

[0196] In this way, personalized learning support is provided based on each student's physiological and emotional state, which can increase student motivation and improve learning outcomes.

[0197] Example 2

[0198] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0199] Traditional education systems have struggled to provide personalized learning support based on each student's physiological and emotional state. This has limited the ability to maximize students' concentration and learning outcomes. Furthermore, there has been a lack of means to provide students in remote locations with the same learning outcomes as those in-person.

[0200] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for receiving data from a sensor, a means for preprocessing the received data, a means for analyzing the preprocessed data using a generative AI model, a means for formatting the analysis results in JSON format, and a means for feeding back the generated analysis results to the interactive agent and the augmented reality system. This makes it possible to provide learning support optimized for each student's physiological and emotional state. Furthermore, the interactive agent provides individually customized feedback, and the augmented reality system provides an interactive learning environment, allowing students in remote locations to experience the same learning effects as those experienced on-site.

[0201] A "sensor" is a device that detects physical information and outputs that information as digital data.

[0202] A "means for receiving data" is a device or program capable of taking data transmitted by a sensor and converting it into a usable format for processing.

[0203] "Data preprocessing" refers to the process of converting received data into an analyzable format, completing incomplete data, and correcting outliers.

[0204] A "generative AI model" is an algorithm that learns from large amounts of training data and analyzes and predicts newly given data.

[0205] "Means of analyzing data" refers to the process of inputting preprocessed data, analyzing it using a generative AI model, and outputting the results.

[0206] "Means for formatting analytical results" refers to the process of converting analytical results obtained from a generative AI model into a usable form and outputting them in an appropriate format.

[0207] An "interactive agent" is a program that uses artificial intelligence to provide information and feedback to a user through dialogue.

[0208] An "augmented reality system" is a system that overlays computer-generated visual information onto real-world visual information to provide an interactive experience.

[0209] "Feedback means" refers to the process of sending the analysis results to interactive agents and augmented reality systems so that these systems can provide appropriate feedback to students.

[0210] This invention is an educational system that analyzes students' situations in real time and provides personalized learning support. This system realizes individual learning support by utilizing sensor data and an emotion engine, and providing feedback to a conversational agent and an augmented reality system.

[0211] 1. Server:

[0212] The server receives sensor data and emotion data sent from the student's device. The sensors detect the student's heart rate, activity level, and emotional state (e.g., joy, sadness, surprise, etc.) and send the data to the server in JSON format. For example, the sensor data is received in the following format: {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}.

[0213] The server preprocesses the received data and formats it for input into the generative AI model. Preprocessing includes checking the data format, filling in incomplete data, and storing it in a database.

[0214] The server passes the preprocessed data to the generative AI model for analysis. The generative AI model has been trained in advance using a large amount of training data and has the ability to comprehensively analyze the student's physiological and emotional state. An example of a prompt to invoke the generative AI model is as follows:

[0215] Input data:

[0216] {

[0217] "heart_rate": 85,

[0218] "activity_level": 6,

[0219] "emotion": "happy"

[0220] }

[0221] Output requirements:

[0222] 1. Student attention level

[0223] 2. Recommended level of study materials

[0224] 3. Emotional Feedback

[0225] The generative AI model performs the analysis, and the server formats the analysis results in JSON format. For example, if the attention level is "high," the recommended material is "advanced," and the feedback is "positive," the server outputs the results in the format {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"} .

[0226] The server feeds the analysis results back to the interactive agent and augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an interactive learning environment tailored to each student's individual situation.

[0227] 2. Terminal:

[0228] The student's device collects data from the sensors, converts it into the required format, and sends it to the server. The device continuously monitors the student's emotional and physiological state while they are learning.

[0229] 3. User (student):

[0230] For example, when a user is participating in an online class, they send sensor data and "happy" emotion data from their device to a server. The server receives this data, analyzes it using a generative AI model, and generates analysis results such as "Attention level: high," "Recommended learning materials: advanced," and "Positive feedback."

[0231] The conversational agent provides advice such as, "Your heart rate is high, so take a short break," as well as positive feedback such as, "Your attitude toward studying is excellent!" The augmented reality system displays new exercises according to the student's situation, providing an optimized learning environment.

[0232] As described above, the present invention provides learning support optimized for each student through collaboration between the server, terminal, and user, thereby improving learning effectiveness and maintaining motivation.

[0233] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0234] Step 1: Collect sensor data and emotion data (device)

[0235] The device collects data using sensors that detect the student's heart rate and activity level, and a system that determines their emotional state. Specifically, the device temporarily stores the data obtained from the sensors in a buffer memory and then collects the data at regular intervals.

[0236] Input: Biological data from sensors (heart rate, activity level) and emotional data

[0237] Output: JSON format data {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}

[0238] Step 2: Send data (terminal)

[0239] The device transmits the collected sensor data and emotion data to the server using a security protocol (e.g., HTTPS).

[0240] Input: JSON format data {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}

[0241] Output: Data received by the server

[0242] Step 3: Data Receipt and Verification (Server)

[0243] The server receives the data sent from the terminal and verifies the format and content of the data, specifically checking whether the structure of the received data is correct and whether there is any incomplete data.

[0244] Input: JSON format data sent from the terminal

[0245] Output: Validated data

[0246] Step 4: Data Preprocessing (Server)

[0247] The server performs preprocessing on the received data, specifically by completing incomplete data, correcting outliers, and formatting it into a format suitable for analysis.

[0248] Input: Validated Data

[0249] Output: Preprocessed data

[0250] Step 5: Analysis using the generative AI model (server)

[0251] The server passes the preprocessed data to the generative AI model for analysis. The following input data is provided as a prompt:

[0252] Input data:

[0253] {

[0254] "heart_rate": 85,

[0255] "activity_level": 6,

[0256] "emotion": "happy"

[0257] }

[0258] Output requirements:

[0259] 1. Student attention level

[0260] 2. Recommended level of study materials

[0261] 3. Emotional Feedback

[0262] The generative AI model analyzes this data and outputs the following analysis results:

[0263] Input: Preprocessed data

[0264] Output: Analysis result {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"}

[0265] Step 6: Formatting the analysis results (server)

[0266] The server formats the analysis results obtained from the generative AI model in JSON format, storing each item in the appropriate field to make it usable for further processing.

[0267] Input: Analysis results

[0268] Output: Parsed results in JSON format {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"}

[0269] Step 7: Feedback to the conversational agent and xR system (server)

[0270] The server then feeds back the formatted analysis results to the interactive agent and the augmented reality system. The interactive agent provides students with data-based advice, and the xR system generates a learning environment that is optimal for the student's situation.

[0271] Input: Formatted analysis results

[0272] Output: Feedback from a conversational agent, an optimized learning environment using an augmented reality system

[0273] Step 8: Running the Example (User and Server)

[0274] For example, while participating in an online class, a user sends data such as a heart rate of 85, activity level of 6, and emotion of "happy." Based on this, the server analyzes and generates high attention levels, advanced learning materials, and positive feedback. The conversational agent advises, "Your heart rate is high, so take a short break," and provides positive feedback, such as, "Your attitude toward learning is excellent!" The augmented reality system displays advanced exercises and provides an optimized learning environment.

[0275] In this way, each processing step works in concert to provide optimal learning support for each student.

[0276] (Application example 2)

[0277] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0278] This invention relates to a system that provides appropriate feedback using sensor data and emotional data. Conventional systems have difficulty providing personalized feedback tailored to the circumstances of individual students or customers, and it has been particularly difficult to provide customer service advice in real time based on the psychological and physiological state of customers in brick-and-mortar stores. Therefore, there is a need for a system that can improve customer satisfaction and provide optimal customer service support.

[0279] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0280] In this invention, the server includes a means for receiving sensor data, a means for analyzing the received sensor data and emotion data using a generative AI model, a means for feeding back the analysis results to the interactive agent and the augmented reality system, and a means for generating personalized customer service advice in the physical store based on the analysis results and displaying it on the smart devices of staff members, thereby enabling real-time analysis and optimal customer service advice based on customers' physiological and emotional data.

[0281] "Sensor data" refers to data obtained from sensors that measure the state of a living body, such as heart rate and activity level.

[0282] "Emotion data" is data that indicates emotional states such as joy, sadness, surprise, etc.

[0283] A "generative AI model" is an artificial intelligence model that is trained based on large amounts of training data and analyzes physiological and emotional data.

[0284] An "interactive agent" is a system that interacts with users and provides appropriate support and feedback.

[0285] An "augmented reality system" is a system that provides an interactive learning environment by overlaying virtual information onto the real world.

[0286] A "brick and mortar store" is a store that exists in a physical location and sells products or provides services.

[0287] "Personalized customer service advice" refers to advice that provides customized customer service methods and suggestions based on the customer's individual circumstances.

[0288] A "smart device" is an electronic device that can connect to the Internet and obtain information and perform functions through applications.

[0289] This invention is a system that provides personalized customer service advice based on physiological and emotional data of customers in brick-and-mortar stores. The system receives sensor data and emotional data, analyzes them with a generative AI model, and feeds the results back to staff members' smart devices.

[0290] Program processing

[0291] The system is programmed using the following hardware and software: The hardware used includes sensors (devices that measure heart rate, activity level, etc.) and smart devices (smartphones, smart glasses, etc.). The software uses an application that receives and analyzes sensor data and displays the analysis results, as well as a generative AI model.

[0292] The sensor data is sent to the smart device in JSON format. The smart device receives it and forwards it to the server. The server then inputs the received data into a generative AI model for analysis. This analysis determines the customer's condition based on their heart rate, activity level, and emotional state.

[0293] Specific examples of data

[0294] For example, if sensor data is sent as follows:

[0295] Sensor data: {"heart_rate": 95, "activity_level": 2, "emotion": "stressed"}

[0296] The server that receives it analyzes it and displays the following advice on the smart device:

[0297] Attention level: High

[0298] Recommendation: Suggest relaxing products

[0299] Emotional feedback: Speaking in a friendly voice

[0300] System flow

[0301] 1. Receiving sensor data:

[0302] The smart device receives real-time data from sensors, capturing heart rate, activity level, and emotional state, and sends it in JSON format to the server.

[0303] 2. Analysis using generative AI models:

[0304] The server inputs the received data into a generative AI model to analyze the customer's physiological and emotional state. For example, if the heart rate is high, activity level is low, or the emotion is analyzed as "stress," it will determine how to respond.

[0305] 3. Display Advice:

[0306] Based on the analysis results, the server displays appropriate customer service advice on the staff's smart device screen, allowing the staff to respond to the customer in accordance with this advice.

[0307] Examples of prompt statements

[0308] The following are examples of prompt sentences:

[0309] Your sensor data: {"heart_rate": 95, "activity_level": 2, "emotion": "stressed"}

[0310] Use this data to generate customer service advice to be displayed on the smart glasses display.

[0311] In this way, the present invention aims to improve customer satisfaction by enabling attentive customer service based on the customer's psychological and physiological state. Real-time analysis and feedback also enable appropriate advice to be provided at the right time. Particularly in brick-and-mortar stores, providing services tailored to the customer's current situation is expected to improve the quality of service.

[0312] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0313] Step 1:

[0314] The terminal (smart device) receives sensor data from the sensor. The sensor data includes heart rate, activity level, and emotional state, and the data is received in JSON format. This input data is used for subsequent analysis.

[0315] Step 2:

[0316] The terminal sends the received sensor data to a server. Specifically, the smart device transfers the sensor data to the server via the Internet. This data becomes the input for the server's analysis process.

[0317] Step 3:

[0318] The server inputs the received sensor data into a generative AI model for analysis. The generative AI model analyzes the heart rate, activity level, and emotional state data to determine the customer's current physiological and emotional state. For example, if the heart rate is high, the activity level is low, and the emotion is "stressed," the model may analyze the data.

[0319] Step 4:

[0320] The generative AI model generates analysis results and sends them back to the server. The analysis results include "attention level," "recommended action," and "emotional feedback," and this information becomes output data for customer support. For example, "attention level: high," "recommended action: suggest a relaxing product," and "emotional feedback: speak to the customer in a friendly manner."

[0321] Step 5:

[0322] The server then feeds back the analysis results to the terminal, which then sends customer service advice based on the analysis results to the smart device, which then displays the advice to the staff.

[0323] Step 6:

[0324] The terminal (smart device) displays the received customer service advice. Based on the displayed advice, the user (staff member) can respond optimally to the customer. For example, they can suggest products with a relaxing effect and take specific actions such as speaking to the customer in a friendly manner.

[0325] This trend will enable the provision of personalized customer service advice in real time based on the customer's physiological and emotional data, thereby improving customer satisfaction and providing optimal service.

[0326] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0327] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0328] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0329] [Second embodiment]

[0330] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0331] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0332] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0333] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0334] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0335] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0336] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0337] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0338] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0339] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0340] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0341] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0342] This invention relates to an educational system that provides personalized learning support to each student by analyzing student data collected using sensors with a generative AI model and feeding the results back to an interactive agent and an augmented reality system.

[0343] Program processing flow

[0344] Receiving sensor data (server)

[0345] The server receives sensor data sent from the student's device. Specifically, it receives data such as heart rate and activity level in JSON format. For example, if a student's heart rate is 85 and their activity level is 6, it stores the data and moves on to the next analysis step.

[0346] Data analysis (server)

[0347] The server analyzes the received sensor data using a generative AI model, which is pre-trained on a large amount of data and evaluates the student's current learning status based on sensor data such as heart rate and activity level.

[0348] For example, if the heart rate is over 80, the attention level is determined to be high, and if the activity level is 6 or higher, advanced learning materials are recommended. In this way, the analyzed data is generated in the form of "Attention level: High" and "Recommended learning materials: Advanced."

[0349] Feedback to agents and xR systems (server)

[0350] The server then feeds the generated analysis results back to the interactive agent and the augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an interactive learning environment tailored to each student's individual situation.

[0351] For example, if the student's attention level is high, the agent will provide advice such as "Take a break to relax." The xR system will also display new exercises as advanced learning materials.

[0352] Specific examples

[0353] Even when students are participating in classes remotely, they can use the system of this invention to achieve the same learning benefits as if they were in a classroom on-site. For example, when a student participates in an online class from home, sensor data is sent to a server in real time, and the generative AI model analyzes it and provides appropriate feedback.

[0354] In this way, the server can provide learning support that is optimized for each student's situation, thereby improving learning effectiveness and maintaining motivation. It is expected that the present invention will greatly improve the diversity and efficiency of education.

[0355] The processing flow will be explained below.

[0356] Step 1:

[0357] The server receives sensor data from the student's device. Specifically, it obtains heart rate and activity level data sent from the student's device in JSON format. This data represents the student's current physiological and behavioral status. For example, the sensor data is received in the format {"heart_rate": 85, "activity_level": 6}.

[0358] Step 2:

[0359] The server passes the received data to a generative AI model. This generative AI model has been trained in advance using a large amount of training data and has the ability to analyze sensor data. The server calls the generative AI model and inputs the sensor data.

[0360] Step 3:

[0361] The server analyzes the sensor data using a generative AI model. Specifically, if the heart rate is over 80, the attention level is determined to be "high," and if the activity level is 6 or higher, the recommended educational material is determined to be "advanced." For example, the analyzed data is generated in the form {"attention_level": "high", "recommended_material": "advanced"}.

[0362] Step 4:

[0363] The server sends the analysis results to the interactive agent system. The server sends JSON data containing the analysis results as a POST request to the specified endpoint of the agent system. The agent system receives this data and generates appropriate advice for the student.

[0364] Step 5:

[0365] The server sends the analysis results to the augmented reality system (xR system). The server sends json data containing the analysis results as a POST request to the specified augmented reality system endpoint. The xR system receives this data and customizes the student's learning environment appropriately.

[0366] Step 6:

[0367] The user (student) receives feedback from the conversational agent and the augmented reality system. For example, the agent may advise, "Your heart rate is high, so take a short break," and the xR system may display advanced learning materials or new exercises. This personalizes the student's learning experience and enables more effective learning.

[0368] Example 1

[0369] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0370] The lack of systems to provide students in remote locations with the same learning outcomes as those in-person and to provide learning support optimized for each individual is particularly problematic, as it is difficult to grasp students' learning status and health status in real time in a remote environment, which can lead to a decline in students' concentration and learning effectiveness.

[0371] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0372] In this invention, the server includes means for collecting and transmitting sensor data from the terminal, means for receiving the sensor data at the server and storing it in a local database, means for analyzing the stored sensor data using a generative AI model, means for feeding back the analyzed results to the interactive agent and the augmented reality system, means for the interactive agent to provide appropriate advice to the user, and means for the augmented reality system to generate an interactive learning environment, thereby making it possible to provide individually customized and effective learning support to students in remote locations.

[0373] A "terminal" is a device used by a student to collect and transmit sensor data, such as a wearable device or smartphone.

[0374] "Sensor data" refers to physiological and behavioral data, such as a student's heart rate and activity level, collected from the device.

[0375] A "server" is a computer system that receives and analyzes sensor data and feeds the results back to other systems.

[0376] A "local database" is a database system for managing sensor data stored on a server.

[0377] A "generative AI model" is an artificial intelligence model that is trained on large amounts of data and analyzes sensor data to evaluate students' learning status.

[0378] An "interactive agent" is software or a system that provides appropriate advice to users and is capable of natural dialogue with humans.

[0379] An "augmented reality system" is a system that provides students with an interactive learning environment and uses technology to overlay digital information onto the real world.

[0380] "Feedback" is the process of providing the analysis results to interactive agents and augmented reality systems to present appropriate advice and educational materials to students.

[0381] This invention is an educational system that analyzes sensor data to provide personalized learning support to each student. Specifically, it uses data analysis with a generative AI model to evaluate students' learning progress in real time and provides appropriate feedback through a conversational agent and an augmented reality system.

[0382] Hardware and Software Configuration

[0383] Device:

[0384] The devices are wearable devices or smartphones used by students. These devices are equipped with heart rate sensors and accelerometers to collect data such as students' heart rates and activity levels.

[0385] server:

[0386] The server is a computer system that receives and analyzes sensor data and feeds the results back to the interactive agent and augmented reality system. The server has a database management system (e.g., MySQL) and a generative AI model (e.g., GPT-4) installed.

[0387] Conversational Agents:

[0388] A conversational agent is software that provides appropriate advice to users. It provides advice and instructions according to the student's learning situation through natural dialogue with the user.

[0389] Augmented Reality Systems (xR Systems):

[0390] Augmented reality systems use technology to overlay digital information onto the real world, providing an interactive learning environment that helps students learn more effectively.

[0391] Data processing and calculation

[0392] Sensor data collection and transmission (terminal)

[0393] The device uses sensors to collect data such as students' heart rate and activity level, which is converted into JSON format in real time and sent to the server.

[0394] Receiving sensor data (server)

[0395] The server receives sensor data in JSON format sent from the student's device and stores it in a local database.

[0396] Data analysis (server)

[0397] The server analyzes the received sensor data using a pre-trained generative AI model, which applies an algorithm to evaluate the student's learning progress.

[0398] Analysis result feedback (server)

[0399] The server feeds the analyzed data back to the interactive agent and augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an appropriate learning environment.

[0400] Specific examples

[0401] For example, if a student is attending a class online from home, the sensor data can be sent to the server in real time, and the generative AI model can analyze it and provide appropriate feedback. You can use the following prompt to have the generative AI model analyze the sensor data:

[0402] Prompt Sentence Examples

[0403] Analyze the data for a student with a heart rate of 85 and activity level of 6. Evaluate the student's current learning situation and recommend appropriate advice and learning materials.

[0404] In this way, the server provides learning support optimized for each student's situation, improving learning effectiveness and maintaining motivation. This makes it possible to provide effective, individually customized learning support even to students in remote locations.

[0405] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0406] Step 1:

[0407] Sensor data collection and transmission (terminal)

[0408] The device collects data such as the student's heart rate and activity level. Specifically, sensors installed in wearable devices and smartphones measure data every second. For example, data such as a heart rate of 85 and an activity level of 6 may be collected. The collected data is converted into JSON format, which is then sent to a server via the Internet.

[0409] Input: Sensor-measured data on student heart rate and activity level

[0410] Output: Sensor data converted to json format

[0411] Step 2:

[0412] Receiving sensor data (server)

[0413] The server receives the sensor data in JSON format sent from the device and stores it in a local database for subsequent analysis.

[0414] Input: JSON formatted sensor data sent from the device

[0415] Output: Sensor data stored in a local database

[0416] Specific behavior:

[0417] The server checks at regular intervals for new data.

[0418] When new data is received, extract the data in json format.

[0419] The extracted data is stored in a database.

[0420] Step 3:

[0421] Data analysis (server)

[0422] The server analyzes the received sensor data using a generative AI model. Specifically, the stored data is input into a generative AI model (e.g., GPT-4) and the data is analyzed based on a prompt. For example, the prompt might be, "Please analyze the data of a student whose heart rate is 85 and whose activity level is 6." The result of this analysis is an evaluation of the student's current learning status.

[0423] Input: Sensor data stored in the local database, prompt text

[0424] Output: Evaluation results of the learning situation analyzed by the generative AI model

[0425] Specific behavior:

[0426] The server retrieves the sensor data from the database.

[0427] Input data into the generative AI model based on the prompt sentence.

[0428] The generative AI model analyzes the input data and generates an evaluation result of the learning situation.

[0429] Step 4:

[0430] Analysis result feedback (server)

[0431] The server feeds back the data analyzed by the generative AI model to the conversational agent and augmented reality system, allowing the agent to provide advice to students through dialogue and the augmented reality system to generate an interactive learning environment.

[0432] Input: Evaluation results of learning status analyzed by the generative AI model

[0433] Output: Feedback data sent to conversational agents and augmented reality systems

[0434] Specific behavior:

[0435] The server sends the analysis results to the interactive agent.

[0436] The server sends the analysis results to the augmented reality system.

[0437] Step 5:

[0438] Providing advice and educational materials to users (users)

[0439] The user receives advice from the interactive agent and learning materials from the augmented reality system. Specifically, the interactive agent provides advice such as "Take a break to relax," and the augmented reality system displays new exercises.

[0440] Input: Feedback data from conversational agents and augmented reality systems

[0441] Output: Advice and educational materials provided to the user

[0442] Specific behavior:

[0443] The user follows the instructions of the interactive agent.

[0444] The user works on exercises displayed by the augmented reality system.

[0445] (Application example 1)

[0446] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0447] In the past, customer feedback in physical stores was mainly provided non-real-time, making it difficult to provide personalized guidance and offers that matched the customer's situation.In addition, to increase customer engagement and satisfaction, it is necessary to provide appropriate information according to each customer's individual situation, but there was a lack of effective ways to achieve this.

[0448] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0449] In this invention, the server includes a means for receiving sensor data, a means for analyzing the received sensor data using a generative AI model, a means for feeding back the analysis results to the application agent and augmented reality system, a means for personalizing guidance for the application based on the fed-back analysis results, and a means for providing application information to a display device based on the fed-back analysis results. This enables brick-and-mortar stores to utilize real-time feedback from customers to provide personalized guidance and offers tailored to each customer's individual status. As a result, customer engagement and satisfaction can be significantly improved.

[0450] "Sensor data" refers to data used to collect biometric and behavioral information about customers.

[0451] A "generative AI model" is an artificial intelligence that is trained on large amounts of data, analyzes input information, and generates useful results.

[0452] "Analysis results" are information obtained from sensor data analyzed by a generative AI model, and may indicate, for example, the psychological state and activity level of customers.

[0453] The "application agent" is an interactive system that provides guidance and offers to customers based on the analysis results.

[0454] An "augmented reality system" is a system that displays digital information overlaid on the real environment.

[0455] "Personalization" refers to providing guidance and offers that are optimized to each individual customer's situation and preferences.

[0456] A "display device" is a device for visually presenting information based on analysis results, including, for example, AR glasses and smartphones.

[0457] To put the present invention into practice, it is specifically realized by the following system configuration.

[0458] First, when a customer enters a store, sensors installed in the store or a smartphone collect the customer's biometric information (such as heart rate and activity level) in real time. The collected sensor data is sent to a server in the store.

[0459] The server uses a pre-trained generative AI model to analyze the received sensor data. This generative AI model is trained on a large amount of data and evaluates the customer's current state from sensor data such as heart rate and activity level. For example, a high heart rate is determined to be a state of tension, and a high activity level is determined to be active behavior.

[0460] The analyzed data is fed back to the agent and the augmented reality system. Based on the analysis results, the agent provides customers with individually customized guidance and offers. For example, it may recommend a relaxing rest area to a nervous customer. The augmented reality system also displays personalized information to customers through AR glasses or their smartphones based on the analysis results. For example, it may display a promotion for a new product collection to an active customer.

[0461] As a specific example, the following prompt sentence is input into the generative AI model for a customer with a heart rate of 90 and an activity level of 7.

[0462] I have a customer with a heart rate of 90 and an activity level of 7. What is their condition?

[0463] Once the generative AI model returns its analysis results, the agent advises, "Try our relaxing rest space," and the AR device displays information such as, "Check out our new collection."

[0464] The server implements a means of receiving sensor data using the Flask framework. This Flask framework processes HTTP requests and receives sensor data in JSON format. The generative AI model operates using the OpenAI API, inputting prompt statements to obtain analysis results. Since the agent and augmented reality system require analysis results to be provided in real time, AR glasses or smartphones are used as display devices.

[0465] This will enable physical stores to utilize real-time feedback from customers and provide personalized guidance and offers, significantly improving customer engagement and satisfaction.

[0466] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0467] Step 1:

[0468] The server receives sensor data from customers. Specifically, biometric information such as heart rate and activity level sent from sensors in the store or smartphones is received as input data in JSON format. The received sensor data is stored in an internal database.

[0469] Step 2:

[0470] The server uses the generative AI model to analyze the received sensor data. In this step, the sensor data (heart rate, activity level, etc.) received as input is sent to the generative AI model as a prompt. For example, a prompt sentence such as "A customer with a heart rate of 90 and an activity level of 7" is created. The generative AI model analyzes this prompt sentence, evaluates the customer's condition as an analysis result, and outputs the evaluation result.

[0471] Step 3:

[0472] The server feeds back the results of the analysis by the generative AI model to the agent and the augmented reality system. The input data for this step are the analysis results from the generative AI model. Based on the analysis results, the server instructs the conversational agent to provide personalized guidance and offers to the store visitor, and at the same time outputs data for displaying information on the augmented reality system.

[0473] Step 4:

[0474] The agent provides individually customized guidance and offers to the customer based on the analysis results. In this step, the agent system receives the analysis result data from the server and generates appropriate advice and offers based on that data. The generated guidance and offers are provided to the customer as voice or text messages.

[0475] Step 5:

[0476] The augmented reality system provides visual information to store visitors based on the analysis results. In this step, the display device (AR glasses or smartphone) receives feedback data from the server and uses that data to display personalized information to store visitors. For example, promotional information for new products or directions to rest areas are visually provided as AR.

[0477] This will allow customers to receive personalized guidance and offers in real time based on their biometric information, improving their in-store experience.

[0478] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0479] The present invention relates to an educational system that provides personalized learning support by analyzing students' situations using sensor data and an emotion engine and providing feedback to an interactive agent and an augmented reality system.

[0480] Program processing flow

[0481] Receiving sensor data and emotion data (server)

[0482] The server receives sensor data and emotion data sent from the student's device. Specifically, in addition to data such as heart rate and activity level, it also obtains the student's emotional state (e.g., joy, sadness, surprise, etc.) in JSON format. For example, the sensor data is received in the following format: {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}.

[0483] Data analysis (server)

[0484] The server passes the received sensor data and emotion data to a generative AI model. This generative AI model has been trained in advance using a large amount of training data and is capable of comprehensively analyzing the student's physiological and emotional state. The server calls the generative AI model and inputs the sensor data and emotion data.

[0485] Generation of analysis results (server)

[0486] The server analyzes the data using a generative AI model. Specifically, if the heart rate is over 80, the attention level is determined to be "high," and if the activity level is 6 or higher, the recommended learning material is "advanced." If the emotion is "happy," positive feedback is recommended. For example, the analysis results are generated in the following format: {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"}.

[0487] Feedback to agents and xR systems (server)

[0488] The server then feeds the generated analysis results back to the interactive agent and the augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an interactive learning environment tailored to each student's individual situation.

[0489] Specific examples

[0490] For example, when a user (student) is participating in an online class, emotional data such as "happy" is sent along with sensor data. This data is received by the server, and the generative AI model generates analysis results such as "Attention level: high," "Recommended learning materials: advanced," and "Positive feedback." The conversational agent provides advice such as "Your heart rate is high, so take a short break," as well as positive feedback such as "Your learning attitude is excellent!" The augmented reality system displays new exercises as advanced learning materials, optimizing the learning environment to the student's situation.

[0491] In this way, the server can provide learning support optimized for each student's physiological and emotional state, thereby improving learning effectiveness and maintaining motivation. The present invention is expected to greatly improve the diversity and efficiency of education.

[0492] The processing flow will be explained below.

[0493] Step 1:

[0494] The server receives sensor data and emotion data from the student's device. The heart rate, activity level, and emotional state are sent in JSON format from the wearable device and emotion recognition camera worn by the user (student). For example, the received data is {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}.

[0495] Step 2:

[0496] The server passes the received sensor data and emotion data to the generative AI model. The server then calls the generative AI model and inputs the acquired data for analysis. This generative AI model has been trained in advance with a large amount of training data, and performs analysis by combining the sensor data and emotion data.

[0497] Step 3:

[0498] The server analyzes sensor data and emotion data using a generative AI model. Specifically, if the heart rate is over 80, the attention level is determined to be "high," and if the activity level is 6 or higher, the recommended learning material is "advanced." If the emotion is "happy," positive feedback is generated. The analysis results are {"attention_level": "high," "recommended_material": "advanced," "emotional_feedback": "positive"}.

[0499] Step 4:

[0500] The server sends the analysis results to the interactive agent system. The analysis results are sent as JSON data to the specified agent system endpoint as a POST request. For example, if the agent system is running at the URL http: / / agent-system / feedback, the analysis results are sent to this URL.

[0501] Step 5:

[0502] The server sends the analysis results to the augmented reality system (xR system). The analysis results are sent as JSON data to the specified xR system endpoint as a POST request. For example, if the xR system is running at the URL http: / / xr-system / feedback, the analysis results are sent to this URL.

[0503] Step 6:

[0504] The user (student) receives feedback from the conversational agent and the xR system. For example, the conversational agent provides advice such as "Your heart rate is high, so take a short break," and also provides positive feedback such as "Your attitude toward studying is excellent!". The augmented reality system also displays recommended "advanced learning materials" to optimize the student's learning environment.

[0505] In this way, personalized learning support is provided based on each student's physiological and emotional state, which can increase student motivation and improve learning outcomes.

[0506] Example 2

[0507] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0508] Traditional education systems have struggled to provide personalized learning support based on each student's physiological and emotional state. This has limited the ability to maximize students' concentration and learning outcomes. Furthermore, there has been a lack of means to provide students in remote locations with the same learning outcomes as those in-person.

[0509] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for receiving data from a sensor, a means for preprocessing the received data, a means for analyzing the preprocessed data using a generative AI model, a means for formatting the analysis results in JSON format, and a means for feeding back the generated analysis results to the interactive agent and the augmented reality system. This makes it possible to provide learning support optimized for each student's physiological and emotional state. Furthermore, the interactive agent provides individually customized feedback, and the augmented reality system provides an interactive learning environment, allowing students in remote locations to experience the same learning effects as those experienced on-site.

[0510] A "sensor" is a device that detects physical information and outputs that information as digital data.

[0511] A "means for receiving data" is a device or program capable of taking data transmitted by a sensor and converting it into a usable format for processing.

[0512] "Data preprocessing" refers to the process of converting received data into an analyzable format, completing incomplete data, and correcting outliers.

[0513] A "generative AI model" is an algorithm that learns from large amounts of training data and analyzes and predicts newly given data.

[0514] "Means of analyzing data" refers to the process of inputting preprocessed data, analyzing it using a generative AI model, and outputting the results.

[0515] "Means for formatting analytical results" refers to the process of converting analytical results obtained from a generative AI model into a usable form and outputting them in an appropriate format.

[0516] An "interactive agent" is a program that uses artificial intelligence to provide information and feedback to a user through dialogue.

[0517] An "augmented reality system" is a system that overlays computer-generated visual information onto real-world visual information to provide an interactive experience.

[0518] "Feedback means" refers to the process of sending the analysis results to interactive agents and augmented reality systems so that these systems can provide appropriate feedback to students.

[0519] This invention is an educational system that analyzes students' situations in real time and provides personalized learning support. This system realizes individual learning support by utilizing sensor data and an emotion engine, and providing feedback to a conversational agent and an augmented reality system.

[0520] 1. Server:

[0521] The server receives sensor data and emotion data sent from the student's device. The sensors detect the student's heart rate, activity level, and emotional state (e.g., joy, sadness, surprise, etc.) and send the data to the server in JSON format. For example, the sensor data is received in the following format: {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}.

[0522] The server preprocesses the received data and formats it for input into the generative AI model. Preprocessing includes checking the data format, filling in incomplete data, and storing it in a database.

[0523] The server passes the preprocessed data to the generative AI model for analysis. The generative AI model has been trained in advance using a large amount of training data and has the ability to comprehensively analyze the student's physiological and emotional state. An example of a prompt to invoke the generative AI model is as follows:

[0524] Input data:

[0525] {

[0526] "heart_rate": 85,

[0527] "activity_level": 6,

[0528] "emotion": "happy"

[0529] }

[0530] Output requirements:

[0531] 1. Student attention level

[0532] 2. Recommended level of study materials

[0533] 3. Emotional Feedback

[0534] The generative AI model performs the analysis, and the server formats the analysis results in JSON format. For example, if the attention level is "high," the recommended material is "advanced," and the feedback is "positive," the server outputs the results in the format {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"} .

[0535] The server feeds the analysis results back to the interactive agent and augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an interactive learning environment tailored to each student's individual situation.

[0536] 2. Terminal:

[0537] The student's device collects data from the sensors, converts it into the required format, and sends it to the server. The device continuously monitors the student's emotional and physiological state while they are learning.

[0538] 3. User (student):

[0539] For example, when a user is participating in an online class, they send sensor data and "happy" emotion data from their device to a server. The server receives this data, analyzes it using a generative AI model, and generates analysis results such as "Attention level: high," "Recommended learning materials: advanced," and "Positive feedback."

[0540] The conversational agent provides advice such as, "Your heart rate is high, so take a short break," as well as positive feedback such as, "Your attitude toward studying is excellent!" The augmented reality system displays new exercises according to the student's situation, providing an optimized learning environment.

[0541] As described above, the present invention provides learning support optimized for each student through collaboration between the server, terminal, and user, thereby improving learning effectiveness and maintaining motivation.

[0542] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0543] Step 1: Collect sensor data and emotion data (device)

[0544] The device collects data using sensors that detect the student's heart rate and activity level, and a system that determines their emotional state. Specifically, the device temporarily stores the data obtained from the sensors in a buffer memory and then collects the data at regular intervals.

[0545] Input: Biological data from sensors (heart rate, activity level) and emotional data

[0546] Output: JSON format data {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}

[0547] Step 2: Send data (terminal)

[0548] The device transmits the collected sensor data and emotion data to the server using a security protocol (e.g., HTTPS).

[0549] Input: JSON format data {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}

[0550] Output: Data received by the server

[0551] Step 3: Data Receipt and Verification (Server)

[0552] The server receives the data sent from the terminal and verifies the format and content of the data, specifically checking whether the structure of the received data is correct and whether there is any incomplete data.

[0553] Input: JSON format data sent from the terminal

[0554] Output: Validated data

[0555] Step 4: Data Preprocessing (Server)

[0556] The server performs preprocessing on the received data, specifically by completing incomplete data, correcting outliers, and formatting it into a format suitable for analysis.

[0557] Input: Validated Data

[0558] Output: Preprocessed data

[0559] Step 5: Analysis using the generative AI model (server)

[0560] The server passes the preprocessed data to the generative AI model for analysis. The following input data is provided as a prompt:

[0561] Input data:

[0562] {

[0563] "heart_rate": 85,

[0564] "activity_level": 6,

[0565] "emotion": "happy"

[0566] }

[0567] Output requirements:

[0568] 1. Student attention level

[0569] 2. Recommended level of study materials

[0570] 3. Emotional Feedback

[0571] The generative AI model analyzes this data and outputs the following analysis results:

[0572] Input: Preprocessed data

[0573] Output: Analysis result {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"}

[0574] Step 6: Formatting the analysis results (server)

[0575] The server formats the analysis results obtained from the generative AI model in JSON format, storing each item in the appropriate field to make it usable for further processing.

[0576] Input: Analysis results

[0577] Output: Parsed results in JSON format {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"}

[0578] Step 7: Feedback to the conversational agent and xR system (server)

[0579] The server then feeds back the formatted analysis results to the interactive agent and the augmented reality system. The interactive agent provides students with data-based advice, and the xR system generates a learning environment that is optimal for the student's situation.

[0580] Input: Formatted analysis results

[0581] Output: Feedback from a conversational agent, an optimized learning environment using an augmented reality system

[0582] Step 8: Running the Example (User and Server)

[0583] For example, while participating in an online class, a user sends data such as a heart rate of 85, activity level of 6, and emotion of "happy." Based on this, the server analyzes and generates high attention levels, advanced learning materials, and positive feedback. The conversational agent advises, "Your heart rate is high, so take a short break," and provides positive feedback, such as, "Your attitude toward learning is excellent!" The augmented reality system displays advanced exercises and provides an optimized learning environment.

[0584] In this way, each processing step works in concert to provide optimal learning support for each student.

[0585] (Application example 2)

[0586] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0587] This invention relates to a system that provides appropriate feedback using sensor data and emotional data. Conventional systems have difficulty providing personalized feedback tailored to the circumstances of individual students or customers, and it has been particularly difficult to provide customer service advice in real time based on the psychological and physiological state of customers in brick-and-mortar stores. Therefore, there is a need for a system that can improve customer satisfaction and provide optimal customer service support.

[0588] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0589] In this invention, the server includes a means for receiving sensor data, a means for analyzing the received sensor data and emotion data using a generative AI model, a means for feeding back the analysis results to the interactive agent and the augmented reality system, and a means for generating personalized customer service advice in the physical store based on the analysis results and displaying it on the smart devices of staff members, thereby enabling real-time analysis and optimal customer service advice based on customers' physiological and emotional data.

[0590] "Sensor data" refers to data obtained from sensors that measure the state of a living body, such as heart rate and activity level.

[0591] "Emotion data" is data that indicates emotional states such as joy, sadness, surprise, etc.

[0592] A "generative AI model" is an artificial intelligence model that is trained based on large amounts of training data and analyzes physiological and emotional data.

[0593] An "interactive agent" is a system that interacts with users and provides appropriate support and feedback.

[0594] An "augmented reality system" is a system that provides an interactive learning environment by overlaying virtual information onto the real world.

[0595] A "brick and mortar store" is a store that exists in a physical location and sells products or provides services.

[0596] "Personalized customer service advice" refers to advice that provides customized customer service methods and suggestions based on the customer's individual circumstances.

[0597] A "smart device" is an electronic device that can connect to the Internet and obtain information and perform functions through applications.

[0598] This invention is a system that provides personalized customer service advice based on physiological and emotional data of customers in brick-and-mortar stores. The system receives sensor data and emotional data, analyzes them with a generative AI model, and feeds the results back to staff members' smart devices.

[0599] Program processing

[0600] The system is programmed using the following hardware and software: The hardware used includes sensors (devices that measure heart rate, activity level, etc.) and smart devices (smartphones, smart glasses, etc.). The software uses an application that receives and analyzes sensor data and displays the analysis results, as well as a generative AI model.

[0601] The sensor data is sent to the smart device in JSON format. The smart device receives it and forwards it to the server. The server then inputs the received data into a generative AI model for analysis. This analysis determines the customer's condition based on their heart rate, activity level, and emotional state.

[0602] Specific examples of data

[0603] For example, if sensor data is sent as follows:

[0604] Sensor data: {"heart_rate": 95, "activity_level": 2, "emotion": "stressed"}

[0605] The server that receives it analyzes it and displays the following advice on the smart device:

[0606] Attention level: High

[0607] Recommendation: Suggest relaxing products

[0608] Emotional feedback: Speaking in a friendly voice

[0609] System flow

[0610] 1. Receiving sensor data:

[0611] The smart device receives real-time data from sensors, capturing heart rate, activity level, and emotional state, and sends it in JSON format to the server.

[0612] 2. Analysis using generative AI models:

[0613] The server inputs the received data into a generative AI model to analyze the customer's physiological and emotional state. For example, if the heart rate is high, activity level is low, or the emotion is analyzed as "stress," it will determine how to respond.

[0614] 3. Display Advice:

[0615] Based on the analysis results, the server displays appropriate customer service advice on the staff's smart device screen, allowing the staff to respond to the customer in accordance with this advice.

[0616] Examples of prompt statements

[0617] The following are examples of prompt sentences:

[0618] Your sensor data: {"heart_rate": 95, "activity_level": 2, "emotion": "stressed"}

[0619] Use this data to generate customer service advice to be displayed on the smart glasses display.

[0620] In this way, the present invention aims to improve customer satisfaction by enabling attentive customer service based on the customer's psychological and physiological state. Real-time analysis and feedback also enable appropriate advice to be provided at the right time. Particularly in brick-and-mortar stores, providing services tailored to the customer's current situation is expected to improve the quality of service.

[0621] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0622] Step 1:

[0623] The terminal (smart device) receives sensor data from the sensor. The sensor data includes heart rate, activity level, and emotional state, and the data is received in JSON format. This input data is used for subsequent analysis.

[0624] Step 2:

[0625] The terminal sends the received sensor data to a server. Specifically, the smart device transfers the sensor data to the server via the Internet. This data becomes the input for the server's analysis process.

[0626] Step 3:

[0627] The server inputs the received sensor data into a generative AI model for analysis. The generative AI model analyzes the heart rate, activity level, and emotional state data to determine the customer's current physiological and emotional state. For example, if the heart rate is high, the activity level is low, and the emotion is "stressed," the model may analyze the data.

[0628] Step 4:

[0629] The generative AI model generates analysis results and sends them back to the server. The analysis results include "attention level," "recommended action," and "emotional feedback," and this information becomes output data for customer support. For example, "attention level: high," "recommended action: suggest a relaxing product," and "emotional feedback: speak to the customer in a friendly manner."

[0630] Step 5:

[0631] The server then feeds back the analysis results to the terminal, which then sends customer service advice based on the analysis results to the smart device, which then displays the advice to the staff.

[0632] Step 6:

[0633] The terminal (smart device) displays the received customer service advice. Based on the displayed advice, the user (staff member) can respond optimally to the customer. For example, they can suggest products with a relaxing effect and take specific actions such as speaking to the customer in a friendly manner.

[0634] This trend will enable the provision of personalized customer service advice in real time based on the customer's physiological and emotional data, thereby improving customer satisfaction and providing optimal service.

[0635] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0636] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0637] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0638] [Third embodiment]

[0639] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0640] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0641] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0642] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0643] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0644] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0645] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0646] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0647] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0648] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0649] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0650] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0651] This invention relates to an educational system that provides personalized learning support to each student by analyzing student data collected using sensors with a generative AI model and feeding the results back to an interactive agent and an augmented reality system.

[0652] Program processing flow

[0653] Receiving sensor data (server)

[0654] The server receives sensor data sent from the student's device. Specifically, it receives data such as heart rate and activity level in JSON format. For example, if a student's heart rate is 85 and their activity level is 6, it stores the data and moves on to the next analysis step.

[0655] Data analysis (server)

[0656] The server analyzes the received sensor data using a generative AI model, which is pre-trained on a large amount of data and evaluates the student's current learning status based on sensor data such as heart rate and activity level.

[0657] For example, if the heart rate is over 80, the attention level is determined to be high, and if the activity level is 6 or higher, advanced learning materials are recommended. In this way, the analyzed data is generated in the form of "Attention level: High" and "Recommended learning materials: Advanced."

[0658] Feedback to agents and xR systems (server)

[0659] The server then feeds the generated analysis results back to the interactive agent and the augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an interactive learning environment tailored to each student's individual situation.

[0660] For example, if the student's attention level is high, the agent will provide advice such as "Take a break to relax." The xR system will also display new exercises as advanced learning materials.

[0661] Specific examples

[0662] Even when students are participating in classes remotely, they can use the system of this invention to achieve the same learning benefits as if they were in a classroom on-site. For example, when a student participates in an online class from home, sensor data is sent to a server in real time, and the generative AI model analyzes it and provides appropriate feedback.

[0663] In this way, the server can provide learning support that is optimized for each student's situation, thereby improving learning effectiveness and maintaining motivation. It is expected that the present invention will greatly improve the diversity and efficiency of education.

[0664] The processing flow will be explained below.

[0665] Step 1:

[0666] The server receives sensor data from the student's device. Specifically, it obtains heart rate and activity level data sent from the student's device in JSON format. This data represents the student's current physiological and behavioral status. For example, the sensor data is received in the format {"heart_rate": 85, "activity_level": 6}.

[0667] Step 2:

[0668] The server passes the received data to a generative AI model. This generative AI model has been trained in advance using a large amount of training data and has the ability to analyze sensor data. The server calls the generative AI model and inputs the sensor data.

[0669] Step 3:

[0670] The server analyzes the sensor data using a generative AI model. Specifically, if the heart rate is over 80, the attention level is determined to be "high," and if the activity level is 6 or higher, the recommended educational material is determined to be "advanced." For example, the analyzed data is generated in the form {"attention_level": "high", "recommended_material": "advanced"}.

[0671] Step 4:

[0672] The server sends the analysis results to the interactive agent system. The server sends JSON data containing the analysis results as a POST request to the specified endpoint of the agent system. The agent system receives this data and generates appropriate advice for the student.

[0673] Step 5:

[0674] The server sends the analysis results to the augmented reality system (xR system). The server sends json data containing the analysis results as a POST request to the specified augmented reality system endpoint. The xR system receives this data and customizes the student's learning environment appropriately.

[0675] Step 6:

[0676] The user (student) receives feedback from the conversational agent and the augmented reality system. For example, the agent may advise, "Your heart rate is high, so take a short break," and the xR system may display advanced learning materials or new exercises. This personalizes the student's learning experience and enables more effective learning.

[0677] Example 1

[0678] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0679] The lack of systems to provide students in remote locations with the same learning outcomes as those in-person and to provide learning support optimized for each individual is particularly problematic, as it is difficult to grasp students' learning status and health status in real time in a remote environment, which can lead to a decline in students' concentration and learning effectiveness.

[0680] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0681] In this invention, the server includes means for collecting and transmitting sensor data from the terminal, means for receiving the sensor data at the server and storing it in a local database, means for analyzing the stored sensor data using a generative AI model, means for feeding back the analyzed results to the interactive agent and the augmented reality system, means for the interactive agent to provide appropriate advice to the user, and means for the augmented reality system to generate an interactive learning environment, thereby making it possible to provide individually customized and effective learning support to students in remote locations.

[0682] A "terminal" is a device used by a student to collect and transmit sensor data, such as a wearable device or smartphone.

[0683] "Sensor data" refers to physiological and behavioral data, such as a student's heart rate and activity level, collected from the device.

[0684] A "server" is a computer system that receives and analyzes sensor data and feeds the results back to other systems.

[0685] A "local database" is a database system for managing sensor data stored on a server.

[0686] A "generative AI model" is an artificial intelligence model that is trained on large amounts of data and analyzes sensor data to evaluate students' learning status.

[0687] An "interactive agent" is software or a system that provides appropriate advice to users and is capable of natural dialogue with humans.

[0688] An "augmented reality system" is a system that provides students with an interactive learning environment and uses technology to overlay digital information onto the real world.

[0689] "Feedback" is the process of providing the analysis results to interactive agents and augmented reality systems to present appropriate advice and educational materials to students.

[0690] This invention is an educational system that analyzes sensor data to provide personalized learning support to each student. Specifically, it uses data analysis with a generative AI model to evaluate students' learning progress in real time and provides appropriate feedback through a conversational agent and an augmented reality system.

[0691] Hardware and Software Configuration

[0692] Device:

[0693] The devices are wearable devices or smartphones used by students. These devices are equipped with heart rate sensors and accelerometers to collect data such as students' heart rates and activity levels.

[0694] server:

[0695] The server is a computer system that receives and analyzes sensor data and feeds the results back to the interactive agent and augmented reality system. The server has a database management system (e.g., MySQL) and a generative AI model (e.g., GPT-4) installed.

[0696] Conversational Agents:

[0697] A conversational agent is software that provides appropriate advice to users. It provides advice and instructions according to the student's learning situation through natural dialogue with the user.

[0698] Augmented Reality Systems (xR Systems):

[0699] Augmented reality systems use technology to overlay digital information onto the real world, providing an interactive learning environment that helps students learn more effectively.

[0700] Data processing and calculation

[0701] Sensor data collection and transmission (terminal)

[0702] The device uses sensors to collect data such as students' heart rate and activity level, which is converted into JSON format in real time and sent to the server.

[0703] Receiving sensor data (server)

[0704] The server receives sensor data in JSON format sent from the student's device and stores it in a local database.

[0705] Data analysis (server)

[0706] The server analyzes the received sensor data using a pre-trained generative AI model, which applies an algorithm to evaluate the student's learning progress.

[0707] Analysis result feedback (server)

[0708] The server feeds the analyzed data back to the interactive agent and augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an appropriate learning environment.

[0709] Specific examples

[0710] For example, if a student is attending a class online from home, the sensor data can be sent to the server in real time, and the generative AI model can analyze it and provide appropriate feedback. You can use the following prompt to have the generative AI model analyze the sensor data:

[0711] Prompt Sentence Examples

[0712] Analyze the data for a student with a heart rate of 85 and activity level of 6. Evaluate the student's current learning situation and recommend appropriate advice and learning materials.

[0713] In this way, the server provides learning support optimized for each student's situation, improving learning effectiveness and maintaining motivation. This makes it possible to provide effective, individually customized learning support even to students in remote locations.

[0714] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0715] Step 1:

[0716] Sensor data collection and transmission (terminal)

[0717] The device collects data such as the student's heart rate and activity level. Specifically, sensors installed in wearable devices and smartphones measure data every second. For example, data such as a heart rate of 85 and an activity level of 6 may be collected. The collected data is converted into JSON format, which is then sent to a server via the Internet.

[0718] Input: Sensor-measured data on student heart rate and activity level

[0719] Output: Sensor data converted to json format

[0720] Step 2:

[0721] Receiving sensor data (server)

[0722] The server receives the sensor data in JSON format sent from the device and stores it in a local database for subsequent analysis.

[0723] Input: JSON formatted sensor data sent from the device

[0724] Output: Sensor data stored in a local database

[0725] Specific behavior:

[0726] The server checks at regular intervals for new data.

[0727] When new data is received, extract the data in json format.

[0728] The extracted data is stored in a database.

[0729] Step 3:

[0730] Data analysis (server)

[0731] The server analyzes the received sensor data using a generative AI model. Specifically, the stored data is input into a generative AI model (e.g., GPT-4) and the data is analyzed based on a prompt. For example, the prompt might be, "Please analyze the data of a student whose heart rate is 85 and whose activity level is 6." The result of this analysis is an evaluation of the student's current learning status.

[0732] Input: Sensor data stored in the local database, prompt text

[0733] Output: Evaluation results of the learning situation analyzed by the generative AI model

[0734] Specific behavior:

[0735] The server retrieves the sensor data from the database.

[0736] Input data into the generative AI model based on the prompt sentence.

[0737] The generative AI model analyzes the input data and generates an evaluation result of the learning situation.

[0738] Step 4:

[0739] Analysis result feedback (server)

[0740] The server feeds back the data analyzed by the generative AI model to the conversational agent and augmented reality system, allowing the agent to provide advice to students through dialogue and the augmented reality system to generate an interactive learning environment.

[0741] Input: Evaluation results of learning status analyzed by the generative AI model

[0742] Output: Feedback data sent to conversational agents and augmented reality systems

[0743] Specific behavior:

[0744] The server sends the analysis results to the interactive agent.

[0745] The server sends the analysis results to the augmented reality system.

[0746] Step 5:

[0747] Providing advice and educational materials to users (users)

[0748] The user receives advice from the interactive agent and learning materials from the augmented reality system. Specifically, the interactive agent provides advice such as "Take a break to relax," and the augmented reality system displays new exercises.

[0749] Input: Feedback data from conversational agents and augmented reality systems

[0750] Output: Advice and educational materials provided to the user

[0751] Specific behavior:

[0752] The user follows the instructions of the interactive agent.

[0753] The user works on exercises displayed by the augmented reality system.

[0754] (Application example 1)

[0755] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0756] In the past, customer feedback in physical stores was mainly provided non-real-time, making it difficult to provide personalized guidance and offers that matched the customer's situation.In addition, to increase customer engagement and satisfaction, it is necessary to provide appropriate information according to each customer's individual situation, but there was a lack of effective ways to achieve this.

[0757] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0758] In this invention, the server includes a means for receiving sensor data, a means for analyzing the received sensor data using a generative AI model, a means for feeding back the analysis results to the application agent and augmented reality system, a means for personalizing guidance for the application based on the fed-back analysis results, and a means for providing application information to a display device based on the fed-back analysis results. This enables brick-and-mortar stores to utilize real-time feedback from customers to provide personalized guidance and offers tailored to each customer's individual status. As a result, customer engagement and satisfaction can be significantly improved.

[0759] "Sensor data" refers to data used to collect biometric and behavioral information about customers.

[0760] A "generative AI model" is an artificial intelligence that is trained on large amounts of data, analyzes input information, and generates useful results.

[0761] "Analysis results" are information obtained from sensor data analyzed by a generative AI model, and may indicate, for example, the psychological state and activity level of customers.

[0762] The "application agent" is an interactive system that provides guidance and offers to customers based on the analysis results.

[0763] An "augmented reality system" is a system that displays digital information overlaid on the real environment.

[0764] "Personalization" refers to providing guidance and offers that are optimized to each individual customer's situation and preferences.

[0765] A "display device" is a device for visually presenting information based on analysis results, including, for example, AR glasses and smartphones.

[0766] To put the present invention into practice, it is specifically realized by the following system configuration.

[0767] First, when a customer enters a store, sensors installed in the store or a smartphone collect the customer's biometric information (such as heart rate and activity level) in real time. The collected sensor data is sent to a server in the store.

[0768] The server uses a pre-trained generative AI model to analyze the received sensor data. This generative AI model is trained on a large amount of data and evaluates the customer's current state from sensor data such as heart rate and activity level. For example, a high heart rate is determined to be a state of tension, and a high activity level is determined to be active behavior.

[0769] The analyzed data is fed back to the agent and the augmented reality system. Based on the analysis results, the agent provides customers with individually customized guidance and offers. For example, it may recommend a relaxing rest area to a nervous customer. The augmented reality system also displays personalized information to customers through AR glasses or their smartphones based on the analysis results. For example, it may display a promotion for a new product collection to an active customer.

[0770] As a specific example, the following prompt sentence is input into the generative AI model for a customer with a heart rate of 90 and an activity level of 7.

[0771] I have a customer with a heart rate of 90 and an activity level of 7. What is their condition?

[0772] Once the generative AI model returns its analysis results, the agent advises, "Try our relaxing rest space," and the AR device displays information such as, "Check out our new collection."

[0773] The server implements a means of receiving sensor data using the Flask framework. This Flask framework processes HTTP requests and receives sensor data in JSON format. The generative AI model operates using the OpenAI API, inputting prompt statements to obtain analysis results. Since the agent and augmented reality system require analysis results to be provided in real time, AR glasses or smartphones are used as display devices.

[0774] This will enable physical stores to utilize real-time feedback from customers and provide personalized guidance and offers, significantly improving customer engagement and satisfaction.

[0775] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0776] Step 1:

[0777] The server receives sensor data from customers. Specifically, biometric information such as heart rate and activity level sent from sensors in the store or smartphones is received as input data in JSON format. The received sensor data is stored in an internal database.

[0778] Step 2:

[0779] The server uses the generative AI model to analyze the received sensor data. In this step, the sensor data (heart rate, activity level, etc.) received as input is sent to the generative AI model as a prompt. For example, a prompt sentence such as "A customer with a heart rate of 90 and an activity level of 7" is created. The generative AI model analyzes this prompt sentence, evaluates the customer's condition as an analysis result, and outputs the evaluation result.

[0780] Step 3:

[0781] The server feeds back the results of the analysis by the generative AI model to the agent and the augmented reality system. The input data for this step are the analysis results from the generative AI model. Based on the analysis results, the server instructs the conversational agent to provide personalized guidance and offers to the store visitor, and at the same time outputs data for displaying information on the augmented reality system.

[0782] Step 4:

[0783] The agent provides individually customized guidance and offers to the customer based on the analysis results. In this step, the agent system receives the analysis result data from the server and generates appropriate advice and offers based on that data. The generated guidance and offers are provided to the customer as voice or text messages.

[0784] Step 5:

[0785] The augmented reality system provides visual information to store visitors based on the analysis results. In this step, the display device (AR glasses or smartphone) receives feedback data from the server and uses that data to display personalized information to store visitors. For example, promotional information for new products or directions to rest areas are visually provided as AR.

[0786] This will allow customers to receive personalized guidance and offers in real time based on their biometric information, improving their in-store experience.

[0787] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0788] The present invention relates to an educational system that provides personalized learning support by analyzing students' situations using sensor data and an emotion engine and providing feedback to an interactive agent and an augmented reality system.

[0789] Program processing flow

[0790] Receiving sensor data and emotion data (server)

[0791] The server receives sensor data and emotion data sent from the student's device. Specifically, in addition to data such as heart rate and activity level, it also obtains the student's emotional state (e.g., joy, sadness, surprise, etc.) in JSON format. For example, the sensor data is received in the following format: {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}.

[0792] Data analysis (server)

[0793] The server passes the received sensor data and emotion data to a generative AI model. This generative AI model has been trained in advance using a large amount of training data and is capable of comprehensively analyzing the student's physiological and emotional state. The server calls the generative AI model and inputs the sensor data and emotion data.

[0794] Generation of analysis results (server)

[0795] The server analyzes the data using a generative AI model. Specifically, if the heart rate is over 80, the attention level is determined to be "high," and if the activity level is 6 or higher, the recommended learning material is "advanced." If the emotion is "happy," positive feedback is recommended. For example, the analysis results are generated in the following format: {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"}.

[0796] Feedback to agents and xR systems (server)

[0797] The server then feeds the generated analysis results back to the interactive agent and the augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an interactive learning environment tailored to each student's individual situation.

[0798] Specific examples

[0799] For example, when a user (student) is participating in an online class, emotional data such as "happy" is sent along with sensor data. This data is received by the server, and the generative AI model generates analysis results such as "Attention level: high," "Recommended learning materials: advanced," and "Positive feedback." The conversational agent provides advice such as "Your heart rate is high, so take a short break," as well as positive feedback such as "Your learning attitude is excellent!" The augmented reality system displays new exercises as advanced learning materials, optimizing the learning environment to the student's situation.

[0800] In this way, the server can provide learning support optimized for each student's physiological and emotional state, thereby improving learning effectiveness and maintaining motivation. The present invention is expected to greatly improve the diversity and efficiency of education.

[0801] The processing flow will be explained below.

[0802] Step 1:

[0803] The server receives sensor data and emotion data from the student's device. The heart rate, activity level, and emotional state are sent in JSON format from the wearable device and emotion recognition camera worn by the user (student). For example, the received data is {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}.

[0804] Step 2:

[0805] The server passes the received sensor data and emotion data to the generative AI model. The server then calls the generative AI model and inputs the acquired data for analysis. This generative AI model has been trained in advance with a large amount of training data, and performs analysis by combining the sensor data and emotion data.

[0806] Step 3:

[0807] The server analyzes sensor data and emotion data using a generative AI model. Specifically, if the heart rate is over 80, the attention level is determined to be "high," and if the activity level is 6 or higher, the recommended learning material is "advanced." If the emotion is "happy," positive feedback is generated. The analysis results are {"attention_level": "high," "recommended_material": "advanced," "emotional_feedback": "positive"}.

[0808] Step 4:

[0809] The server sends the analysis results to the interactive agent system. The analysis results are sent as JSON data to the specified agent system endpoint as a POST request. For example, if the agent system is running at the URL http: / / agent-system / feedback, the analysis results are sent to this URL.

[0810] Step 5:

[0811] The server sends the analysis results to the augmented reality system (xR system). The analysis results are sent as JSON data to the specified xR system endpoint as a POST request. For example, if the xR system is running at the URL http: / / xr-system / feedback, the analysis results are sent to this URL.

[0812] Step 6:

[0813] The user (student) receives feedback from the conversational agent and the xR system. For example, the conversational agent provides advice such as "Your heart rate is high, so take a short break," and also provides positive feedback such as "Your attitude toward studying is excellent!". The augmented reality system also displays recommended "advanced learning materials" to optimize the student's learning environment.

[0814] In this way, personalized learning support is provided based on each student's physiological and emotional state, which can increase student motivation and improve learning outcomes.

[0815] Example 2

[0816] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0817] Traditional education systems have struggled to provide personalized learning support based on each student's physiological and emotional state. This has limited the ability to maximize students' concentration and learning outcomes. Furthermore, there has been a lack of means to provide students in remote locations with the same learning outcomes as those in-person.

[0818] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for receiving data from a sensor, a means for preprocessing the received data, a means for analyzing the preprocessed data using a generative AI model, a means for formatting the analysis results in JSON format, and a means for feeding back the generated analysis results to the interactive agent and the augmented reality system. This makes it possible to provide learning support optimized for each student's physiological and emotional state. Furthermore, the interactive agent provides individually customized feedback, and the augmented reality system provides an interactive learning environment, allowing students in remote locations to experience the same learning effects as those experienced on-site.

[0819] A "sensor" is a device that detects physical information and outputs that information as digital data.

[0820] A "means for receiving data" is a device or program capable of taking data transmitted by a sensor and converting it into a usable format for processing.

[0821] "Data preprocessing" refers to the process of converting received data into an analyzable format, completing incomplete data, and correcting outliers.

[0822] A "generative AI model" is an algorithm that learns from large amounts of training data and analyzes and predicts newly given data.

[0823] "Means of analyzing data" refers to the process of inputting preprocessed data, analyzing it using a generative AI model, and outputting the results.

[0824] "Means for formatting analytical results" refers to the process of converting analytical results obtained from a generative AI model into a usable form and outputting them in an appropriate format.

[0825] An "interactive agent" is a program that uses artificial intelligence to provide information and feedback to a user through dialogue.

[0826] An "augmented reality system" is a system that overlays computer-generated visual information onto real-world visual information to provide an interactive experience.

[0827] "Feedback means" refers to the process of sending the analysis results to interactive agents and augmented reality systems so that these systems can provide appropriate feedback to students.

[0828] This invention is an educational system that analyzes students' situations in real time and provides personalized learning support. This system realizes individual learning support by utilizing sensor data and an emotion engine, and providing feedback to a conversational agent and an augmented reality system.

[0829] 1. Server:

[0830] The server receives sensor data and emotion data sent from the student's device. The sensors detect the student's heart rate, activity level, and emotional state (e.g., joy, sadness, surprise, etc.) and send the data to the server in JSON format. For example, the sensor data is received in the following format: {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}.

[0831] The server preprocesses the received data and formats it for input into the generative AI model. Preprocessing includes checking the data format, filling in incomplete data, and storing it in a database.

[0832] The server passes the preprocessed data to the generative AI model for analysis. The generative AI model has been trained in advance using a large amount of training data and has the ability to comprehensively analyze the student's physiological and emotional state. An example of a prompt to invoke the generative AI model is as follows:

[0833] Input data:

[0834] {

[0835] "heart_rate": 85,

[0836] "activity_level": 6,

[0837] "emotion": "happy"

[0838] }

[0839] Output requirements:

[0840] 1. Student attention level

[0841] 2. Recommended level of study materials

[0842] 3. Emotional Feedback

[0843] The generative AI model performs the analysis, and the server formats the analysis results in JSON format. For example, if the attention level is "high," the recommended material is "advanced," and the feedback is "positive," the server outputs the results in the format {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"} .

[0844] The server feeds the analysis results back to the interactive agent and augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an interactive learning environment tailored to each student's individual situation.

[0845] 2. Terminal:

[0846] The student's device collects data from the sensors, converts it into the required format, and sends it to the server. The device continuously monitors the student's emotional and physiological state while they are learning.

[0847] 3. User (student):

[0848] For example, when a user is participating in an online class, they send sensor data and "happy" emotion data from their device to a server. The server receives this data, analyzes it using a generative AI model, and generates analysis results such as "Attention level: high," "Recommended learning materials: advanced," and "Positive feedback."

[0849] The conversational agent provides advice such as, "Your heart rate is high, so take a short break," as well as positive feedback such as, "Your attitude toward studying is excellent!" The augmented reality system displays new exercises according to the student's situation, providing an optimized learning environment.

[0850] As described above, the present invention provides learning support optimized for each student through collaboration between the server, terminal, and user, thereby improving learning effectiveness and maintaining motivation.

[0851] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0852] Step 1: Collect sensor data and emotion data (device)

[0853] The device collects data using sensors that detect the student's heart rate and activity level, and a system that determines their emotional state. Specifically, the device temporarily stores the data obtained from the sensors in a buffer memory and then collects the data at regular intervals.

[0854] Input: Biological data from sensors (heart rate, activity level) and emotional data

[0855] Output: JSON format data {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}

[0856] Step 2: Send data (terminal)

[0857] The device transmits the collected sensor data and emotion data to the server using a security protocol (e.g., HTTPS).

[0858] Input: JSON format data {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}

[0859] Output: Data received by the server

[0860] Step 3: Data Receipt and Verification (Server)

[0861] The server receives the data sent from the terminal and verifies the format and content of the data, specifically checking whether the structure of the received data is correct and whether there is any incomplete data.

[0862] Input: JSON format data sent from the terminal

[0863] Output: Validated data

[0864] Step 4: Data Preprocessing (Server)

[0865] The server performs preprocessing on the received data, specifically by completing incomplete data, correcting outliers, and formatting it into a format suitable for analysis.

[0866] Input: Validated Data

[0867] Output: Preprocessed data

[0868] Step 5: Analysis using the generative AI model (server)

[0869] The server passes the preprocessed data to the generative AI model for analysis. The following input data is provided as a prompt:

[0870] Input data:

[0871] {

[0872] "heart_rate": 85,

[0873] "activity_level": 6,

[0874] "emotion": "happy"

[0875] }

[0876] Output requirements:

[0877] 1. Student attention level

[0878] 2. Recommended level of study materials

[0879] 3. Emotional Feedback

[0880] The generative AI model analyzes this data and outputs the following analysis results:

[0881] Input: Preprocessed data

[0882] Output: Analysis result {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"}

[0883] Step 6: Formatting the analysis results (server)

[0884] The server formats the analysis results obtained from the generative AI model in JSON format, storing each item in the appropriate field to make it usable for further processing.

[0885] Input: Analysis results

[0886] Output: Parsed results in JSON format {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"}

[0887] Step 7: Feedback to the conversational agent and xR system (server)

[0888] The server then feeds back the formatted analysis results to the interactive agent and the augmented reality system. The interactive agent provides students with data-based advice, and the xR system generates a learning environment that is optimal for the student's situation.

[0889] Input: Formatted analysis results

[0890] Output: Feedback from a conversational agent, an optimized learning environment using an augmented reality system

[0891] Step 8: Running the Example (User and Server)

[0892] For example, while participating in an online class, a user sends data such as a heart rate of 85, activity level of 6, and emotion of "happy." Based on this, the server analyzes and generates high attention levels, advanced learning materials, and positive feedback. The conversational agent advises, "Your heart rate is high, so take a short break," and provides positive feedback, such as, "Your attitude toward learning is excellent!" The augmented reality system displays advanced exercises and provides an optimized learning environment.

[0893] In this way, each processing step works in concert to provide optimal learning support for each student.

[0894] (Application example 2)

[0895] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0896] This invention relates to a system that provides appropriate feedback using sensor data and emotional data. Conventional systems have difficulty providing personalized feedback tailored to the circumstances of individual students or customers, and it has been particularly difficult to provide customer service advice in real time based on the psychological and physiological state of customers in brick-and-mortar stores. Therefore, there is a need for a system that can improve customer satisfaction and provide optimal customer service support.

[0897] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0898] In this invention, the server includes a means for receiving sensor data, a means for analyzing the received sensor data and emotion data using a generative AI model, a means for feeding back the analysis results to the interactive agent and the augmented reality system, and a means for generating personalized customer service advice in the physical store based on the analysis results and displaying it on the smart devices of staff members, thereby enabling real-time analysis and optimal customer service advice based on customers' physiological and emotional data.

[0899] "Sensor data" refers to data obtained from sensors that measure the state of a living body, such as heart rate and activity level.

[0900] "Emotion data" is data that indicates emotional states such as joy, sadness, surprise, etc.

[0901] A "generative AI model" is an artificial intelligence model that is trained based on large amounts of training data and analyzes physiological and emotional data.

[0902] An "interactive agent" is a system that interacts with users and provides appropriate support and feedback.

[0903] An "augmented reality system" is a system that provides an interactive learning environment by overlaying virtual information onto the real world.

[0904] A "brick and mortar store" is a store that exists in a physical location and sells products or provides services.

[0905] "Personalized customer service advice" refers to advice that provides customized customer service methods and suggestions based on the customer's individual circumstances.

[0906] A "smart device" is an electronic device that can connect to the Internet and obtain information and perform functions through applications.

[0907] This invention is a system that provides personalized customer service advice based on physiological and emotional data of customers in brick-and-mortar stores. The system receives sensor data and emotional data, analyzes them with a generative AI model, and feeds the results back to staff members' smart devices.

[0908] Program processing

[0909] The system is programmed using the following hardware and software: The hardware used includes sensors (devices that measure heart rate, activity level, etc.) and smart devices (smartphones, smart glasses, etc.). The software uses an application that receives and analyzes sensor data and displays the analysis results, as well as a generative AI model.

[0910] The sensor data is sent to the smart device in JSON format. The smart device receives it and forwards it to the server. The server then inputs the received data into a generative AI model for analysis. This analysis determines the customer's condition based on their heart rate, activity level, and emotional state.

[0911] Specific examples of data

[0912] For example, if sensor data is sent as follows:

[0913] Sensor data: {"heart_rate": 95, "activity_level": 2, "emotion": "stressed"}

[0914] The server that receives it analyzes it and displays the following advice on the smart device:

[0915] Attention level: High

[0916] Recommendation: Suggest relaxing products

[0917] Emotional feedback: Speaking in a friendly voice

[0918] System flow

[0919] 1. Receiving sensor data:

[0920] The smart device receives real-time data from sensors, capturing heart rate, activity level, and emotional state, and sends it in JSON format to the server.

[0921] 2. Analysis using generative AI models:

[0922] The server inputs the received data into a generative AI model to analyze the customer's physiological and emotional state. For example, if the heart rate is high, activity level is low, or the emotion is analyzed as "stress," it will determine how to respond.

[0923] 3. Display Advice:

[0924] Based on the analysis results, the server displays appropriate customer service advice on the staff's smart device screen, allowing the staff to respond to the customer in accordance with this advice.

[0925] Examples of prompt statements

[0926] The following are examples of prompt sentences:

[0927] Your sensor data: {"heart_rate": 95, "activity_level": 2, "emotion": "stressed"}

[0928] Use this data to generate customer service advice to be displayed on the smart glasses display.

[0929] In this way, the present invention aims to improve customer satisfaction by enabling attentive customer service based on the customer's psychological and physiological state. Real-time analysis and feedback also enable appropriate advice to be provided at the right time. Particularly in brick-and-mortar stores, providing services tailored to the customer's current situation is expected to improve the quality of service.

[0930] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0931] Step 1:

[0932] The terminal (smart device) receives sensor data from the sensor. The sensor data includes heart rate, activity level, and emotional state, and the data is received in JSON format. This input data is used for subsequent analysis.

[0933] Step 2:

[0934] The terminal sends the received sensor data to a server. Specifically, the smart device transfers the sensor data to the server via the Internet. This data becomes the input for the server's analysis process.

[0935] Step 3:

[0936] The server inputs the received sensor data into a generative AI model for analysis. The generative AI model analyzes the heart rate, activity level, and emotional state data to determine the customer's current physiological and emotional state. For example, if the heart rate is high, the activity level is low, and the emotion is "stressed," the model may analyze the data.

[0937] Step 4:

[0938] The generative AI model generates analysis results and sends them back to the server. The analysis results include "attention level," "recommended action," and "emotional feedback," and this information becomes output data for customer support. For example, "attention level: high," "recommended action: suggest a relaxing product," and "emotional feedback: speak to the customer in a friendly manner."

[0939] Step 5:

[0940] The server then feeds back the analysis results to the terminal, which then sends customer service advice based on the analysis results to the smart device, which then displays the advice to the staff.

[0941] Step 6:

[0942] The terminal (smart device) displays the received customer service advice. Based on the displayed advice, the user (staff member) can respond optimally to the customer. For example, they can suggest products with a relaxing effect and take specific actions such as speaking to the customer in a friendly manner.

[0943] This trend will enable the provision of personalized customer service advice in real time based on the customer's physiological and emotional data, thereby improving customer satisfaction and providing optimal service.

[0944] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0945] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0946] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0947] [Fourth embodiment]

[0948] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0949] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0950] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0951] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0952] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0953] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0954] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0955] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0956] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0957] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0958] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0959] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0960] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0961] This invention relates to an educational system that provides personalized learning support to each student by analyzing student data collected using sensors with a generative AI model and feeding the results back to an interactive agent and an augmented reality system.

[0962] Program processing flow

[0963] Receiving sensor data (server)

[0964] The server receives sensor data sent from the student's device. Specifically, it receives data such as heart rate and activity level in JSON format. For example, if a student's heart rate is 85 and their activity level is 6, it stores the data and moves on to the next analysis step.

[0965] Data analysis (server)

[0966] The server analyzes the received sensor data using a generative AI model, which is pre-trained on a large amount of data and evaluates the student's current learning status based on sensor data such as heart rate and activity level.

[0967] For example, if the heart rate is over 80, the attention level is determined to be high, and if the activity level is 6 or higher, advanced learning materials are recommended. In this way, the analyzed data is generated in the form of "Attention level: High" and "Recommended learning materials: Advanced."

[0968] Feedback to agents and xR systems (server)

[0969] The server then feeds the generated analysis results back to the interactive agent and the augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an interactive learning environment tailored to each student's individual situation.

[0970] For example, if the student's attention level is high, the agent will provide advice such as "Take a break to relax." The xR system will also display new exercises as advanced learning materials.

[0971] Specific examples

[0972] Even when students are participating in classes remotely, they can use the system of this invention to achieve the same learning benefits as if they were in a classroom on-site. For example, when a student participates in an online class from home, sensor data is sent to a server in real time, and the generative AI model analyzes it and provides appropriate feedback.

[0973] In this way, the server can provide learning support that is optimized for each student's situation, thereby improving learning effectiveness and maintaining motivation. It is expected that the present invention will greatly improve the diversity and efficiency of education.

[0974] The processing flow will be explained below.

[0975] Step 1:

[0976] The server receives sensor data from the student's device. Specifically, it obtains heart rate and activity level data sent from the student's device in JSON format. This data represents the student's current physiological and behavioral status. For example, the sensor data is received in the format {"heart_rate": 85, "activity_level": 6}.

[0977] Step 2:

[0978] The server passes the received data to a generative AI model. This generative AI model has been trained in advance using a large amount of training data and has the ability to analyze sensor data. The server calls the generative AI model and inputs the sensor data.

[0979] Step 3:

[0980] The server analyzes the sensor data using a generative AI model. Specifically, if the heart rate is over 80, the attention level is determined to be "high," and if the activity level is 6 or higher, the recommended educational material is determined to be "advanced." For example, the analyzed data is generated in the form {"attention_level": "high", "recommended_material": "advanced"}.

[0981] Step 4:

[0982] The server sends the analysis results to the interactive agent system. The server sends JSON data containing the analysis results as a POST request to the specified endpoint of the agent system. The agent system receives this data and generates appropriate advice for the student.

[0983] Step 5:

[0984] The server sends the analysis results to the augmented reality system (xR system). The server sends json data containing the analysis results as a POST request to the specified augmented reality system endpoint. The xR system receives this data and customizes the student's learning environment appropriately.

[0985] Step 6:

[0986] The user (student) receives feedback from the conversational agent and the augmented reality system. For example, the agent may advise, "Your heart rate is high, so take a short break," and the xR system may display advanced learning materials or new exercises. This personalizes the student's learning experience and enables more effective learning.

[0987] Example 1

[0988] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0989] The lack of systems to provide students in remote locations with the same learning outcomes as those in-person and to provide learning support optimized for each individual is particularly problematic, as it is difficult to grasp students' learning status and health status in real time in a remote environment, which can lead to a decline in students' concentration and learning effectiveness.

[0990] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0991] In this invention, the server includes means for collecting and transmitting sensor data from the terminal, means for receiving the sensor data at the server and storing it in a local database, means for analyzing the stored sensor data using a generative AI model, means for feeding back the analyzed results to the interactive agent and the augmented reality system, means for the interactive agent to provide appropriate advice to the user, and means for the augmented reality system to generate an interactive learning environment, thereby making it possible to provide individually customized and effective learning support to students in remote locations.

[0992] A "terminal" is a device used by a student to collect and transmit sensor data, such as a wearable device or smartphone.

[0993] "Sensor data" refers to physiological and behavioral data, such as a student's heart rate and activity level, collected from the device.

[0994] A "server" is a computer system that receives and analyzes sensor data and feeds the results back to other systems.

[0995] A "local database" is a database system for managing sensor data stored on a server.

[0996] A "generative AI model" is an artificial intelligence model that is trained on large amounts of data and analyzes sensor data to evaluate students' learning status.

[0997] An "interactive agent" is software or a system that provides appropriate advice to users and is capable of natural dialogue with humans.

[0998] An "augmented reality system" is a system that provides students with an interactive learning environment and uses technology to overlay digital information onto the real world.

[0999] "Feedback" is the process of providing the analysis results to interactive agents and augmented reality systems to present appropriate advice and educational materials to students.

[1000] This invention is an educational system that analyzes sensor data to provide personalized learning support to each student. Specifically, it uses data analysis with a generative AI model to evaluate students' learning progress in real time and provides appropriate feedback through a conversational agent and an augmented reality system.

[1001] Hardware and Software Configuration

[1002] Device:

[1003] The devices are wearable devices or smartphones used by students. These devices are equipped with heart rate sensors and accelerometers to collect data such as students' heart rates and activity levels.

[1004] server:

[1005] The server is a computer system that receives and analyzes sensor data and feeds the results back to the interactive agent and augmented reality system. The server has a database management system (e.g., MySQL) and a generative AI model (e.g., GPT-4) installed.

[1006] Conversational Agents:

[1007] A conversational agent is software that provides appropriate advice to users. It provides advice and instructions according to the student's learning situation through natural dialogue with the user.

[1008] Augmented Reality Systems (xR Systems):

[1009] Augmented reality systems use technology to overlay digital information onto the real world, providing an interactive learning environment that helps students learn more effectively.

[1010] Data processing and calculation

[1011] Sensor data collection and transmission (terminal)

[1012] The device uses sensors to collect data such as students' heart rate and activity level, which is converted into JSON format in real time and sent to the server.

[1013] Receiving sensor data (server)

[1014] The server receives sensor data in JSON format sent from the student's device and stores it in a local database.

[1015] Data analysis (server)

[1016] The server analyzes the received sensor data using a pre-trained generative AI model, which applies an algorithm to evaluate the student's learning progress.

[1017] Analysis result feedback (server)

[1018] The server feeds the analyzed data back to the interactive agent and augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an appropriate learning environment.

[1019] Specific examples

[1020] For example, if a student is attending a class online from home, the sensor data can be sent to the server in real time, and the generative AI model can analyze it and provide appropriate feedback. You can use the following prompt to have the generative AI model analyze the sensor data:

[1021] Prompt Sentence Examples

[1022] Analyze the data for a student with a heart rate of 85 and activity level of 6. Evaluate the student's current learning situation and recommend appropriate advice and learning materials.

[1023] In this way, the server provides learning support optimized for each student's situation, improving learning effectiveness and maintaining motivation. This makes it possible to provide effective, individually customized learning support even to students in remote locations.

[1024] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1025] Step 1:

[1026] Sensor data collection and transmission (terminal)

[1027] The device collects data such as the student's heart rate and activity level. Specifically, sensors installed in wearable devices and smartphones measure data every second. For example, data such as a heart rate of 85 and an activity level of 6 may be collected. The collected data is converted into JSON format, which is then sent to a server via the Internet.

[1028] Input: Sensor-measured data on student heart rate and activity level

[1029] Output: Sensor data converted to json format

[1030] Step 2:

[1031] Receiving sensor data (server)

[1032] The server receives the sensor data in JSON format sent from the device and stores it in a local database for subsequent analysis.

[1033] Input: JSON formatted sensor data sent from the device

[1034] Output: Sensor data stored in a local database

[1035] Specific behavior:

[1036] The server checks at regular intervals for new data.

[1037] When new data is received, extract the data in json format.

[1038] The extracted data is stored in a database.

[1039] Step 3:

[1040] Data analysis (server)

[1041] The server analyzes the received sensor data using a generative AI model. Specifically, the stored data is input into a generative AI model (e.g., GPT-4) and the data is analyzed based on a prompt. For example, the prompt might be, "Please analyze the data of a student whose heart rate is 85 and whose activity level is 6." The result of this analysis is an evaluation of the student's current learning status.

[1042] Input: Sensor data stored in the local database, prompt text

[1043] Output: Evaluation results of the learning situation analyzed by the generative AI model

[1044] Specific behavior:

[1045] The server retrieves the sensor data from the database.

[1046] Input data into the generative AI model based on the prompt sentence.

[1047] The generative AI model analyzes the input data and generates an evaluation result of the learning situation.

[1048] Step 4:

[1049] Analysis result feedback (server)

[1050] The server feeds back the data analyzed by the generative AI model to the conversational agent and augmented reality system, allowing the agent to provide advice to students through dialogue and the augmented reality system to generate an interactive learning environment.

[1051] Input: Evaluation results of learning status analyzed by the generative AI model

[1052] Output: Feedback data sent to conversational agents and augmented reality systems

[1053] Specific behavior:

[1054] The server sends the analysis results to the interactive agent.

[1055] The server sends the analysis results to the augmented reality system.

[1056] Step 5:

[1057] Providing advice and educational materials to users (users)

[1058] The user receives advice from the interactive agent and learning materials from the augmented reality system. Specifically, the interactive agent provides advice such as "Take a break to relax," and the augmented reality system displays new exercises.

[1059] Input: Feedback data from conversational agents and augmented reality systems

[1060] Output: Advice and educational materials provided to the user

[1061] Specific behavior:

[1062] The user follows the instructions of the interactive agent.

[1063] The user works on exercises displayed by the augmented reality system.

[1064] (Application example 1)

[1065] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1066] In the past, customer feedback in physical stores was mainly provided non-real-time, making it difficult to provide personalized guidance and offers that matched the customer's situation.In addition, to increase customer engagement and satisfaction, it is necessary to provide appropriate information according to each customer's individual situation, but there was a lack of effective ways to achieve this.

[1067] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1068] In this invention, the server includes a means for receiving sensor data, a means for analyzing the received sensor data using a generative AI model, a means for feeding back the analysis results to the application agent and augmented reality system, a means for personalizing guidance for the application based on the fed-back analysis results, and a means for providing application information to a display device based on the fed-back analysis results. This enables brick-and-mortar stores to utilize real-time feedback from customers to provide personalized guidance and offers tailored to each customer's individual status. As a result, customer engagement and satisfaction can be significantly improved.

[1069] "Sensor data" refers to data used to collect biometric and behavioral information about customers.

[1070] A "generative AI model" is an artificial intelligence that is trained on large amounts of data, analyzes input information, and generates useful results.

[1071] "Analysis results" are information obtained from sensor data analyzed by a generative AI model, and may indicate, for example, the psychological state and activity level of customers.

[1072] The "application agent" is an interactive system that provides guidance and offers to customers based on the analysis results.

[1073] An "augmented reality system" is a system that displays digital information overlaid on the real environment.

[1074] "Personalization" refers to providing guidance and offers that are optimized to each individual customer's situation and preferences.

[1075] A "display device" is a device for visually presenting information based on analysis results, including, for example, AR glasses and smartphones.

[1076] To put the present invention into practice, it is specifically realized by the following system configuration.

[1077] First, when a customer enters a store, sensors installed in the store or a smartphone collect the customer's biometric information (such as heart rate and activity level) in real time. The collected sensor data is sent to a server in the store.

[1078] The server uses a pre-trained generative AI model to analyze the received sensor data. This generative AI model is trained on a large amount of data and evaluates the customer's current state from sensor data such as heart rate and activity level. For example, a high heart rate is determined to be a state of tension, and a high activity level is determined to be active behavior.

[1079] The analyzed data is fed back to the agent and the augmented reality system. Based on the analysis results, the agent provides customers with individually customized guidance and offers. For example, it may recommend a relaxing rest area to a nervous customer. The augmented reality system also displays personalized information to customers through AR glasses or their smartphones based on the analysis results. For example, it may display a promotion for a new product collection to an active customer.

[1080] As a specific example, the following prompt sentence is input into the generative AI model for a customer with a heart rate of 90 and an activity level of 7.

[1081] I have a customer with a heart rate of 90 and an activity level of 7. What is their condition?

[1082] Once the generative AI model returns its analysis results, the agent advises, "Try our relaxing rest space," and the AR device displays information such as, "Check out our new collection."

[1083] The server implements a means of receiving sensor data using the Flask framework. This Flask framework processes HTTP requests and receives sensor data in JSON format. The generative AI model operates using the OpenAI API, inputting prompt statements to obtain analysis results. Since the agent and augmented reality system require analysis results to be provided in real time, AR glasses or smartphones are used as display devices.

[1084] This will enable physical stores to utilize real-time feedback from customers and provide personalized guidance and offers, significantly improving customer engagement and satisfaction.

[1085] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1086] Step 1:

[1087] The server receives sensor data from customers. Specifically, biometric information such as heart rate and activity level sent from sensors in the store or smartphones is received as input data in JSON format. The received sensor data is stored in an internal database.

[1088] Step 2:

[1089] The server uses the generative AI model to analyze the received sensor data. In this step, the sensor data (heart rate, activity level, etc.) received as input is sent to the generative AI model as a prompt. For example, a prompt sentence such as "A customer with a heart rate of 90 and an activity level of 7" is created. The generative AI model analyzes this prompt sentence, evaluates the customer's condition as an analysis result, and outputs the evaluation result.

[1090] Step 3:

[1091] The server feeds back the results of the analysis by the generative AI model to the agent and the augmented reality system. The input data for this step are the analysis results from the generative AI model. Based on the analysis results, the server instructs the conversational agent to provide personalized guidance and offers to the store visitor, and at the same time outputs data for displaying information on the augmented reality system.

[1092] Step 4:

[1093] The agent provides individually customized guidance and offers to the customer based on the analysis results. In this step, the agent system receives the analysis result data from the server and generates appropriate advice and offers based on that data. The generated guidance and offers are provided to the customer as voice or text messages.

[1094] Step 5:

[1095] The augmented reality system provides visual information to store visitors based on the analysis results. In this step, the display device (AR glasses or smartphone) receives feedback data from the server and uses that data to display personalized information to store visitors. For example, promotional information for new products or directions to rest areas are visually provided as AR.

[1096] This will allow customers to receive personalized guidance and offers in real time based on their biometric information, improving their in-store experience.

[1097] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1098] The present invention relates to an educational system that provides personalized learning support by analyzing students' situations using sensor data and an emotion engine and providing feedback to an interactive agent and an augmented reality system.

[1099] Program processing flow

[1100] Receiving sensor data and emotion data (server)

[1101] The server receives sensor data and emotion data sent from the student's device. Specifically, in addition to data such as heart rate and activity level, it also obtains the student's emotional state (e.g., joy, sadness, surprise, etc.) in JSON format. For example, the sensor data is received in the following format: {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}.

[1102] Data analysis (server)

[1103] The server passes the received sensor data and emotion data to a generative AI model. This generative AI model has been trained in advance using a large amount of training data and is capable of comprehensively analyzing the student's physiological and emotional state. The server calls the generative AI model and inputs the sensor data and emotion data.

[1104] Generation of analysis results (server)

[1105] The server analyzes the data using a generative AI model. Specifically, if the heart rate is over 80, the attention level is determined to be "high," and if the activity level is 6 or higher, the recommended learning material is "advanced." If the emotion is "happy," positive feedback is recommended. For example, the analysis results are generated in the following format: {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"}.

[1106] Feedback to agents and xR systems (server)

[1107] The server then feeds the generated analysis results back to the interactive agent and the augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an interactive learning environment tailored to each student's individual situation.

[1108] Specific examples

[1109] For example, when a user (student) is participating in an online class, emotional data such as "happy" is sent along with sensor data. This data is received by the server, and the generative AI model generates analysis results such as "Attention level: high," "Recommended learning materials: advanced," and "Positive feedback." The conversational agent provides advice such as "Your heart rate is high, so take a short break," as well as positive feedback such as "Your learning attitude is excellent!" The augmented reality system displays new exercises as advanced learning materials, optimizing the learning environment to the student's situation.

[1110] In this way, the server can provide learning support optimized for each student's physiological and emotional state, thereby improving learning effectiveness and maintaining motivation. The present invention is expected to greatly improve the diversity and efficiency of education.

[1111] The processing flow will be explained below.

[1112] Step 1:

[1113] The server receives sensor data and emotion data from the student's device. The heart rate, activity level, and emotional state are sent in JSON format from the wearable device and emotion recognition camera worn by the user (student). For example, the received data is {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}.

[1114] Step 2:

[1115] The server passes the received sensor data and emotion data to the generative AI model. The server then calls the generative AI model and inputs the acquired data for analysis. This generative AI model has been trained in advance with a large amount of training data, and performs analysis by combining the sensor data and emotion data.

[1116] Step 3:

[1117] The server analyzes sensor data and emotion data using a generative AI model. Specifically, if the heart rate is over 80, the attention level is determined to be "high," and if the activity level is 6 or higher, the recommended learning material is "advanced." If the emotion is "happy," positive feedback is generated. The analysis results are {"attention_level": "high," "recommended_material": "advanced," "emotional_feedback": "positive"}.

[1118] Step 4:

[1119] The server sends the analysis results to the interactive agent system. The analysis results are sent as JSON data to the specified agent system endpoint as a POST request. For example, if the agent system is running at the URL http: / / agent-system / feedback, the analysis results are sent to this URL.

[1120] Step 5:

[1121] The server sends the analysis results to the augmented reality system (xR system). The analysis results are sent as JSON data to the specified xR system endpoint as a POST request. For example, if the xR system is running at the URL http: / / xr-system / feedback, the analysis results are sent to this URL.

[1122] Step 6:

[1123] The user (student) receives feedback from the conversational agent and the xR system. For example, the conversational agent provides advice such as "Your heart rate is high, so take a short break," and also provides positive feedback such as "Your attitude toward studying is excellent!". The augmented reality system also displays recommended "advanced learning materials" to optimize the student's learning environment.

[1124] In this way, personalized learning support is provided based on each student's physiological and emotional state, which can increase student motivation and improve learning outcomes.

[1125] Example 2

[1126] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1127] Traditional education systems have struggled to provide personalized learning support based on each student's physiological and emotional state. This has limited the ability to maximize students' concentration and learning outcomes. Furthermore, there has been a lack of means to provide students in remote locations with the same learning outcomes as those in-person.

[1128] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for receiving data from a sensor, a means for preprocessing the received data, a means for analyzing the preprocessed data using a generative AI model, a means for formatting the analysis results in JSON format, and a means for feeding back the generated analysis results to the interactive agent and the augmented reality system. This makes it possible to provide learning support optimized for each student's physiological and emotional state. Furthermore, the interactive agent provides individually customized feedback, and the augmented reality system provides an interactive learning environment, allowing students in remote locations to experience the same learning effects as those experienced on-site.

[1129] A "sensor" is a device that detects physical information and outputs that information as digital data.

[1130] A "means for receiving data" is a device or program capable of taking data transmitted by a sensor and converting it into a usable format for processing.

[1131] "Data preprocessing" refers to the process of converting received data into an analyzable format, completing incomplete data, and correcting outliers.

[1132] A "generative AI model" is an algorithm that learns from large amounts of training data and analyzes and predicts newly given data.

[1133] "Means of analyzing data" refers to the process of inputting preprocessed data, analyzing it using a generative AI model, and outputting the results.

[1134] "Means for formatting analytical results" refers to the process of converting analytical results obtained from a generative AI model into a usable form and outputting them in an appropriate format.

[1135] An "interactive agent" is a program that uses artificial intelligence to provide information and feedback to a user through dialogue.

[1136] An "augmented reality system" is a system that overlays computer-generated visual information onto real-world visual information to provide an interactive experience.

[1137] "Feedback means" refers to the process of sending the analysis results to interactive agents and augmented reality systems so that these systems can provide appropriate feedback to students.

[1138] This invention is an educational system that analyzes students' situations in real time and provides personalized learning support. This system realizes individual learning support by utilizing sensor data and an emotion engine, and providing feedback to a conversational agent and an augmented reality system.

[1139] 1. Server:

[1140] The server receives sensor data and emotion data sent from the student's device. The sensors detect the student's heart rate, activity level, and emotional state (e.g., joy, sadness, surprise, etc.) and send the data to the server in JSON format. For example, the sensor data is received in the following format: {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}.

[1141] The server preprocesses the received data and formats it for input into the generative AI model. Preprocessing includes checking the data format, filling in incomplete data, and storing it in a database.

[1142] The server passes the preprocessed data to the generative AI model for analysis. The generative AI model has been trained in advance using a large amount of training data and has the ability to comprehensively analyze the student's physiological and emotional state. An example of a prompt to invoke the generative AI model is as follows:

[1143] Input data:

[1144] {

[1145] "heart_rate": 85,

[1146] "activity_level": 6,

[1147] "emotion": "happy"

[1148] }

[1149] Output requirements:

[1150] 1. Student attention level

[1151] 2. Recommended level of study materials

[1152] 3. Emotional Feedback

[1153] The generative AI model performs the analysis, and the server formats the analysis results in JSON format. For example, if the attention level is "high," the recommended material is "advanced," and the feedback is "positive," the server outputs the results in the format {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"} .

[1154] The server feeds the analysis results back to the interactive agent and augmented reality system, allowing the agent to provide specific advice to students and the xR system to generate an interactive learning environment tailored to each student's individual situation.

[1155] 2. Terminal:

[1156] The student's device collects data from the sensors, converts it into the required format, and sends it to the server. The device continuously monitors the student's emotional and physiological state while they are learning.

[1157] 3. User (student):

[1158] For example, when a user is participating in an online class, they send sensor data and "happy" emotion data from their device to a server. The server receives this data, analyzes it using a generative AI model, and generates analysis results such as "Attention level: high," "Recommended learning materials: advanced," and "Positive feedback."

[1159] The conversational agent provides advice such as, "Your heart rate is high, so take a short break," as well as positive feedback such as, "Your attitude toward studying is excellent!" The augmented reality system displays new exercises according to the student's situation, providing an optimized learning environment.

[1160] As described above, the present invention provides learning support optimized for each student through collaboration between the server, terminal, and user, thereby improving learning effectiveness and maintaining motivation.

[1161] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1162] Step 1: Collect sensor data and emotion data (device)

[1163] The device collects data using sensors that detect the student's heart rate and activity level, and a system that determines their emotional state. Specifically, the device temporarily stores the data obtained from the sensors in a buffer memory and then collects the data at regular intervals.

[1164] Input: Biological data from sensors (heart rate, activity level) and emotional data

[1165] Output: JSON format data {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}

[1166] Step 2: Send data (terminal)

[1167] The device transmits the collected sensor data and emotion data to the server using a security protocol (e.g., HTTPS).

[1168] Input: JSON format data {"heart_rate": 85, "activity_level": 6, "emotion": "happy"}

[1169] Output: Data received by the server

[1170] Step 3: Data Receipt and Verification (Server)

[1171] The server receives the data sent from the terminal and verifies the format and content of the data, specifically checking whether the structure of the received data is correct and whether there is any incomplete data.

[1172] Input: JSON format data sent from the terminal

[1173] Output: Validated data

[1174] Step 4: Data Preprocessing (Server)

[1175] The server performs preprocessing on the received data, specifically by completing incomplete data, correcting outliers, and formatting it into a format suitable for analysis.

[1176] Input: Validated Data

[1177] Output: Preprocessed data

[1178] Step 5: Analysis using the generative AI model (server)

[1179] The server passes the preprocessed data to the generative AI model for analysis. The following input data is provided as a prompt:

[1180] Input data:

[1181] {

[1182] "heart_rate": 85,

[1183] "activity_level": 6,

[1184] "emotion": "happy"

[1185] }

[1186] Output requirements:

[1187] 1. Student attention level

[1188] 2. Recommended level of study materials

[1189] 3. Emotional Feedback

[1190] The generative AI model analyzes this data and outputs the following analysis results:

[1191] Input: Preprocessed data

[1192] Output: Analysis result {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"}

[1193] Step 6: Formatting the analysis results (server)

[1194] The server formats the analysis results obtained from the generative AI model in JSON format, storing each item in the appropriate field to make it usable for further processing.

[1195] Input: Analysis results

[1196] Output: Parsed results in JSON format {"attention_level": "high", "recommended_material": "advanced", "emotional_feedback": "positive"}

[1197] Step 7: Feedback to the conversational agent and xR system (server)

[1198] The server then feeds back the formatted analysis results to the interactive agent and the augmented reality system. The interactive agent provides students with data-based advice, and the xR system generates a learning environment that is optimal for the student's situation.

[1199] Input: Formatted analysis results

[1200] Output: Feedback from a conversational agent, an optimized learning environment using an augmented reality system

[1201] Step 8: Running the Example (User and Server)

[1202] For example, while participating in an online class, a user sends data such as a heart rate of 85, activity level of 6, and emotion of "happy." Based on this, the server analyzes and generates high attention levels, advanced learning materials, and positive feedback. The conversational agent advises, "Your heart rate is high, so take a short break," and provides positive feedback, such as, "Your attitude toward learning is excellent!" The augmented reality system displays advanced exercises and provides an optimized learning environment.

[1203] In this way, each processing step works in concert to provide optimal learning support for each student.

[1204] (Application example 2)

[1205] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1206] This invention relates to a system that provides appropriate feedback using sensor data and emotional data. Conventional systems have difficulty providing personalized feedback tailored to the circumstances of individual students or customers, and it has been particularly difficult to provide customer service advice in real time based on the psychological and physiological state of customers in brick-and-mortar stores. Therefore, there is a need for a system that can improve customer satisfaction and provide optimal customer service support.

[1207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1208] In this invention, the server includes a means for receiving sensor data, a means for analyzing the received sensor data and emotion data using a generative AI model, a means for feeding back the analysis results to the interactive agent and the augmented reality system, and a means for generating personalized customer service advice in the physical store based on the analysis results and displaying it on the smart devices of staff members, thereby enabling real-time analysis and optimal customer service advice based on customers' physiological and emotional data.

[1209] "Sensor data" refers to data obtained from sensors that measure the state of a living body, such as heart rate and activity level.

[1210] "Emotion data" is data that indicates emotional states such as joy, sadness, surprise, etc.

[1211] A "generative AI model" is an artificial intelligence model that is trained based on large amounts of training data and analyzes physiological and emotional data.

[1212] An "interactive agent" is a system that interacts with users and provides appropriate support and feedback.

[1213] An "augmented reality system" is a system that provides an interactive learning environment by overlaying virtual information onto the real world.

[1214] A "brick and mortar store" is a store that exists in a physical location and sells products or provides services.

[1215] "Personalized customer service advice" refers to advice that provides customized customer service methods and suggestions based on the customer's individual circumstances.

[1216] A "smart device" is an electronic device that can connect to the Internet and obtain information and perform functions through applications.

[1217] This invention is a system that provides personalized customer service advice based on physiological and emotional data of customers in brick-and-mortar stores. The system receives sensor data and emotional data, analyzes them with a generative AI model, and feeds the results back to staff members' smart devices.

[1218] Program processing

[1219] The system is programmed using the following hardware and software: The hardware used includes sensors (devices that measure heart rate, activity level, etc.) and smart devices (smartphones, smart glasses, etc.). The software uses an application that receives and analyzes sensor data and displays the analysis results, as well as a generative AI model.

[1220] The sensor data is sent to the smart device in JSON format. The smart device receives it and forwards it to the server. The server then inputs the received data into a generative AI model for analysis. This analysis determines the customer's condition based on their heart rate, activity level, and emotional state.

[1221] Specific examples of data

[1222] For example, if sensor data is sent as follows:

[1223] Sensor data: {"heart_rate": 95, "activity_level": 2, "emotion": "stressed"}

[1224] The server that receives it analyzes it and displays the following advice on the smart device:

[1225] Attention level: High

[1226] Recommendation: Suggest relaxing products

[1227] Emotional feedback: Speaking in a friendly voice

[1228] System flow

[1229] 1. Receiving sensor data:

[1230] The smart device receives real-time data from sensors, capturing heart rate, activity level, and emotional state, and sends it in JSON format to the server.

[1231] 2. Analysis using generative AI models:

[1232] The server inputs the received data into a generative AI model to analyze the customer's physiological and emotional state. For example, if the heart rate is high, activity level is low, or the emotion is analyzed as "stress," it will determine how to respond.

[1233] 3. Display Advice:

[1234] Based on the analysis results, the server displays appropriate customer service advice on the staff's smart device screen, allowing the staff to respond to the customer in accordance with this advice.

[1235] Examples of prompt statements

[1236] The following are examples of prompt sentences:

[1237] Your sensor data: {"heart_rate": 95, "activity_level": 2, "emotion": "stressed"}

[1238] Use this data to generate customer service advice to be displayed on the smart glasses display.

[1239] In this way, the present invention aims to improve customer satisfaction by enabling attentive customer service based on the customer's psychological and physiological state. Real-time analysis and feedback also enable appropriate advice to be provided at the right time. Particularly in brick-and-mortar stores, providing services tailored to the customer's current situation is expected to improve the quality of service.

[1240] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1241] Step 1:

[1242] The terminal (smart device) receives sensor data from the sensor. The sensor data includes heart rate, activity level, and emotional state, and the data is received in JSON format. This input data is used for subsequent analysis.

[1243] Step 2:

[1244] The terminal sends the received sensor data to a server. Specifically, the smart device transfers the sensor data to the server via the Internet. This data becomes the input for the server's analysis process.

[1245] Step 3:

[1246] The server inputs the received sensor data into a generative AI model for analysis. The generative AI model analyzes the heart rate, activity level, and emotional state data to determine the customer's current physiological and emotional state. For example, if the heart rate is high, the activity level is low, and the emotion is "stressed," the model may analyze the data.

[1247] Step 4:

[1248] The generative AI model generates analysis results and sends them back to the server. The analysis results include "attention level," "recommended action," and "emotional feedback," and this information becomes output data for customer support. For example, "attention level: high," "recommended action: suggest a relaxing product," and "emotional feedback: speak to the customer in a friendly manner."

[1249] Step 5:

[1250] The server then feeds back the analysis results to the terminal, which then sends customer service advice based on the analysis results to the smart device, which then displays the advice to the staff.

[1251] Step 6:

[1252] The terminal (smart device) displays the received customer service advice. Based on the displayed advice, the user (staff member) can respond optimally to the customer. For example, they can suggest products with a relaxing effect and take specific actions such as speaking to the customer in a friendly manner.

[1253] This trend will enable the provision of personalized customer service advice in real time based on the customer's physiological and emotional data, thereby improving customer satisfaction and providing optimal service.

[1254] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1255] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1256] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1257] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1258] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1259] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1260] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1261] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1262] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1263] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1264] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1265] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1266] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1267] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1268] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1269] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1270] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1271] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1272] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1273] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1274] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1275] The following is further disclosed regarding the above embodiment.

[1276] (Claim 1)

[1277] means for receiving sensor data;

[1278] A means for analyzing the received sensor data using a generative AI model; and

[1279] The system includes a means for feeding back the results of the analysis to an interactive agent and an augmented reality system.

[1280] (Claim 2)

[1281] 10. The system of claim 1, wherein the interactive agent includes means for providing individually customized learning assistance based on the student's current learning situation.

[1282] (Claim 3)

[1283] 10. The system of claim 1, wherein the augmented reality system includes means for providing on-site learning benefits to students in remote locations.

[1284] "Example 1"

[1285] (Claim 1)

[1286] means for collecting and transmitting sensor data from the device;

[1287] a means for receiving the sensor data at a server and storing it in a local database;

[1288] A means for analyzing the stored sensor data using a generative AI model; and

[1289] a means for feeding back the analyzed results to the interactive agent and the augmented reality system;

[1290] A means for the conversational agent to provide appropriate advice to the user;

[1291] A system including means for generating an interactive learning environment by an augmented reality system.

[1292] (Claim 2)

[1293] 10. The system of claim 1, wherein the interactive agent includes means for providing individually customized learning assistance based on the student's current learning situation.

[1294] (Claim 3)

[1295] 10. The system of claim 1, wherein the augmented reality system includes means for providing on-site learning benefits to students in remote locations.

[1296] "Application Example 1"

[1297] (Claim 1)

[1298] means for receiving sensor data;

[1299] A means for analyzing the received sensor data using a generative AI model; and

[1300] A means for feeding back the analysis results to the agent and augmented reality system to which the analysis results are applied;

[1301] A means for personalizing guidance on applications based on the analysis results that have been fed back;

[1302] a means for providing application information to a display device based on the analysis results fed back;

[1303] A system including:

[1304] (Claim 2)

[1305] 10. The system of claim 1, further comprising means for the agent to provide personalized guidance and offers.

[1306] (Claim 3)

[1307] 2. The system according to claim 1, wherein the augmented reality system includes means for presenting information according to the individual situation at the application site.

[1308] "Example 2: Combining Emotion Engines"

[1309] (Claim 1)

[1310] means for receiving data from the sensor;

[1311] means for preprocessing the received data;

[1312] a means for analyzing the preprocessed data with a generative AI model;

[1313] A means to format the analysis results in JSON format,

[1314] a means for feeding back the generated analysis results to the interactive agent and the augmented reality system;

[1315] A system including:

[1316] (Claim 2)

[1317] 10. The system of claim 1, wherein the interactive agent includes means for providing individually customized learning assistance based on the student's current physiological and emotional state.

[1318] (Claim 3)

[1319] 10. The system of claim 1, wherein the augmented reality system includes means for providing an interactive learning environment that is sensitive to the physiological and emotional state of the student.

[1320] "Application example 2 when combining emotion engines"

[1321] (Claim 1)

[1322] means for receiving sensor data;

[1323] a means for analyzing the received sensor data and emotion data using a generative AI model;

[1324] a means for feeding back the analysis results to the interactive agent and the augmented reality system;

[1325] A method for generating personalized customer service advice in physical stores based on the analysis results and displaying it on staff's smart devices;

[1326] A system including:

[1327] (Claim 2)

[1328] 10. The system of claim 1, wherein the interactive agent includes means for providing individually customized support based on the student's or customer's current situation.

[1329] (Claim 3)

[1330] 10. The system of claim 1, wherein the augmented reality system includes means for providing a local experience to students or customers in remote locations. [Explanation of symbols]

[1331] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving sensor data; A means for analyzing the received sensor data using a generative AI model; and The system includes a means for feeding back the results of the analysis to an interactive agent and an augmented reality system.

2. 10. The system of claim 1, wherein the interactive agent includes means for providing individually customized learning assistance based on the student's current learning situation.

3. 10. The system of claim 1, wherein the augmented reality system includes means for providing on-site learning effects to students in remote locations.

Images