system

Abstract

We provide the system. [Solution] In an information processing device, a means for digitizing information written by a learner, A means for analyzing the digitized information and estimating the emotional state, A means for analyzing the aforementioned changes in emotional state and detecting abnormalities, A means of sending a notification signal to the education officer when an anomaly is detected, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional educational field, the detection and response to mental stress rely on human observation and reports, and it has been difficult to detect changes in mental state at an early stage. As a result, problems such as stress, bullying, and school refusal cannot be dealt with promptly, which may have an adverse impact on the health of children and students and the learning environment. In order to solve such problems, there is a demand for a system that can accurately detect subtle mental changes in children and students and enable immediate response.

Means for Solving the Problems

[0005] This invention provides a means for digitizing information written by learners using an information processing device, and for estimating their emotional state by analyzing that digitized information. Furthermore, it includes a means for analyzing changes in emotional state and detecting abnormalities. When an abnormality is detected, a notification signal is sent to the educator, thereby providing a system that can detect learners' mental stress early and prompt appropriate countermeasures.

[0006] An "information processing device" is a device that performs information input, analysis, and output, and is responsible for processing digital data.

[0007] A "learner" is a person who receives instruction or training in an educational setting, or a student who is in the process of receiving such instruction or training.

[0008] "Means of digitization" refer to methods and devices for converting analog information into digital data, and specifically include hardware such as scanners and cameras.

[0009] "Means of analysis" refer to processes and devices for extracting specific meanings or features from input information, particularly those using natural language processing or image recognition technologies.

[0010] "Emotional state" is an indicator that represents the mental and emotional condition of a learner, and is an internal state estimated from observed data.

[0011] "Means for detecting abnormalities" refer to methods or devices that monitor changes in emotional states and identify fluctuations that exceed the normal range.

[0012] A "notification signal" is a communication signal transmitted to indicate the occurrence of an anomaly, and is intended to prompt the training staff to take action.

[0013] An "educational professional" refers to an educator or counselor whose role is to guide, support, and promote the growth of learners. [Brief explanation of the drawing]

[0014] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention provides a system for continuously monitoring the mental state of learners in educational settings and detecting problems early. Learners keep notes or diaries during their daily learning activities, and the contents are digitized by an information processing device.

[0036] 1. Data Collection

[0037] Notes taken by users (learners) during class are photographed using a scanner or camera-equipped device. This digitizes the handwritten content and extracts it as text data.

[0038] 2. Data Analysis

[0039] The device transmits the digitized notebook information to the server. The server analyzes this data using natural language processing techniques to estimate the emotional state. The emotional state is inferred from the keywords and phrases written by the learner in the notebook content.

[0040] 3. Anomaly detection

[0041] The server compares the analyzed emotional data with historically accumulated data to track changes in emotions. If any abnormalities indicating increased stress or anxiety are detected during this process, a notification is sent to the educator.

[0042] 4. Notification and Response

[0043] If the server detects an anomaly, it immediately sends a notification signal to the educator. This signal includes the name of the learner who detected the anomaly, as well as detailed information about the detected increase in stress or emotion. Recommended countermeasures may also be provided at the same time.

[0044] 5. Specific Examples

[0045] If a student writes in their diary, "Recent lessons are difficult, and I can't keep up," the server analyzes the text and determines that the student is experiencing stress. If the stress level exceeds a set threshold, a notification is sent to the instructor, allowing for prompt action.

[0046] This system allows for the rapid detection and response to changes in learners' emotions, thereby maintaining their mental health and preventing problems such as bullying and school absenteeism.

[0047] The following describes the processing flow.

[0048] Step 1:

[0049] Users (learners) take notes or write diaries during class and save them as digital images using the device's camera or scanner. The device uses optical character recognition (OCR) technology to convert this image data into text data.

[0050] Step 2:

[0051] The terminal transfers text data and digital image data to the server via the network. This data includes all handwritten input from the learner.

[0052] Step 3:

[0053] The server analyzes the received text data using natural language processing (NLP) techniques. This extracts keywords and phrases related to emotions and stress from the recorded content and estimates the emotional state.

[0054] Step 4:

[0055] The server uses image recognition technology to evaluate the irregularities and patterns in handwritten characters. If irregularities in characters occur frequently, this information is used as an indicator of stress or emotional instability.

[0056] Step 5:

[0057] The server compares the analyzed emotional state with existing stored data over time to understand changes in emotional state. In this process, any emotional fluctuations that exceed a certain threshold are detected as abnormal.

[0058] Step 6:

[0059] When the server detects an anomaly, it automatically sends a notification signal to the educator. This notification includes identification of the learner in question, details of the emotional fluctuations, and recommendations for action.

[0060] Step 7:

[0061] Educators receive notifications and take appropriate action or conduct interviews with learners. Users (learners) receive feedback and can get support as needed. This enables early detection and resolution of problems.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] In educational settings, there is a challenge in appropriately monitoring changes in learners' mental states and detecting problems early to take countermeasures. In particular, the lack of means to grasp learners' emotional states in detail increases the risk of overlooking signs of stress and anxiety. Furthermore, immediate notification is necessary when an anomaly is detected so that educators can respond quickly.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes means for converting handwritten information into digital data using information acquired by the learner with an input device, means for transmitting the digital data to a processing device, and means for the processing device to analyze the digital data using natural language processing technology and estimate the learner's emotional state. This enables rapid and accurate detection of changes in the learner's emotional state, allowing educators to take appropriate action.

[0067] An "input device" is a device used by learners to convert their handwritten information into digital data, and includes equipment such as scanners and cameras.

[0068] "Handwritten information" refers to information such as characters and drawings that learners have written by hand on paper or other media, and is the type of information that is to be converted into digital data.

[0069] "Digital data" refers to data that has been converted from handwritten information using an input device and then processed in a format that can be used by a computer.

[0070] A "processing device" is a device used to analyze and process digital data, and includes equipment such as servers and computers.

[0071] "Natural language processing technology" is a technique that analyzes digitized text to understand the emotions and intentions contained within it, and it utilizes machine learning and artificial intelligence algorithms.

[0072] "Emotional state" refers to the learner's psychological state inferred from the text contained in digital data, and includes classifications such as positive, negative, and neutral.

[0073] "Anomaly detection" is a process that identifies unusual changes or trends based on the results of an analysis of emotional states, and serves as a criterion for notifying educators of the results.

[0074] An "educational administrator" is a person who receives notification when an anomaly is detected and takes measures to improve the learner's condition, and this includes teachers, counselors, and others.

[0075] This system is primarily intended to monitor the mental state of learners in educational environments and to detect abnormalities early. The following hardware and software are used to implement this invention.

[0076] First, the user (learner) acquires handwritten information, such as notes or diaries, using an input device. Scanners or cameras are used as input devices. This converts the handwritten information into digital data.

[0077] Next, the terminal plays the role of transmitting the acquired digital data to the server. A dedicated application is installed on the terminal, and this application automatically manages the data transmission. The data is securely transferred to the server via the internet.

[0078] The server processes the received digital data for analysis. Specifically, it uses natural language processing technology running on the server to estimate emotional states from text written by learners. This analysis utilizes sentiment dictionaries and machine learning models. The server detects anomalies by comparing them with past data and notifies educational administrators as needed.

[0079] As a concrete example, consider a case where a learner writes in their diary, "I have a lot of assignments every day, and I feel pressured." The server, based on this text, determines that the learner is experiencing stress, and if the stress level exceeds a certain threshold, a notification is sent to the education administrator. This notification may include specific details about the problem and recommended coping strategies.

[0080] Furthermore, as an example of a prompt using a generative AI model, the model can be given instructions such as, "Please input the contents of a student's diary and provide the emotions and stress indicators you perceive from the text." Such prompts enable the generative AI model to perform appropriate emotion analysis and respond quickly.

[0081] This system effectively supports learners' mental health and enables early detection and response to problems.

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The user (learner) takes notes or diary entries during class and inputs them into an input device. The input device used is either a scanner or a terminal with a camera. The input is handwritten information, and the output is that information converted into a digital image. Specifically, the user takes a picture of a page of their notebook using the terminal, and the image file is saved to the terminal.

[0085] Step 2:

[0086] The device converts captured digital images into text data. This process utilizes OCR (Optical Character Recognition) software. The input is a digital image, and the output is text data extracted from that image. Specifically, the device automatically performs OCR, reads the characters contained in the image as text, and saves them to a text file.

[0087] Step 3:

[0088] The terminal sends the generated text data to the server. The input is the text data obtained in the previous step, and the output is the uploading of the text data to the server. Specifically, a dedicated application on the terminal is launched, and the text data is uploaded to the server using a secure protocol.

[0089] Step 4:

[0090] The server performs natural language processing to analyze the received text data. The input is text data sent from the terminal, and the output is the learner's estimated emotional state. Specifically, a natural language processing algorithm on the server analyzes the text data, and a generative AI model is used to determine emotions and psychological states.

[0091] Step 5:

[0092] The server detects anomalies based on the analysis results. Inputs include the current emotional state determined by text analysis and past data, while output is the result of the anomaly detection. Specifically, the server compares the current emotional state with a set baseline value to identify emotional deviations or abnormal changes.

[0093] Step 6:

[0094] The server performs the action of notifying the education administrator of the results of anomaly detection. The inputs are the anomaly detection results and learner information, and the output is a notification to the education administrator. Specifically, the server provides the education administrator with details of the anomaly and recommended countermeasures using email or a notification application.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] There is a need to provide an environment that effectively utilizes the time learners spend in autonomous vehicles and promotes learning while appropriately monitoring their mental state. However, conventional educational systems have difficulty monitoring learners' emotional states in real time and taking appropriate action. This presents a challenge in early detection of learners' anxiety and stress and taking swift countermeasures.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes processing means for converting information entered by the user into a digital format, processing means for analyzing the digitized information and inferring the emotional state, and processing means for sending a warning signal to an educational support worker when an abnormality is detected. This enables real-time monitoring of the learner's mental state inside an autonomous vehicle and allows for a rapid response when an abnormality is detected.

[0100] "Users" refer to individuals who utilize the learning environment within an autonomous vehicle, and whose learning progress and mental state are subject to monitoring.

[0101] "Digital format" refers to a format in which handwritten or printed information is converted into electronic information that can be processed by a computer.

[0102] "Processing means" refers to a device or program for handling, analyzing, transforming, or transmitting information, and is a technical means for performing operations necessary to solve a problem.

[0103] "Emotional state" refers to information that indicates the user's psychological or mental state, and particularly includes stress, anxiety, and satisfaction during learning activities.

[0104] A "warning signal" is a notification sent to educational support staff or others to draw their attention when a deviation from the expected normal state is detected.

[0105] An "educational supporter" is a person whose role is to monitor learners' learning and mental state, and to provide guidance or support as needed.

[0106] An "autonomous vehicle" refers to a vehicle that operates autonomously using artificial intelligence and sensor technology, minimizing human intervention during operation.

[0107] This invention is a system that provides a safe and secure learning environment for learners inside autonomous vehicles. The system continuously monitors the learner's mental state and detects any abnormalities early, thereby providing optimal support.

[0108] The server receives information transmitted from a device such as a tablet or smart glasses used by the learner and uses OCR technology to digitize that information. Specifically, it uses Google® Cloud Vision API or similar to convert handwritten information into electronic data. The server then analyzes this electronic data and estimates the learner's emotional state using the natural language processing library NLTK and an AI model that enables sentiment analysis (e.g., using TENSORFLOW®).

[0109] When fluctuations in emotional state are detected, the server uses time-series data to identify anomalies. Once an anomaly is confirmed, it utilizes email APIs and notification systems to quickly send warning signals to educational support staff. This process enables appropriate support for learners and improves the learning environment.

[0110] For example, if a learner writes "Today's exam was very difficult" on an electronic tablet while in an autonomous vehicle, the server analyzes this statement and detects a potential stress level. An AI model then evaluates the situation, and if the learner's stress level exceeds a certain threshold, it notifies the educational support staff.

[0111] An example of a prompt would be: "Evaluate the situation based on the note data and sentiment analysis results written by the student inside the autonomous vehicle, and propose learning support measures. This time, create a prompt to support a student who is experiencing stress from the assignment." In this way, the system has a function to support the mental health of learners.

[0112] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0113] Step 1:

[0114] The user inputs text and drawings during learning using a tablet or smart glasses. Image data is obtained by taking a picture of the handwritten notes with the device's built-in camera. In this step, the input is handwritten notes, and the output is image data.

[0115] Step 2:

[0116] The device converts the acquired image data into text data using OCR technology. Specifically, it uses the Google Cloud Vision API to recognize characters in the image and extracts them as digital text. The input for this step is image data, and the output is text data.

[0117] Step 3:

[0118] The device sends text data to a cloud server. The server analyzes the received text data using the natural language processing library NLTK and performs sentiment analysis. In this step, the emotional state is estimated based on keywords and phrases in the text data. The input is text data, and the output is estimated sentiment data.

[0119] Step 4:

[0120] The server compares estimated emotion data with historically stored data. Based on the dataset of past emotional states, it detects anomalies in the current emotional state. Specifically, it uses an AI model (e.g., TensorFlow) to evaluate emotional fluctuations. The inputs are emotion data and historical data, and the output is whether or not an anomaly is present.

[0121] Step 5:

[0122] If an anomaly is detected, the server sends a warning signal to the educational support staff. This process utilizes an email API or notification system, providing specific emotional states and recommended actions as warnings. The input is the result of the anomaly detection, and the output is the warning signal.

[0123] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0124] This invention relates to a system for accurately understanding the mental state of learners in educational settings and detecting abnormalities early. It is particularly characterized by its incorporation of an emotion engine to recognize learners' emotions. Learners keep notes and diaries during their daily learning activities, and these are digitized by an information processing device.

[0125] 1. Information gathering and digitization

[0126] Users (learners) take notes during class and scan those notes using a device. This scanning process digitizes the handwritten content, which is then converted into text data using optical character recognition (OCR) technology.

[0127] 2. Data transmission and initial analysis

[0128] The device sends digitized text and image data to the server. The server analyzes the received data using natural language processing (NLP) to estimate the user's basic emotional state.

[0129] 3. Reinforcement estimation using an emotion engine

[0130] The server utilizes an emotion engine to further enhance the estimated emotional state. The emotion engine detects more refined emotional patterns and recognizes subtle emotional nuances that cannot be captured by conventional natural language processing techniques.

[0131] 4. Anomaly detection and notification coordination

[0132] The server utilizes information obtained from the emotion engine to detect an anomaly when the emotional state deviates significantly from the normal range. Based on the output of the emotion engine, the server adjusts the content of the notification signal when an anomaly occurs and notifies the training staff, including the urgency and recommended countermeasures.

[0133] 5. Specific Examples

[0134] Suppose a learner writes in their notebook, "I'm very confused and don't know what to do next." The server's natural language processing system references the keyword "confused," and the emotion engine further estimates the level of stress and anxiety underlying the statement. As a result, it determines that the stress level exceeds a certain threshold and sends a notification to the educator stating, "The learner is experiencing a high level of stress. Please consider a support session."

[0135] This system enables faster and more accurate detection of changes in learners' emotions, facilitating early problem resolution.

[0136] The following describes the processing flow.

[0137] Step 1:

[0138] The user (learner) takes notes or writes a diary during class and captures them as digital images using the device's camera or scanner. The device then uses character recognition technology to convert these images into text data.

[0139] Step 2:

[0140] The terminal sends the generated text and image data to the server. During this process, it verifies that the data is properly formatted and adds any necessary metadata.

[0141] Step 3:

[0142] The server analyzes the received text data using natural language processing techniques to estimate basic emotional states. This analysis classifies emotions based on specific keywords or phrases.

[0143] Step 4:

[0144] The server activates the emotion engine to perform further estimation of emotional states. The emotion engine detects subtle emotional nuances from the text written by the learner and reinforces the intensity and type of emotion.

[0145] Step 5:

[0146] Based on the analysis results, the server compares the learner's emotional changes with past data to detect anomalies. If an anomaly is found, it analyzes its nature and evaluates its severity and scope of impact.

[0147] Step 6:

[0148] If an anomaly is detected, the server sends a notification signal to the educator. This signal includes learner identification information, a specific description of the emotional anomaly, and recommended actions. The notification content is adjusted based on the output of the emotion engine.

[0149] Step 7:

[0150] Instructors receive notifications from the server, provide feedback to learners as needed, and conduct counseling and support activities. Users (learners) can resolve problems through the support provided by instructors.

[0151] (Example 2)

[0152] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0153] In educational settings, accurately understanding learners' mental states and detecting abnormalities early is crucial for providing appropriate support to individual learners. Traditional methods struggle to quickly and accurately detect emotional problems faced by learners, potentially delaying appropriate intervention. Therefore, a system is needed that can more accurately assess learners' emotional states and immediately detect abnormalities.

[0154] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0155] In this invention, the server includes means for digitizing information created by learners, means for estimating basic emotional states using natural language processing technology, means for analyzing subtle emotional nuances using an emotion engine, and means for detecting anomalies and transmitting notification signals. This makes it possible to quickly and accurately detect subtle changes in the learner's emotional state and to immediately provide appropriate educational interventions.

[0156] An "information processing device" is a device equipped with the function of analyzing digitized information and performing specific processing.

[0157] "Digitalization" is the process of converting analog data into an electronically processable format.

[0158] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0159] An "emotion engine" is a technology that analyzes human emotions and recognizes subtle emotional nuances.

[0160] "Anomaly detection" is the process of identifying conditions that deviate from the normal range.

[0161] A "notification signal" is a notification or alert that is sent when an anomaly is detected.

[0162] This invention is a system for accurately understanding the emotional state of learners in educational settings and detecting abnormalities early. The system is centered around an information processing device and is composed of the following hardware and software components.

[0163] Users (learners) record information in paper or digital notebooks during their daily learning activities. The recorded content is scanned by a device with a dedicated application installed and saved as a digital image. This device includes optical character recognition (OCR) software, which converts the image data into text data.

[0164] The converted text data is transmitted to the server via wireless communication. The server analyzes the text data using natural language processing (NLP) techniques to estimate the learner's basic emotional state. Furthermore, it utilizes an emotion engine to extract subtle emotional nuances from this basic emotional state. This process enables a deeper understanding of emotions compared to conventional natural language processing.

[0165] The server monitors changes in learners' emotional states based on emotional information obtained from the emotion engine, and generates a notification signal when an anomaly is detected. This signal is configured to be automatically sent to the educator depending on the type and urgency of the anomaly, allowing the educator to take prompt action.

[0166] As a concrete example, consider a case where a learner writes in their notebook, "I'm very confused and don't know what the next step is." The device scans this information and converts it into text data using OCR. Then, the server's natural language processing and sentiment engine estimates that the learner is in a highly confused state. It then sends a notification to the educator stating, "The learner is at a high stress level. Please consider a support meeting."

[0167] An example of a prompt for a generative AI model is: "Using the emotion engine, estimate a specific emotion from the sentence written in the note: 'I'm very confused and don't know what to do next.'"

[0168] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0169] Step 1:

[0170] Users write information in paper or digital notebooks during learning activities. If this information is handwritten, the device scans the notebook using a dedicated application. The handwritten notebook is provided as input. Specifically, an image of the notebook is taken using a camera and digitized. The scanned digital image is generated as output.

[0171] Step 2:

[0172] The terminal extracts text data from scanned digital images using optical character recognition (OCR) technology. The input is the digital image generated in step 1. The OCR software analyzes the characters in the image and outputs the character information as text data. Specifically, it identifies each character and converts it to its corresponding character code.

[0173] Step 3:

[0174] The terminal transmits text data generated by OCR to the server via wireless communication. Text data is used as input. The specific operation of sending data to the server is performed using a communication module. The output is the text data received by the server.

[0175] Step 4:

[0176] The server analyzes the received text data using natural language processing (NLP) to estimate the learner's basic emotional state. The input is the text data from step 3. NLP techniques are used to extract emotion-related keywords and phrases. The output is an estimate of the basic emotional state. Specifically, the server analyzes the text context and calculates an emotion vector.

[0177] Step 5:

[0178] The server uses an emotion engine to reinforce the basic emotional states estimated by NLP. The input includes the basic emotional states obtained in step 4. The emotion engine consults a database and analyzes the subtle nuances of the emotions. The output is the detailed emotional state analysis. Specifically, the emotion engine detects multiple emotional patterns and evaluates their relationships.

[0179] Step 6:

[0180] The server evaluates changes in emotional state and detects anomalies based on the analysis results of the emotion engine. The input is the detailed emotional state from step 5. Based on the analysis results, the anomaly detection algorithm is executed. The output is whether or not an anomaly was detected and its detailed information. Specifically, it compares the result to a baseline value to determine the likelihood of an anomaly.

[0181] Step 7:

[0182] The server generates a notification signal when an anomaly is detected and sends this signal to the training staff. The input contains the anomaly information obtained in step 6. The notification system generates a signal as an alert message. The output is the notification sent to the training staff. Specifically, it generates message content according to the urgency and sends it through the appropriate channel.

[0183] (Application Example 2)

[0184] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0185] In today's brick-and-mortar retail environment, understanding customers' emotional states in real time and responding appropriately is crucial for improving customer satisfaction and increasing sales. However, traditional systems fail to fully utilize customer feedback, making rapid responses difficult. Furthermore, the low accuracy of emotion analysis limits the available response methods.

[0186] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0187] In this invention, the server includes means for digitizing information provided by a user in an information processing device, means for analyzing the digitized information and estimating the emotional state, and means for digitizing user feedback information in a physical store environment, analyzing it to estimate the emotional state, and providing appropriate response methods. This makes it possible to quickly and accurately grasp the emotional state of customers in physical stores and provide effective support by specialist staff.

[0188] An "information processing device" is an electronic device used for collecting, analyzing, digitizing, and managing data.

[0189] "Digitalization" is the process of converting information in analog format into a digital format that can be processed by electronic devices.

[0190] "Emotional state" refers to the user's mental state and emotional condition, and is estimated using natural language processing technology.

[0191] "Analysis" is the process of performing computational processing on digitized information to interpret and evaluate its content.

[0192] An "abnormality" refers to an emotional or behavioral state that deviates from the normal range, and is detected based on the system's judgment.

[0193] A "notification signal" refers to a notification or warning sent to the relevant personnel when an anomaly is detected.

[0194] A "physical store environment" refers to a physical business location where customers visit in person to receive services or products.

[0195] "Feedback information" refers to text or audio recordings of opinions, impressions, questions, etc., provided by users.

[0196] "Response methods" refer to specific action guidelines and solutions provided based on the user's emotional state and feedback.

[0197] "Specialized staff" refers to employees who have received special training to handle customer service and provide product information.

[0198] The system for implementing this invention is centered around a server acting as an information processing device. The server receives customer feedback information transmitted from terminals installed in a physical store environment and performs the necessary processing. The terminals are equipped with a scanner and optical character recognition (OCR) software to scan handwritten notes and feedback forms provided by customers and convert them into digital data. For this OCR technology, for example, the Google Cloud Vision API can be used.

[0199] The server uses natural language processing (NLP) techniques to analyze the customer's emotional state based on the received digitized data. Examples of NLP engines that can be used include spaCy and NLTK. Subsequently, an emotion engine is used for deeper analysis to detect detailed emotional states and abnormal emotional patterns. Based on these analysis results, the server considers appropriate responses to the customer's emotions and sends a notification signal to the specialist staff at the physical store.

[0200] As a concrete example, suppose a customer writes in a feedback form, "I'm interested in the new product, but I'm unsure which to choose." This handwritten comment is scanned by a terminal in the store and converted into text via OCR. This information is sent to a server, where NLP and an emotion engine analyze it to detect emotions such as "hesitation" or "anxiety." Based on this information, the server can send a notification to the store staff such as, "The customer is feeling unsure about their product choice. A detailed explanation from a specialist is recommended."

[0201] Examples of prompt messages include the following:

[0202] "Analyze the feedback content and estimate the customer's emotional state. Based on this, propose appropriate responses to the store staff."

[0203] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0204] Step 1:

[0205] The terminal scans handwritten feedback information provided by the user in the store. The input is the handwritten feedback, and the output is scanned image data. This image data is then passed to OCR software.

[0206] Step 2:

[0207] The device extracts text data from image data using OCR (Optical Character Recognition) technology. The input is a scanned image, and the output is text data. At this stage, handwritten characters are analyzed using OCR technology (e.g., Google Cloud Vision API) and converted into digital text.

[0208] Step 3:

[0209] The terminal sends text data to the server. The input is digitized text data, and the output is a transmission completion notification. The server prepares to receive this text data.

[0210] Step 4:

[0211] The server performs natural language processing (NLP) based on the received text data. The input is text data, and the output is analyzed base sentiment information. An NLP engine (e.g., spaCy, NLTK) is used to analyze the text content and estimate the basic sentiment state.

[0212] Step 5:

[0213] The server uses an emotion engine to perform more detailed emotional analysis. The input is basic emotional information from NLP, and the output is a detailed emotional pattern. The emotion engine analyzes the user's emotional state in detail and detects abnormal emotions.

[0214] Step 6:

[0215] The server generates notification signals for in-store specialists based on the sentiment analysis results. The input is a detailed sentiment pattern, and the output is a notification signal. This notification is configured to include recommended customer service actions.

[0216] Step 7:

[0217] The server sends a notification signal to the store staff's terminal. The input is the notification signal that was created, and the output is a notification that the signal has been sent. The store staff checks this notification and prepares to provide appropriate customer service.

[0218] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0219] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0220] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0221] [Second Embodiment]

[0222] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0223] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0224] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0225] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0226] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0227] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0228] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0229] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0230] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0231] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0232] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0233] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0234] This invention provides a system for continuously monitoring the mental state of learners in educational settings and detecting problems early. Learners keep notes or diaries during their daily learning activities, and the contents are digitized by an information processing device.

[0235] 1. Data Collection

[0236] Notes taken by users (learners) during class are photographed using a scanner or camera-equipped device. This digitizes the handwritten content and extracts it as text data.

[0237] 2. Data Analysis

[0238] The device transmits the digitized notebook information to the server. The server analyzes this data using natural language processing techniques to estimate the emotional state. The emotional state is inferred from the keywords and phrases written by the learner in the notebook content.

[0239] 3. Anomaly detection

[0240] The server compares the analyzed emotional data with historically accumulated data to track changes in emotions. If any abnormalities indicating increased stress or anxiety are detected during this process, a notification is sent to the educator.

[0241] 4. Notification and Response

[0242] If the server detects an anomaly, it immediately sends a notification signal to the educator. This signal includes the name of the learner who detected the anomaly, as well as detailed information about the detected increase in stress or emotion. Recommended countermeasures may also be provided at the same time.

[0243] 5. Specific Examples

[0244] If a student writes in their diary, "Recent lessons are difficult, and I can't keep up," the server analyzes the text and determines that the student is experiencing stress. If the stress level exceeds a set threshold, a notification is sent to the instructor, allowing for prompt action.

[0245] This system allows for the rapid detection and response to changes in learners' emotions, thereby maintaining their mental health and preventing problems such as bullying and school absenteeism.

[0246] The following describes the processing flow.

[0247] Step 1:

[0248] Users (learners) take notes or write diaries during class and save them as digital images using the device's camera or scanner. The device uses optical character recognition (OCR) technology to convert this image data into text data.

[0249] Step 2:

[0250] The terminal transfers text data and digital image data to the server via the network. This data includes all handwritten input from the learner.

[0251] Step 3:

[0252] The server analyzes the received text data using natural language processing (NLP) techniques. This extracts keywords and phrases related to emotions and stress from the recorded content and estimates the emotional state.

[0253] Step 4:

[0254] The server uses image recognition technology to evaluate the irregularities and patterns in handwritten characters. If irregularities in characters occur frequently, this information is used as an indicator of stress or emotional instability.

[0255] Step 5:

[0256] The server compares the analyzed emotional state with existing stored data over time to understand changes in emotional state. In this process, any emotional fluctuations that exceed a certain threshold are detected as abnormal.

[0257] Step 6:

[0258] When the server detects an anomaly, it automatically sends a notification signal to the educator. This notification includes identification of the learner in question, details of the emotional fluctuations, and recommendations for action.

[0259] Step 7:

[0260] Educators receive notifications and take appropriate action or conduct interviews with learners. Users (learners) receive feedback and can get support as needed. This enables early detection and resolution of problems.

[0261] (Example 1)

[0262] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0263] In educational settings, there is a challenge in appropriately monitoring changes in learners' mental states and detecting problems early to take countermeasures. In particular, the lack of means to grasp learners' emotional states in detail increases the risk of overlooking signs of stress and anxiety. Furthermore, immediate notification is necessary when an anomaly is detected so that educators can respond quickly.

[0264] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0265] In this invention, the server includes means for converting handwritten information into digital data using information acquired by the learner with an input device, means for transmitting the digital data to a processing device, and means for the processing device to analyze the digital data using natural language processing technology and estimate the learner's emotional state. This enables rapid and accurate detection of changes in the learner's emotional state, allowing educators to take appropriate action.

[0266] An "input device" is a device used by learners to convert their handwritten information into digital data, and includes equipment such as scanners and cameras.

[0267] "Handwritten information" refers to information such as characters and drawings that learners have written by hand on paper or other media, and is the type of information that is to be converted into digital data.

[0268] "Digital data" refers to data that has been converted from handwritten information using an input device and then processed in a format that can be used by a computer.

[0269] A "processing device" is a device used to analyze and process digital data, and includes equipment such as servers and computers.

[0270] "Natural language processing technology" is a technique that analyzes digitized text to understand the emotions and intentions contained within it, and it utilizes machine learning and artificial intelligence algorithms.

[0271] "Emotional state" refers to the learner's psychological state inferred from the text contained in digital data, and includes classifications such as positive, negative, and neutral.

[0272] "Anomaly detection" is a process that identifies unusual changes or trends based on the results of an analysis of emotional states, and serves as a criterion for notifying educators of the results.

[0273] An "educational administrator" is a person who receives notification when an anomaly is detected and takes measures to improve the learner's condition, and this includes teachers, counselors, and others.

[0274] This system is primarily intended to monitor the mental state of learners in educational environments and to detect abnormalities early. The following hardware and software are used to implement this invention.

[0275] First, the user (learner) acquires handwritten information, such as notes or diaries, using an input device. Scanners or cameras are used as input devices. This converts the handwritten information into digital data.

[0276] Next, the terminal plays the role of transmitting the acquired digital data to the server. A dedicated application is installed on the terminal, and this application automatically manages the data transmission. The data is securely transferred to the server via the internet.

[0277] The server processes the received digital data for analysis. Specifically, it uses natural language processing technology running on the server to estimate emotional states from text written by learners. This analysis utilizes sentiment dictionaries and machine learning models. The server detects anomalies by comparing them with past data and notifies educational administrators as needed.

[0278] As a concrete example, consider a case where a learner writes in their diary, "I have a lot of assignments every day, and I feel pressured." The server, based on this text, determines that the learner is experiencing stress, and if the stress level exceeds a certain threshold, a notification is sent to the education administrator. This notification may include specific details about the problem and recommended coping strategies.

[0279] Furthermore, as an example of a prompt using a generative AI model, the model can be given instructions such as, "Please input the contents of a student's diary and provide the emotions and stress indicators you perceive from the text." Such prompts enable the generative AI model to perform appropriate emotion analysis and respond quickly.

[0280] This system effectively supports learners' mental health and enables early detection and response to problems.

[0281] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0282] Step 1:

[0283] The user (learner) performs an operation of importing notes and diaries during the class into the input device. The input device used is a scanner or a terminal with a camera. The input is handwritten information, and the output is that the information is converted into a digital image. Specifically, the user uses the terminal to take a picture of the note page, and the image file is saved in the terminal.

[0284] Step 2:

[0285] The terminal performs an operation of converting the captured digital image into text data. OCR (Optical Character Recognition) software is used for this process. The input is a digital image, and the output is the text data extracted from the image. Specifically, the terminal automatically executes OCR, reads the characters contained in the image as text, and saves them in a text file.

[0286] Step 3:

[0287] The terminal sends the generated text data to the server. The input is the text data obtained in the previous step, and the output is the upload of the text data to the server. Specifically, the dedicated application of the terminal is launched, and the text data is uploaded to the server using a secure protocol.

[0288] Step 4:<000091十二>

[0289] The server executes natural language processing to analyze the received text data. The input is the text data sent from the terminal, and the output is the estimated emotional state of the learner. Specifically, the natural language processing algorithm on the server analyzes the text data, and the generated AI model is used to judge emotions and psychological states.

[0290] Step 5:

[0291] The server detects anomalies based on the analysis results. Inputs include the current emotional state determined by text analysis and past data, while output is the result of the anomaly detection. Specifically, the server compares the current emotional state with a set baseline value to identify emotional deviations or abnormal changes.

[0292] Step 6:

[0293] The server performs the action of notifying the education administrator of the results of anomaly detection. The inputs are the anomaly detection results and learner information, and the output is a notification to the education administrator. Specifically, the server provides the education administrator with details of the anomaly and recommended countermeasures using email or a notification application.

[0294] (Application Example 1)

[0295] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0296] There is a need to provide an environment that effectively utilizes the time learners spend in autonomous vehicles and promotes learning while appropriately monitoring their mental state. However, conventional educational systems have difficulty monitoring learners' emotional states in real time and taking appropriate action. This presents a challenge in early detection of learners' anxiety and stress and taking swift countermeasures.

[0297] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0298] In this invention, the server includes processing means for converting information entered by the user into a digital format, processing means for analyzing the digitized information and inferring the emotional state, and processing means for sending a warning signal to an educational support worker when an abnormality is detected. This enables real-time monitoring of the learner's mental state inside an autonomous vehicle and allows for a rapid response when an abnormality is detected.

[0299] "Users" refer to individuals who utilize the learning environment within an autonomous vehicle, and whose learning progress and mental state are subject to monitoring.

[0300] "Digital format" refers to a format in which handwritten or printed information is converted into electronic information that can be processed by a computer.

[0301] "Processing means" refers to a device or program for handling, analyzing, transforming, or transmitting information, and is a technical means for performing operations necessary to solve a problem.

[0302] "Emotional state" refers to information that indicates the user's psychological or mental state, and particularly includes stress, anxiety, and satisfaction during learning activities.

[0303] A "warning signal" is a notification sent to educational support staff or others to draw their attention when a deviation from the expected normal state is detected.

[0304] An "educational supporter" is a person whose role is to monitor learners' learning and mental state, and to provide guidance or support as needed.

[0305] An "autonomous vehicle" refers to a vehicle that operates autonomously using artificial intelligence and sensor technology, minimizing human intervention during operation.

[0306] This invention is a system that provides a safe and secure learning environment for learners inside autonomous vehicles. The system continuously monitors the learner's mental state and detects any abnormalities early, thereby providing optimal support.

[0307] The server receives information transmitted from terminals such as tablets or smart glasses used by learners, and uses OCR technology to digitize this information. Specifically, it uses Google Cloud Vision API or the like to convert handwritten information into electronic data. The server analyzes this electronic data and estimates the emotional state of the learner using natural language processing libraries such as NLTK and AI models (e.g., using TensorFlow) that enable sentiment analysis.

[0308] When fluctuations in the emotional state are detected, the server uses time-series data to identify anomalies. When an anomaly is confirmed, it utilizes the mail API or notification system to promptly send a warning signal to the education supporter. This process enables appropriate support for learners and improves the learning environment.

[0309] As a specific example, when a learner enters "Today's test was very difficult" on an electronic tablet inside an autonomous vehicle, the server analyzes this description and detects a potential stress state. Subsequently, the AI model evaluates it and notifies the education supporter if the learner's stress exceeds the standard.

[0310] An example of a prompt sentence is: "Please evaluate the situation based on the note data entered by the student in the autonomous vehicle and the sentiment analysis result, and propose learning support measures. This time, we will create a prompt for supporting students who feel stressed about the assignment." In this way, the system has a function to support the mental health of learners.

[0311] The flow of the specific process in Application Example 1 will be described using Figure 12.

[0312] Step 1: <00009​​The user inputs text and drawings during learning using a tablet or smart glasses. Image data is obtained by taking a picture of the handwritten notes with the device's built-in camera. In this step, the input is handwritten notes, and the output is image data.

[0314] Step 2:

[0315] The device converts the acquired image data into text data using OCR technology. Specifically, it uses the Google Cloud Vision API to recognize characters in the image and extracts them as digital text. The input for this step is image data, and the output is text data.

[0316] Step 3:

[0317] The device sends text data to a cloud server. The server analyzes the received text data using the natural language processing library NLTK and performs sentiment analysis. In this step, the emotional state is estimated based on keywords and phrases in the text data. The input is text data, and the output is estimated sentiment data.

[0318] Step 4:

[0319] The server compares estimated emotion data with historically stored data. Based on the dataset of past emotional states, it detects anomalies in the current emotional state. Specifically, it uses an AI model (e.g., TensorFlow) to evaluate emotional fluctuations. The inputs are emotion data and historical data, and the output is whether or not an anomaly is present.

[0320] Step 5:

[0321] If an anomaly is detected, the server sends a warning signal to the educational support staff. This process utilizes an email API or notification system, providing specific emotional states and recommended actions as warnings. The input is the result of the anomaly detection, and the output is the warning signal.

[0322] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0323] This invention relates to a system for accurately understanding the mental state of learners in educational settings and detecting abnormalities early. It is particularly characterized by its incorporation of an emotion engine to recognize learners' emotions. Learners keep notes and diaries during their daily learning activities, and these are digitized by an information processing device.

[0324] 1. Information gathering and digitization

[0325] Users (learners) take notes during class and scan those notes using a device. This scanning process digitizes the handwritten content, which is then converted into text data using optical character recognition (OCR) technology.

[0326] 2. Data transmission and initial analysis

[0327] The device sends digitized text and image data to the server. The server analyzes the received data using natural language processing (NLP) to estimate the user's basic emotional state.

[0328] 3. Reinforcement estimation using an emotion engine

[0329] The server utilizes an emotion engine to further enhance the estimated emotional state. The emotion engine detects more refined emotional patterns and recognizes subtle emotional nuances that cannot be captured by conventional natural language processing techniques.

[0330] 4. Anomaly detection and notification coordination

[0331] The server utilizes information obtained from the emotion engine to detect an anomaly when the emotional state deviates significantly from the normal range. Based on the output of the emotion engine, the server adjusts the content of the notification signal when an anomaly occurs and notifies the training staff, including the urgency and recommended countermeasures.

[0332] 5. Specific Examples

[0333] Suppose a learner writes in their notebook, "I'm very confused and don't know what to do next." The server's natural language processing system references the keyword "confused," and the emotion engine further estimates the level of stress and anxiety underlying the statement. As a result, it determines that the stress level exceeds a certain threshold and sends a notification to the educator stating, "The learner is experiencing a high level of stress. Please consider a support session."

[0334] This system enables faster and more accurate detection of changes in learners' emotions, facilitating early problem resolution.

[0335] The following describes the processing flow.

[0336] Step 1:

[0337] The user (learner) takes notes or writes a diary during class and captures them as digital images using the device's camera or scanner. The device then uses character recognition technology to convert these images into text data.

[0338] Step 2:

[0339] The terminal sends the generated text and image data to the server. During this process, it verifies that the data is properly formatted and adds any necessary metadata.

[0340] Step 3:

[0341] The server analyzes the received text data using natural language processing techniques to estimate basic emotional states. This analysis classifies emotions based on specific keywords or phrases.

[0342] Step 4:

[0343] The server activates the emotion engine to perform further estimation of emotional states. The emotion engine detects subtle emotional nuances from the text written by the learner and reinforces the intensity and type of emotion.

[0344] Step 5:

[0345] Based on the analysis results, the server compares the learner's emotional changes with past data to detect anomalies. If an anomaly is found, it analyzes its nature and evaluates its severity and scope of impact.

[0346] Step 6:

[0347] If an anomaly is detected, the server sends a notification signal to the educator. This signal includes learner identification information, a specific description of the emotional anomaly, and recommended actions. The notification content is adjusted based on the output of the emotion engine.

[0348] Step 7:

[0349] Instructors receive notifications from the server, provide feedback to learners as needed, and conduct counseling and support activities. Users (learners) can resolve problems through the support provided by instructors.

[0350] (Example 2)

[0351] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0352] In educational settings, accurately understanding learners' mental states and detecting abnormalities early is crucial for providing appropriate support to individual learners. Traditional methods struggle to quickly and accurately detect emotional problems faced by learners, potentially delaying appropriate intervention. Therefore, a system is needed that can more accurately assess learners' emotional states and immediately detect abnormalities.

[0353] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0354] In this invention, the server includes means for digitizing information created by learners, means for estimating basic emotional states using natural language processing technology, means for analyzing subtle emotional nuances using an emotion engine, and means for detecting anomalies and transmitting notification signals. This makes it possible to quickly and accurately detect subtle changes in the learner's emotional state and to immediately provide appropriate educational interventions.

[0355] An "information processing device" is a device equipped with the function of analyzing digitized information and performing specific processing.

[0356] "Digitalization" is the process of converting analog data into an electronically processable format.

[0357] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0358] An "emotion engine" is a technology that analyzes human emotions and recognizes subtle emotional nuances.

[0359] "Anomaly detection" is the process of identifying conditions that deviate from the normal range.

[0360] A "notification signal" is a notification or alert that is sent when an anomaly is detected.

[0361] This invention is a system for accurately understanding the emotional state of learners in educational settings and detecting abnormalities early. The system is centered around an information processing device and is composed of the following hardware and software components.

[0362] Users (learners) record information in paper or digital notebooks during their daily learning activities. The recorded content is scanned by a device with a dedicated application installed and saved as a digital image. This device includes optical character recognition (OCR) software, which converts the image data into text data.

[0363] The converted text data is transmitted to the server via wireless communication. The server analyzes the text data using natural language processing (NLP) techniques to estimate the learner's basic emotional state. Furthermore, it utilizes an emotion engine to extract subtle emotional nuances from this basic emotional state. This process enables a deeper understanding of emotions compared to conventional natural language processing.

[0364] The server monitors changes in learners' emotional states based on emotional information obtained from the emotion engine, and generates a notification signal when an anomaly is detected. This signal is configured to be automatically sent to the educator depending on the type and urgency of the anomaly, allowing the educator to take prompt action.

[0365] As a concrete example, consider a case where a learner writes in their notebook, "I'm very confused and don't know what the next step is." The device scans this information and converts it into text data using OCR. Then, the server's natural language processing and sentiment engine estimates that the learner is in a highly confused state. It then sends a notification to the educator stating, "The learner is at a high stress level. Please consider a support meeting."

[0366] An example of a prompt for a generative AI model is: "Using the emotion engine, estimate a specific emotion from the sentence written in the note: 'I'm very confused and don't know what to do next.'"

[0367] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0368] Step 1:

[0369] Users write information in paper or digital notebooks during learning activities. If this information is handwritten, the device scans the notebook using a dedicated application. The handwritten notebook is provided as input. Specifically, an image of the notebook is taken using a camera and digitized. The scanned digital image is generated as output.

[0370] Step 2:

[0371] The terminal extracts text data from scanned digital images using optical character recognition (OCR) technology. The input is the digital image generated in step 1. The OCR software analyzes the characters in the image and outputs the character information as text data. Specifically, it identifies each character and converts it to its corresponding character code.

[0372] Step 3:

[0373] The terminal transmits text data generated by OCR to the server via wireless communication. Text data is used as input. The specific operation of sending data to the server is performed using a communication module. The output is the text data received by the server.

[0374] Step 4:

[0375] The server analyzes the received text data using natural language processing (NLP) to estimate the learner's basic emotional state. The input is the text data from step 3. NLP techniques are used to extract emotion-related keywords and phrases. The output is an estimate of the basic emotional state. Specifically, the server analyzes the text context and calculates an emotion vector.

[0376] Step 5:

[0377] The server uses an emotion engine to reinforce the basic emotional states estimated by NLP. The input includes the basic emotional states obtained in step 4. The emotion engine consults a database and analyzes the subtle nuances of the emotions. The output is the detailed emotional state analysis. Specifically, the emotion engine detects multiple emotional patterns and evaluates their relationships.

[0378] Step 6:

[0379] The server evaluates changes in emotional state and detects anomalies based on the analysis results of the emotion engine. The input is the detailed emotional state from step 5. Based on the analysis results, the anomaly detection algorithm is executed. The output is whether or not an anomaly was detected and its detailed information. Specifically, it compares the result to a baseline value to determine the likelihood of an anomaly.

[0380] Step 7:

[0381] The server generates a notification signal when an anomaly is detected and sends this signal to the training staff. The input contains the anomaly information obtained in step 6. The notification system generates a signal as an alert message. The output is the notification sent to the training staff. Specifically, it generates message content according to the urgency and sends it through the appropriate channel.

[0382] (Application Example 2)

[0383] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0384] In today's brick-and-mortar retail environment, understanding customers' emotional states in real time and responding appropriately is crucial for improving customer satisfaction and increasing sales. However, traditional systems fail to fully utilize customer feedback, making rapid responses difficult. Furthermore, the low accuracy of emotion analysis limits the available response methods.

[0385] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0386] In this invention, the server includes means for digitizing information provided by a user in an information processing device, means for analyzing the digitized information and estimating the emotional state, and means for digitizing user feedback information in a physical store environment, analyzing it to estimate the emotional state, and providing appropriate response methods. This makes it possible to quickly and accurately grasp the emotional state of customers in physical stores and provide effective support by specialist staff.

[0387] An "information processing device" is an electronic device used for collecting, analyzing, digitizing, and managing data.

[0388] "Digitalization" is the process of converting information in analog format into a digital format that can be processed by electronic devices.

[0389] "Emotional state" refers to the user's mental state and emotional condition, and is estimated using natural language processing technology.

[0390] "Analysis" is the process of performing computational processing on digitized information to interpret and evaluate its content.

[0391] An "abnormality" refers to an emotional or behavioral state that deviates from the normal range, and is detected based on the system's judgment.

[0392] A "notification signal" refers to a notification or warning sent to the relevant personnel when an anomaly is detected.

[0393] A "physical store environment" refers to a physical business location where customers visit in person to receive services or products.

[0394] "Feedback information" refers to text or audio recordings of opinions, impressions, questions, etc., provided by users.

[0395] "Response methods" refer to specific action guidelines and solutions provided based on the user's emotional state and feedback.

[0396] "Specialized staff" refers to employees who have received special training to handle customer service and provide product information.

[0397] The system for implementing this invention is centered around a server acting as an information processing device. The server receives customer feedback information transmitted from terminals installed in a physical store environment and performs the necessary processing. The terminals are equipped with a scanner and optical character recognition (OCR) software to scan handwritten notes and feedback forms provided by customers and convert them into digital data. For this OCR technology, for example, the Google Cloud Vision API can be used.

[0398] The server uses natural language processing (NLP) techniques to analyze the customer's emotional state based on the received digitized data. Examples of NLP engines that can be used include spaCy and NLTK. Subsequently, an emotion engine is used for deeper analysis to detect detailed emotional states and abnormal emotional patterns. Based on these analysis results, the server considers appropriate responses to the customer's emotions and sends a notification signal to the specialist staff at the physical store.

[0399] As a concrete example, suppose a customer writes in a feedback form, "I'm interested in the new product, but I'm unsure which to choose." This handwritten comment is scanned by a terminal in the store and converted into text via OCR. This information is sent to a server, where NLP and an emotion engine analyze it to detect emotions such as "hesitation" or "anxiety." Based on this information, the server can send a notification to the store staff such as, "The customer is feeling unsure about their product choice. A detailed explanation from a specialist is recommended."

[0400] Examples of prompt messages include the following:

[0401] "Analyze the feedback content and estimate the customer's emotional state. Based on this, propose appropriate responses to the store staff."

[0402] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0403] Step 1:

[0404] The terminal scans handwritten feedback information provided by the user in the store. The input is the handwritten feedback, and the output is scanned image data. This image data is then passed to OCR software.

[0405] Step 2:

[0406] The device extracts text data from image data using OCR (Optical Character Recognition) technology. The input is a scanned image, and the output is text data. At this stage, handwritten characters are analyzed using OCR technology (e.g., Google Cloud Vision API) and converted into digital text.

[0407] Step 3:

[0408] The terminal sends text data to the server. The input is digitized text data, and the output is a transmission completion notification. The server prepares to receive this text data.

[0409] Step 4:

[0410] The server performs natural language processing (NLP) based on the received text data. The input is text data, and the output is analyzed base sentiment information. An NLP engine (e.g., spaCy, NLTK) is used to analyze the text content and estimate the basic sentiment state.

[0411] Step 5:

[0412] The server uses an emotion engine to perform more detailed emotional analysis. The input is basic emotional information from NLP, and the output is a detailed emotional pattern. The emotion engine analyzes the user's emotional state in detail and detects abnormal emotions.

[0413] Step 6:

[0414] The server generates notification signals for in-store specialists based on the sentiment analysis results. The input is a detailed sentiment pattern, and the output is a notification signal. This notification is configured to include recommended customer service actions.

[0415] Step 7:

[0416] The server sends a notification signal to the store staff's terminal. The input is the notification signal that was created, and the output is a notification that the signal has been sent. The store staff checks this notification and prepares to provide appropriate customer service.

[0417] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0418] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0419] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0420] [Third Embodiment]

[0421] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0422] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0423] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0424] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0425] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0426] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0427] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0428] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0429] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0430] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0431] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0432] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0433] This invention provides a system for continuously monitoring the mental state of learners in educational settings and detecting problems early. Learners keep notes or diaries during their daily learning activities, and the contents are digitized by an information processing device.

[0434] 1. Data Collection

[0435] Notes taken by users (learners) during class are photographed using a scanner or camera-equipped device. This digitizes the handwritten content and extracts it as text data.

[0436] 2. Data Analysis

[0437] The device transmits the digitized notebook information to the server. The server analyzes this data using natural language processing techniques to estimate the emotional state. The emotional state is inferred from the keywords and phrases written by the learner in the notebook content.

[0438] 3. Anomaly detection

[0439] The server compares the analyzed emotional data with historically accumulated data to track changes in emotions. If any abnormalities indicating increased stress or anxiety are detected during this process, a notification is sent to the educator.

[0440] 4. Notification and Response

[0441] If the server detects an anomaly, it immediately sends a notification signal to the educator. This signal includes the name of the learner who detected the anomaly, as well as detailed information about the detected increase in stress or emotion. Recommended countermeasures may also be provided at the same time.

[0442] 5. Specific Examples

[0443] If a student writes in their diary, "Recent lessons are difficult, and I can't keep up," the server analyzes the text and determines that the student is experiencing stress. If the stress level exceeds a set threshold, a notification is sent to the instructor, allowing for prompt action.

[0444] This system allows for the rapid detection and response to changes in learners' emotions, thereby maintaining their mental health and preventing problems such as bullying and school absenteeism.

[0445] The following describes the processing flow.

[0446] Step 1:

[0447] Users (learners) take notes or write diaries during class and save them as digital images using the device's camera or scanner. The device uses optical character recognition (OCR) technology to convert this image data into text data.

[0448] Step 2:

[0449] The terminal transfers text data and digital image data to the server via the network. This data includes all handwritten input from the learner.

[0450] Step 3:

[0451] The server analyzes the received text data using natural language processing (NLP) techniques. This extracts keywords and phrases related to emotions and stress from the recorded content and estimates the emotional state.

[0452] Step 4:

[0453] The server uses image recognition technology to evaluate the irregularities and patterns in handwritten characters. If irregularities in characters occur frequently, this information is used as an indicator of stress or emotional instability.

[0454] Step 5:

[0455] The server compares the analyzed emotional state with existing stored data over time to understand changes in emotional state. In this process, any emotional fluctuations that exceed a certain threshold are detected as abnormal.

[0456] Step 6:

[0457] When the server detects an anomaly, it automatically sends a notification signal to the educator. This notification includes identification of the learner in question, details of the emotional fluctuations, and recommendations for action.

[0458] Step 7:

[0459] Educators receive notifications and take appropriate action or conduct interviews with learners. Users (learners) receive feedback and can get support as needed. This enables early detection and resolution of problems.

[0460] (Example 1)

[0461] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0462] In educational settings, there is a challenge in appropriately monitoring changes in learners' mental states and detecting problems early to take countermeasures. In particular, the lack of means to grasp learners' emotional states in detail increases the risk of overlooking signs of stress and anxiety. Furthermore, immediate notification is necessary when an anomaly is detected so that educators can respond quickly.

[0463] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0464] In this invention, the server includes means for converting handwritten information into digital data using information acquired by the learner with an input device, means for transmitting the digital data to a processing device, and means for the processing device to analyze the digital data using natural language processing technology and estimate the learner's emotional state. This enables rapid and accurate detection of changes in the learner's emotional state, allowing educators to take appropriate action.

[0465] An "input device" is a device used by learners to convert their handwritten information into digital data, and includes equipment such as scanners and cameras.

[0466] "Handwritten information" refers to information such as characters and drawings that learners have written by hand on paper or other media, and is the type of information that is to be converted into digital data.

[0467] "Digital data" refers to data that has been converted from handwritten information using an input device and then processed in a format that can be used by a computer.

[0468] A "processing device" is a device used to analyze and process digital data, and includes equipment such as servers and computers.

[0469] "Natural language processing technology" is a technique that analyzes digitized text to understand the emotions and intentions contained within it, and it utilizes machine learning and artificial intelligence algorithms.

[0470] "Emotional state" refers to the learner's psychological state inferred from the text contained in digital data, and includes classifications such as positive, negative, and neutral.

[0471] "Anomaly detection" is a process that identifies unusual changes or trends based on the results of an analysis of emotional states, and serves as a criterion for notifying educators of the results.

[0472] An "educational administrator" is a person who receives notification when an anomaly is detected and takes measures to improve the learner's condition, and this includes teachers, counselors, and others.

[0473] This system is primarily intended to monitor the mental state of learners in educational environments and to detect abnormalities early. The following hardware and software are used to implement this invention.

[0474] First, the user (learner) acquires handwritten information, such as notes or diaries, using an input device. Scanners or cameras are used as input devices. This converts the handwritten information into digital data.

[0475] Next, the terminal plays the role of transmitting the acquired digital data to the server. A dedicated application is installed on the terminal, and this application automatically manages the data transmission. The data is securely transferred to the server via the internet.

[0476] The server processes the received digital data for analysis. Specifically, it uses natural language processing technology running on the server to estimate emotional states from text written by learners. This analysis utilizes sentiment dictionaries and machine learning models. The server detects anomalies by comparing them with past data and notifies educational administrators as needed.

[0477] As a concrete example, consider a case where a learner writes in their diary, "I have a lot of assignments every day, and I feel pressured." The server, based on this text, determines that the learner is experiencing stress, and if the stress level exceeds a certain threshold, a notification is sent to the education administrator. This notification may include specific details about the problem and recommended coping strategies.

[0478] Furthermore, as an example of a prompt using a generative AI model, the model can be given instructions such as, "Please input the contents of a student's diary and provide the emotions and stress indicators you perceive from the text." Such prompts enable the generative AI model to perform appropriate emotion analysis and respond quickly.

[0479] This system effectively supports learners' mental health and enables early detection and response to problems.

[0480] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0481] Step 1:

[0482] The user (learner) takes notes or diary entries during class and inputs them into an input device. The input device used is either a scanner or a terminal with a camera. The input is handwritten information, and the output is that information converted into a digital image. Specifically, the user takes a picture of a page of their notebook using the terminal, and the image file is saved to the terminal.

[0483] Step 2:

[0484] The device converts captured digital images into text data. This process utilizes OCR (Optical Character Recognition) software. The input is a digital image, and the output is text data extracted from that image. Specifically, the device automatically performs OCR, reads the characters contained in the image as text, and saves them to a text file.

[0485] Step 3:

[0486] The terminal sends the generated text data to the server. The input is the text data obtained in the previous step, and the output is the uploading of the text data to the server. Specifically, a dedicated application on the terminal is launched, and the text data is uploaded to the server using a secure protocol.

[0487] Step 4:

[0488] The server performs natural language processing to analyze the received text data. The input is text data sent from the terminal, and the output is the learner's estimated emotional state. Specifically, a natural language processing algorithm on the server analyzes the text data, and a generative AI model is used to determine emotions and psychological states.

[0489] Step 5:

[0490] The server detects anomalies based on the analysis results. Inputs include the current emotional state determined by text analysis and past data, while output is the result of the anomaly detection. Specifically, the server compares the current emotional state with a set baseline value to identify emotional deviations or abnormal changes.

[0491] Step 6:

[0492] The server performs the action of notifying the education administrator of the results of anomaly detection. The inputs are the anomaly detection results and learner information, and the output is a notification to the education administrator. Specifically, the server provides the education administrator with details of the anomaly and recommended countermeasures using email or a notification application.

[0493] (Application Example 1)

[0494] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0495] There is a need to provide an environment that effectively utilizes the time learners spend in autonomous vehicles and promotes learning while appropriately monitoring their mental state. However, conventional educational systems have difficulty monitoring learners' emotional states in real time and taking appropriate action. This presents a challenge in early detection of learners' anxiety and stress and taking swift countermeasures.

[0496] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0497] In this invention, the server includes processing means for converting information entered by the user into a digital format, processing means for analyzing the digitized information and inferring the emotional state, and processing means for sending a warning signal to an educational support worker when an abnormality is detected. This enables real-time monitoring of the learner's mental state inside an autonomous vehicle and allows for a rapid response when an abnormality is detected.

[0498] "Users" refer to individuals who utilize the learning environment within an autonomous vehicle, and whose learning progress and mental state are subject to monitoring.

[0499] "Digital format" refers to a format in which handwritten or printed information is converted into electronic information that can be processed by a computer.

[0500] "Processing means" refers to a device or program for handling, analyzing, transforming, or transmitting information, and is a technical means for performing operations necessary to solve a problem.

[0501] "Emotional state" refers to information that indicates the user's psychological or mental state, and particularly includes stress, anxiety, and satisfaction during learning activities.

[0502] A "warning signal" is a notification sent to educational support staff or others to draw their attention when a deviation from the expected normal state is detected.

[0503] An "educational supporter" is a person whose role is to monitor learners' learning and mental state, and to provide guidance or support as needed.

[0504] An "autonomous vehicle" refers to a vehicle that operates autonomously using artificial intelligence and sensor technology, minimizing human intervention during operation.

[0505] This invention is a system that provides a safe and secure learning environment for learners inside autonomous vehicles. The system continuously monitors the learner's mental state and detects any abnormalities early, thereby providing optimal support.

[0506] The server receives information transmitted from a device used by the learner, such as a tablet or smart glasses, and uses OCR technology to digitize that information. Specifically, it uses the Google Cloud Vision API or similar to convert handwritten information into electronic data. The server then analyzes this electronic data and estimates the learner's emotional state using the natural language processing library NLTK and an AI model that enables sentiment analysis (e.g., using TensorFlow).

[0507] When fluctuations in emotional state are detected, the server uses time-series data to identify anomalies. Once an anomaly is confirmed, it utilizes email APIs and notification systems to quickly send warning signals to educational support staff. This process enables appropriate support for learners and improves the learning environment.

[0508] For example, if a learner writes "Today's exam was very difficult" on an electronic tablet while in an autonomous vehicle, the server analyzes this statement and detects a potential stress level. An AI model then evaluates the situation, and if the learner's stress level exceeds a certain threshold, it notifies the educational support staff.

[0509] An example of a prompt is: "Evaluate the situation based on the note data and sentiment analysis results written by the student inside the autonomous vehicle, and propose learning support measures. This time, create a prompt to support a student who is experiencing stress from the assignment." In this way, the system has a function to support the mental health of learners.

[0510] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0511] Step 1:

[0512] The user inputs text and drawings during learning using a tablet or smart glasses. Image data is obtained by taking a picture of the handwritten notes with the device's built-in camera. In this step, the input is handwritten notes, and the output is image data.

[0513] Step 2:

[0514] The device converts the acquired image data into text data using OCR technology. Specifically, it uses the Google Cloud Vision API to recognize characters in the image and extracts them as digital text. The input for this step is image data, and the output is text data.

[0515] Step 3:

[0516] The device sends text data to a cloud server. The server analyzes the received text data using the natural language processing library NLTK and performs sentiment analysis. In this step, the emotional state is estimated based on keywords and phrases in the text data. The input is text data, and the output is estimated sentiment data.

[0517] Step 4:

[0518] The server compares estimated emotion data with historically stored data. Based on the dataset of past emotional states, it detects anomalies in the current emotional state. Specifically, it uses an AI model (e.g., TensorFlow) to evaluate emotional fluctuations. The inputs are emotion data and historical data, and the output is whether or not an anomaly is present.

[0519] Step 5:

[0520] If an anomaly is detected, the server sends a warning signal to the educational support staff. This process utilizes an email API or notification system, providing specific emotional states and recommended actions as warnings. The input is the result of the anomaly detection, and the output is the warning signal.

[0521] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0522] This invention relates to a system for accurately understanding the mental state of learners in educational settings and detecting abnormalities early. It is particularly characterized by its incorporation of an emotion engine to recognize learners' emotions. Learners keep notes and diaries during their daily learning activities, and these are digitized by an information processing device.

[0523] 1. Information gathering and digitization

[0524] Users (learners) take notes during class and scan those notes using a device. This scanning process digitizes the handwritten content, which is then converted into text data using optical character recognition (OCR) technology.

[0525] 2. Data transmission and initial analysis

[0526] The device sends digitized text and image data to the server. The server analyzes the received data using natural language processing (NLP) to estimate the user's basic emotional state.

[0527] 3. Reinforcement estimation using an emotion engine

[0528] The server utilizes an emotion engine to further enhance the estimated emotional state. The emotion engine detects more refined emotional patterns and recognizes subtle emotional nuances that cannot be captured by conventional natural language processing techniques.

[0529] 4. Anomaly detection and notification coordination

[0530] The server utilizes information obtained from the emotion engine to detect an anomaly when the emotional state deviates significantly from the normal range. Based on the output of the emotion engine, the server adjusts the content of the notification signal when an anomaly occurs and notifies the training staff, including the urgency and recommended countermeasures.

[0531] 5. Specific Examples

[0532] Suppose a learner writes in their notebook, "I'm very confused and don't know what to do next." The server's natural language processing system references the keyword "confused," and the emotion engine further estimates the level of stress and anxiety underlying the statement. As a result, it determines that the stress level exceeds a certain threshold and sends a notification to the educator stating, "The learner is experiencing a high level of stress. Please consider a support session."

[0533] This system enables faster and more accurate detection of changes in learners' emotions, facilitating early problem resolution.

[0534] The following describes the processing flow.

[0535] Step 1:

[0536] The user (learner) takes notes or writes a diary during class and captures them as digital images using the device's camera or scanner. The device then uses character recognition technology to convert these images into text data.

[0537] Step 2:

[0538] The terminal sends the generated text and image data to the server. During this process, it verifies that the data is properly formatted and adds any necessary metadata.

[0539] Step 3:

[0540] The server analyzes the received text data using natural language processing techniques to estimate basic emotional states. This analysis classifies emotions based on specific keywords or phrases.

[0541] Step 4:

[0542] The server activates the emotion engine to perform further estimation of emotional states. The emotion engine detects subtle emotional nuances from the text written by the learner and reinforces the intensity and type of emotion.

[0543] Step 5:

[0544] Based on the analysis results, the server compares the learner's emotional changes with past data to detect anomalies. If an anomaly is found, it analyzes its nature and evaluates its severity and scope of impact.

[0545] Step 6:

[0546] If an anomaly is detected, the server sends a notification signal to the educator. This signal includes learner identification information, a specific description of the emotional anomaly, and recommended actions. The notification content is adjusted based on the output of the emotion engine.

[0547] Step 7:

[0548] Instructors receive notifications from the server, provide feedback to learners as needed, and conduct counseling and support activities. Users (learners) can resolve problems through the support provided by instructors.

[0549] (Example 2)

[0550] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0551] In educational settings, accurately understanding learners' mental states and detecting abnormalities early is crucial for providing appropriate support to individual learners. Traditional methods struggle to quickly and accurately detect emotional problems faced by learners, potentially delaying appropriate intervention. Therefore, a system is needed that can more accurately assess learners' emotional states and immediately detect abnormalities.

[0552] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0553] In this invention, the server includes means for digitizing information created by learners, means for estimating basic emotional states using natural language processing technology, means for analyzing subtle emotional nuances using an emotion engine, and means for detecting anomalies and transmitting notification signals. This makes it possible to quickly and accurately detect subtle changes in the learner's emotional state and to immediately provide appropriate educational interventions.

[0554] An "information processing device" is a device equipped with the function of analyzing digitized information and performing specific processing.

[0555] "Digitalization" is the process of converting analog data into an electronically processable format.

[0556] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0557] An "emotion engine" is a technology that analyzes human emotions and recognizes subtle emotional nuances.

[0558] "Anomaly detection" is the process of identifying conditions that deviate from the normal range.

[0559] A "notification signal" is a notification or alert that is sent when an anomaly is detected.

[0560] This invention is a system for accurately understanding the emotional state of learners in educational settings and detecting abnormalities early. The system is centered around an information processing device and is composed of the following hardware and software components.

[0561] Users (learners) record information in paper or digital notebooks during their daily learning activities. The recorded content is scanned by a device with a dedicated application installed and saved as a digital image. This device includes optical character recognition (OCR) software, which converts the image data into text data.

[0562] The converted text data is transmitted to the server via wireless communication. The server analyzes the text data using natural language processing (NLP) techniques to estimate the learner's basic emotional state. Furthermore, it utilizes an emotion engine to extract subtle emotional nuances from this basic emotional state. This process enables a deeper understanding of emotions compared to conventional natural language processing.

[0563] The server monitors changes in learners' emotional states based on emotional information obtained from the emotion engine, and generates a notification signal when an anomaly is detected. This signal is configured to be automatically sent to the educator depending on the type and urgency of the anomaly, allowing the educator to take prompt action.

[0564] As a concrete example, consider a case where a learner writes in their notebook, "I'm very confused and don't know what the next step is." The device scans this information and converts it into text data using OCR. Then, the server's natural language processing and sentiment engine estimates that the learner is in a highly confused state. It then sends a notification to the educator stating, "The learner is at a high stress level. Please consider a support meeting."

[0565] An example of a prompt for a generative AI model is: "Using the emotion engine, estimate a specific emotion from the sentence written in the note: 'I'm very confused and don't know what to do next.'"

[0566] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0567] Step 1:

[0568] Users write information in paper or digital notebooks during learning activities. If this information is handwritten, the device scans the notebook using a dedicated application. The handwritten notebook is provided as input. Specifically, an image of the notebook is taken using a camera and digitized. The scanned digital image is generated as output.

[0569] Step 2:

[0570] The terminal extracts text data from scanned digital images using optical character recognition (OCR) technology. The input is the digital image generated in step 1. The OCR software analyzes the characters in the image and outputs the character information as text data. Specifically, it identifies each character and converts it to its corresponding character code.

[0571] Step 3:

[0572] The terminal transmits text data generated by OCR to the server via wireless communication. Text data is used as input. The specific operation of sending data to the server is performed using a communication module. The output is the text data received by the server.

[0573] Step 4:

[0574] The server analyzes the received text data using natural language processing (NLP) to estimate the learner's basic emotional state. The input is the text data from step 3. NLP techniques are used to extract emotion-related keywords and phrases. The output is an estimate of the basic emotional state. Specifically, the server analyzes the text context and calculates an emotion vector.

[0575] Step 5:

[0576] The server uses an emotion engine to reinforce the basic emotional states estimated by NLP. The input includes the basic emotional states obtained in step 4. The emotion engine consults a database and analyzes the subtle nuances of the emotions. The output is the detailed emotional state analysis. Specifically, the emotion engine detects multiple emotional patterns and evaluates their relationships.

[0577] Step 6:

[0578] The server evaluates changes in emotional state and detects anomalies based on the analysis results of the emotion engine. The input is the detailed emotional state from step 5. Based on the analysis results, the anomaly detection algorithm is executed. The output is whether or not an anomaly was detected and its detailed information. Specifically, it compares the result to a baseline value to determine the likelihood of an anomaly.

[0579] Step 7:

[0580] The server generates a notification signal when an anomaly is detected and sends this signal to the training staff. The input contains the anomaly information obtained in step 6. The notification system generates a signal as an alert message. The output is the notification sent to the training staff. Specifically, it generates message content according to the urgency and sends it through the appropriate channel.

[0581] (Application Example 2)

[0582] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0583] In today's brick-and-mortar retail environment, understanding customers' emotional states in real time and responding appropriately is crucial for improving customer satisfaction and increasing sales. However, traditional systems fail to fully utilize customer feedback, making rapid responses difficult. Furthermore, the low accuracy of emotion analysis limits the available response methods.

[0584] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0585] In this invention, the server includes means for digitizing information provided by a user in an information processing device, means for analyzing the digitized information and estimating the emotional state, and means for digitizing user feedback information in a physical store environment, analyzing it to estimate the emotional state, and providing appropriate response methods. This makes it possible to quickly and accurately grasp the emotional state of customers in physical stores and provide effective support by specialist staff.

[0586] An "information processing device" is an electronic device used for collecting, analyzing, digitizing, and managing data.

[0587] "Digitalization" is the process of converting information in analog format into a digital format that can be processed by electronic devices.

[0588] "Emotional state" refers to the user's mental state and emotional condition, and is estimated using natural language processing technology.

[0589] "Analysis" is the process of performing computational processing on digitized information to interpret and evaluate its content.

[0590] An "abnormality" refers to an emotional or behavioral state that deviates from the normal range, and is detected based on the system's judgment.

[0591] A "notification signal" refers to a notification or warning sent to the relevant personnel when an anomaly is detected.

[0592] A "physical store environment" refers to a physical business location where customers visit in person to receive services or products.

[0593] "Feedback information" refers to text or audio recordings of opinions, impressions, questions, etc., provided by users.

[0594] "Response methods" refer to specific action guidelines and solutions provided based on the user's emotional state and feedback.

[0595] "Specialized staff" refers to employees who have received special training to handle customer service and provide product information.

[0596] The system for implementing this invention is centered around a server acting as an information processing device. The server receives customer feedback information transmitted from terminals installed in a physical store environment and performs the necessary processing. The terminals are equipped with a scanner and optical character recognition (OCR) software to scan handwritten notes and feedback forms provided by customers and convert them into digital data. For this OCR technology, for example, the Google Cloud Vision API can be used.

[0597] The server uses natural language processing (NLP) techniques to analyze the customer's emotional state based on the received digitized data. Examples of NLP engines that can be used include spaCy and NLTK. Subsequently, an emotion engine is used for deeper analysis to detect detailed emotional states and abnormal emotional patterns. Based on these analysis results, the server considers appropriate responses to the customer's emotions and sends a notification signal to the specialist staff at the physical store.

[0598] As a concrete example, suppose a customer writes in a feedback form, "I'm interested in the new product, but I'm unsure which to choose." This handwritten comment is scanned by a terminal in the store and converted into text via OCR. This information is sent to a server, where NLP and an emotion engine analyze it to detect emotions such as "hesitation" or "anxiety." Based on this information, the server can send a notification to the store staff such as, "The customer is feeling unsure about their product choice. A detailed explanation from a specialist is recommended."

[0599] Examples of prompt messages include the following:

[0600] "Analyze the feedback content and estimate the customer's emotional state. Based on this, propose appropriate responses to the store staff."

[0601] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0602] Step 1:

[0603] The terminal scans handwritten feedback information provided by the user in the store. The input is the handwritten feedback, and the output is scanned image data. This image data is then passed to OCR software.

[0604] Step 2:

[0605] The device extracts text data from image data using OCR (Optical Character Recognition) technology. The input is a scanned image, and the output is text data. At this stage, handwritten characters are analyzed using OCR technology (e.g., Google Cloud Vision API) and converted into digital text.

[0606] Step 3:

[0607] The terminal sends text data to the server. The input is digitized text data, and the output is a transmission completion notification. The server prepares to receive this text data.

[0608] Step 4:

[0609] The server performs natural language processing (NLP) based on the received text data. The input is text data, and the output is analyzed base sentiment information. An NLP engine (e.g., spaCy, NLTK) is used to analyze the text content and estimate the basic sentiment state.

[0610] Step 5:

[0611] The server uses an emotion engine to perform more detailed emotional analysis. The input is basic emotional information from NLP, and the output is a detailed emotional pattern. The emotion engine analyzes the user's emotional state in detail and detects abnormal emotions.

[0612] Step 6:

[0613] The server generates notification signals for in-store specialists based on the sentiment analysis results. The input is a detailed sentiment pattern, and the output is a notification signal. This notification is configured to include recommended customer service actions.

[0614] Step 7:

[0615] The server sends a notification signal to the store staff's terminal. The input is the notification signal that was created, and the output is a notification that the signal has been sent. The store staff checks this notification and prepares to provide appropriate customer service.

[0616] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0617] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0618] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0619] [Fourth Embodiment]

[0620] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0621] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0622] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0623] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0624] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0625] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0626] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0627] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0628] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0629] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0630] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0631] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0632] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0633] This invention provides a system for continuously monitoring the mental state of learners in educational settings and detecting problems early. Learners keep notes or diaries during their daily learning activities, and the contents are digitized by an information processing device.

[0634] 1. Data Collection

[0635] Notes taken by users (learners) during class are photographed using a scanner or camera-equipped device. This digitizes the handwritten content and extracts it as text data.

[0636] 2. Data Analysis

[0637] The device transmits the digitized notebook information to the server. The server analyzes this data using natural language processing techniques to estimate the emotional state. The emotional state is inferred from the keywords and phrases written by the learner in the notebook content.

[0638] 3. Anomaly detection

[0639] The server compares the analyzed emotional data with historically accumulated data to track changes in emotions. If any abnormalities indicating increased stress or anxiety are detected during this process, a notification is sent to the educator.

[0640] 4. Notification and Response

[0641] If the server detects an anomaly, it immediately sends a notification signal to the educator. This signal includes the name of the learner who detected the anomaly, as well as detailed information about the detected increase in stress or emotion. Recommended countermeasures may also be provided at the same time.

[0642] 5. Specific Examples

[0643] If a student writes in their diary, "Recent lessons are difficult, and I can't keep up," the server analyzes the text and determines that the student is experiencing stress. If the stress level exceeds a set threshold, a notification is sent to the instructor, allowing for prompt action.

[0644] This system allows for the rapid detection and response to changes in learners' emotions, thereby maintaining their mental health and preventing problems such as bullying and school absenteeism.

[0645] The following describes the processing flow.

[0646] Step 1:

[0647] Users (learners) take notes or write diaries during class and save them as digital images using the device's camera or scanner. The device uses optical character recognition (OCR) technology to convert this image data into text data.

[0648] Step 2:

[0649] The terminal transfers text data and digital image data to the server via the network. This data includes all handwritten input from the learner.

[0650] Step 3:

[0651] The server analyzes the received text data using natural language processing (NLP) techniques. This extracts keywords and phrases related to emotions and stress from the recorded content and estimates the emotional state.

[0652] Step 4:

[0653] The server uses image recognition technology to evaluate the irregularities and patterns in handwritten characters. If irregularities in characters occur frequently, this information is used as an indicator of stress or emotional instability.

[0654] Step 5:

[0655] The server compares the analyzed emotional state with existing stored data over time to understand changes in emotional state. In this process, any emotional fluctuations that exceed a certain threshold are detected as abnormal.

[0656] Step 6:

[0657] When the server detects an anomaly, it automatically sends a notification signal to the educator. This notification includes identification of the learner in question, details of the emotional fluctuations, and recommendations for action.

[0658] Step 7:

[0659] Educators receive notifications and take appropriate action or conduct interviews with learners. Users (learners) receive feedback and can get support as needed. This enables early detection and resolution of problems.

[0660] (Example 1)

[0661] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0662] In educational settings, there is a challenge in appropriately monitoring changes in learners' mental states and detecting problems early to take countermeasures. In particular, the lack of means to grasp learners' emotional states in detail increases the risk of overlooking signs of stress and anxiety. Furthermore, immediate notification is necessary when an anomaly is detected so that educators can respond quickly.

[0663] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0664] In this invention, the server includes means for converting handwritten information into digital data using information acquired by the learner with an input device, means for transmitting the digital data to a processing device, and means for the processing device to analyze the digital data using natural language processing technology and estimate the learner's emotional state. This enables rapid and accurate detection of changes in the learner's emotional state, allowing educators to take appropriate action.

[0665] An "input device" is a device used by learners to convert their handwritten information into digital data, and includes equipment such as scanners and cameras.

[0666] "Handwritten information" refers to information such as characters and drawings that learners have written by hand on paper or other media, and is the type of information that is to be converted into digital data.

[0667] "Digital data" refers to data that has been converted from handwritten information using an input device and then processed in a format that can be used by a computer.

[0668] A "processing device" is a device used to analyze and process digital data, and includes equipment such as servers and computers.

[0669] "Natural language processing technology" is a technique that analyzes digitized text to understand the emotions and intentions contained within it, and it utilizes machine learning and artificial intelligence algorithms.

[0670] "Emotional state" refers to the learner's psychological state inferred from the text contained in digital data, and includes classifications such as positive, negative, and neutral.

[0671] "Anomaly detection" is a process that identifies unusual changes or trends based on the results of an analysis of emotional states, and serves as a criterion for notifying educators of the results.

[0672] An "educational administrator" is a person who receives notification when an anomaly is detected and takes measures to improve the learner's condition, and this includes teachers, counselors, and others.

[0673] This system is primarily intended to monitor the mental state of learners in educational environments and to detect abnormalities early. The following hardware and software are used to implement this invention.

[0674] First, the user (learner) acquires handwritten information, such as notes or diaries, using an input device. Scanners or cameras are used as input devices. This converts the handwritten information into digital data.

[0675] Next, the terminal plays the role of transmitting the acquired digital data to the server. A dedicated application is installed on the terminal, and this application automatically manages the data transmission. The data is securely transferred to the server via the internet.

[0676] The server processes the received digital data for analysis. Specifically, it uses natural language processing technology running on the server to estimate emotional states from text written by learners. This analysis utilizes sentiment dictionaries and machine learning models. The server detects anomalies by comparing them with past data and notifies educational administrators as needed.

[0677] As a concrete example, consider a case where a learner writes in their diary, "I have a lot of assignments every day, and I feel pressured." The server, based on this text, determines that the learner is experiencing stress, and if the stress level exceeds a certain threshold, a notification is sent to the education administrator. This notification may include specific details about the problem and recommended coping strategies.

[0678] Furthermore, as an example of a prompt using a generative AI model, the model can be given instructions such as, "Please input the contents of a student's diary and provide the emotions and stress indicators you perceive from the text." Such prompts enable the generative AI model to perform appropriate emotion analysis and respond quickly.

[0679] This system effectively supports learners' mental health and enables early detection and response to problems.

[0680] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0681] Step 1:

[0682] The user (learner) takes notes or diary entries during class and inputs them into an input device. The input device used is either a scanner or a terminal with a camera. The input is handwritten information, and the output is that information converted into a digital image. Specifically, the user takes a picture of a page of their notebook using the terminal, and the image file is saved to the terminal.

[0683] Step 2:

[0684] The device converts captured digital images into text data. This process utilizes OCR (Optical Character Recognition) software. The input is a digital image, and the output is text data extracted from that image. Specifically, the device automatically performs OCR, reads the characters contained in the image as text, and saves them to a text file.

[0685] Step 3:

[0686] The terminal sends the generated text data to the server. The input is the text data obtained in the previous step, and the output is the uploading of the text data to the server. Specifically, a dedicated application on the terminal is launched, and the text data is uploaded to the server using a secure protocol.

[0687] Step 4:

[0688] The server performs natural language processing to analyze the received text data. The input is text data sent from the terminal, and the output is the learner's estimated emotional state. Specifically, a natural language processing algorithm on the server analyzes the text data, and a generative AI model is used to determine emotions and psychological states.

[0689] Step 5:

[0690] The server detects anomalies based on the analysis results. Inputs include the current emotional state determined by text analysis and past data, while output is the result of the anomaly detection. Specifically, the server compares the current emotional state with a set baseline value to identify emotional deviations or abnormal changes.

[0691] Step 6:

[0692] The server performs the action of notifying the education administrator of the results of anomaly detection. The inputs are the anomaly detection results and learner information, and the output is a notification to the education administrator. Specifically, the server provides the education administrator with details of the anomaly and recommended countermeasures using email or a notification application.

[0693] (Application Example 1)

[0694] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0695] There is a need to provide an environment that effectively utilizes the time learners spend in autonomous vehicles and promotes learning while appropriately monitoring their mental state. However, conventional educational systems have difficulty monitoring learners' emotional states in real time and taking appropriate action. This presents a challenge in early detection of learners' anxiety and stress and taking swift countermeasures.

[0696] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0697] In this invention, the server includes processing means for converting information entered by the user into a digital format, processing means for analyzing the digitized information and inferring the emotional state, and processing means for sending a warning signal to an educational support worker when an abnormality is detected. This enables real-time monitoring of the learner's mental state inside an autonomous vehicle and allows for a rapid response when an abnormality is detected.

[0698] "Users" refer to individuals who utilize the learning environment within an autonomous vehicle, and whose learning progress and mental state are subject to monitoring.

[0699] "Digital format" refers to a format in which handwritten or printed information is converted into electronic information that can be processed by a computer.

[0700] "Processing means" refers to a device or program for handling, analyzing, transforming, or transmitting information, and is a technical means for performing operations necessary to solve a problem.

[0701] "Emotional state" refers to information that indicates the user's psychological or mental state, and particularly includes stress, anxiety, and satisfaction during learning activities.

[0702] A "warning signal" is a notification sent to educational support staff or others to draw their attention when a deviation from the expected normal state is detected.

[0703] An "educational supporter" is a person whose role is to monitor learners' learning and mental state, and to provide guidance or support as needed.

[0704] An "autonomous vehicle" refers to a vehicle that operates autonomously using artificial intelligence and sensor technology, minimizing human intervention during operation.

[0705] This invention is a system that provides a safe and secure learning environment for learners inside autonomous vehicles. The system continuously monitors the learner's mental state and detects any abnormalities early, thereby providing optimal support.

[0706] The server receives information transmitted from a device used by the learner, such as a tablet or smart glasses, and uses OCR technology to digitize that information. Specifically, it uses the Google Cloud Vision API or similar to convert handwritten information into electronic data. The server then analyzes this electronic data and estimates the learner's emotional state using the natural language processing library NLTK and an AI model that enables sentiment analysis (e.g., using TensorFlow).

[0707] When fluctuations in emotional state are detected, the server uses time-series data to identify anomalies. Once an anomaly is confirmed, it utilizes email APIs and notification systems to quickly send warning signals to educational support staff. This process enables appropriate support for learners and improves the learning environment.

[0708] For example, if a learner writes "Today's exam was very difficult" on an electronic tablet while in an autonomous vehicle, the server analyzes this statement and detects a potential stress level. An AI model then evaluates the situation, and if the learner's stress level exceeds a certain threshold, it notifies the educational support staff.

[0709] An example of a prompt is: "Evaluate the situation based on the note data and sentiment analysis results written by the student inside the autonomous vehicle, and propose learning support measures. This time, create a prompt to support a student who is experiencing stress from the assignment." In this way, the system has a function to support the mental health of learners.

[0710] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0711] Step 1:

[0712] The user inputs text and drawings during learning using a tablet or smart glasses. Image data is obtained by taking a picture of the handwritten notes with the device's built-in camera. In this step, the input is handwritten notes, and the output is image data.

[0713] Step 2:

[0714] The device converts the acquired image data into text data using OCR technology. Specifically, it uses the Google Cloud Vision API to recognize characters in the image and extracts them as digital text. The input for this step is image data, and the output is text data.

[0715] Step 3:

[0716] The device sends text data to a cloud server. The server analyzes the received text data using the natural language processing library NLTK and performs sentiment analysis. In this step, the emotional state is estimated based on keywords and phrases in the text data. The input is text data, and the output is estimated sentiment data.

[0717] Step 4:

[0718] The server compares estimated emotion data with historically stored data. Based on the dataset of past emotional states, it detects anomalies in the current emotional state. Specifically, it uses an AI model (e.g., TensorFlow) to evaluate emotional fluctuations. The inputs are emotion data and historical data, and the output is whether or not an anomaly is present.

[0719] Step 5:

[0720] If an anomaly is detected, the server sends a warning signal to the educational support staff. This process utilizes an email API or notification system, providing specific emotional states and recommended actions as warnings. The input is the result of the anomaly detection, and the output is the warning signal.

[0721] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0722] This invention relates to a system for accurately understanding the mental state of learners in educational settings and detecting abnormalities early. It is particularly characterized by its incorporation of an emotion engine to recognize learners' emotions. Learners keep notes and diaries during their daily learning activities, and these are digitized by an information processing device.

[0723] 1. Information gathering and digitization

[0724] Users (learners) take notes during class and scan those notes using a device. This scanning process digitizes the handwritten content, which is then converted into text data using optical character recognition (OCR) technology.

[0725] 2. Data transmission and initial analysis

[0726] The device sends digitized text and image data to the server. The server analyzes the received data using natural language processing (NLP) to estimate the user's basic emotional state.

[0727] 3. Reinforcement estimation using an emotion engine

[0728] The server utilizes an emotion engine to further enhance the estimated emotional state. The emotion engine detects more refined emotional patterns and recognizes subtle emotional nuances that cannot be captured by conventional natural language processing techniques.

[0729] 4. Anomaly detection and notification coordination

[0730] The server utilizes information obtained from the emotion engine to detect an anomaly when the emotional state deviates significantly from the normal range. Based on the output of the emotion engine, the server adjusts the content of the notification signal when an anomaly occurs and notifies the training staff, including the urgency and recommended countermeasures.

[0731] 5. Specific Examples

[0732] Suppose a learner writes in their notebook, "I'm very confused and don't know what to do next." The server's natural language processing system references the keyword "confused," and the emotion engine further estimates the level of stress and anxiety underlying the statement. As a result, it determines that the stress level exceeds a certain threshold and sends a notification to the educator stating, "The learner is experiencing a high level of stress. Please consider a support session."

[0733] This system enables faster and more accurate detection of changes in learners' emotions, facilitating early problem resolution.

[0734] The following describes the processing flow.

[0735] Step 1:

[0736] The user (learner) takes notes or writes a diary during class and captures them as digital images using the device's camera or scanner. The device then uses character recognition technology to convert these images into text data.

[0737] Step 2:

[0738] The terminal sends the generated text and image data to the server. During this process, it verifies that the data is properly formatted and adds any necessary metadata.

[0739] Step 3:

[0740] The server analyzes the received text data using natural language processing techniques to estimate basic emotional states. This analysis classifies emotions based on specific keywords or phrases.

[0741] Step 4:

[0742] The server activates the emotion engine to perform further estimation of emotional states. The emotion engine detects subtle emotional nuances from the text written by the learner and reinforces the intensity and type of emotion.

[0743] Step 5:

[0744] Based on the analysis results, the server compares the learner's emotional changes with past data to detect anomalies. If an anomaly is found, it analyzes its nature and evaluates its severity and scope of impact.

[0745] Step 6:

[0746] If an anomaly is detected, the server sends a notification signal to the educator. This signal includes learner identification information, a specific description of the emotional anomaly, and recommended actions. The notification content is adjusted based on the output of the emotion engine.

[0747] Step 7:

[0748] Instructors receive notifications from the server, provide feedback to learners as needed, and conduct counseling and support activities. Users (learners) can resolve problems through the support provided by instructors.

[0749] (Example 2)

[0750] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0751] In educational settings, accurately understanding learners' mental states and detecting abnormalities early is crucial for providing appropriate support to individual learners. Traditional methods struggle to quickly and accurately detect emotional problems faced by learners, potentially delaying appropriate intervention. Therefore, a system is needed that can more accurately assess learners' emotional states and immediately detect abnormalities.

[0752] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0753] In this invention, the server includes means for digitizing information created by learners, means for estimating basic emotional states using natural language processing technology, means for analyzing subtle emotional nuances using an emotion engine, and means for detecting anomalies and transmitting notification signals. This makes it possible to quickly and accurately detect subtle changes in the learner's emotional state and to immediately provide appropriate educational interventions.

[0754] An "information processing device" is a device equipped with the function of analyzing digitized information and performing specific processing.

[0755] "Digitalization" is the process of converting analog data into an electronically processable format.

[0756] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0757] An "emotion engine" is a technology that analyzes human emotions and recognizes subtle emotional nuances.

[0758] "Anomaly detection" is the process of identifying conditions that deviate from the normal range.

[0759] A "notification signal" is a notification or alert that is sent when an anomaly is detected.

[0760] This invention is a system for accurately understanding the emotional state of learners in educational settings and detecting abnormalities early. The system is centered around an information processing device and is composed of the following hardware and software components.

[0761] Users (learners) record information in paper or digital notebooks during their daily learning activities. The recorded content is scanned by a device with a dedicated application installed and saved as a digital image. This device includes optical character recognition (OCR) software, which converts the image data into text data.

[0762] The converted text data is transmitted to the server via wireless communication. The server analyzes the text data using natural language processing (NLP) techniques to estimate the learner's basic emotional state. Furthermore, it utilizes an emotion engine to extract subtle emotional nuances from this basic emotional state. This process enables a deeper understanding of emotions compared to conventional natural language processing.

[0763] The server monitors changes in learners' emotional states based on emotional information obtained from the emotion engine, and generates a notification signal when an anomaly is detected. This signal is configured to be automatically sent to the educator depending on the type and urgency of the anomaly, allowing the educator to take prompt action.

[0764] As a concrete example, consider a case where a learner writes in their notebook, "I'm very confused and don't know what the next step is." The device scans this information and converts it into text data using OCR. Then, the server's natural language processing and sentiment engine estimates that the learner is in a highly confused state. It then sends a notification to the educator stating, "The learner is at a high stress level. Please consider a support meeting."

[0765] An example of a prompt for a generative AI model is: "Using the emotion engine, estimate a specific emotion from the sentence written in the note: 'I'm very confused and don't know what to do next.'"

[0766] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0767] Step 1:

[0768] Users write information in paper or digital notebooks during learning activities. If this information is handwritten, the device scans the notebook using a dedicated application. The handwritten notebook is provided as input. Specifically, an image of the notebook is taken using a camera and digitized. The scanned digital image is generated as output.

[0769] Step 2:

[0770] The terminal extracts text data from scanned digital images using optical character recognition (OCR) technology. The input is the digital image generated in step 1. The OCR software analyzes the characters in the image and outputs the character information as text data. Specifically, it identifies each character and converts it to its corresponding character code.

[0771] Step 3:

[0772] The terminal transmits text data generated by OCR to the server via wireless communication. Text data is used as input. The specific operation of sending data to the server is performed using a communication module. The output is the text data received by the server.

[0773] Step 4:

[0774] The server analyzes the received text data using natural language processing (NLP) to estimate the learner's basic emotional state. The input is the text data from step 3. NLP techniques are used to extract emotion-related keywords and phrases. The output is an estimate of the basic emotional state. Specifically, the server analyzes the text context and calculates an emotion vector.

[0775] Step 5:

[0776] The server uses an emotion engine to reinforce the basic emotional states estimated by NLP. The input includes the basic emotional states obtained in step 4. The emotion engine consults a database and analyzes the subtle nuances of the emotions. The output is the detailed emotional state analysis. Specifically, the emotion engine detects multiple emotional patterns and evaluates their relationships.

[0777] Step 6:

[0778] The server evaluates changes in emotional state and detects anomalies based on the analysis results of the emotion engine. The input is the detailed emotional state from step 5. Based on the analysis results, the anomaly detection algorithm is executed. The output is whether or not an anomaly was detected and its detailed information. Specifically, it compares the result to a baseline value to determine the likelihood of an anomaly.

[0779] Step 7:

[0780] The server generates a notification signal when an anomaly is detected and sends this signal to the training staff. The input contains the anomaly information obtained in step 6. The notification system generates a signal as an alert message. The output is the notification sent to the training staff. Specifically, it generates message content according to the urgency and sends it through the appropriate channel.

[0781] (Application Example 2)

[0782] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0783] In today's brick-and-mortar retail environment, understanding customers' emotional states in real time and responding appropriately is crucial for improving customer satisfaction and increasing sales. However, traditional systems fail to fully utilize customer feedback, making rapid responses difficult. Furthermore, the low accuracy of emotion analysis limits the available response methods.

[0784] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0785] In this invention, the server includes means for digitizing information provided by a user in an information processing device, means for analyzing the digitized information and estimating the emotional state, and means for digitizing user feedback information in a physical store environment, analyzing it to estimate the emotional state, and providing appropriate response methods. This makes it possible to quickly and accurately grasp the emotional state of customers in physical stores and provide effective support by specialist staff.

[0786] An "information processing device" is an electronic device used for collecting, analyzing, digitizing, and managing data.

[0787] "Digitalization" is the process of converting information in analog format into a digital format that can be processed by electronic devices.

[0788] "Emotional state" refers to the user's mental state and emotional condition, and is estimated using natural language processing technology.

[0789] "Analysis" is the process of performing computational processing on digitized information to interpret and evaluate its content.

[0790] An "abnormality" refers to an emotional or behavioral state that deviates from the normal range, and is detected based on the system's judgment.

[0791] A "notification signal" refers to a notification or warning sent to the relevant personnel when an anomaly is detected.

[0792] A "physical store environment" refers to a physical business location where customers visit in person to receive services or products.

[0793] "Feedback information" refers to text or audio recordings of opinions, impressions, questions, etc., provided by users.

[0794] "Response methods" refer to specific action guidelines and solutions provided based on the user's emotional state and feedback.

[0795] "Specialized staff" refers to employees who have received special training to handle customer service and provide product information.

[0796] The system for implementing this invention is centered around a server acting as an information processing device. The server receives customer feedback information transmitted from terminals installed in a physical store environment and performs the necessary processing. The terminals are equipped with a scanner and optical character recognition (OCR) software to scan handwritten notes and feedback forms provided by customers and convert them into digital data. For this OCR technology, for example, the Google Cloud Vision API can be used.

[0797] The server uses natural language processing (NLP) techniques to analyze the customer's emotional state based on the received digitized data. Examples of NLP engines that can be used include spaCy and NLTK. Subsequently, an emotion engine is used for deeper analysis to detect detailed emotional states and abnormal emotional patterns. Based on these analysis results, the server considers appropriate responses to the customer's emotions and sends a notification signal to the specialist staff at the physical store.

[0798] As a concrete example, suppose a customer writes in a feedback form, "I'm interested in the new product, but I'm unsure which to choose." This handwritten comment is scanned by a terminal in the store and converted into text via OCR. This information is sent to a server, where NLP and an emotion engine analyze it to detect emotions such as "hesitation" or "anxiety." Based on this information, the server can send a notification to the store staff such as, "The customer is feeling unsure about their product choice. A detailed explanation from a specialist is recommended."

[0799] Examples of prompt messages include the following:

[0800] "Analyze the feedback content and estimate the customer's emotional state. Based on this, propose appropriate responses to the store staff."

[0801] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0802] Step 1:

[0803] The terminal scans handwritten feedback information provided by the user in the store. The input is the handwritten feedback, and the output is scanned image data. This image data is then passed to OCR software.

[0804] Step 2:

[0805] The device extracts text data from image data using OCR (Optical Character Recognition) technology. The input is a scanned image, and the output is text data. At this stage, handwritten characters are analyzed using OCR technology (e.g., Google Cloud Vision API) and converted into digital text.

[0806] Step 3:

[0807] The terminal sends text data to the server. The input is digitized text data, and the output is a transmission completion notification. The server prepares to receive this text data.

[0808] Step 4:

[0809] The server performs natural language processing (NLP) based on the received text data. The input is text data, and the output is analyzed base sentiment information. An NLP engine (e.g., spaCy, NLTK) is used to analyze the text content and estimate the basic sentiment state.

[0810] Step 5:

[0811] The server uses an emotion engine to perform more detailed emotional analysis. The input is basic emotional information from NLP, and the output is a detailed emotional pattern. The emotion engine analyzes the user's emotional state in detail and detects abnormal emotions.

[0812] Step 6:

[0813] The server generates notification signals for in-store specialists based on the sentiment analysis results. The input is a detailed sentiment pattern, and the output is a notification signal. This notification is configured to include recommended customer service actions.

[0814] Step 7:

[0815] The server sends a notification signal to the store staff's terminal. The input is the notification signal that was created, and the output is a notification that the signal has been sent. The store staff checks this notification and prepares to provide appropriate customer service.

[0816] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0817] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0818] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0819] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0820] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0821] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0822] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0823] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0824] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0825] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0826] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0827] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0828] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0829] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0830] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0831] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0832] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0833] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0834] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0835] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0836] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0837] The following is further disclosed regarding the embodiments described above.

[0838] (Claim 1)

[0839] In an information processing device, a means for digitizing information written by a learner,

[0840] A means for analyzing the digitized information and estimating the emotional state,

[0841] A means for analyzing the aforementioned changes in emotional state and detecting abnormalities,

[0842] A means of sending a notification signal to the education officer when an anomaly is detected,

[0843] A system that includes this.

[0844] (Claim 2)

[0845] The system according to claim 1, further comprising means for using natural language processing technology in the aforementioned analysis.

[0846] (Claim 3)

[0847] The system according to claim 1, further comprising means for evaluating the character distortion of the information written by the learner using image recognition technology.

[0848] "Example 1"

[0849] (Claim 1)

[0850] A means of converting handwritten information into digital data using information acquired by learners with an input device,

[0851] means for transmitting the aforementioned digital data to a processing device,

[0852] The processing apparatus includes means for analyzing the digital data using natural language processing technology and estimating the emotional state,

[0853] A means for comparing the estimated emotional state with past data and detecting anomalies,

[0854] A means of notifying the education administrator of the anomaly detection result via communication means,

[0855] A system that includes this.

[0856] (Claim 2)

[0857] The system according to claim 1, comprising means for analyzing the aforementioned digital data and using emotional fluctuations as an indicator.

[0858] (Claim 3)

[0859] The system according to claim 1, further comprising means for evaluating the character shape of information acquired by the learner using an input device with image processing technology.

[0860] "Application Example 1"

[0861] (Claim 1)

[0862] A processing means for converting information entered by a user into a digital format,

[0863] A processing means for analyzing the digitized information and inferring the emotional state,

[0864] A processing means for analyzing fluctuations in the aforementioned emotional state and identifying abnormalities,

[0865] A processing means that transmits a warning signal to an educational support worker when an abnormality is detected,

[0866] A processing means that provides an environment for users to facilitate learning within a vehicle,

[0867] A system that includes this.

[0868] (Claim 2)

[0869] The system according to claim 1, further comprising a processing means that utilizes language analysis technology in the aforementioned analysis.

[0870] (Claim 3)

[0871] The system according to claim 1, further comprising processing means for evaluating the character shape of the information entered by the user using visual recognition technology.

[0872] "Example 2 of combining an emotion engine"

[0873] (Claim 1)

[0874] In an information processing device, a means for digitizing information created by learners,

[0875] A means for analyzing the digitized information using natural language processing technology and estimating basic emotional states,

[0876] A means for analyzing subtle emotional nuances by utilizing an emotional engine to further reinforce the aforementioned basic emotional state,

[0877] A means for analyzing the aforementioned changes in emotional state and detecting abnormalities,

[0878] A means of sending a notification signal to the training staff, including the type of anomaly and its urgency, when an anomaly is detected,

[0879] A system that includes this.

[0880] (Claim 2)

[0881] The system according to claim 1, further comprising means for identifying the intensity and relevance of an emotion based on a specific emotion category using the emotion engine.

[0882] (Claim 3)

[0883] The system according to claim 1, further comprising means for extracting text data from information created by the learner using optical character recognition technology.

[0884] "Application example 2 when combining with an emotional engine"

[0885] (Claim 1)

[0886] In an information processing device, a means of digitizing information provided by the user,

[0887] A means for analyzing the digitized information and estimating the emotional state,

[0888] A means for analyzing the aforementioned changes in emotional state and detecting abnormalities,

[0889] A means of transmitting an alert signal to the relevant personnel when an anomaly is detected,

[0890] A means of digitizing customer feedback information in a physical store environment, analyzing it to estimate emotional states, and providing appropriate response methods.

[0891] A system that includes this.

[0892] (Claim 2)

[0893] The system according to claim 1, comprising means for using natural language processing technology in the analysis, and means for proposing a response method based on the emotional state of the user in the physical store environment.

[0894] (Claim 3)

[0895] The system according to claim 1, further comprising means for evaluating the character distortion of the information entered by the user using image recognition technology, and further comprising means for proposing support by specialist staff at the physical store. [Explanation of Symbols]

[0896] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. In an information processing device, a means for digitizing information written by a learner, A means for analyzing the digitized information and estimating the emotional state, A means for analyzing the aforementioned changes in emotional state and detecting abnormalities, A means of sending a notification signal to the education officer when an anomaly is detected, A system that includes this.

2. The system according to claim 1, further comprising means for using natural language processing technology in the aforementioned analysis.

3. The system according to claim 1, further comprising means for evaluating the character distortion of the information written by the learner using image recognition technology.

Images