system

Abstract

We provide the system. [Solution] It was established with the aim of learning alongside human students, and includes data analysis tools for analyzing student behavior in the classroom, A behavior generation means that generates appropriate educational support behaviors based on analyzed data, A control means that controls a robot student based on generated behavior and allows it to interact with a human student, A feedback collection method that records interaction results and analyzes them to improve the educational environment, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern educational settings, there are serious problems such as a shortage of teachers and class breakdown, which lead to a decline in the quality of education. Also, bullying, insufficient communication among students, and inadequate exploration of the potential abilities of individual students are reasons why improvement of the educational environment is required. It is necessary to effectively solve such educational problems and provide a sustainable educational environment.

Means for Solving the Problems

[0005] This invention solves the above problems by providing a robot student system that can learn alongside human students. This system includes data analysis means for analyzing student behavior in the classroom and behavior generation means for generating appropriate educational support actions according to the situation of each student. It also includes control means for controlling the robot student based on the generated actions and enabling natural interaction with human students. Furthermore, it includes feedback collection means for recording the interaction results and providing feedback for improving the educational environment. This makes it possible to prevent bullying, strengthen class unity, discover students' potential abilities, and reduce the burden on teachers.

[0006] "Data analysis means" refers to a device or system that has the function of analyzing student behavior data acquired in the classroom in real time and understanding the situation and needs of each student.

[0007] A "behavior generation tool" is a system that generates specific action plans and messages to achieve effective educational support and communication based on analyzed data.

[0008] "Control means" refers to a device or system that has the function of operating the robot according to the generated action plan and performing smooth interaction with human students.

[0009] A "feedback collection method" is a system that records the results of interactions between robots and students, and uses this data to improve the educational environment and provide feedback to teachers.

[0010] A "robot student" is an artificial being that can learn and interact with human students, and is controlled by a program designed to support education. [Brief explanation of the drawing]

[0011] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0012] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0013] First, the terms used in the following description will be explained.

[0014] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0015] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0016] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0017] In the following embodiments, a labeled communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0019] [First Embodiment]

[0020] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0021] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0024] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0027] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0031] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0032] To implement this invention, effective operation of robot students within the classroom is required. This system mainly consists of a server and terminals, and each component functions in cooperation with others.

[0033] The server first collects behavioral data from each student in the classroom using various sensors and analyzes it in real time. This includes facial recognition, voice analysis, and tracking of behavioral patterns. Based on the data obtained in this way, it forms a foundation for directing support actions for teachers and robot students.

[0034] Next, the server generates the optimal communication method for a specific student based on the analyzed data. This generative AI plans the conversation content and actions necessary to enhance the effectiveness of education. Specifically, this includes questions and explanations to answer students' questions, and suggestions to encourage cooperation with classmates.

[0035] The terminal controls the robot students based on instructions from the server, enabling interaction with the students. The robot students follow these instructions, allowing for natural dialogue and emotional expression within the classroom. For example, in a math class, they might gently advise a student struggling with a calculation, saying, "Let's try using a basic formula," and then answer further questions in a casual tone.

[0036] Furthermore, through a feedback recording system, teachers are provided with detailed information on the results of interactions after each lesson. Based on this, teachers can easily create instructional plans tailored to each student's situation. This makes it possible to provide education that is appropriate for each student and improve the learning efficiency of the entire class.

[0037] Specific examples of this invention include preventing bullying, deterring classroom disruption, and assisting students who speak different languages. By utilizing the above mechanism, it is expected that a better educational environment will be realized by drawing out the potential of human students while robot students support their learning.

[0038] The following describes the processing flow.

[0039] Step 1:

[0040] The server collects data in real time from sensors such as cameras and microphones installed in the classroom. This allows it to capture each student's facial expressions, voice, and behavioral patterns, and prepare them for analysis.

[0041] Step 2:

[0042] The server uses the collected data to run an AI model that analyzes students' psychological state and learning progress. This analysis determines which students need what kind of support.

[0043] Step 3:

[0044] The server generates individualized student support plans based on the analysis results. For example, for a student working on a math problem, it generates a message suggesting "which formula should you use next?"

[0045] Step 4:

[0046] The server sends the generated instructions and messages to the robot student. These instructions include specific actions and messages to be delivered to the student.

[0047] Step 5:

[0048] The terminal (robot student) follows instructions from the server and performs actions for designated students or the entire class. It can provide explanations via voice, approach specific students and speak to them individually, and engage in other forms of interaction.

[0049] Step 6:

[0050] The user (student) deepens their learning through interaction with the robot student. If the student responds or asks questions during this process, that information is sent back to the server.

[0051] Step 7:

[0052] The server receives feedback after the interaction and evaluates the educational effectiveness. This allows for analysis of what types of instruction were effective, which is then used to improve teaching methods for teachers and schools.

[0053] (Example 1)

[0054] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0055] In traditional educational environments, providing flexible support tailored to the individual needs and circumstances of each student was difficult. Furthermore, a lack of communication, particularly among students with different language backgrounds, hindered learning efficiency. Additionally, the time and effort required for teachers to thoroughly understand each student's learning process and provide individualized instruction based on that understanding created a need for efficient feedback.

[0056] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0057] In this invention, the server includes information analysis means, behavior generation means, and generation means for generating communication methods using a generational AI model. This enables support tailored to the individual needs of learners within an educational environment, facilitates dialogue between learners with different languages, and provides effective feedback to teachers.

[0058] "Information analysis tools" are technologies for collecting and analyzing learners' behavior, statements, and ecological information within an educational environment.

[0059] "Action generation means" refers to technology that determines the optimal learning support action for a learner based on analyzed information and generates the content of that action.

[0060] A "generative AI model" refers to an algorithm that learns from accumulated data and generates communication methods and actions appropriate to new situations.

[0061] "Generative means" refers to technologies that utilize generative AI models to create communication methods tailored to the individual circumstances and needs of learners.

[0062] "Control means" refers to technology that operates automated devices based on generated action plans and enables interaction with human learners.

[0063] "Result collection methods" refer to technologies that collect data necessary to record the results of interactions with learners and provide that data to teachers as feedback.

[0064] This system is designed to provide support tailored to the individual needs of learners in an educational environment. Specifically, it collects and analyzes student behavior data through the cooperation of a server and terminals, and then provides students with optimized teaching activities based on that data through an automated system.

[0065] The server acquires behavioral data in real time through hardware such as facial recognition cameras, voice-collecting microphones, and motion sensors installed within the educational space. The server processes this collected data using information analysis tools to evaluate learners' comprehension and interest. Facial recognition technology can quantify the degree to which students are concentrating on the lesson, and voice analysis can measure the amount of conversation and participation.

[0066] Subsequently, the server uses an action generation mechanism to determine learning support actions based on the analysis results, and then uses a generative AI model to generate appropriate communication content. For example, a prompt such as "What kind of support should be provided if the student cannot answer the question?" is input to the AI, and specific instructions are formulated based on the response obtained.

[0067] The terminal controls automated devices according to instructions received from the server, enabling interaction with human learners. This device functions as a robotic student, guiding learners through natural conversation and instructions. For example, it can prompt a student with a weak understanding of a math problem by saying, "Let's think about what's important here," and then ask additional questions to deepen their understanding.

[0068] The teacher, as a user, uses the feedback provided by the device to evaluate each student's learning progress and develop a teaching plan for the next lesson. This allows the teacher to take an approach that is tailored to each individual student, thereby improving learning efficiency.

[0069] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0070] Step 1:

[0071] The server receives real-time input of student behavior data from facial recognition cameras, audio microphones, and motion sensors placed throughout the classroom. This allows the server to collect data recording student seating arrangements, classroom behavior, and spoken content.

[0072] Step 2:

[0073] The server processes the behavioral data obtained in step 1 using information analysis tools. Specifically, it analyzes the students' level of concentration using facial recognition technology and evaluates the number of times they speak and their pitch through voice analysis. This data is integrated and analyzed to generate an output that quantifies each student's level of interest and understanding.

[0074] Step 3:

[0075] The server uses the analysis results as input to generate optimal learning support actions using an action generation mechanism. Using a generation AI model, it formulates the optimal communication method and instructional content based on prompt sentences. At this stage, prompts such as "How should we approach a student who is having trouble asking a question?" are used. This results in the output of an action plan that includes specific instructions.

[0076] Step 4:

[0077] The terminal controls the automated device using the output of the action plan from the server as input. The robot student prompts and responds to students during class. For example, to a student struggling with a math problem, it might instruct, "Let's go back to the basics and think about this step." This establishes a dialogue and enables the robot to take specific actions to support the student in solving the problem.

[0078] Step 5:

[0079] The teacher, as the user, receives feedback provided from the device. This feedback includes statistics to evaluate the effectiveness of the interaction and progress reports for each learner. Based on this output, the teacher can develop individualized instruction plans as input and optimize the educational content by adjusting students' learning plans.

[0080] (Application Example 1)

[0081] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0082] In the manufacturing sector, there is a need to improve worker efficiency and optimize the work environment. In particular, when introducing new machinery or adapting to different work methods, it is difficult for workers to immediately learn the optimal operating procedures, which can result in decreased productivity and safety issues. It is necessary to overcome these problems and create an efficient and safe work environment.

[0083] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0084] In this invention, the server includes data analysis means for analyzing worker work data, action generation means for generating appropriate work support actions based on the analyzed data, control means for controlling work support machines and interacting with workers based on the generated actions, and feedback collection means for recording interaction results and analyzing them to improve the work environment. This enables real-time work support for workers, improving work efficiency and ensuring safety.

[0085] "Worker" refers to an individual engaged in work related to manufacturing or production.

[0086] "Work data" refers to data collected during work, including information about the actions and movements of workers.

[0087] "Data analysis means" refers to a system that processes collected data and evaluates the work efficiency and performance of workers.

[0088] "Action generation means" refers to a system that determines the optimal method of work support for workers based on analyzed data.

[0089] "Work support machinery" refers to robots and other devices used to assist workers in their tasks.

[0090] "Control means" refers to a system that has the function of operating work assistance machinery based on the generated support actions.

[0091] "Interaction results" refer to data showing workers' responses to the support actions of work assistance machines and the effects of those responses.

[0092] A "feedback collection method" refers to a system that records interaction results and collects data for later analysis.

[0093] To implement this invention, a comprehensive system is needed to support workers in a factory environment. This system includes data analysis means, action generation means, control means, and feedback collection means.

[0094] First, the server uses sensors installed throughout the factory to collect real-time data on the workers' activities. This data includes details such as their work style, speed, and environmental conditions. Based on this information, the data analysis system performs calculations necessary to analyze the efficiency and safety of each worker.

[0095] Next, the action generation system determines the optimal work support action based on the analysis results. This is done using a generative AI model, which plans specific support methods tailored to the worker's abilities and the nature of the work. For example, it can provide step-by-step guidance to a worker unfamiliar with operating new machinery.

[0096] Subsequently, the control system operates the work assistance machinery based on the generated actions and interacts effectively with the worker. This ensures that the guidance the worker receives is applied to the real environment and is reliably useful on the spot.

[0097] Furthermore, feedback collection mechanisms record the results of interactions. This data is used for analysis on the server and helps in the continuous improvement of the work environment. This also enables workers to perform tasks safely and efficiently.

[0098] As a concrete example, a prompt message such as "Do you need support while working with the new equipment?" might be displayed via smartphone to a worker unfamiliar with new operating procedures. This prompt message provides quick support for any challenges the worker may face.

[0099] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0100] Step 1:

[0101] The server utilizes sensors within the factory to collect worker activity data in real time. The input is signals from the sensors, which are converted into digital data. The output is raw data indicating the worker's activity status and environmental conditions. This raw data specifically includes information about each worker's position, movement, and speed.

[0102] Step 2:

[0103] The server processes the collected work data using data analysis tools. The input is the raw data collected in step 1. Here, analysis algorithms are used to detect data trends and anomalies, and to evaluate work efficiency and safety. The output is the analysis results for generating work support actions. This includes performance indicators and required support points for each worker.

[0104] Step 3:

[0105] The action generation mechanism uses the server's analysis results to generate optimal work support actions for each worker. The input is the analysis results from step 2. The generation AI model is used to perform data calculations to create a plan for support actions. The output is a support scenario for guidance at a specific work step or for new operating procedures.

[0106] Step 4:

[0107] The terminal receives the generated support actions and controls the work assistance machine. The input is the support action generated in step 3. It is converted into specific operating instructions and communicated to the machine in a way that is easy for the worker to understand. The output is the specific operation of the machine, which includes new machine operation guidance and warning messages.

[0108] Step 5:

[0109] The feedback collection mechanism allows the terminal to record the results of interactions with the work assistance machine. The input is interaction data between the machine and the worker. This data includes the worker's reactions and the machine's actual operation history. The output is a detailed interaction report sent back to the server. This report is used for further analysis to improve the work environment.

[0110] Step 6:

[0111] Users improve their work environment and operating procedures based on the feedback they receive. The input is interaction reports received from the server. The data is used to develop improvement strategies to clearly identify areas for improvement. The output is the improved work procedures and environment settings.

[0112] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0113] This invention relates to a robotic student system that functions effectively in a classroom and aims to recognize user emotions and provide appropriate educational support by combining it with an emotion engine. The system consists of a server and robotic terminals, and each component operates in conjunction with the others.

[0114] The server first collects facial expressions and voice data from sensor devices installed in the classroom. The collected data is analyzed by the system's emotion engine to understand students' emotional states and psychological needs in real time. This analysis provides information such as whether a particular student is experiencing anxiety or showing interest in the lesson.

[0115] Next, the server uses the information obtained from the emotion engine to formulate educational support methods using behavior generation mechanisms. For example, for a student who is feeling anxious, it generates a plan in which a robot student offers words of encouragement in a gentle voice. These plans are designed to be flexible enough to accommodate students who require support in different languages.

[0116] The robot student, acting as a terminal, receives instructions from the server and then performs the designated actions within the classroom. For example, if a student appears depressed, the robot student will approach that student and ask, "Is there anything I can do to help?", providing support through friendly interaction.

[0117] Teachers and other students who are users of these robot students can improve the learning environment for the entire class with their support. In particular, the emotion recognition-based feedback function allows teachers to understand the emotional state of each student and use that information to devise more appropriate teaching methods.

[0118] As a concrete example, consider a scenario in a language class where a student with a different cultural background experiences language barriers and feels isolated. The emotion engine analyzes the student's anxious facial expressions. Based on this, the robot student can provide support in the student's native language and create opportunities for interaction with other students.

[0119] Thus, the robot student system of the present invention achieves effects such as improving interpersonal relationships in the classroom, preventing bullying, and deterring classroom disruption, as well as drawing out potential abilities.

[0120] The following describes the processing flow.

[0121] Step 1:

[0122] The server continuously collects students' facial expressions and voice data from sensor devices such as cameras and microphones installed in the classroom. The collected data is temporarily stored and prepared for analysis.

[0123] Step 2:

[0124] The server uses an emotion engine to analyze the collected data. It estimates students' emotional states from their facial expressions and voice tone, and monitors changes in anxiety, excitement, and concentration levels in real time.

[0125] Step 3:

[0126] The server generates educational support behaviors tailored to individual students based on emotional data estimated by the emotion engine. Specifically, it creates support plans that include encouraging words in a gentle voice and questions that engage the student's interest.

[0127] Step 4:

[0128] The server sends the generated educational support actions as instructions to the robot student. These instructions include specific conversation topics with the student and guidelines for action.

[0129] Step 5:

[0130] The terminal (robot student) moves around the classroom based on instructions received from the server and makes contact with the target student. The robot student speaks to the student using designated phrases and gestures and provides follow-up.

[0131] Step 6:

[0132] The user (student) interacts with the robot student after receiving contact from it. The student's responses and new emotional states are recorded as additional data.

[0133] Step 7:

[0134] The server evaluates the interaction between robot students and human students and generates feedback from the recorded data to help improve the educational environment and methods. Users (teachers) can then use this feedback to develop more appropriate teaching strategies.

[0135] (Example 2)

[0136] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0137] Traditional educational support systems have been insufficient in real-time emotional recognition and providing appropriate feedback to individual learners, making it difficult to flexibly accommodate learners with diverse linguistic and cultural backgrounds. As a result, there is a challenge in that learners' psychological needs cannot be met, hindering the improvement of learning effectiveness.

[0138] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0139] In this invention, the server includes information analysis means, action planning means, and emotion analysis means. This makes it possible to grasp the emotional state of human learners in a learning space in real time and provide support tailored to their individual educational needs.

[0140] An "information analysis tool" is a device that collects and analyzes learners' behavior, facial expressions, and voice data to identify their emotional state and psychological needs in real time.

[0141] An "action planning tool" is a device that generates a plan to design appropriate educational activities and effectively intervene in the learning environment based on analyzed information.

[0142] A "control device" is a device that operates a machine learning device based on a generated action plan and interacts with the learner.

[0143] A "feedback acquisition method" is a device that records the results of conversations and analyzes the collected information in order to improve the learning environment.

[0144] An "emotion analysis device" is a device that analyzes a learner's facial expressions and voice data to determine their psychological state in real time.

[0145] A "generative AI model" is a model that uses machine learning algorithms to flexibly adjust action plans, and is a device that supports decision-making within a system.

[0146] This system is an educational support system that uses machine learning devices to function effectively within a learning environment. The server collects learners' facial expressions and voice data through sensor devices installed in the classroom. This data is analyzed in real time by emotion analysis capabilities on the server to determine the learners' emotional state and psychological needs. The hardware used includes sensor devices such as cameras and recording devices like microphones, and the server requires powerful computing resources to process the data from these devices.

[0147] Based on the information obtained through emotion analysis, the server generates individual support plans using an action planning mechanism. The action planning mechanism uses a generative AI model to design optimal educational support from the analysis results. For example, if emotion analysis determines that a learner is experiencing stress, the generative AI model will suggest ways to talk to that learner to help them relax. An example of a prompt in this context would be, "Learner A appears to be experiencing stress. What should I say to them?"

[0148] The generated action plan is sent to the robot student, which acts as a terminal, and implemented in the classroom as an actual educational support activity. The robot student follows the instructions received and provides gentle encouragement and specific support to the learners. These activities are carried out through control mechanisms, which play a role in improving the quality of interaction with learners.

[0149] Furthermore, the user, the teacher, records the results of these interactions using feedback mechanisms and utilizes them to improve the learning environment. As described above, this system has a mechanism to provide more appropriate and flexible educational support through the use of sentiment analysis and generative AI models.

[0150] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0151] Step 1:

[0152] The server collects learners' facial expressions and voice data from sensor devices installed in the classroom. Inputs are camera and microphone data, and this information is aggregated on the server as output. This data collection enables real-time emotion analysis. Specifically, the server acquires this data at regular intervals and stores it in a buffer.

[0153] Step 2:

[0154] The server passes the collected facial and audio data to the emotion analysis system. The input is the student's facial and audio data collected in step 1, and the output is the analysis result identifying the student's emotional state. The server uses machine learning algorithms to analyze the data and determine emotions such as "anxiety," "interest," and "confusion" in real time. Specific operations include data formatting and feature extraction during the analysis.

[0155] Step 3:

[0156] The server generates a support plan via an action planning mechanism based on the results of the emotion analysis. The input is the emotion analysis results from step 2, and the output is a specific educational support plan. This plan is formulated using a generative AI model and is output in the form of, for example, "Learner B is feeling anxious, so offer words of encouragement." The specific actions here are inputting prompt sentences to the generative AI model and generating support suggestions.

[0157] Step 4:

[0158] The server sends the generated support plan to the robot terminal. The input is the support plan generated in step 3, and the output is the instructions to the robot terminal. The server sends the instructions based on the plan to the robot wirelessly and prepares the robot to execute them at the appropriate time. Specific actions include communication execution and serialization of instruction data.

[0159] Step 5:

[0160] The robot, acting as the terminal, performs interactions within the classroom based on instructions received from the server. The input is instruction data from the server, and the output is direct dialogue with the learner. Specifically, the robot might approach and say, "Let's solve this problem together."

[0161] Step 6:

[0162] The user, the teacher, observes the interaction between the robot and the learners and records the results using feedback mechanisms. The input is classroom events and learners' responses, and the output is recorded feedback data. This data will be used to improve future educational support. Specific actions include inputting data into record sheets or digital forms.

[0163] (Application Example 2)

[0164] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0165] In today's work environment, the mental and physical health of workers is directly linked to work efficiency and safety. However, many factories and workplaces lack mechanisms to properly monitor workers' emotional states and stress levels and provide support based on that information. This situation can lead to decreased worker productivity and safety problems. Therefore, there is a need for a system that can improve the work environment by understanding workers' emotional states in real time and providing optimal support.

[0166] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes data analysis means for analyzing the emotional state of the worker, action generation means for generating appropriate support actions based on the analyzed emotional state, and control means for controlling the support device and interacting with the worker based on the generated actions. As a result, the support device in the workplace can analyze the emotional state of the worker in real time and provide appropriate verbal encouragement and advice, thereby improving the work environment.

[0167] A "data analysis device" is a device equipped with the function of collecting and analyzing information necessary to understand the emotional state of workers.

[0168] A "behavior generation device" is a device that has the function of designing appropriate support actions for workers based on analyzed data.

[0169] A "control means" is a device that operates a support device according to the generated support action and has the function of interacting with the worker.

[0170] A "feedback collection device" is a device that records the results of interactions with support devices and uses them for analysis to further improve the work environment.

[0171] A "support device" is a device that can concretize and actually implement support actions based on the emotional state of the worker.

[0172] In the system for realizing this invention, a server plays a central role. The server acquires facial expressions and voice data of workers from various sensors installed in the work environment. Hardware such as cameras and microphones are used for this purpose. The acquired data is analyzed in real time using emotion analysis software installed on the server. The emotion analysis software can, for example, utilize a Python®-based emotion analysis library.

[0173] Once the emotional state of the user (worker) is analyzed, the server uses an action generation algorithm to formulate appropriate support actions. This algorithm designs break suggestions and work-related advice based on the worker's stress and fatigue levels. Communication in different languages ​​is also possible, utilizing a multilingual speech generation engine.

[0174] Subsequently, the server operates the support device via a control system, interacting with the worker. The control system includes actuators and speakers, providing voice output for natural dialogue. The results of the interactions performed by the support device are recorded by a feedback collection system and analyzed for further optimization of the work environment.

[0175] As a concrete example, considering its use in a manufacturing plant, if a sensor detects worker fatigue, the server can provide support such as, "You seem tired. Would you like to take a short break?" An example of a prompt message used in this case would be, "We analyze the worker's emotional state in real time and provide appropriate break suggestions if stress or fatigue is detected. This is supported in Japanese or other languages ​​as needed."

[0176] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0177] Step 1:

[0178] The server captures the worker's facial expressions and voice using cameras and microphones installed in the work environment. The input is raw data obtained through the cameras and microphones, which is then sent to the server. The output consists of analyzable image and audio data.

[0179] Step 2:

[0180] The server uses an emotion analysis library to analyze the captured image and audio data. Here, facial features are extracted from the image data, and voice tone and speed are analyzed from the audio data. The input is the data from step 1, and the output is an evaluation score indicating the worker's emotional state. This process quantifies, for example, levels of stress and fatigue.

[0181] Step 3:

[0182] The server executes an action generation algorithm based on the evaluation score. This algorithm uses predefined rules and machine learning models to formulate the optimal support actions to provide to the worker. The input is the evaluation score obtained in step 2, and the output is a specific action plan such as "suggestion for a break" or "advice to improve work efficiency."

[0183] Step 4:

[0184] The server instructs the support device, via a control mechanism, on the action plan formulated by the action generation algorithm. The support device uses a speaker to perform voice actions and attempts to engage in natural dialogue with the worker. The input is the action plan from step 3, and the output is the voice action.

[0185] Step 5:

[0186] The terminal records the results of the interaction in the feedback collection system. This recorded data is then analyzed by the server for future system improvements. The input is the details of the actual interaction by the support device, and the output is analyzable feedback data.

[0187] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0188] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0189] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0190] [Second Embodiment]

[0191] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0192] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0193] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0194] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0195] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0196] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0197] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0198] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0199] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0200] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0201] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0202] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0203] To implement this invention, effective operation of robot students within the classroom is required. This system mainly consists of a server and terminals, and each component functions in cooperation with others.

[0204] The server first collects behavioral data from each student in the classroom using various sensors and analyzes it in real time. This includes facial recognition, voice analysis, and tracking of behavioral patterns. Based on the data obtained in this way, it forms a foundation for directing support actions for teachers and robot students.

[0205] Next, the server generates the optimal communication method for a specific student based on the analyzed data. This generative AI plans the conversation content and actions necessary to enhance the effectiveness of education. Specifically, this includes questions and explanations to answer students' questions, and suggestions to encourage cooperation with classmates.

[0206] The terminal controls the robot students based on instructions from the server, enabling interaction with the students. The robot students follow these instructions, allowing for natural dialogue and emotional expression within the classroom. For example, in a math class, they might gently advise a student struggling with a calculation, saying, "Let's try using a basic formula," and then answer further questions in a casual tone.

[0207] Furthermore, through a feedback recording system, teachers are provided with detailed information on the results of interactions after each lesson. Based on this, teachers can easily create instructional plans tailored to each student's situation. This makes it possible to provide education that is appropriate for each student and improve the learning efficiency of the entire class.

[0208] Specific examples of this invention include preventing bullying, deterring classroom disruption, and assisting students who speak different languages. By utilizing the above mechanism, it is expected that a better educational environment will be realized by drawing out the potential of human students while robot students support their learning.

[0209] The following describes the processing flow.

[0210] Step 1:

[0211] The server collects data in real time from sensors such as cameras and microphones installed in the classroom. This allows it to capture each student's facial expressions, voice, and behavioral patterns, and prepare them for analysis.

[0212] Step 2:

[0213] The server uses the collected data to run an AI model that analyzes students' psychological state and learning progress. This analysis determines which students need what kind of support.

[0214] Step 3:

[0215] The server generates individualized student support plans based on the analysis results. For example, for a student working on a math problem, it generates a message suggesting "which formula should you use next?"

[0216] Step 4:

[0217] The server sends the generated instructions and messages to the robot student. These instructions include specific actions and messages to be delivered to the student.

[0218] Step 5:

[0219] The terminal (robot student) follows instructions from the server and performs actions for designated students or the entire class. It can provide explanations via voice, approach specific students and speak to them individually, and engage in other forms of interaction.

[0220] Step 6:

[0221] The user (student) deepens their learning through interaction with the robot student. If the student responds or asks questions during this process, that information is sent back to the server.

[0222] Step 7:

[0223] The server receives feedback after the interaction and evaluates the educational effectiveness. This allows for analysis of what types of instruction were effective, which is then used to improve teaching methods for teachers and schools.

[0224] (Example 1)

[0225] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0226] In traditional educational environments, providing flexible support tailored to the individual needs and circumstances of each student was difficult. Furthermore, a lack of communication, particularly among students with different language backgrounds, hindered learning efficiency. Additionally, the time and effort required for teachers to thoroughly understand each student's learning process and provide individualized instruction based on that understanding created a need for efficient feedback.

[0227] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0228] In this invention, the server includes information analysis means, behavior generation means, and generation means for generating communication methods using a generational AI model. This enables support tailored to the individual needs of learners within an educational environment, facilitates dialogue between learners with different languages, and provides effective feedback to teachers.

[0229] "Information analysis tools" are technologies for collecting and analyzing learners' behavior, statements, and ecological information within an educational environment.

[0230] "Action generation means" refers to technology that determines the optimal learning support action for a learner based on analyzed information and generates the content of that action.

[0231] A "generative AI model" refers to an algorithm that learns from accumulated data and generates communication methods and actions appropriate to new situations.

[0232] "Generative means" refers to technologies that utilize generative AI models to create communication methods tailored to the individual circumstances and needs of learners.

[0233] "Control means" refers to technology that operates automated devices based on generated action plans and enables interaction with human learners.

[0234] "Result collection methods" refer to technologies that collect data necessary to record the results of interactions with learners and provide that data to teachers as feedback.

[0235] This system is designed to provide support tailored to the individual needs of learners in an educational environment. Specifically, it collects and analyzes student behavior data through the cooperation of a server and terminals, and then provides students with optimized teaching activities based on that data through an automated system.

[0236] The server acquires behavioral data in real time through hardware such as facial recognition cameras, voice-collecting microphones, and motion sensors installed within the educational space. The server processes this collected data using information analysis tools to evaluate learners' comprehension and interest. Facial recognition technology can quantify the degree to which students are concentrating on the lesson, and voice analysis can measure the amount of conversation and participation.

[0237] Subsequently, the server uses an action generation mechanism to determine learning support actions based on the analysis results, and then uses a generative AI model to generate appropriate communication content. For example, a prompt such as "What kind of support should be provided if the student cannot answer the question?" is input to the AI, and specific instructions are formulated based on the response obtained.

[0238] The terminal controls automated devices according to instructions received from the server, enabling interaction with human learners. This device functions as a robotic student, guiding learners through natural conversation and instructions. For example, it can prompt a student with a weak understanding of a math problem by saying, "Let's think about what's important here," and then ask additional questions to deepen their understanding.

[0239] The teacher, as a user, uses the feedback provided by the device to evaluate each student's learning progress and develop a teaching plan for the next lesson. This allows the teacher to take an approach that is tailored to each individual student, thereby improving learning efficiency.

[0240] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0241] Step 1:

[0242] The server receives real-time input of student behavior data from facial recognition cameras, audio microphones, and motion sensors placed throughout the classroom. This allows the server to collect data recording student seating arrangements, classroom behavior, and spoken content.

[0243] Step 2:

[0244] The server processes the behavioral data obtained in step 1 using information analysis tools. Specifically, it analyzes the students' level of concentration using facial recognition technology and evaluates the number of times they speak and their pitch through voice analysis. This data is integrated and analyzed to generate an output that quantifies each student's level of interest and understanding.

[0245] Step 3:

[0246] The server uses the analysis results as input to generate optimal learning support actions using an action generation mechanism. Using a generation AI model, it formulates the optimal communication method and instructional content based on prompt sentences. At this stage, prompts such as "How should we approach a student who is having trouble asking a question?" are used. This results in the output of an action plan that includes specific instructions.

[0247] Step 4:

[0248] The terminal controls the automated device using the output of the action plan from the server as input. The robot student prompts and responds to students during class. For example, to a student struggling with a math problem, it might instruct, "Let's go back to the basics and think about this step." This establishes a dialogue and enables the robot to take specific actions to support the student in solving the problem.

[0249] Step 5:

[0250] The teacher, as the user, receives feedback provided from the device. This feedback includes statistics to evaluate the effectiveness of the interaction and progress reports for each learner. Based on this output, the teacher can develop individualized instruction plans as input and optimize the educational content by adjusting students' learning plans.

[0251] (Application Example 1)

[0252] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0253] In the manufacturing sector, there is a need to improve worker efficiency and optimize the work environment. In particular, when introducing new machinery or adapting to different work methods, it is difficult for workers to immediately learn the optimal operating procedures, which can result in decreased productivity and safety issues. It is necessary to overcome these problems and create an efficient and safe work environment.

[0254] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0255] In this invention, the server includes data analysis means for analyzing worker work data, action generation means for generating appropriate work support actions based on the analyzed data, control means for controlling work support machines and interacting with workers based on the generated actions, and feedback collection means for recording interaction results and analyzing them to improve the work environment. This enables real-time work support for workers, improving work efficiency and ensuring safety.

[0256] "Worker" refers to an individual engaged in work related to manufacturing or production.

[0257] "Work data" refers to data collected during work, including information about the actions and movements of workers.

[0258] "Data analysis means" refers to a system that processes collected data and evaluates the work efficiency and performance of workers.

[0259] "Action generation means" refers to a system that determines the optimal method of work support for workers based on analyzed data.

[0260] "Work support machinery" refers to robots and other devices used to assist workers in their tasks.

[0261] "Control means" refers to a system that has the function of operating work assistance machinery based on the generated support actions.

[0262] "Interaction results" refer to data showing workers' responses to the support actions of work assistance machines and the effects of those responses.

[0263] A "feedback collection method" refers to a system that records interaction results and collects data for later analysis.

[0264] To implement this invention, a comprehensive system is needed to support workers in a factory environment. This system includes data analysis means, action generation means, control means, and feedback collection means.

[0265] First, the server uses sensors installed throughout the factory to collect real-time data on the workers' activities. This data includes details such as their work style, speed, and environmental conditions. Based on this information, the data analysis system performs calculations necessary to analyze the efficiency and safety of each worker.

[0266] Next, the action generation system determines the optimal work support action based on the analysis results. This is done using a generative AI model, which plans specific support methods tailored to the worker's abilities and the nature of the work. For example, it can provide step-by-step guidance to a worker unfamiliar with operating new machinery.

[0267] Subsequently, the control system operates the work assistance machinery based on the generated actions and interacts effectively with the worker. This ensures that the guidance the worker receives is applied to the real environment and is reliably useful on the spot.

[0268] Furthermore, feedback collection mechanisms record the results of interactions. This data is used for analysis on the server and helps in the continuous improvement of the work environment. This also enables workers to perform tasks safely and efficiently.

[0269] As a concrete example, a prompt message such as "Do you need support while working with the new equipment?" might be displayed via smartphone to a worker unfamiliar with new operating procedures. This prompt message provides quick support for any challenges the worker may face.

[0270] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0271] Step 1:

[0272] The server utilizes sensors within the factory to collect worker activity data in real time. The input is signals from the sensors, which are converted into digital data. The output is raw data indicating the worker's activity status and environmental conditions. This raw data specifically includes information about each worker's position, movement, and speed.

[0273] Step 2:

[0274] The server processes the collected work data using data analysis tools. The input is the raw data collected in step 1. Here, analysis algorithms are used to detect data trends and anomalies, and to evaluate work efficiency and safety. The output is the analysis results for generating work support actions. This includes performance indicators and required support points for each worker.

[0275] Step 3:

[0276] The action generation mechanism uses the server's analysis results to generate optimal work support actions for each worker. The input is the analysis results from step 2. The generation AI model is used to perform data calculations to create a plan for support actions. The output is a support scenario for guidance at a specific work step or for new operating procedures.

[0277] Step 4:

[0278] The terminal receives the generated support actions and controls the work assistance machine. The input is the support action generated in step 3. It is converted into specific operating instructions and communicated to the machine in a way that is easy for the worker to understand. The output is the specific operation of the machine, which includes new machine operation guidance and warning messages.

[0279] Step 5:

[0280] Through the feedback collection means, the terminal records the interaction results with the work assistance machine. The input is the interaction data between the machine and the worker. This data includes the worker's reactions and the actual operation history of the machine. The output is a detailed interaction report that is sent back to the server. This report is used when conducting new analyses to improve the working environment.

[0281] Step 6:

[0282] Based on the provided feedback, the user improves the working environment and operation procedures. The input is the interaction report received from the server. To clearly indicate the points to be improved, the data is utilized to formulate improvement measures. The output is the improved working procedures and environment settings.

[0283] Furthermore, an emotion engine for estimating the user's emotions may be combined. That is, the specific processing unit 290 may estimate the user's emotions using the emotion recognition model 59 and perform specific processing using the user's emotions.

[0284] The present invention is a robot student system that functions effectively in a classroom, and aims to recognize the user's emotions and provide appropriate educational support by combining an emotion engine. This system is composed of a server and a robot terminal, and each component operates in联动.

[0285] The server first collects the facial expressions and voice data of the students from the sensor devices installed in the classroom. The obtained data is analyzed by the emotion engine within the system to grasp the emotional state and psychological needs of the students in real time. Through this analysis, information such as whether a specific student is experiencing anxiety or showing interest in the lesson can be obtained.

[0286] Next, based on the information obtained from the emotion engine, the server formulates an education support method using the action generation means. For example, for a student feeling anxious, a plan is generated for the robot student to encourage the student with kind words. Such a plan can flexibly respond to students who need support in different languages.

[0287] Upon receiving an instruction from the server, the robot student acting as the terminal performs the specified action in the classroom. For example, if a certain student seems to be depressed, the robot student goes near the student and asks, "Is there anything I can help with?" to provide support through friendly interaction.

[0288] Users such as teachers and other students can improve the learning environment of the entire class while receiving such support from the robot student. In particular, with the feedback function based on emotion recognition, teachers can understand the mental state of each student and use it to devise more suitable teaching methods.

[0289] As a specific example, considering a scenario in a language class, when students with different cultural backgrounds feel isolated due to language barriers, the emotion engine analyzes the anxious expressions of those students. Based on this, the robot student can provide support in the native language to those students and create an opportunity for interaction with other students.

[0290] In this way, the robot student system of the present invention realizes effects such as improving human relationships in the classroom, preventing bullying, suppressing class breakdown, and bringing out potential abilities.

[0291] The following explains the processing flow.

[0292] Step 1:

[0293] The server continuously collects students' facial expressions and voice data from sensor devices such as cameras and microphones installed in the classroom. The collected data is temporarily stored and prepared for analysis.

[0294] Step 2:

[0295] The server uses an emotion engine to analyze the collected data. It estimates students' emotional states from their facial expressions and voice tone, and monitors changes in anxiety, excitement, and concentration levels in real time.

[0296] Step 3:

[0297] The server generates educational support behaviors tailored to individual students based on emotional data estimated by the emotion engine. Specifically, it creates support plans that include encouraging words in a gentle voice and questions that are engaging.

[0298] Step 4:

[0299] The server sends the generated educational support actions as instructions to the robot student. These instructions include specific conversation topics with the student and guidelines for action.

[0300] Step 5:

[0301] The terminal (robot student) moves around the classroom based on instructions received from the server and makes contact with the target student. The robot student speaks to the student using designated phrases and gestures and provides follow-up.

[0302] Step 6:

[0303] The user (student) interacts with the robot student after receiving contact from it. The student's responses and new emotional states are recorded as additional data.

[0304] Step 7:

[0305] The server evaluates the interaction results between robot students and human students and generates feedback useful for improving the educational environment and methods from the recorded data. The user (teacher) can use this to formulate more appropriate teaching strategies.

[0306] (Example 2)

[0307] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0308] In conventional educational support systems, real-time emotion recognition for individual learners and the provision of appropriate feedback are insufficient, and it is difficult to flexibly respond to learners with diverse languages and cultural backgrounds. As a result, there are problems in that the psychological needs of learners cannot be met and the improvement of learning effects is hindered.

[0309] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0310] In this invention, the server includes an information analysis means, an action planning means, and an emotion analysis means. Thereby, it becomes possible to grasp the emotional state of human learners in the learning space in real time and provide support according to individual educational needs.

[0311] The "information analysis means" is a device for specifying the emotional state and psychological needs in real time by collecting and analyzing the actions, expressions, and voice data of learners.

[0312] The "action planning means" is a device for designing appropriate educational activities and generating a plan for effectively intervening in the learning environment based on the analyzed information.

[0313] The "control means" is a device for operating the machine learning device based on the generated action plan and interacting with the learner.

[0314] A "feedback acquisition method" is a device that records the results of conversations and analyzes the collected information in order to improve the learning environment.

[0315] An "emotion analysis device" is a device that analyzes a learner's facial expressions and voice data to determine their psychological state in real time.

[0316] A "generative AI model" is a model that uses machine learning algorithms to flexibly adjust action plans, and is a device that supports decision-making within a system.

[0317] This system is an educational support system that uses machine learning devices to function effectively within a learning environment. The server collects learners' facial expressions and voice data through sensor devices installed in the classroom. This data is analyzed in real time by emotion analysis capabilities on the server to determine the learners' emotional state and psychological needs. The hardware used includes sensor devices such as cameras and recording devices like microphones, and the server requires powerful computing resources to process the data from these devices.

[0318] Based on the information obtained through emotion analysis, the server generates individual support plans using an action planning mechanism. The action planning mechanism uses a generative AI model to design optimal educational support from the analysis results. For example, if emotion analysis determines that a learner is experiencing stress, the generative AI model will suggest ways to talk to that learner to help them relax. An example of a prompt in this context would be, "Learner A appears to be experiencing stress. What should I say to them?"

[0319] The generated action plan is sent to the robot student, which acts as a terminal, and implemented in the classroom as an actual educational support activity. The robot student follows the instructions received and provides gentle encouragement and specific support to the learners. These activities are carried out through control mechanisms, which play a role in improving the quality of interaction with learners.

[0320] Furthermore, the user, the teacher, records the results of these interactions using feedback mechanisms and utilizes them to improve the learning environment. As described above, this system has a mechanism to provide more appropriate and flexible educational support through the use of sentiment analysis and generative AI models.

[0321] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0322] Step 1:

[0323] The server collects learners' facial expressions and voice data from sensor devices installed in the classroom. Inputs are camera and microphone data, and this information is aggregated on the server as output. This data collection enables real-time emotion analysis. Specifically, the server acquires this data at regular intervals and stores it in a buffer.

[0324] Step 2:

[0325] The server passes the collected facial and audio data to the emotion analysis system. The input is the student's facial and audio data collected in step 1, and the output is the analysis result identifying the student's emotional state. The server uses machine learning algorithms to analyze the data and determine emotions such as "anxiety," "interest," and "confusion" in real time. Specific operations include data formatting and feature extraction during the analysis.

[0326] Step 3:

[0327] The server generates a support plan via an action planning mechanism based on the results of the emotion analysis. The input is the emotion analysis results from step 2, and the output is a specific educational support plan. This plan is formulated using a generative AI model and is output in the form of, for example, "Learner B is feeling anxious, so offer words of encouragement." The specific actions here are inputting prompt sentences to the generative AI model and generating support suggestions.

[0328] Step 4:

[0329] The server sends the generated support plan to the robot terminal. The input is the support plan generated in step 3, and the output is the instructions to the robot terminal. The server sends the instructions based on the plan to the robot wirelessly and prepares the robot to execute them at the appropriate time. Specific actions include communication execution and serialization of instruction data.

[0330] Step 5:

[0331] The robot, acting as the terminal, performs interactions within the classroom based on instructions received from the server. The input is instruction data from the server, and the output is direct dialogue with the learner. Specifically, the robot might approach and say, "Let's solve this problem together."

[0332] Step 6:

[0333] The user, the teacher, observes the interaction between the robot and the learners and records the results using feedback mechanisms. The input is classroom events and learners' responses, and the output is recorded feedback data. This data will be used to improve future educational support. Specific actions include inputting data into record sheets or digital forms.

[0334] (Application Example 2)

[0335] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0336] In today's work environment, the mental and physical health of workers is directly linked to work efficiency and safety. However, many factories and workplaces lack mechanisms to properly monitor workers' emotional states and stress levels and provide support based on that information. This situation can lead to decreased worker productivity and safety problems. Therefore, there is a need for a system that can improve the work environment by understanding workers' emotional states in real time and providing optimal support.

[0337] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes data analysis means for analyzing the emotional state of the worker, action generation means for generating appropriate support actions based on the analyzed emotional state, and control means for controlling the support device and interacting with the worker based on the generated actions. As a result, the support device in the workplace can analyze the emotional state of the worker in real time and provide appropriate verbal encouragement and advice, thereby improving the work environment.

[0338] A "data analysis device" is a device equipped with the function of collecting and analyzing information necessary to understand the emotional state of workers.

[0339] A "behavior generation device" is a device that has the function of designing appropriate support actions for workers based on analyzed data.

[0340] A "control means" is a device that operates a support device according to the generated support action and has the function of interacting with the worker.

[0341] A "feedback collection device" is a device that records the results of interactions with support devices and uses them for analysis to further improve the work environment.

[0342] A "support device" is a device that can concretize and actually implement support actions based on the emotional state of the worker.

[0343] In the system for realizing this invention, a server plays a central role. The server acquires facial expressions and voice data of workers from various sensors installed in the work environment. Hardware such as cameras and microphones are used for this purpose. The acquired data is analyzed in real time using emotion analysis software installed on the server. The emotion analysis software can utilize, for example, a Python-based emotion analysis library.

[0344] Once the emotional state of the user (worker) is analyzed, the server uses an action generation algorithm to formulate appropriate support actions. This algorithm designs break suggestions and work-related advice based on the worker's stress and fatigue levels. Communication in different languages ​​is also possible, utilizing a multilingual speech generation engine.

[0345] Subsequently, the server operates the support device via a control system, interacting with the worker. The control system includes actuators and speakers, providing voice output for natural dialogue. The results of the interactions performed by the support device are recorded by a feedback collection system and analyzed for further optimization of the work environment.

[0346] As a concrete example, considering its use in a manufacturing plant, if a sensor detects worker fatigue, the server can provide support such as, "You seem tired. Would you like to take a short break?" An example of a prompt message used in this case would be, "We analyze the worker's emotional state in real time and provide appropriate break suggestions if stress or fatigue is detected. This is supported in Japanese or other languages ​​as needed."

[0347] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0348] Step 1:

[0349] The server captures the worker's facial expressions and voice using cameras and microphones installed in the work environment. The input is raw data obtained through the cameras and microphones, which is then sent to the server. The output consists of analyzable image and audio data.

[0350] Step 2:

[0351] The server uses an emotion analysis library to analyze the captured image and audio data. Here, facial features are extracted from the image data, and voice tone and speed are analyzed from the audio data. The input is the data from step 1, and the output is an evaluation score indicating the worker's emotional state. This process quantifies, for example, levels of stress and fatigue.

[0352] Step 3:

[0353] The server executes an action generation algorithm based on the evaluation score. This algorithm uses predefined rules and machine learning models to formulate the optimal support actions to provide to the worker. The input is the evaluation score obtained in step 2, and the output is a specific action plan such as "suggestion for a break" or "advice to improve work efficiency."

[0354] Step 4:

[0355] The server instructs the support device, via a control mechanism, on the action plan formulated by the action generation algorithm. The support device uses a speaker to perform voice actions and attempts to engage in natural dialogue with the worker. The input is the action plan from step 3, and the output is the voice action.

[0356] Step 5:

[0357] The terminal records the results of the interaction in the feedback collection system. This recorded data is then analyzed by the server for future system improvements. The input is the details of the actual interaction by the support device, and the output is analyzable feedback data.

[0358] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0359] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0360] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0361] [Third Embodiment]

[0362] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0363] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0364] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0365] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0366] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0367] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0368] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0369] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0370] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0371] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0372] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0373] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0374] To implement this invention, effective operation of robot students within the classroom is required. This system mainly consists of a server and terminals, and each component functions in cooperation with others.

[0375] The server first collects behavioral data from each student in the classroom using various sensors and analyzes it in real time. This includes facial recognition, voice analysis, and tracking of behavioral patterns. Based on the data obtained in this way, it forms a foundation for directing support actions for teachers and robot students.

[0376] Next, the server generates the optimal communication method for a specific student based on the analyzed data. This generative AI plans the conversation content and actions necessary to enhance the effectiveness of education. Specifically, this includes questions and explanations to answer students' questions, and suggestions to encourage cooperation with classmates.

[0377] The terminal controls the robot students based on instructions from the server, enabling interaction with the students. The robot students follow these instructions, allowing for natural dialogue and emotional expression within the classroom. For example, in a math class, they might gently advise a student struggling with a calculation, saying, "Let's try using a basic formula," and then answer further questions in a casual tone.

[0378] Furthermore, through a feedback recording system, teachers are provided with detailed information on the results of interactions after each lesson. Based on this, teachers can easily create instructional plans tailored to each student's situation. This makes it possible to provide education that is appropriate for each student and improve the learning efficiency of the entire class.

[0379] Specific examples of this invention include preventing bullying, deterring classroom disruption, and assisting students who speak different languages. By utilizing the above mechanism, it is expected that a better educational environment will be realized by drawing out the potential of human students while robot students support their learning.

[0380] The following describes the processing flow.

[0381] Step 1:

[0382] The server collects data in real time from sensors such as cameras and microphones installed in the classroom. This allows it to capture each student's facial expressions, voice, and behavioral patterns, and prepare them for analysis.

[0383] Step 2:

[0384] The server uses the collected data to run an AI model that analyzes students' psychological state and learning progress. This analysis determines which students need what kind of support.

[0385] Step 3:

[0386] The server generates individualized student support plans based on the analysis results. For example, for a student working on a math problem, it generates a message suggesting "which formula should you use next?"

[0387] Step 4:

[0388] The server sends the generated instructions and messages to the robot student. These instructions include specific actions and messages to be delivered to the student.

[0389] Step 5:

[0390] The terminal (robot student) follows instructions from the server and performs actions for designated students or the entire class. It can provide explanations via voice, approach specific students and speak to them individually, and engage in other forms of interaction.

[0391] Step 6:

[0392] The user (student) deepens their learning through interaction with the robot student. If the student responds or asks questions during this process, that information is sent back to the server.

[0393] Step 7:

[0394] The server receives feedback after the interaction and evaluates the educational effectiveness. This allows for analysis of what types of instruction were effective, which is then used to improve teaching methods for teachers and schools.

[0395] (Example 1)

[0396] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0397] In traditional educational environments, providing flexible support tailored to the individual needs and circumstances of each student was difficult. Furthermore, a lack of communication, particularly among students with different language backgrounds, hindered learning efficiency. Additionally, the time and effort required for teachers to thoroughly understand each student's learning process and provide individualized instruction based on that understanding created a need for efficient feedback.

[0398] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0399] In this invention, the server includes information analysis means, behavior generation means, and generation means for generating communication methods using a generational AI model. This enables support tailored to the individual needs of learners within an educational environment, facilitates dialogue between learners with different languages, and provides effective feedback to teachers.

[0400] "Information analysis tools" are technologies for collecting and analyzing learners' behavior, statements, and ecological information within an educational environment.

[0401] "Action generation means" refers to technology that determines the optimal learning support action for a learner based on analyzed information and generates the content of that action.

[0402] A "generative AI model" refers to an algorithm that learns from accumulated data and generates communication methods and actions appropriate to new situations.

[0403] "Generative means" refers to technologies that utilize generative AI models to create communication methods tailored to the individual circumstances and needs of learners.

[0404] "Control means" refers to technology that operates automated devices based on generated action plans and enables interaction with human learners.

[0405] "Result collection methods" refer to technologies that collect data necessary to record the results of interactions with learners and provide that data to teachers as feedback.

[0406] This system is designed to provide support tailored to the individual needs of learners in an educational environment. Specifically, it collects and analyzes student behavior data through the cooperation of a server and terminals, and then provides students with optimized teaching activities based on that data through an automated system.

[0407] The server acquires behavioral data in real time through hardware such as facial recognition cameras, voice-collecting microphones, and motion sensors installed within the educational space. The server processes this collected data using information analysis tools to evaluate learners' comprehension and interest. Facial recognition technology can quantify the degree to which students are concentrating on the lesson, and voice analysis can measure the amount of conversation and participation.

[0408] Subsequently, the server uses an action generation mechanism to determine learning support actions based on the analysis results, and then uses a generative AI model to generate appropriate communication content. For example, a prompt such as "What kind of support should be provided if the student cannot answer the question?" is input to the AI, and specific instructions are formulated based on the response obtained.

[0409] The terminal controls automated devices according to instructions received from the server, enabling interaction with human learners. This device functions as a robotic student, guiding learners through natural conversation and instructions. For example, it can prompt a student with a weak understanding of a math problem by saying, "Let's think about what's important here," and then ask additional questions to deepen their understanding.

[0410] The teacher, as a user, uses the feedback provided by the device to evaluate each student's learning progress and develop a teaching plan for the next lesson. This allows the teacher to take an approach that is tailored to each individual student, thereby improving learning efficiency.

[0411] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0412] Step 1:

[0413] The server receives real-time input of student behavior data from facial recognition cameras, audio microphones, and motion sensors placed throughout the classroom. This allows the server to collect data recording student seating arrangements, classroom behavior, and spoken content.

[0414] Step 2:

[0415] The server processes the behavioral data obtained in step 1 using information analysis tools. Specifically, it analyzes the students' level of concentration using facial recognition technology and evaluates the number of times they speak and their pitch through voice analysis. This data is integrated and analyzed to generate an output that quantifies each student's level of interest and understanding.

[0416] Step 3:

[0417] The server uses the analysis results as input to generate optimal learning support actions using an action generation mechanism. Using a generation AI model, it formulates the optimal communication method and instructional content based on prompt sentences. At this stage, prompts such as "How should we approach a student who is having trouble asking a question?" are used. This results in the output of an action plan that includes specific instructions.

[0418] Step 4:

[0419] The terminal controls the automated device using the output of the action plan from the server as input. The robot student prompts and responds to students during class. For example, to a student struggling with a math problem, it might instruct, "Let's go back to the basics and think about this step." This establishes a dialogue and enables the robot to take specific actions to support the student in solving the problem.

[0420] Step 5:

[0421] The teacher, as the user, receives feedback provided from the device. This feedback includes statistics to evaluate the effectiveness of the interaction and progress reports for each learner. Based on this output, the teacher can develop individualized instruction plans as input and optimize the educational content by adjusting students' learning plans.

[0422] (Application Example 1)

[0423] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0424] In the manufacturing sector, there is a need to improve worker efficiency and optimize the work environment. In particular, when introducing new machinery or adapting to different work methods, it is difficult for workers to immediately learn the optimal operating procedures, which can result in decreased productivity and safety issues. It is necessary to overcome these problems and create an efficient and safe work environment.

[0425] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0426] In this invention, the server includes data analysis means for analyzing worker work data, action generation means for generating appropriate work support actions based on the analyzed data, control means for controlling work support machines and interacting with workers based on the generated actions, and feedback collection means for recording interaction results and analyzing them to improve the work environment. This enables real-time work support for workers, improving work efficiency and ensuring safety.

[0427] "Worker" refers to an individual engaged in work related to manufacturing or production.

[0428] "Work data" refers to data collected during work, including information about the actions and movements of workers.

[0429] "Data analysis means" refers to a system that processes collected data and evaluates the work efficiency and performance of workers.

[0430] "Action generation means" refers to a system that determines the optimal method of work support for workers based on analyzed data.

[0431] "Work support machinery" refers to robots and other devices used to assist workers in their tasks.

[0432] "Control means" refers to a system that has the function of operating work assistance machinery based on the generated support actions.

[0433] "Interaction results" refer to data showing workers' responses to the support actions of work assistance machines and the effects of those responses.

[0434] A "feedback collection method" refers to a system that records interaction results and collects data for later analysis.

[0435] To implement this invention, a comprehensive system is needed to support workers in a factory environment. This system includes data analysis means, action generation means, control means, and feedback collection means.

[0436] First, the server uses sensors installed throughout the factory to collect real-time data on the workers' activities. This data includes details such as their work style, speed, and environmental conditions. Based on this information, the data analysis system performs calculations necessary to analyze the efficiency and safety of each worker.

[0437] Next, the action generation system determines the optimal work support action based on the analysis results. This is done using a generative AI model, which plans specific support methods tailored to the worker's abilities and the nature of the work. For example, it can provide step-by-step guidance to a worker unfamiliar with operating new machinery.

[0438] Subsequently, the control system operates the work assistance machinery based on the generated actions and interacts effectively with the worker. This ensures that the guidance the worker receives is applied to the real environment and is reliably useful on the spot.

[0439] Furthermore, feedback collection mechanisms record the results of interactions. This data is used for analysis on the server and helps in the continuous improvement of the work environment. This also enables workers to perform tasks safely and efficiently.

[0440] As a concrete example, a prompt message such as "Do you need support while working with the new equipment?" might be displayed via smartphone to a worker unfamiliar with new operating procedures. This prompt message provides quick support for any challenges the worker may face.

[0441] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0442] Step 1:

[0443] The server utilizes sensors within the factory to collect worker activity data in real time. The input is signals from the sensors, which are converted into digital data. The output is raw data indicating the worker's activity status and environmental conditions. This raw data specifically includes information about each worker's position, movement, and speed.

[0444] Step 2:

[0445] The server processes the collected work data using data analysis tools. The input is the raw data collected in step 1. Here, analysis algorithms are used to detect data trends and anomalies, and to evaluate work efficiency and safety. The output is the analysis results for generating work support actions. This includes performance indicators and required support points for each worker.

[0446] Step 3:

[0447] The action generation mechanism uses the server's analysis results to generate optimal work support actions for each worker. The input is the analysis results from step 2. The generation AI model is used to perform data calculations to create a plan for support actions. The output is a support scenario for guidance at a specific work step or for new operating procedures.

[0448] Step 4:

[0449] The terminal receives the generated support actions and controls the work assistance machine. The input is the support action generated in step 3. It is converted into specific operating instructions and communicated to the machine in a way that is easy for the worker to understand. The output is the specific operation of the machine, which includes new machine operation guidance and warning messages.

[0450] Step 5:

[0451] The feedback collection mechanism allows the terminal to record the results of interactions with the work assistance machine. The input is interaction data between the machine and the worker. This data includes the worker's reactions and the machine's actual operation history. The output is a detailed interaction report sent back to the server. This report is used for further analysis to improve the work environment.

[0452] Step 6:

[0453] Users improve their work environment and operating procedures based on the feedback they receive. The input is interaction reports received from the server. The data is used to develop improvement strategies to clearly identify areas for improvement. The output is the improved work procedures and environment settings.

[0454] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0455] This invention relates to a robotic student system that functions effectively in a classroom and aims to recognize user emotions and provide appropriate educational support by combining it with an emotion engine. The system consists of a server and robotic terminals, and each component operates in conjunction with the others.

[0456] The server first collects facial expressions and voice data from sensor devices installed in the classroom. The collected data is analyzed by the system's emotion engine to understand students' emotional states and psychological needs in real time. This analysis provides information such as whether a particular student is experiencing anxiety or showing interest in the lesson.

[0457] Next, the server uses the information obtained from the emotion engine to formulate educational support methods using behavior generation mechanisms. For example, for a student who is feeling anxious, it generates a plan in which a robot student offers words of encouragement in a gentle voice. These plans are designed to be flexible enough to accommodate students who require support in different languages.

[0458] The robot student, acting as a terminal, receives instructions from the server and then performs the designated actions within the classroom. For example, if a student appears depressed, the robot student will approach that student and ask, "Is there anything I can do to help?", providing support through friendly interaction.

[0459] Teachers and other students who are users of these robot students can improve the learning environment for the entire class with their support. In particular, the emotion recognition-based feedback function allows teachers to understand the emotional state of each student and use that information to devise more appropriate teaching methods.

[0460] As a concrete example, consider a scenario in a language class where a student with a different cultural background experiences language barriers and feels isolated. The emotion engine analyzes the student's anxious facial expressions. Based on this, the robot student can provide support in the student's native language and create opportunities for interaction with other students.

[0461] Thus, the robot student system of the present invention achieves effects such as improving interpersonal relationships in the classroom, preventing bullying, and deterring classroom disruption, as well as drawing out potential abilities.

[0462] The following describes the processing flow.

[0463] Step 1:

[0464] The server continuously collects students' facial expressions and voice data from sensor devices such as cameras and microphones installed in the classroom. The collected data is temporarily stored and prepared for analysis.

[0465] Step 2:

[0466] The server uses an emotion engine to analyze the collected data. It estimates students' emotional states from their facial expressions and voice tone, and monitors changes in anxiety, excitement, and concentration levels in real time.

[0467] Step 3:

[0468] The server generates educational support behaviors tailored to individual students based on emotional data estimated by the emotion engine. Specifically, it creates support plans that include encouraging words in a gentle voice and questions that are engaging.

[0469] Step 4:

[0470] The server sends the generated educational support actions as instructions to the robot student. These instructions include specific conversation topics with the student and guidelines for action.

[0471] Step 5:

[0472] The terminal (robot student) moves around the classroom based on instructions received from the server and makes contact with the target student. The robot student speaks to the student using designated phrases and gestures and provides follow-up.

[0473] Step 6:

[0474] The user (student) interacts with the robot student after receiving contact from it. The student's responses and new emotional states are recorded as additional data.

[0475] Step 7:

[0476] The server evaluates the interaction between robot students and human students and generates feedback from the recorded data to help improve the educational environment and methods. Users (teachers) can then use this feedback to develop more appropriate teaching strategies.

[0477] (Example 2)

[0478] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0479] Traditional educational support systems have been insufficient in real-time emotional recognition and providing appropriate feedback to individual learners, making it difficult to flexibly accommodate learners with diverse linguistic and cultural backgrounds. As a result, there is a challenge in that learners' psychological needs cannot be met, hindering the improvement of learning effectiveness.

[0480] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0481] In this invention, the server includes information analysis means, action planning means, and emotion analysis means. This makes it possible to grasp the emotional state of human learners in a learning space in real time and provide support tailored to their individual educational needs.

[0482] An "information analysis tool" is a device that collects and analyzes learners' behavior, facial expressions, and voice data to identify their emotional state and psychological needs in real time.

[0483] An "action planning tool" is a device that generates a plan to design appropriate educational activities and effectively intervene in the learning environment based on analyzed information.

[0484] A "control device" is a device that operates a machine learning device based on a generated action plan and interacts with the learner.

[0485] A "feedback acquisition method" is a device that records the results of conversations and analyzes the collected information in order to improve the learning environment.

[0486] An "emotion analysis device" is a device that analyzes a learner's facial expressions and voice data to determine their psychological state in real time.

[0487] A "generative AI model" is a model that uses machine learning algorithms to flexibly adjust action plans, and is a device that supports decision-making within a system.

[0488] This system is an educational support system that uses machine learning devices to function effectively within a learning environment. The server collects learners' facial expressions and voice data through sensor devices installed in the classroom. This data is analyzed in real time by emotion analysis capabilities on the server to determine the learners' emotional state and psychological needs. The hardware used includes sensor devices such as cameras and recording devices like microphones, and the server requires powerful computing resources to process the data from these devices.

[0489] Based on the information obtained through emotion analysis, the server generates individual support plans using an action planning mechanism. The action planning mechanism uses a generative AI model to design optimal educational support from the analysis results. For example, if emotion analysis determines that a learner is experiencing stress, the generative AI model will suggest ways to talk to that learner to help them relax. An example of a prompt in this context would be, "Learner A appears to be experiencing stress. What should I say to them?"

[0490] The generated action plan is sent to the robot student, which acts as a terminal, and implemented in the classroom as an actual educational support activity. The robot student follows the instructions received and provides gentle encouragement and specific support to the learners. These activities are carried out through control mechanisms, which play a role in improving the quality of interaction with learners.

[0491] Furthermore, the user, the teacher, records the results of these interactions using feedback mechanisms and utilizes them to improve the learning environment. As described above, this system has a mechanism to provide more appropriate and flexible educational support through the use of sentiment analysis and generative AI models.

[0492] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0493] Step 1:

[0494] The server collects learners' facial expressions and voice data from sensor devices installed in the classroom. Inputs are camera and microphone data, and this information is aggregated on the server as output. This data collection enables real-time emotion analysis. Specifically, the server acquires this data at regular intervals and stores it in a buffer.

[0495] Step 2:

[0496] The server passes the collected facial and audio data to the emotion analysis system. The input is the student's facial and audio data collected in step 1, and the output is the analysis result identifying the student's emotional state. The server uses machine learning algorithms to analyze the data and determine emotions such as "anxiety," "interest," and "confusion" in real time. Specific operations include data formatting and feature extraction during the analysis.

[0497] Step 3:

[0498] The server generates a support plan via an action planning mechanism based on the results of the emotion analysis. The input is the emotion analysis results from step 2, and the output is a specific educational support plan. This plan is formulated using a generative AI model and is output in the form of, for example, "Learner B is feeling anxious, so offer words of encouragement." The specific actions here are inputting prompt sentences to the generative AI model and generating support suggestions.

[0499] Step 4:

[0500] The server sends the generated support plan to the robot terminal. The input is the support plan generated in step 3, and the output is the instructions to the robot terminal. The server sends the instructions based on the plan to the robot wirelessly and prepares the robot to execute them at the appropriate time. Specific actions include communication execution and serialization of instruction data.

[0501] Step 5:

[0502] The robot, acting as the terminal, performs interactions within the classroom based on instructions received from the server. The input is instruction data from the server, and the output is direct dialogue with the learner. Specifically, the robot might approach and say, "Let's solve this problem together."

[0503] Step 6:

[0504] The user, the teacher, observes the interaction between the robot and the learners and records the results using feedback mechanisms. The input is classroom events and learners' responses, and the output is recorded feedback data. This data will be used to improve future educational support. Specific actions include inputting data into record sheets or digital forms.

[0505] (Application Example 2)

[0506] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0507] In today's work environment, the mental and physical health of workers is directly linked to work efficiency and safety. However, many factories and workplaces lack mechanisms to properly monitor workers' emotional states and stress levels and provide support based on that information. This situation can lead to decreased worker productivity and safety problems. Therefore, there is a need for a system that can improve the work environment by understanding workers' emotional states in real time and providing optimal support.

[0508] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes data analysis means for analyzing the emotional state of the worker, action generation means for generating appropriate support actions based on the analyzed emotional state, and control means for controlling the support device and interacting with the worker based on the generated actions. As a result, the support device in the workplace can analyze the emotional state of the worker in real time and provide appropriate verbal encouragement and advice, thereby improving the work environment.

[0509] A "data analysis device" is a device equipped with the function of collecting and analyzing information necessary to understand the emotional state of workers.

[0510] A "behavior generation device" is a device that has the function of designing appropriate support actions for workers based on analyzed data.

[0511] A "control means" is a device that operates a support device according to the generated support action and has the function of interacting with the worker.

[0512] A "feedback collection device" is a device that records the results of interactions with support devices and uses them for analysis to further improve the work environment.

[0513] A "support device" is a device that can concretize and actually implement support actions based on the emotional state of the worker.

[0514] In the system for realizing this invention, a server plays a central role. The server acquires facial expressions and voice data of workers from various sensors installed in the work environment. Hardware such as cameras and microphones are used for this purpose. The acquired data is analyzed in real time using emotion analysis software installed on the server. The emotion analysis software can utilize, for example, a Python-based emotion analysis library.

[0515] Once the emotional state of the user (worker) is analyzed, the server uses an action generation algorithm to formulate appropriate support actions. This algorithm designs break suggestions and work-related advice based on the worker's stress and fatigue levels. Communication in different languages ​​is also possible, utilizing a multilingual speech generation engine.

[0516] Subsequently, the server operates the support device via a control system, interacting with the worker. The control system includes actuators and speakers, providing voice output for natural dialogue. The results of the interactions performed by the support device are recorded by a feedback collection system and analyzed for further optimization of the work environment.

[0517] As a concrete example, considering its use in a manufacturing plant, if a sensor detects worker fatigue, the server can provide support such as, "You seem tired. Would you like to take a short break?" An example of a prompt message used in this case would be, "We analyze the worker's emotional state in real time and provide appropriate break suggestions if stress or fatigue is detected. This is supported in Japanese or other languages ​​as needed."

[0518] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0519] Step 1:

[0520] The server captures the worker's facial expressions and voice using cameras and microphones installed in the work environment. The input is raw data obtained through the cameras and microphones, which is then sent to the server. The output consists of analyzable image and audio data.

[0521] Step 2:

[0522] The server uses an emotion analysis library to analyze the captured image and audio data. Here, facial features are extracted from the image data, and voice tone and speed are analyzed from the audio data. The input is the data from step 1, and the output is an evaluation score indicating the worker's emotional state. This process quantifies, for example, levels of stress and fatigue.

[0523] Step 3:

[0524] The server executes an action generation algorithm based on the evaluation score. This algorithm uses predefined rules and machine learning models to formulate the optimal support actions to provide to the worker. The input is the evaluation score obtained in step 2, and the output is a specific action plan such as "suggestion for a break" or "advice to improve work efficiency."

[0525] Step 4:

[0526] The server instructs the support device, via a control mechanism, on the action plan formulated by the action generation algorithm. The support device uses a speaker to perform voice actions and attempts to engage in natural dialogue with the worker. The input is the action plan from step 3, and the output is the voice action.

[0527] Step 5:

[0528] The terminal records the results of the interaction in the feedback collection system. This recorded data is then analyzed by the server for future system improvements. The input is the details of the actual interaction by the support device, and the output is analyzable feedback data.

[0529] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0530] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0531] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0532] [Fourth Embodiment]

[0533] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0534] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0535] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0536] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0537] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0538] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0539] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0540] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0541] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0542] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0543] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0544] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0545] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0546] To implement this invention, effective operation of robot students within the classroom is required. This system mainly consists of a server and terminals, and each component functions in cooperation with others.

[0547] The server first collects behavioral data from each student in the classroom using various sensors and analyzes it in real time. This includes facial recognition, voice analysis, and tracking of behavioral patterns. Based on the data obtained in this way, it forms a foundation for directing support actions for teachers and robot students.

[0548] Next, the server generates the optimal communication method for a specific student based on the analyzed data. This generative AI plans the conversation content and actions necessary to enhance the effectiveness of education. Specifically, this includes questions and explanations to answer students' questions, and suggestions to encourage cooperation with classmates.

[0549] The terminal controls the robot students based on instructions from the server, enabling interaction with the students. The robot students follow these instructions, allowing for natural dialogue and emotional expression within the classroom. For example, in a math class, they might gently advise a student struggling with a calculation, saying, "Let's try using a basic formula," and then answer further questions in a casual tone.

[0550] Furthermore, through a feedback recording system, teachers are provided with detailed information on the results of interactions after each lesson. Based on this, teachers can easily create instructional plans tailored to each student's situation. This makes it possible to provide education that is appropriate for each student and improve the learning efficiency of the entire class.

[0551] Specific examples of this invention include preventing bullying, deterring classroom disruption, and assisting students who speak different languages. By utilizing the above mechanism, it is expected that a better educational environment will be realized by drawing out the potential of human students while robot students support their learning.

[0552] The following describes the processing flow.

[0553] Step 1:

[0554] The server collects data in real time from sensors such as cameras and microphones installed in the classroom. This allows it to capture each student's facial expressions, voice, and behavioral patterns, and prepare them for analysis.

[0555] Step 2:

[0556] The server uses the collected data to run an AI model that analyzes students' psychological state and learning progress. This analysis determines which students need what kind of support.

[0557] Step 3:

[0558] The server generates individualized student support plans based on the analysis results. For example, for a student working on a math problem, it generates a message suggesting "which formula should you use next?"

[0559] Step 4:

[0560] The server sends the generated instructions and messages to the robot student. These instructions include specific actions and messages to be delivered to the student.

[0561] Step 5:

[0562] The terminal (robot student) follows instructions from the server and performs actions for designated students or the entire class. It can provide explanations via voice, approach specific students and speak to them individually, and engage in other forms of interaction.

[0563] Step 6:

[0564] The user (student) deepens their learning through interaction with the robot student. If the student responds or asks questions during this process, that information is sent back to the server.

[0565] Step 7:

[0566] The server receives feedback after the interaction and evaluates the educational effectiveness. This allows for analysis of what types of instruction were effective, which is then used to improve teaching methods for teachers and schools.

[0567] (Example 1)

[0568] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0569] In traditional educational environments, providing flexible support tailored to the individual needs and circumstances of each student was difficult. Furthermore, a lack of communication, particularly among students with different language backgrounds, hindered learning efficiency. Additionally, the time and effort required for teachers to thoroughly understand each student's learning process and provide individualized instruction based on that understanding created a need for efficient feedback.

[0570] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0571] In this invention, the server includes information analysis means, behavior generation means, and generation means for generating communication methods using a generational AI model. This enables support tailored to the individual needs of learners within an educational environment, facilitates dialogue between learners with different languages, and provides effective feedback to teachers.

[0572] "Information analysis tools" are technologies for collecting and analyzing learners' behavior, statements, and ecological information within an educational environment.

[0573] "Action generation means" refers to technology that determines the optimal learning support action for a learner based on analyzed information and generates the content of that action.

[0574] A "generative AI model" refers to an algorithm that learns from accumulated data and generates communication methods and actions appropriate to new situations.

[0575] "Generative means" refers to technologies that utilize generative AI models to create communication methods tailored to the individual circumstances and needs of learners.

[0576] "Control means" refers to technology that operates automated devices based on generated action plans and enables interaction with human learners.

[0577] "Result collection methods" refer to technologies that collect data necessary to record the results of interactions with learners and provide that data to teachers as feedback.

[0578] This system is designed to provide support tailored to the individual needs of learners in an educational environment. Specifically, it collects and analyzes student behavior data through the cooperation of a server and terminals, and then provides students with optimized teaching activities based on that data through an automated system.

[0579] The server acquires behavioral data in real time through hardware such as facial recognition cameras, voice-collecting microphones, and motion sensors installed within the educational space. The server processes this collected data using information analysis tools to evaluate learners' comprehension and interest. Facial recognition technology can quantify the degree to which students are concentrating on the lesson, and voice analysis can measure the amount of conversation and participation.

[0580] Subsequently, the server uses an action generation mechanism to determine learning support actions based on the analysis results, and then uses a generative AI model to generate appropriate communication content. For example, a prompt such as "What kind of support should be provided if the student cannot answer the question?" is input to the AI, and specific instructions are formulated based on the response obtained.

[0581] The terminal controls automated devices according to instructions received from the server, enabling interaction with human learners. This device functions as a robotic student, guiding learners through natural conversation and instructions. For example, it can prompt a student with a weak understanding of a math problem by saying, "Let's think about what's important here," and then ask additional questions to deepen their understanding.

[0582] The teacher, as a user, uses the feedback provided by the device to evaluate each student's learning progress and develop a teaching plan for the next lesson. This allows the teacher to take an approach that is tailored to each individual student, thereby improving learning efficiency.

[0583] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0584] Step 1:

[0585] The server receives real-time input of student behavior data from facial recognition cameras, audio microphones, and motion sensors placed throughout the classroom. This allows the server to collect data recording student seating arrangements, classroom behavior, and spoken content.

[0586] Step 2:

[0587] The server processes the behavioral data obtained in step 1 using information analysis tools. Specifically, it analyzes the students' level of concentration using facial recognition technology and evaluates the number of times they speak and their pitch through voice analysis. This data is integrated and analyzed to generate an output that quantifies each student's level of interest and understanding.

[0588] Step 3:

[0589] The server uses the analysis results as input to generate optimal learning support actions using an action generation mechanism. Using a generation AI model, it formulates the optimal communication method and instructional content based on prompt sentences. At this stage, prompts such as "How should we approach a student who is having trouble asking a question?" are used. This results in the output of an action plan that includes specific instructions.

[0590] Step 4:

[0591] The terminal controls the automated device using the output of the action plan from the server as input. The robot student prompts and responds to students during class. For example, to a student struggling with a math problem, it might instruct, "Let's go back to the basics and think about this step." This establishes a dialogue and enables the robot to take specific actions to support the student in solving the problem.

[0592] Step 5:

[0593] The teacher, as the user, receives feedback provided from the device. This feedback includes statistics to evaluate the effectiveness of the interaction and progress reports for each learner. Based on this output, the teacher can develop individualized instruction plans as input and optimize the educational content by adjusting students' learning plans.

[0594] (Application Example 1)

[0595] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0596] In the manufacturing sector, there is a need to improve worker efficiency and optimize the work environment. In particular, when introducing new machinery or adapting to different work methods, it is difficult for workers to immediately learn the optimal operating procedures, which can result in decreased productivity and safety issues. It is necessary to overcome these problems and create an efficient and safe work environment.

[0597] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0598] In this invention, the server includes data analysis means for analyzing worker work data, action generation means for generating appropriate work support actions based on the analyzed data, control means for controlling work support machines and interacting with workers based on the generated actions, and feedback collection means for recording interaction results and analyzing them to improve the work environment. This enables real-time work support for workers, improving work efficiency and ensuring safety.

[0599] "Worker" refers to an individual engaged in work related to manufacturing or production.

[0600] "Work data" refers to data collected during work, including information about the actions and movements of workers.

[0601] "Data analysis means" refers to a system that processes collected data and evaluates the work efficiency and performance of workers.

[0602] "Action generation means" refers to a system that determines the optimal method of work support for workers based on analyzed data.

[0603] "Work support machinery" refers to robots and other devices used to assist workers in their tasks.

[0604] "Control means" refers to a system that has the function of operating work assistance machinery based on the generated support actions.

[0605] "Interaction results" refer to data showing workers' responses to the support actions of work assistance machines and the effects of those responses.

[0606] A "feedback collection method" refers to a system that records interaction results and collects data for later analysis.

[0607] To implement this invention, a comprehensive system is needed to support workers in a factory environment. This system includes data analysis means, action generation means, control means, and feedback collection means.

[0608] First, the server uses sensors installed throughout the factory to collect real-time data on the workers' activities. This data includes details such as their work style, speed, and environmental conditions. Based on this information, the data analysis system performs calculations necessary to analyze the efficiency and safety of each worker.

[0609] Next, the action generation system determines the optimal work support action based on the analysis results. This is done using a generative AI model, which plans specific support methods tailored to the worker's abilities and the nature of the work. For example, it can provide step-by-step guidance to a worker unfamiliar with operating new machinery.

[0610] Subsequently, the control system operates the work assistance machinery based on the generated actions and interacts effectively with the worker. This ensures that the guidance the worker receives is applied to the real environment and is reliably useful on the spot.

[0611] Furthermore, feedback collection mechanisms record the results of interactions. This data is used for analysis on the server and helps in the continuous improvement of the work environment. This also enables workers to perform tasks safely and efficiently.

[0612] As a concrete example, a prompt message such as "Do you need support while working with the new equipment?" might be displayed via smartphone to a worker unfamiliar with new operating procedures. This prompt message provides quick support for any challenges the worker may face.

[0613] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0614] Step 1:

[0615] The server utilizes sensors within the factory to collect worker activity data in real time. The input is signals from the sensors, which are converted into digital data. The output is raw data indicating the worker's activity status and environmental conditions. This raw data specifically includes information about each worker's position, movement, and speed.

[0616] Step 2:

[0617] The server processes the collected work data using data analysis tools. The input is the raw data collected in step 1. Here, analysis algorithms are used to detect data trends and anomalies, and to evaluate work efficiency and safety. The output is the analysis results for generating work support actions. This includes performance indicators and required support points for each worker.

[0618] Step 3:

[0619] The action generation mechanism uses the server's analysis results to generate optimal work support actions for each worker. The input is the analysis results from step 2. The generation AI model is used to perform data calculations to create a plan for support actions. The output is a support scenario for guidance at a specific work step or for new operating procedures.

[0620] Step 4:

[0621] The terminal receives the generated support actions and controls the work assistance machine. The input is the support action generated in step 3. It is converted into specific operating instructions and communicated to the machine in a way that is easy for the worker to understand. The output is the specific operation of the machine, which includes new machine operation guidance and warning messages.

[0622] Step 5:

[0623] The feedback collection mechanism allows the terminal to record the results of interactions with the work assistance machine. The input is interaction data between the machine and the worker. This data includes the worker's reactions and the machine's actual operation history. The output is a detailed interaction report sent back to the server. This report is used for further analysis to improve the work environment.

[0624] Step 6:

[0625] Users improve their work environment and operating procedures based on the feedback they receive. The input is interaction reports received from the server. The data is used to develop improvement strategies to clearly identify areas for improvement. The output is the improved work procedures and environment settings.

[0626] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0627] This invention relates to a robotic student system that functions effectively in a classroom and aims to recognize user emotions and provide appropriate educational support by combining it with an emotion engine. The system consists of a server and robotic terminals, and each component operates in conjunction with the others.

[0628] The server first collects facial expressions and voice data from sensor devices installed in the classroom. The collected data is analyzed by the system's emotion engine to understand students' emotional states and psychological needs in real time. This analysis provides information such as whether a particular student is experiencing anxiety or showing interest in the lesson.

[0629] Next, the server uses the information obtained from the emotion engine to formulate educational support methods using behavior generation mechanisms. For example, for a student who is feeling anxious, it generates a plan in which a robot student offers words of encouragement in a gentle voice. These plans are designed to be flexible enough to accommodate students who require support in different languages.

[0630] The robot student, acting as a terminal, receives instructions from the server and then performs the designated actions within the classroom. For example, if a student appears depressed, the robot student will approach that student and ask, "Is there anything I can do to help?", providing support through friendly interaction.

[0631] Teachers and other students who are users of these robot students can improve the learning environment for the entire class with their support. In particular, the emotion recognition-based feedback function allows teachers to understand the emotional state of each student and use that information to devise more appropriate teaching methods.

[0632] As a concrete example, consider a scenario in a language class where a student with a different cultural background experiences language barriers and feels isolated. The emotion engine analyzes the student's anxious facial expressions. Based on this, the robot student can provide support in the student's native language and create opportunities for interaction with other students.

[0633] Thus, the robot student system of the present invention achieves effects such as improving interpersonal relationships in the classroom, preventing bullying, and deterring classroom disruption, as well as drawing out potential abilities.

[0634] The following describes the processing flow.

[0635] Step 1:

[0636] The server continuously collects students' facial expressions and voice data from sensor devices such as cameras and microphones installed in the classroom. The collected data is temporarily stored and prepared for analysis.

[0637] Step 2:

[0638] The server uses an emotion engine to analyze the collected data. It estimates students' emotional states from their facial expressions and voice tone, and monitors changes in anxiety, excitement, and concentration levels in real time.

[0639] Step 3:

[0640] The server generates educational support behaviors tailored to individual students based on emotional data estimated by the emotion engine. Specifically, it creates support plans that include encouraging words in a gentle voice and questions that are engaging.

[0641] Step 4:

[0642] The server sends the generated educational support actions as instructions to the robot student. These instructions include specific conversation topics with the student and guidelines for action.

[0643] Step 5:

[0644] The terminal (robot student) moves around the classroom based on instructions received from the server and makes contact with the target student. The robot student speaks to the student using designated phrases and gestures and provides follow-up.

[0645] Step 6:

[0646] The user (student) interacts with the robot student after receiving contact from it. The student's responses and new emotional states are recorded as additional data.

[0647] Step 7:

[0648] The server evaluates the interaction between robot students and human students and generates feedback from the recorded data to help improve the educational environment and methods. Users (teachers) can then use this feedback to develop more appropriate teaching strategies.

[0649] (Example 2)

[0650] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0651] Traditional educational support systems have been insufficient in real-time emotional recognition and providing appropriate feedback to individual learners, making it difficult to flexibly accommodate learners with diverse linguistic and cultural backgrounds. As a result, there is a challenge in that learners' psychological needs cannot be met, hindering the improvement of learning effectiveness.

[0652] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0653] In this invention, the server includes information analysis means, action planning means, and emotion analysis means. This makes it possible to grasp the emotional state of human learners in a learning space in real time and provide support tailored to their individual educational needs.

[0654] An "information analysis tool" is a device that collects and analyzes learners' behavior, facial expressions, and voice data to identify their emotional state and psychological needs in real time.

[0655] An "action planning tool" is a device that generates a plan to design appropriate educational activities and effectively intervene in the learning environment based on analyzed information.

[0656] A "control device" is a device that operates a machine learning device based on a generated action plan and interacts with the learner.

[0657] A "feedback acquisition method" is a device that records the results of conversations and analyzes the collected information in order to improve the learning environment.

[0658] An "emotion analysis device" is a device that analyzes a learner's facial expressions and voice data to determine their psychological state in real time.

[0659] A "generative AI model" is a model that uses machine learning algorithms to flexibly adjust action plans, and is a device that supports decision-making within a system.

[0660] This system is an educational support system that uses machine learning devices to function effectively within a learning environment. The server collects learners' facial expressions and voice data through sensor devices installed in the classroom. This data is analyzed in real time by emotion analysis capabilities on the server to determine the learners' emotional state and psychological needs. The hardware used includes sensor devices such as cameras and recording devices like microphones, and the server requires powerful computing resources to process the data from these devices.

[0661] Based on the information obtained through emotion analysis, the server generates individual support plans using an action planning mechanism. The action planning mechanism uses a generative AI model to design optimal educational support from the analysis results. For example, if emotion analysis determines that a learner is experiencing stress, the generative AI model will suggest ways to talk to that learner to help them relax. An example of a prompt in this context would be, "Learner A appears to be experiencing stress. What should I say to them?"

[0662] The generated action plan is sent to the robot student, which acts as a terminal, and implemented in the classroom as an actual educational support activity. The robot student follows the instructions received and provides gentle encouragement and specific support to the learners. These activities are carried out through control mechanisms, which play a role in improving the quality of interaction with learners.

[0663] Furthermore, the user, the teacher, records the results of these interactions using feedback mechanisms and utilizes them to improve the learning environment. As described above, this system has a mechanism to provide more appropriate and flexible educational support through the use of sentiment analysis and generative AI models.

[0664] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0665] Step 1:

[0666] The server collects learners' facial expressions and voice data from sensor devices installed in the classroom. Inputs are camera and microphone data, and this information is aggregated on the server as output. This data collection enables real-time emotion analysis. Specifically, the server acquires this data at regular intervals and stores it in a buffer.

[0667] Step 2:

[0668] The server passes the collected facial and audio data to the emotion analysis system. The input is the student's facial and audio data collected in step 1, and the output is the analysis result identifying the student's emotional state. The server uses machine learning algorithms to analyze the data and determine emotions such as "anxiety," "interest," and "confusion" in real time. Specific operations include data formatting and feature extraction during the analysis.

[0669] Step 3:

[0670] The server generates a support plan via an action planning mechanism based on the results of the emotion analysis. The input is the emotion analysis results from step 2, and the output is a specific educational support plan. This plan is formulated using a generative AI model and is output in the form of, for example, "Learner B is feeling anxious, so offer words of encouragement." The specific actions here are inputting prompt sentences to the generative AI model and generating support suggestions.

[0671] Step 4:

[0672] The server sends the generated support plan to the robot terminal. The input is the support plan generated in step 3, and the output is the instructions to the robot terminal. The server sends the instructions based on the plan to the robot wirelessly and prepares the robot to execute them at the appropriate time. Specific actions include communication execution and serialization of instruction data.

[0673] Step 5:

[0674] The robot, acting as the terminal, performs interactions within the classroom based on instructions received from the server. The input is instruction data from the server, and the output is direct dialogue with the learner. Specifically, the robot might approach and say, "Let's solve this problem together."

[0675] Step 6:

[0676] The user, the teacher, observes the interaction between the robot and the learners and records the results using feedback mechanisms. The input is classroom events and learners' responses, and the output is recorded feedback data. This data will be used to improve future educational support. Specific actions include inputting data into record sheets or digital forms.

[0677] (Application Example 2)

[0678] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0679] In today's work environment, the mental and physical health of workers is directly linked to work efficiency and safety. However, many factories and workplaces lack mechanisms to properly monitor workers' emotional states and stress levels and provide support based on that information. This situation can lead to decreased worker productivity and safety problems. Therefore, there is a need for a system that can improve the work environment by understanding workers' emotional states in real time and providing optimal support.

[0680] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes data analysis means for analyzing the emotional state of the worker, action generation means for generating appropriate support actions based on the analyzed emotional state, and control means for controlling the support device and interacting with the worker based on the generated actions. As a result, the support device in the workplace can analyze the emotional state of the worker in real time and provide appropriate verbal encouragement and advice, thereby improving the work environment.

[0681] A "data analysis device" is a device equipped with the function of collecting and analyzing information necessary to understand the emotional state of workers.

[0682] A "behavior generation device" is a device that has the function of designing appropriate support actions for workers based on analyzed data.

[0683] A "control means" is a device that operates a support device according to the generated support action and has the function of interacting with the worker.

[0684] A "feedback collection device" is a device that records the results of interactions with support devices and uses them for analysis to further improve the work environment.

[0685] A "support device" is a device that can concretize and actually implement support actions based on the emotional state of the worker.

[0686] In the system for realizing this invention, a server plays a central role. The server acquires facial expressions and voice data of workers from various sensors installed in the work environment. Hardware such as cameras and microphones are used for this purpose. The acquired data is analyzed in real time using emotion analysis software installed on the server. The emotion analysis software can utilize, for example, a Python-based emotion analysis library.

[0687] Once the emotional state of the user (worker) is analyzed, the server uses an action generation algorithm to formulate appropriate support actions. This algorithm designs break suggestions and work-related advice based on the worker's stress and fatigue levels. Communication in different languages ​​is also possible, utilizing a multilingual speech generation engine.

[0688] Subsequently, the server operates the support device via a control system, interacting with the worker. The control system includes actuators and speakers, providing voice output for natural dialogue. The results of the interactions performed by the support device are recorded by a feedback collection system and analyzed for further optimization of the work environment.

[0689] As a concrete example, considering its use in a manufacturing plant, if a sensor detects worker fatigue, the server can provide support such as, "You seem tired. Would you like to take a short break?" An example of a prompt message used in this case would be, "We analyze the worker's emotional state in real time and provide appropriate break suggestions if stress or fatigue is detected. This is supported in Japanese or other languages ​​as needed."

[0690] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0691] Step 1:

[0692] The server captures the worker's facial expressions and voice using cameras and microphones installed in the work environment. The input is raw data obtained through the cameras and microphones, which is then sent to the server. The output consists of analyzable image and audio data.

[0693] Step 2:

[0694] The server uses an emotion analysis library to analyze the captured image and audio data. Here, facial features are extracted from the image data, and voice tone and speed are analyzed from the audio data. The input is the data from step 1, and the output is an evaluation score indicating the worker's emotional state. This process quantifies, for example, levels of stress and fatigue.

[0695] Step 3:

[0696] The server executes an action generation algorithm based on the evaluation score. This algorithm uses predefined rules and machine learning models to formulate the optimal support actions to provide to the worker. The input is the evaluation score obtained in step 2, and the output is a specific action plan such as "suggestion for a break" or "advice to improve work efficiency."

[0697] Step 4:

[0698] The server instructs the support device, via a control mechanism, on the action plan formulated by the action generation algorithm. The support device uses a speaker to perform voice actions and attempts to engage in natural dialogue with the worker. The input is the action plan from step 3, and the output is the voice action.

[0699] Step 5:

[0700] The terminal records the results of the interaction in the feedback collection system. This recorded data is then analyzed by the server for future system improvements. The input is the details of the actual interaction by the support device, and the output is analyzable feedback data.

[0701] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0702] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0703] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0704] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0705] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0706] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0707] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0708] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0709] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0710] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0711] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0712] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0713] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0714] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0715] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0716] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0717] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0718] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0719] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0720] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0721] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0722] The following is further disclosed regarding the embodiments described above.

[0723] (Claim 1)

[0724] It was established with the aim of learning alongside human students, and includes data analysis tools for analyzing student behavior in the classroom,

[0725] A behavior generation means that generates appropriate educational support behaviors based on analyzed data,

[0726] A control means that controls a robot student based on generated behavior and allows it to interact with a human student,

[0727] A feedback collection method that records interaction results and analyzes them to improve the educational environment,

[0728] A system that includes this.

[0729] (Claim 2)

[0730] The system according to claim 1, wherein the behavior generation means further includes means for enabling communication in different languages.

[0731] (Claim 3)

[0732] The system according to claim 1, wherein the control means further includes means for generating emotional responses based on the facial expressions and voices of human students and causing them to perform empathetic behavior.

[0733] "Example 1"

[0734] (Claim 1)

[0735] It was established with the aim of learning alongside human learners, and provides information analysis tools for analyzing learner behavior within the educational environment,

[0736] A behavior generation means that generates appropriate learning support behaviors based on analyzed information,

[0737] A generation method that generates communication methods based on learner needs using a generative AI model,

[0738] A control means that controls an automated device based on generated behavior and interacts with a human learner,

[0739] A means for collecting results that record interaction outcomes and analyze them for improvement of the educational environment,

[0740] A system that includes this.

[0741] (Claim 2)

[0742] The system according to claim 1, wherein the generating means further includes a medium that enables communication in different languages.

[0743] (Claim 3)

[0744] The system according to claim 1, further comprising a method for the control means to generate emotional responses based on the emotions and voice of a human learner and to cause the learner to perform empathetic behavior.

[0745] "Application Example 1"

[0746] (Claim 1)

[0747] A data analysis method for analyzing worker work data,

[0748] Action generation means for generating appropriate work support actions based on analyzed data,

[0749] A control means that controls work-assisting machines based on generated actions and interacts with workers,

[0750] A feedback collection method that records interaction results and analyzes them for improvement of the work environment,

[0751] A system that includes this.

[0752] (Claim 2)

[0753] The system according to claim 1, wherein the action generation means further includes means for supporting different work methods.

[0754] (Claim 3)

[0755] The system according to claim 1, wherein the control means further includes means for causing the worker to perform supportive actions based on the worker's movements and reactions.

[0756] "Example 2 of combining an emotion engine"

[0757] (Claim 1)

[0758] An information analysis tool that operates in cooperation with human learners within a learning space and analyzes learner behavior,

[0759] Based on the analyzed information, a means of creating an action plan for developing appropriate educational support actions,

[0760] A control means for operating a machine learning device based on the created action plan and conducting dialogue with the learner,

[0761] A means of obtaining feedback by recording the results of the dialogue and analyzing them to improve the learning environment,

[0762] A means of analyzing emotions to determine psychological state in real time,

[0763] A means of using a generative AI model to flexibly adjust the action plan within the system,

[0764] A system that includes this.

[0765] (Claim 2)

[0766] The system according to claim 1, wherein the action planning means further includes means for enabling communication in different languages.

[0767] (Claim 3)

[0768] The system according to claim 1, wherein the information analysis means further includes means for calculating a psychological response based on the learner's facial expressions and voice, and causing the learner to perform an action that indicates emotion.

[0769] "Application example 2 when combining with an emotional engine"

[0770] (Claim 1)

[0771] A data analysis method for analyzing the emotional state of human workers,

[0772] A behavior generation means that generates appropriate supportive behaviors based on the analyzed emotional state,

[0773] A control means that controls the support device based on the generated actions and interacts with the worker,

[0774] A feedback collection method that records interaction results and analyzes them for improvement of the work environment,

[0775] A system that includes this.

[0776] (Claim 2)

[0777] The system according to claim 1, wherein the behavior generation means further includes means for enabling communication in different languages.

[0778] (Claim 3)

[0779] The system according to claim 1, wherein the control means further includes means for generating an emotional response based on the worker's facial expressions and voice, and causing the worker to perform empathetic actions. [Explanation of Symbols]

[0780] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. It was established with the aim of learning alongside human students, and includes data analysis tools for analyzing student behavior in the classroom, A behavior generation means that generates appropriate educational support behaviors based on analyzed data, A control means that controls a robot student based on generated behavior and allows it to interact with a human student, A feedback collection method that records interaction results and analyzes them to improve the educational environment, A system that includes this.

2. The system according to claim 1, wherein the behavior generation means further includes means for enabling communication in different languages.

3. The system according to claim 1, wherein the control means further includes means for generating emotional responses based on the facial expressions and voices of human students and causing them to perform empathetic behavior.

Images