system

Abstract

We provide the system. [Solution] A data processing means for collecting user profile information and generating educational content based on the user's learning tendencies and interests, A means of providing content to users using virtual reality or augmented reality technology to create an immersive learning environment, An emotion recognition processing means for detecting the user's emotional state and adjusting learning support accordingly, A means of gamifying the learning process to motivate learning and managing user feedback, A system including a means of communication with a mechanical device for physically performing tasks within a virtual scenario.

Description

Technical Field

[0004] , , , ,

[0005] , , , , ,

[0003] , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern education and corporate training settings, there is a demand to provide personalized education according to the characteristics and abilities of individual learners. However, existing systems have problems such as difficulty in generating appropriate content based on data, difficulty in creating an immersive experience and maintaining learning motivation for a long time. In addition, the mechanism for providing support according to the emotional state during the learning process is insufficient, and furthermore, there is a lack of physical experience for connecting the learning content to reality.

Means for Solving the Problems

[0005] Note: The number in the Japanese Patent Application Laid-Open No. 2022-**** is replaced with ****, because the specific number is not provided in the original text.The present invention includes data processing means for collecting user profile information and generating educational content based on individual learning tendencies and interests. It also includes means for providing users with an immersive learning environment using virtual reality or augmented reality technology. Furthermore, it detects the user's emotional state using emotion recognition technology and adjusts learning support accordingly. It integrates gamification techniques for learning motivation and enhances user feedback to encourage continued learning. In addition, it includes means for communicating with mechanical devices for physically executing tasks within virtual scenarios, enabling the connection of learning content to actual work and real-world experiences.

[0006] "User profile information" refers to data about a user's learning history, interests, abilities, and other individual characteristics.

[0007] "Educational content" refers to educational materials and learning resources that users use for learning and skill improvement.

[0008] "Data processing means" refers to methods and technologies for analyzing collected data and generating, transforming, storing, and providing information based on that analysis.

[0009] "Virtual reality technology" refers to technology that allows users to experience a virtual environment created by computer technology as if it were reality.

[0010] Augmented reality technology refers to the technology that combines and displays information from the real world with digital information generated by computers.

[0011] An "immersive learning environment" refers to an interactive and dynamic learning space designed to allow users to fully concentrate on their learning.

[0012] "Emotion recognition technology" refers to technology that identifies a user's emotional state from their facial expressions, voice, and behavior.

[0013] "Gamification techniques" refer to strategies that enhance user motivation and enrich the learning experience by incorporating game elements into the learning process.

[0014] A "virtual scenario" refers to a hypothetical situation or scenario prepared as part of learning or training.

[0015] "Mechanical device" refers to a machine or device used to perform the physical tasks related to the learning content. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12]It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the language used in the following description will be explained.

[0019] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a RAM (Random Access Memory) with a reference numeral is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention is a system for providing users with a personalized learning experience in educational and corporate training settings. The system consists of a server, terminals, and users.

[0038] The server first collects user profile information. This profile information includes past learning history, interests, and abilities. Based on this data, the server analyzes the user's learning tendencies and generates educational content optimized for the user. The generated content is then sent from the server to the device.

[0039] The device presents educational content received from the server to the user using virtual reality or augmented reality technology. This allows the user to learn in an immersive learning environment. The device also tracks the user's location and movements in real time and adjusts the educational content accordingly.

[0040] The system further monitors the user's emotional state using emotion recognition technology. The server provides support and adaptive instructions tailored to the user's emotions, reducing user stress and improving learning efficiency.

[0041] Furthermore, the server applies gamification techniques to enhance motivation for learning. By providing feedback such as points and badges, it encourages continued learning by giving users a sense of accomplishment.

[0042] As a concrete example, consider a scenario involving new employee training at a certain company. In this case, the server creates a profile of the new employee and generates an individualized learning plan. The terminal uses virtual reality technology to provide virtual training that closely resembles actual work. As the user progresses through the learning process, the server uses emotion recognition technology to understand the new employee's stress level and provides support as needed. Furthermore, the server awards points based on the learning progress and displays them on the terminal to further increase motivation.

[0043] Ultimately, the server controls mechanical devices to physically execute tasks within the virtual scenario, enabling users to apply what they've learned in real-world work environments. This makes it easier to translate learning outcomes into reality.

[0044] The following describes the processing flow.

[0045] Step 1:

[0046] The server collects profile information, learning history, and interest data submitted by the user. This data is stored in a database and used for later analysis.

[0047] Step 2:

[0048] The server runs an algorithm that analyzes the user's learning tendencies based on the collected data. This analysis provides the information necessary to create educational content optimized for the user.

[0049] Step 3:

[0050] The server generates individually optimized educational content based on the analysis results. This content is tailored to each user's abilities and interests and is then sent from the server to their device.

[0051] Step 4:

[0052] The device presents educational content transmitted from the server to the user using virtual reality or augmented reality technology. This allows the user to have an immersive learning experience.

[0053] Step 5:

[0054] The user engages in learning activities within a virtual or augmented reality environment, following interactive instructions presented by the device. During this time, the device tracks the user's location and movements in real time.

[0055] Step 6:

[0056] The server monitors the user's emotional state using emotion recognition technology. If the user is experiencing stress, the server adaptively modifies the learning plan and provides appropriate support.

[0057] Step 7:

[0058] The server applies gamification techniques to enhance learning motivation. Points and badges are awarded based on the user's progress, and the device displays these to provide visual feedback on the user's progress.

[0059] Step 8:

[0060] The server communicates with the machine to handle tasks within the virtual scenario. This allows users to perform tasks relevant to their real-world work environment and apply what they have learned to real-world situations.

[0061] (Example 1)

[0062] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0063] Modern education and corporate training require the provision of individualized learning experiences that cater to each learner's characteristics and interests. However, conventional systems struggle to provide optimal educational information based on individual learner profiles, adjust support according to their emotions, and appropriately motivate them to learn. Furthermore, applying activities in a virtual environment to a real-world work environment is not easy. This invention aims to solve these problems.

[0064] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0065] In this invention, the server includes information processing means for collecting user attribute information and generating educational information based on the user's learning tendencies and interests; means for providing information to the user using extended environment technology to construct a virtual environment; and emotion recognition means for detecting the user's emotions and adjusting support accordingly. This enables the provision of personalized learning information and the optimization of an adaptive learning experience in real time.

[0066] "User attribute information" is a general term for information about an individual user, such as their learning history, interests, and skill level.

[0067] "Educational information" refers to learning materials and content that are customized based on the user's learning needs and goals.

[0068] "Information processing means" refers to methods for collecting, analyzing, and generating data, and specifically, to algorithms and software that operate on computer systems.

[0069] A "virtual environment" refers to an artificial environment created using computer technology, an immersive space that allows users to experience things beyond physical limitations.

[0070] "Augmented environment technology" refers to technology that overlays virtual information onto the real world, allowing users to perceive both reality and the virtual simultaneously.

[0071] "Emotion recognition means" refers to technologies that grasp a user's emotional state from their facial expressions, voice, and actions, and are a means of estimating the user's psychological state.

[0072] "Personalized learning information" refers to learning content tailored to each user's characteristics, and is information provided according to the user's specific needs and goals.

[0073] An "adaptive learning experience" refers to a learning process in which the content and methods are dynamically adjusted according to the learner's real-time situation and needs.

[0074] This invention is a system for providing a user-optimized learning experience, involving three parties: a server, a terminal, and the user. Specific embodiments of this system are described below.

[0075] The server collects user attribute information and analyzes it using information processing tools. A database management system is used to aggregate data on the user's learning history, interests, and skill level. Then, a generative AI model is used to generate personalized educational information for each user. This generation process employs natural language processing algorithms and machine learning techniques to create optimal content tailored to each user's needs.

[0076] The generated educational information is sent from the server to the user's device. The device then presents the content to the user using augmented reality (AR) technology. Specifically, by using AR headsets or VR devices, users can experience a real-time, interactive learning environment. This technology allows users to engage in practical learning in a virtual space without being tied to a physical location.

[0077] Furthermore, the server uses emotion recognition to detect the user's emotions. Based on information obtained from the user's facial expressions and voice, the user's psychological state is analyzed. Based on this data, the server can adjust the content of learning support, reducing the user's burden while providing effective education.

[0078] Furthermore, by employing gamification techniques, the server is designed to enhance user motivation for learning. A point system and badge system are incorporated, providing real-time feedback based on learning progress. This allows users to experience a sense of accomplishment and encourages them to continue learning.

[0079] As a concrete example, in new employee training, the server creates a profile based on the new employee's information and generates an individualized training plan. Based on the generated plan, the terminal uses AR technology to provide hands-on training that closely resembles actual work.

[0080] An example of an input prompt for a generative AI model is: "Design a customized virtual training material for new employees. Incorporate an appropriate point system based on their past learning history and current skills."

[0081] As described above, the entire system works together to provide users with a personalized learning experience.

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The server collects user attribute information. This input includes data such as the user's learning history, interests, and skill level. The server stores this data in a database and, as a concrete action to prepare for later analysis, provides an online user profile creation form and stores the information entered by the user with security in mind.

[0085] Step 2:

[0086] The server analyzes learning trends based on the collected user attribute information. In this step, a data analysis algorithm is applied to the profile information as input, and the output is a quantified result of the user's learning style and interests. Specifically, a machine learning model is used to cluster the user dataset and identify patterns of different learning styles.

[0087] Step 3:

[0088] The server generates educational information using a generative AI model. In this step, the AI ​​model uses analyzed learning tendency data as input to design optimal educational content. The output is educational information customized specifically for that user. Specifically, this involves using natural language processing techniques to generate text and interactive learning materials.

[0089] Step 4:

[0090] The server sends the generated educational information to the terminal. The input is content information generated by AI, and the terminal receives that information as output. Specifically, the process involves securely transferring data over the network using encryption technology.

[0091] Step 5:

[0092] The terminal presents the received educational information to the user. In this step, the input is the educational information received from the server, and the output is the information the user receives visually or haptically through the interface. Specifically, an AR device using augmented reality technology displays the content, allowing the user to interact with it.

[0093] Step 6:

[0094] The server collects and analyzes emotional data from the terminal in real time. The input is the user's facial expressions and voice data, and the output is the analyzed emotional state. Specifically, an emotion recognition algorithm evaluates changes in voice tone and facial expressions to determine stress and interest levels.

[0095] Step 7:

[0096] The server provides the user with feedback and support based on emotional data. The input for this step is data obtained through emotion recognition, and the output is learning content and support methods tailored based on that analysis. A concrete example of its operation is when the user is experiencing stress; in such cases, relaxation music might be played.

[0097] (Application Example 1)

[0098] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0099] To promote effective learning in educational and corporate training settings, it is crucial to provide instruction tailored to individual characteristics and to acquire practical skills. However, conventional methods struggle to provide an adaptive learning environment that reflects the tendencies and emotional states of individual learners, and there are also challenges in how to apply the learned content to actual work environments. This invention aims to solve these problems and provide a more efficient and practical learning experience.

[0100] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0101] In this invention, the server comprises information processing means for collecting user characteristic information and generating educational materials based on the user's learning tendencies and interests; means for providing materials to the user using virtual reality or augmented reality technology and constructing an immersive learning environment; emotion recognition processing means for detecting the user's emotional state and adjusting learning support accordingly; means for evaluating the user's actions and state and providing immediate support for tasks in the virtual environment; and means for providing instruction related to the real world in order to practically learn skills in a work environment. This makes it possible to provide each individual learner with an appropriate learning experience and to streamline the acquisition of skills that can be immediately applied in actual workplaces.

[0102] "User characteristic information" refers to information necessary to provide a personalized learning experience, such as each learner's learning history, interests, and abilities.

[0103] "Virtual reality technology" is a technology that presents a computer-generated virtual environment visually and aurally, enabling users to become immersed in it.

[0104] Augmented reality technology is a technology that extends the real environment by overlaying digital information onto real-world visual information.

[0105] "Emotion recognition processing means" refers to a technical means that analyzes the user's emotional state and selects and provides an appropriate response according to that state.

[0106] An "immersive learning environment" is a learning environment that allows users to deeply concentrate on the subject matter and provides an experience that feels as if they are actually present within it.

[0107] "Information processing means" is a general term for methods and technologies used to analyze and interpret collected data and generate desired deliverables.

[0108] "Immediate support" refers to providing real-time feedback and guidance to users regarding problems and questions they encounter during the learning process.

[0109] "Instruction related to the real world" refers to providing specific and practical guidance that enables users to apply what they have learned in the virtual environment to their actual work environment.

[0110] This invention provides a system for personalizing learning experiences in education and corporate training. This system mainly consists of a server, terminals, and users.

[0111] The server first collects user characteristic information, including learning history, interests, and abilities. Using information processing tools, the server analyzes the user's learning tendencies from the collected data and generates optimal educational materials.

[0112] The generated educational materials are delivered to the user via a terminal using virtual reality or augmented reality technology. The user progresses through this immersive learning environment, and their emotional state is continuously monitored in real time by an emotion recognition processing system. Based on this information, the server dynamically adjusts learning support, extracting and providing gamified elements and immediate assistance to enhance the user's motivation and understanding.

[0113] As a concrete example, consider a scenario where a new employee at a manufacturing company undergoes practical training. The user operates a virtual production line via a head-mounted display, learning the safe handling and efficient operation methods of specific machinery. The system provides personalized guidance based on the user's actions and emotions, supporting the learning process.

[0114] Furthermore, when using generative AI models to create real-time scenarios that provide user performance feedback and suggest next steps, the following prompt is used: "Create a scenario in which a new employee simulates a manufacturing line in a VR environment, using sentiment analysis and gamification elements to support learning."

[0115] This system provides a flexible learning environment that takes individual characteristics into account, enabling the practical application of skills.

[0116] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0117] Step 1:

[0118] The server collects user characteristic information. Specifically, it retrieves data from a database based on user input regarding the user's past learning activities, areas of interest, and abilities. This information is processed to analyze the user's past learning trends and used as input data for the next step.

[0119] Step 2:

[0120] The server generates optimal educational materials based on the user's characteristics information obtained in Step 1. Using data processing tools, the server assembles the relevant educational content in digital format and designs an immersive learning experience using virtual reality or augmented reality technology. This generated educational material becomes the output data for the terminal.

[0121] Step 3:

[0122] The device provides the user with educational materials received from the server. The device utilizes virtual reality or augmented reality technology to present the content in an interactive and immersive way. The user begins learning in this environment, and the device senses the user's input and provides immediate feedback.

[0123] Step 4:

[0124] The server monitors the user's emotional state using emotion recognition technology. It analyzes the user's facial expressions and voice as input to identify their stress levels and comprehension level. Based on these analysis results, it adjusts the content and difficulty level of learning support and outputs it to the terminal.

[0125] Step 5:

[0126] The server uses a generative AI model to provide real-time learning feedback and suggestions for the next steps. It processes user performance data as input and generates gamified elements (badges, points, etc.) and specific improvement suggestions. This result is then displayed on the user's device as final feedback.

[0127] Step 6:

[0128] Users apply what they've learned to real-world tasks through their terminals. Practical guidance from terminals and servers is integrated, making it possible to reproduce skills acquired in the virtual environment in real-world situations. Specific operating procedures and safety instructions are output, which users use to perform their tasks.

[0129] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0130] This invention provides a system for providing users with personalized learning experiences in education and corporate training. It incorporates an emotion engine to understand the user's emotional state and utilize that information to optimize the learning process. The system consists of a server, terminals, and the emotion engine.

[0131] The server first collects user profile information and analyzes the user's learning tendencies using learning history, interests, and past performance data. Based on this analysis, it generates individually optimized educational content and sends this content to the user's device.

[0132] The device presents educational content provided by the server to the user using virtual reality or augmented reality technology. Users can gain an immersive learning experience through the device, and the system tracks the user's location and movements in real time. This tracking data helps the system further refine the educational content.

[0133] The emotion engine analyzes the user's facial expressions, voice, and biometric information to recognize the user's emotional state in real time. Based on this emotional state, the server dynamically adjusts the user's learning process. For example, if the server determines that the user is experiencing stress, it can temporarily suspend the learning plan or take measures to alleviate the stress. Furthermore, based on the emotion engine's data, the device provides personalized feedback to the user and evaluates their learning progress.

[0134] As a concrete example, if this system were implemented in a company's new employee training program, the new employees would receive instruction through virtual reality technology. During this time, the emotion engine would continuously monitor each employee's emotional state. For instance, if an employee's attention becomes distracted during learning, the system would detect this and suggest that the employee take a break or present the content in a simplified form. In this way, new employees can learn at their own pace and efficiently acquire the information.

[0135] Ultimately, the server comprehensively analyzes the data obtained upon the user's completion of the learning process and provides the results to the training staff, which can then be used to improve subsequent educational programs. This allows for continuous support in improving learner performance.

[0136] The following describes the processing flow.

[0137] Step 1:

[0138] The server collects profile information from users. This includes learning history, interests, and past performance data, which is stored in a database.

[0139] Step 2:

[0140] The server analyzes the collected profile information and generates educational content based on the user's learning tendencies and interests. This content is customized and optimized for each user.

[0141] Step 3:

[0142] The server sends the generated educational content to the device. The device receives this content and prepares to present it to the user.

[0143] Step 4:

[0144] The device uses virtual reality or augmented reality technology to present educational content to the user, allowing them to begin an immersive learning experience.

[0145] Step 5:

[0146] The emotion engine analyzes the user's facial expressions, voice, and biometric information in real time to determine their emotional state. This data is then sent to the server.

[0147] Step 6:

[0148] The server dynamically adjusts the user's learning process based on emotional data obtained from the emotion engine. For example, if it determines that the user is fatigued, it suggests taking a break.

[0149] Step 7:

[0150] As users engage in learning activities, the device tracks their location and movements, sending this data to the server. This allows for further adjustments to the content.

[0151] Step 8:

[0152] The server provides users with gamified feedback to enhance learning motivation. Points and badges are awarded and displayed on the device based on achievements.

[0153] Step 9:

[0154] Upon completion of the training, the server comprehensively analyzes all training data and evaluates the user's learning performance. The results are then saved as data for further improvement.

[0155] (Example 2)

[0156] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0157] In today's educational and training environments, providing flexible learning plans tailored to the individual characteristics and emotional states of learners, and ensuring their efficient application, presents a significant challenge. In particular, because learners have diverse learning paces and styles, uniform educational content makes effective learning difficult. Furthermore, learning processes that disregard emotional states can lead to decreased learner motivation. Therefore, addressing these challenges is essential to maximizing learning outcomes.

[0158] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0159] In this invention, the server includes information processing means for collecting and generating user characteristic information, means for providing content to the user using virtual representation technology or extended representation technology, and emotion recognition processing means for detecting and adjusting the user's emotional state. This enables the provision of an optimal learning environment for each learner in real time, thereby improving learning effectiveness.

[0160] "Characteristic information" refers to specific information about the user, including learning history, interests, and individual learning style.

[0161] "Information processing means" refers to hardware and software systems for collecting, analyzing, and generating educational content.

[0162] "Virtual representation technology" refers to technology that allows users to immerse themselves in a computer-generated three-dimensional environment, thereby enabling a realistic learning experience.

[0163] "Augmented representation technology" refers to technology that overlays digital information onto the real world, providing an intuitive learning experience in a real-world environment.

[0164] "Emotion recognition processing means" refers to technology that analyzes a user's facial expressions, voice, and biometric data to determine their emotional state in real time.

[0165] A "learner" refers to an individual who utilizes an educational system, and whose learning environment and content are individually optimized for that person.

[0166] This invention provides a specific form for implementing a system that provides individualized learning experiences in education and corporate training. The components of this system and their operation are described below.

[0167] System configuration:

[0168] 1. Server:

[0169] The server collects user characteristic information and performs analysis using information processing tools. This requires various databases and a computing platform to run machine learning algorithms. The content generation engine generates personalized educational content based on the analysis results.

[0170] The server can use emotion recognition processing to analyze biometric data provided by the user, determine the emotional state in real time, and reflect it in the learning content.

[0171] 2. Terminal:

[0172] A terminal is a device used to implement virtual or augmented representation technologies. This includes VR headsets and AR-enabled mobile devices. The terminal presents received educational content to the user in an immersive manner and tracks the user's actions in real time.

[0173] This tracking function enables the analysis of the user's location and movements, allowing for dynamic adjustments to the display of educational content.

[0174] 3. User:

[0175] A user is an individual who experiences the learning content through their device. Users can progress through their learning at their own pace through feedback from the system.

[0176] Users can also receive personalized content from the system based on their emotional state and learning outcomes.

[0177] Specific example:

[0178] For example, if this system is used for new employee training, new employees will receive simulation training in a workplace environment recreated in virtual reality. If a user experiences stress during the learning process, the server will detect this and suggest a break or flexibly adjust the learning content.

[0179] Example of a prompt:

[0180] "Please explain how the system responds when a user's attention becomes sluggish during a training session using virtual reality technology for new employees."

[0181] This format allows us to address the individual learning needs of each user and provide an efficient and effective learning experience.

[0182] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0183] Step 1:

[0184] The server collects user characteristic information. The input is profile data provided by the user during registration. This data includes learning history, areas of interest, and past evaluations. The server stores this input data in a database and manages it as structured data, representing the user profile. Specifically, the server receives and stores data using a secure communication protocol.

[0185] Step 2:

[0186] The server generates educational content based on collected characteristic information. The input is a user profile obtained from a characteristic information database. The server uses machine learning algorithms to analyze this input data and generate individually optimized educational content. The output is user-optimized educational content, consisting of videos, text, and interactive quizzes. Specifically, the server's content generation engine uses an AI model to create learning materials.

[0187] Step 3:

[0188] The server sends the generated educational content to the terminal. The input is optimized educational content generated within the server. The output is content data that can be displayed on the user's terminal. The server uses an end-to-end encryption protocol to securely deliver the content to the terminal. Specifically, the server uses the HTTPS protocol to deliver data to the terminal.

[0189] Step 4:

[0190] The device presents educational content to the user using virtual representation technology. The input is optimized educational content sent from the server. The device displays the received content as a VR or AR device, providing the user with an immersive learning environment. The output is the visual / auditory data experienced by the user. Specifically, the device renders a 3D virtual classroom using a VR headset.

[0191] Step 5:

[0192] The emotion engine monitors the user's emotional state. Inputs include user actions, facial expressions, and voice data. The emotion engine analyzes these inputs in real time to determine the user's emotional state. Outputs are status information related to the user's emotional state. Specifically, the emotion engine collects data through the camera and microphone and processes it using AI analysis tools.

[0193] Step 6:

[0194] The server adjusts the learning experience based on information obtained from the emotion engine. The input is status information regarding the user's emotional state. The server uses this to re-evaluate the learning plan and make changes as needed. The output is the adjusted educational content and learning plan, which may include relaxing music and pace adjustments. Specifically, the server restructures the content to reduce the user's emotional burden.

[0195] Step 7:

[0196] Users progress through customized learning content via their devices. Inputs consist of customized educational content and feedback. Users utilize this for self-study and improve their learning process based on the feedback. Outputs are feedback regarding learning progress and achievement. Specifically, users follow instructions on their devices and solve interactive problems.

[0197] (Application Example 2)

[0198] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0199] Traditional educational programs and skills training systems have struggled to adapt to individual users' emotional states and learning paces by providing uniform content. In particular, training for technicians to improve their machine operation and maintenance skills within factories requires individualized attention, but while technologies exist to grasp and adjust to diverse emotional states in real time, they are far from practical. Furthermore, when conducting training in a virtual environment, the inability to dynamically adjust content according to the user's emotional state hinders effective learning.

[0200] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0201] In this invention, the server includes information processing means for collecting basic user information and generating instructional content based on the user's learning patterns and interests; means for constructing an immersive educational environment by providing content to the user using virtual environment technology or extended environment technology; and emotion recognition processing means for understanding the user's emotional state and adjusting instructional support accordingly. This makes it possible to adjust the educational process to match the user's individual emotional state, enabling efficient and effective learning and training.

[0202] "User basic information" is a general term for information necessary to provide an individualized educational experience, including the user's characteristics and preferences, past learning history and achievements.

[0203] "Information processing means for generating instructional content" refers to processing devices and software that analyze a user's basic information and create and provide optimal educational content.

[0204] "Virtual environment technology or augmented environment technology" refers to technologies that use digital technology to create and provide users with an environment that does not actually exist but can be experienced visually.

[0205] An "immersive educational environment" refers to a learning space that provides an immersive educational experience designed to give users a sense of being in a place equivalent to physical reality.

[0206] "Emotion recognition processing means" refers to a device or algorithm used to analyze and determine a user's emotional state in real time.

[0207] "Adjusting instructional support" refers to the process of dynamically changing the educational process and content based on the user's emotional state and learning progress to ensure that learning progresses smoothly.

[0208] This invention is a system for providing users with an immersive, personalized learning experience. It aggregates user profile information and dynamically generates educational content. The server, as an information processing tool, collects basic user information and generates optimal instructional content based on that information. This process utilizes the user's past learning data and profile information. The server employs a powerful processor and data storage to analyze this data in real time.

[0209] The device provides users with a visual and interactive educational experience using virtual or augmented reality technologies. Specifically, it presents dynamically generated educational content using virtual reality (VR) or augmented reality (AR) devices. This allows users to learn at their own pace and immerse themselves in content tailored to their interests. This can be easily implemented using devices such as Oculus Quest 2 or AR applications on smartphones.

[0210] The user's emotional state is monitored by sensors and cameras on the device and analyzed through emotion recognition software such as the Affectiva SDK. Based on this real-time emotional data, the server adjusts the instruction content; for example, if the user is stressed, the content is made simpler or adjusted to be more relaxing. This maximizes the user's learning effectiveness.

[0211] For example, when a technician is learning to operate a specific machine in a factory, if emotional data detects a lack of concentration, the system will simplify the training content and help the technician regain confidence in their ability to concentrate again.

[0212] Examples of prompt statements in the AI ​​model generated by this system are as follows:

[0213] "For engineers with decreased concentration, how can we dynamically adjust VR training programs to optimize learning efficiency? Please advise."

[0214] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0215] Step 1:

[0216] The server collects basic user information. This includes user personal information, past learning history, and data on interests. This data is integrated to create profiles for generating educational content. The profiles stored in the database form the basis for providing instruction optimized for each user.

[0217] Step 2:

[0218] The server uses the generated profile to create instructional content tailored to the user. The input is the profile data obtained in step 1, and the output is optimized educational content. The generation AI model is used to create customized learning materials based on the user's learning tendencies. Specifically, the AI ​​model determines the difficulty level and theme of the content.

[0219] Step 3:

[0220] The device receives educational content provided by the server and presents it to the user using virtual environment technology. The input is the educational content from the server, and the output is a visual and interactive learning experience for the user. The device uses Oculus Quest 2 and AR applications to provide the user with an immersive educational environment.

[0221] Step 4:

[0222] The device monitors the user's emotional state in real time. Input is biometric information from sensors and cameras installed in the device, and output is the result of emotion recognition analysis. Analysis is performed using Affectiva SDK, etc., to determine the user's emotional state, such as whether they are focused or stressed.

[0223] Step 5:

[0224] The server adjusts the educational content as needed based on the results of the emotion recognition analysis. The input is the emotion data from step 4, and the output is the adjusted educational content. Specifically, if the user is feeling stressed, the server will either simplify the content or suggest a break for relaxation.

[0225] Step 6:

[0226] The user continues with the educational content based on feedback from the device. The input is the adjusted educational content and feedback from the device, and the output is the improvement in the user's learning progress. The device provides feedback visually or audibly to help the user learn more effectively.

[0227] Through the above processing steps, users can efficiently acquire knowledge and skills in a learning environment that best suits their own pace and emotional state.

[0228] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0229] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0230] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0231] [Second Embodiment]

[0232] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0233] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0234] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0235] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0236] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0237] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0238] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0239] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0240] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0241] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0242] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0243] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0244] This invention is a system for providing users with a personalized learning experience in educational and corporate training settings. The system consists of a server, terminals, and users.

[0245] The server first collects user profile information. This profile information includes past learning history, interests, and abilities. Based on this data, the server analyzes the user's learning tendencies and generates educational content optimized for the user. The generated content is then sent from the server to the device.

[0246] The device presents educational content received from the server to the user using virtual reality or augmented reality technology. This allows the user to learn in an immersive learning environment. The device also tracks the user's location and movements in real time and adjusts the educational content accordingly.

[0247] The system further monitors the user's emotional state using emotion recognition technology. The server provides support and adaptive instructions tailored to the user's emotions, reducing user stress and improving learning efficiency.

[0248] Furthermore, the server applies gamification techniques to enhance motivation for learning. By providing feedback such as points and badges, it encourages continued learning by giving users a sense of accomplishment.

[0249] As a concrete example, consider a scenario involving new employee training at a certain company. In this case, the server creates a profile of the new employee and generates an individualized learning plan. The terminal uses virtual reality technology to provide virtual training that closely resembles actual work. As the user progresses through the learning process, the server uses emotion recognition technology to understand the new employee's stress level and provides support as needed. Furthermore, the server awards points based on the learning progress and displays them on the terminal to further increase motivation.

[0250] Ultimately, the server controls mechanical devices to physically execute tasks within the virtual scenario, enabling users to apply what they've learned in real-world work environments. This makes it easier to translate learning outcomes into reality.

[0251] The following describes the processing flow.

[0252] Step 1:

[0253] The server collects profile information, learning history, and interest data submitted by the user. This data is stored in a database and used for later analysis.

[0254] Step 2:

[0255] The server runs an algorithm that analyzes the user's learning tendencies based on the collected data. This analysis provides the information necessary to create educational content optimized for the user.

[0256] Step 3:

[0257] The server generates individually optimized educational content based on the analysis results. This content is tailored to each user's abilities and interests and is then sent from the server to their device.

[0258] Step 4:

[0259] The device presents educational content transmitted from the server to the user using virtual reality or augmented reality technology. This allows the user to have an immersive learning experience.

[0260] Step 5:

[0261] The user engages in learning activities within a virtual or augmented reality environment, following interactive instructions presented by the device. During this time, the device tracks the user's location and movements in real time.

[0262] Step 6:

[0263] The server monitors the user's emotional state using emotion recognition technology. If the user is experiencing stress, the server adaptively modifies the learning plan and provides appropriate support.

[0264] Step 7:

[0265] The server applies gamification techniques to enhance learning motivation. Points and badges are awarded based on the user's progress, and the device displays these to provide visual feedback on the user's progress.

[0266] Step 8:

[0267] The server communicates with the machine to handle tasks within the virtual scenario. This allows users to perform tasks relevant to their real-world work environment and apply what they have learned to real-world situations.

[0268] (Example 1)

[0269] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0270] Modern education and corporate training require the provision of individualized learning experiences that cater to each learner's characteristics and interests. However, conventional systems struggle to provide optimal educational information based on individual learner profiles, adjust support according to their emotions, and appropriately motivate them to learn. Furthermore, applying activities in a virtual environment to a real-world work environment is not easy. This invention aims to solve these problems.

[0271] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0272] In this invention, the server includes information processing means for collecting user attribute information and generating educational information based on the user's learning tendencies and interests; means for providing information to the user using extended environment technology to construct a virtual environment; and emotion recognition means for detecting the user's emotions and adjusting support accordingly. This enables the provision of personalized learning information and the optimization of an adaptive learning experience in real time.

[0273] "User attribute information" is a general term for information about an individual user, such as their learning history, interests, and skill level.

[0274] "Educational information" refers to learning materials and content that are customized based on the user's learning needs and goals.

[0275] "Information processing means" refers to methods for collecting, analyzing, and generating data, and specifically, to algorithms and software that operate on computer systems.

[0276] A "virtual environment" refers to an artificial environment created using computer technology, an immersive space that allows users to experience things beyond physical limitations.

[0277] "Augmented environment technology" refers to technology that overlays virtual information onto the real world, allowing users to perceive both reality and the virtual simultaneously.

[0278] "Emotion recognition means" refers to technologies that grasp a user's emotional state from their facial expressions, voice, and actions, and are a means of estimating the user's psychological state.

[0279] "Personalized learning information" refers to learning content tailored to each user's characteristics, and is information provided according to the user's specific needs and goals.

[0280] An "adaptive learning experience" refers to a learning process in which the content and methods are dynamically adjusted according to the learner's real-time situation and needs.

[0281] This invention is a system for providing a user-optimized learning experience, involving three parties: a server, a terminal, and the user. Specific embodiments of this system are described below.

[0282] The server collects the user's attribute information and analyzes it using information processing means. Here, a database management system is used to aggregate data on the user's learning history, interests, and ability levels. Then, by leveraging a generative AI model, customized educational information for the user is generated. In this generation process, natural language processing algorithms and machine learning techniques are used to create optimal content tailored to each user's needs.

[0283] The generated educational information is transmitted from the server to the user's terminal. The terminal uses augmented environment technology to present content to the user. Specifically, by using an AR headset or VR device, the user can experience an interactive learning environment in real time. With this technology, the user can engage in practical learning in a virtual space without being restricted by a physical location.

[0284] Also, the server uses emotion recognition means to detect the user's emotions. Based on information obtained from the user's expressions and voice, the user's mental state is analyzed. Based on this data, the server can adjust the content of learning support and implement effective education while reducing the user's burden.

[0285] Furthermore, by using gamification techniques, the server is designed to enhance the user's motivation for learning. A point system and badge system are incorporated, and real-time feedback according to the progress of learning is provided. This enables the user to gain a sense of achievement and promotes the willingness to continue learning continuously.

[0286] As a specific example, in new employee training, the server creates a profile based on the information of new employees and generates an individual education plan for each employee. The terminal provides experience-based training similar to actual work using AR technology based on the generated plan.

[0287] An example of an input prompt for a generative AI model is: "Design a customized virtual training material for new employees. Incorporate an appropriate point system based on their past learning history and current skills."

[0288] As described above, the entire system works together to provide users with a personalized learning experience.

[0289] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0290] Step 1:

[0291] The server collects user attribute information. This input includes data such as the user's learning history, interests, and skill level. The server stores this data in a database and, as a concrete action to prepare for later analysis, provides an online user profile creation form and stores the information entered by the user with security in mind.

[0292] Step 2:

[0293] The server analyzes learning trends based on the collected user attribute information. In this step, a data analysis algorithm is applied to the profile information as input, and the output is a quantified result of the user's learning style and interests. Specifically, a machine learning model is used to cluster the user dataset and identify patterns of different learning styles.

[0294] Step 3:

[0295] The server generates educational information using a generative AI model. In this step, the AI ​​model uses analyzed learning tendency data as input to design optimal educational content. The output is educational information customized specifically for that user. Specifically, this involves using natural language processing techniques to generate text and interactive learning materials.

[0296] Step 4:

[0297] The server sends the generated educational information to the terminal. The input is content information generated by AI, and the terminal receives that information as output. Specifically, the process involves securely transferring data over the network using encryption technology.

[0298] Step 5:

[0299] The terminal presents the received educational information to the user. In this step, the input is the educational information received from the server, and the output is the information the user receives visually or haptically through the interface. Specifically, an AR device using augmented reality technology displays the content, allowing the user to interact with it.

[0300] Step 6:

[0301] The server collects and analyzes emotional data from the terminal in real time. The input is the user's facial expressions and voice data, and the output is the analyzed emotional state. Specifically, an emotion recognition algorithm evaluates changes in voice tone and facial expressions to determine stress and interest levels.

[0302] Step 7:

[0303] The server provides the user with feedback and support based on emotional data. The input for this step is data obtained through emotion recognition, and the output is learning content and support methods tailored based on that analysis. A concrete example of its operation is when the user is experiencing stress; in such cases, relaxation music might be played.

[0304] (Application Example 1)

[0305] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0306] In order to promote effective learning in educational and corporate training settings, it is important to provide guidance tailored to an individual's characteristics and to acquire practical skills. However, with conventional methods, it is difficult to provide an adaptive learning environment that reflects the tendencies and emotional states of individual learners, and there are also issues regarding how to apply the learning content in an actual work environment. The present invention aims to solve these problems and provide a more efficient and practical learning experience.

[0307] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0308] In this invention, the server includes information processing means for collecting user characteristic information and generating educational materials based on the user's learning tendencies and interests, means for using virtual reality technology or augmented reality technology to provide materials to the user and construct an immersive learning environment, emotional recognition processing means for detecting the user's emotional state and adjusting learning support accordingly, means for evaluating the user's actions and states and providing immediate support for tasks within the virtual environment, and means for providing guidance related to the real world for practical learning of skills in a work environment. This makes it possible to provide an appropriate learning experience for each individual learner and to improve the efficiency of acquiring skills that can be immediately applied even in an actual workplace.

[0309] The "user characteristic information" is information necessary to provide a personalized learning experience, such as the learning history, interests, and abilities of individual learners.

[0310] "Virtual reality technology" is a technology that visually and auditorily presents a virtual environment generated by a computer and enables the user to immerse themselves in it.

[0311] Augmented reality technology is a technology that extends the real environment by overlaying digital information onto real-world visual information.

[0312] "Emotion recognition processing means" refers to a technical means that analyzes the user's emotional state and selects and provides an appropriate response according to that state.

[0313] An "immersive learning environment" is a learning environment that allows users to deeply concentrate on the subject matter and provides an experience that feels as if they are actually present within it.

[0314] "Information processing means" is a general term for methods and technologies used to analyze and interpret collected data and generate desired deliverables.

[0315] "Immediate support" refers to providing real-time feedback and guidance to users regarding problems and questions they encounter during the learning process.

[0316] "Instruction related to the real world" refers to providing specific and practical guidance that enables users to apply what they have learned in the virtual environment to their actual work environment.

[0317] This invention provides a system for personalizing learning experiences in education and corporate training. This system mainly consists of a server, terminals, and users.

[0318] The server first collects user characteristic information, including learning history, interests, and abilities. Using information processing tools, the server analyzes the user's learning tendencies from the collected data and generates optimal educational materials.

[0319] The generated educational materials are delivered to the user via a terminal using virtual reality or augmented reality technology. The user progresses through this immersive learning environment, and their emotional state is continuously monitored in real time by an emotion recognition processing system. Based on this information, the server dynamically adjusts learning support, extracting and providing gamified elements and immediate assistance to enhance the user's motivation and understanding.

[0320] As a concrete example, consider a scenario where a new employee at a manufacturing company undergoes practical training. The user operates a virtual production line via a head-mounted display, learning the safe handling and efficient operation methods of specific machinery. The system provides personalized guidance based on the user's actions and emotions, supporting the learning process.

[0321] Furthermore, when using generative AI models to create real-time scenarios that provide user performance feedback and suggest next steps, the following prompt is used: "Create a scenario in which a new employee simulates a manufacturing line in a VR environment, using sentiment analysis and gamification elements to support learning."

[0322] This system provides a flexible learning environment that takes individual characteristics into account, enabling the practical application of skills.

[0323] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0324] Step 1:

[0325] The server collects user characteristic information. Specifically, it retrieves data from a database based on user input regarding the user's past learning activities, areas of interest, and abilities. This information is processed to analyze the user's past learning trends and used as input data for the next step.

[0326] Step 2:

[0327] The server generates optimal educational materials based on the user's characteristics information obtained in Step 1. Using data processing tools, the server assembles the relevant educational content in digital format and designs an immersive learning experience using virtual reality or augmented reality technology. This generated educational material becomes the output data for the terminal.

[0328] Step 3:

[0329] The device provides the user with educational materials received from the server. The device utilizes virtual reality or augmented reality technology to present the content in an interactive and immersive way. The user begins learning in this environment, and the device senses the user's input and provides immediate feedback.

[0330] Step 4:

[0331] The server monitors the user's emotional state using emotion recognition technology. It analyzes the user's facial expressions and voice as input to identify their stress levels and comprehension level. Based on these analysis results, it adjusts the content and difficulty level of learning support and outputs it to the terminal.

[0332] Step 5:

[0333] The server uses a generative AI model to provide real-time learning feedback and suggestions for the next steps. It processes user performance data as input and generates gamified elements (badges, points, etc.) and specific improvement suggestions. This result is then displayed on the user's device as final feedback.

[0334] Step 6:

[0335] Users apply what they've learned to real-world tasks through their terminals. Practical guidance from terminals and servers is integrated, making it possible to reproduce skills acquired in the virtual environment in real-world situations. Specific operating procedures and safety instructions are output, which users use to perform their tasks.

[0336] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0337] This invention provides a system for providing users with personalized learning experiences in education and corporate training. It incorporates an emotion engine to understand the user's emotional state and utilize that information to optimize the learning process. The system consists of a server, terminals, and the emotion engine.

[0338] The server first collects user profile information and analyzes the user's learning tendencies using learning history, interests, and past performance data. Based on this analysis, it generates individually optimized educational content and sends this content to the user's device.

[0339] The device presents educational content provided by the server to the user using virtual reality or augmented reality technology. Users can gain an immersive learning experience through the device, and the system tracks the user's location and movements in real time. This tracking data helps the system further refine the educational content.

[0340] The emotion engine analyzes the user's facial expressions, voice, and biometric information to recognize the user's emotional state in real time. Based on this emotional state, the server dynamically adjusts the user's learning process. For example, if the server determines that the user is experiencing stress, it can temporarily suspend the learning plan or take measures to alleviate the stress. Furthermore, based on the emotion engine's data, the device provides personalized feedback to the user and evaluates their learning progress.

[0341] As a concrete example, if this system were implemented in a company's new employee training program, the new employees would receive instruction through virtual reality technology. During this time, the emotion engine would continuously monitor each employee's emotional state. For instance, if an employee's attention becomes distracted during learning, the system would detect this and suggest that the employee take a break or present the content in a simplified form. In this way, new employees can learn at their own pace and efficiently acquire the information.

[0342] Ultimately, the server comprehensively analyzes the data obtained upon the user's completion of the learning process and provides the results to the training staff, which can then be used to improve subsequent educational programs. This allows for continuous support in improving learner performance.

[0343] The following describes the processing flow.

[0344] Step 1:

[0345] The server collects profile information from users. This includes learning history, interests, and past performance data, which is stored in a database.

[0346] Step 2:

[0347] The server analyzes the collected profile information and generates educational content based on the user's learning tendencies and interests. This content is customized and optimized for each user.

[0348] Step 3:

[0349] The server sends the generated educational content to the device. The device receives this content and prepares to present it to the user.

[0350] Step 4:

[0351] The device uses virtual reality or augmented reality technology to present educational content to the user, allowing them to begin an immersive learning experience.

[0352] Step 5:

[0353] The emotion engine analyzes the user's facial expressions, voice, and biometric information in real time to determine their emotional state. This data is then sent to the server.

[0354] Step 6:

[0355] The server dynamically adjusts the user's learning process based on emotional data obtained from the emotion engine. For example, if it determines that the user is fatigued, it suggests taking a break.

[0356] Step 7:

[0357] As users engage in learning activities, the device tracks their location and movements, sending this data to the server. This allows for further adjustments to the content.

[0358] Step 8:

[0359] The server provides users with gamified feedback to enhance learning motivation. Points and badges are awarded and displayed on the device based on achievements.

[0360] Step 9:

[0361] Upon completion of the training, the server comprehensively analyzes all training data and evaluates the user's learning performance. The results are then saved as data for further improvement.

[0362] (Example 2)

[0363] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0364] In today's educational and training environments, providing flexible learning plans tailored to the individual characteristics and emotional states of learners, and ensuring their efficient application, presents a significant challenge. In particular, because learners have diverse learning paces and styles, uniform educational content makes effective learning difficult. Furthermore, learning processes that disregard emotional states can lead to decreased learner motivation. Therefore, addressing these challenges is essential to maximizing learning outcomes.

[0365] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0366] In this invention, the server includes information processing means for collecting and generating user characteristic information, means for providing content to the user using virtual representation technology or extended representation technology, and emotion recognition processing means for detecting and adjusting the user's emotional state. This enables the provision of an optimal learning environment for each learner in real time, thereby improving learning effectiveness.

[0367] "Characteristic information" refers to specific information about the user, including learning history, interests, and individual learning style.

[0368] "Information processing means" refers to hardware and software systems for collecting, analyzing, and generating educational content.

[0369] "Virtual representation technology" refers to technology that allows users to immerse themselves in a computer-generated three-dimensional environment, thereby enabling a realistic learning experience.

[0370] "Augmented representation technology" refers to technology that overlays digital information onto the real world, providing an intuitive learning experience in a real-world environment.

[0371] "Emotion recognition processing means" refers to technology that analyzes a user's facial expressions, voice, and biometric data to determine their emotional state in real time.

[0372] A "learner" refers to an individual who utilizes an educational system, and whose learning environment and content are individually optimized for that person.

[0373] This invention provides a specific form for implementing a system that provides individualized learning experiences in education and corporate training. The components of this system and their operation are described below.

[0374] System configuration:

[0375] 1. Server:

[0376] The server collects user characteristic information and performs analysis using information processing tools. This requires various databases and a computing platform to run machine learning algorithms. The content generation engine generates personalized educational content based on the analysis results.

[0377] The server can use emotion recognition processing to analyze biometric data provided by the user, determine the emotional state in real time, and reflect it in the learning content.

[0378] 2. Terminal:

[0379] A terminal is a device used to implement virtual or augmented representation technologies. This includes VR headsets and AR-enabled mobile devices. The terminal presents received educational content to the user in an immersive manner and tracks the user's actions in real time.

[0380] This tracking function enables the analysis of the user's location and movements, allowing for dynamic adjustments to the display of educational content.

[0381] 3. User:

[0382] A user is an individual who experiences the learning content through their device. Users can progress through their learning at their own pace through feedback from the system.

[0383] Users can also receive personalized content from the system based on their emotional state and learning outcomes.

[0384] Specific example:

[0385] For example, if this system is used for new employee training, new employees will receive simulation training in a workplace environment recreated in virtual reality. If a user experiences stress during the learning process, the server will detect this and suggest a break or flexibly adjust the learning content.

[0386] Example of a prompt:

[0387] "Please explain how the system responds when a user's attention becomes sluggish during a training session using virtual reality technology for new employees."

[0388] This format allows us to address the individual learning needs of each user and provide an efficient and effective learning experience.

[0389] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0390] Step 1:

[0391] The server collects user characteristic information. The input is profile data provided by the user during registration. This data includes learning history, areas of interest, and past evaluations. The server stores this input data in a database and manages it as structured data, representing the user profile. Specifically, the server receives and stores data using a secure communication protocol.

[0392] Step 2:

[0393] The server generates educational content based on collected characteristic information. The input is a user profile obtained from a characteristic information database. The server uses machine learning algorithms to analyze this input data and generate individually optimized educational content. The output is user-optimized educational content, consisting of videos, text, and interactive quizzes. Specifically, the server's content generation engine uses an AI model to create learning materials.

[0394] Step 3:

[0395] The server sends the generated educational content to the terminal. The input is optimized educational content generated within the server. The output is content data that can be displayed on the user's terminal. The server uses an end-to-end encryption protocol to securely deliver the content to the terminal. Specifically, the server uses the HTTPS protocol to deliver data to the terminal.

[0396] Step 4:

[0397] The device presents educational content to the user using virtual representation technology. The input is optimized educational content sent from the server. The device displays the received content as a VR or AR device, providing the user with an immersive learning environment. The output is the visual / auditory data experienced by the user. Specifically, the device renders a 3D virtual classroom using a VR headset.

[0398] Step 5:

[0399] The emotion engine monitors the user's emotional state. Inputs include user actions, facial expressions, and voice data. The emotion engine analyzes these inputs in real time to determine the user's emotional state. Outputs are status information related to the user's emotional state. Specifically, the emotion engine collects data through the camera and microphone and processes it using AI analysis tools.

[0400] Step 6:

[0401] The server adjusts the learning experience based on information obtained from the emotion engine. The input is status information regarding the user's emotional state. The server uses this to re-evaluate the learning plan and make changes as needed. The output is the adjusted educational content and learning plan, which may include relaxing music and pace adjustments. Specifically, the server restructures the content to reduce the user's emotional burden.

[0402] Step 7:

[0403] Users progress through customized learning content via their devices. Inputs consist of customized educational content and feedback. Users utilize this for self-study and improve their learning process based on the feedback. Outputs are feedback regarding learning progress and achievement. Specifically, users follow instructions on their devices and solve interactive problems.

[0404] (Application Example 2)

[0405] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0406] Traditional educational programs and skills training systems have struggled to adapt to individual users' emotional states and learning paces by providing uniform content. In particular, training for technicians to improve their machine operation and maintenance skills within factories requires individualized attention, but while technologies exist to grasp and adjust to diverse emotional states in real time, they are far from practical. Furthermore, when conducting training in a virtual environment, the inability to dynamically adjust content according to the user's emotional state hinders effective learning.

[0407] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0408] In this invention, the server includes information processing means for collecting basic user information and generating instructional content based on the user's learning patterns and interests; means for constructing an immersive educational environment by providing content to the user using virtual environment technology or extended environment technology; and emotion recognition processing means for understanding the user's emotional state and adjusting instructional support accordingly. This makes it possible to adjust the educational process to match the user's individual emotional state, enabling efficient and effective learning and training.

[0409] "User basic information" is a general term for information necessary to provide an individualized educational experience, including the user's characteristics and preferences, past learning history and achievements.

[0410] "Information processing means for generating instructional content" refers to processing devices and software that analyze a user's basic information and create and provide optimal educational content.

[0411] "Virtual environment technology or augmented environment technology" refers to technologies that use digital technology to create and provide users with an environment that does not actually exist but can be experienced visually.

[0412] An "immersive educational environment" refers to a learning space that provides an immersive educational experience designed to give users a sense of being in a place equivalent to physical reality.

[0413] "Emotion recognition processing means" refers to a device or algorithm used to analyze and determine a user's emotional state in real time.

[0414] "Adjusting instructional support" refers to the process of dynamically changing the educational process and content based on the user's emotional state and learning progress to ensure that learning progresses smoothly.

[0415] This invention is a system for providing users with an immersive, personalized learning experience. It aggregates user profile information and dynamically generates educational content. The server, as an information processing tool, collects basic user information and generates optimal instructional content based on that information. This process utilizes the user's past learning data and profile information. The server employs a powerful processor and data storage to analyze this data in real time.

[0416] The device provides users with a visual and interactive educational experience using virtual or augmented reality technologies. Specifically, it presents dynamically generated educational content using virtual reality (VR) or augmented reality (AR) devices. This allows users to learn at their own pace and immerse themselves in content tailored to their interests. This can be easily implemented using devices such as Oculus Quest 2 or AR applications on smartphones.

[0417] The user's emotional state is monitored by sensors and cameras on the device and analyzed through emotion recognition software such as the Affectiva SDK. Based on this real-time emotional data, the server adjusts the instruction content; for example, if the user is stressed, the content is made simpler or adjusted to be more relaxing. This maximizes the user's learning effectiveness.

[0418] For example, when a technician is learning to operate a specific machine in a factory, if emotional data detects a lack of concentration, the system will simplify the training content and help the technician regain confidence in their ability to concentrate again.

[0419] Examples of prompt statements in the AI ​​model generated by this system are as follows:

[0420] "For engineers with decreased concentration, how can we dynamically adjust VR training programs to optimize learning efficiency? Please advise."

[0421] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0422] Step 1:

[0423] The server collects basic user information. This includes user personal information, past learning history, and data on interests. This data is integrated to create profiles for generating educational content. The profiles stored in the database form the basis for providing instruction optimized for each user.

[0424] Step 2:

[0425] The server uses the generated profile to create instructional content tailored to the user. The input is the profile data obtained in step 1, and the output is optimized educational content. The generation AI model is used to create customized learning materials based on the user's learning tendencies. Specifically, the AI ​​model determines the difficulty level and theme of the content.

[0426] Step 3:

[0427] The device receives educational content provided by the server and presents it to the user using virtual environment technology. The input is the educational content from the server, and the output is a visual and interactive learning experience for the user. The device uses Oculus Quest 2 and AR applications to provide the user with an immersive educational environment.

[0428] Step 4:

[0429] The device monitors the user's emotional state in real time. Input is biometric information from sensors and cameras installed in the device, and output is the result of emotion recognition analysis. Analysis is performed using Affectiva SDK, etc., to determine the user's emotional state, such as whether they are focused or stressed.

[0430] Step 5:

[0431] The server adjusts the educational content as needed based on the results of the emotion recognition analysis. The input is the emotion data from step 4, and the output is the adjusted educational content. Specifically, if the user is feeling stressed, the server will either simplify the content or suggest a break for relaxation.

[0432] Step 6:

[0433] The user continues with the educational content based on feedback from the device. The input is the adjusted educational content and feedback from the device, and the output is the improvement in the user's learning progress. The device provides feedback visually or audibly to help the user learn more effectively.

[0434] Through the above processing steps, users can efficiently acquire knowledge and skills in a learning environment that best suits their own pace and emotional state.

[0435] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0436] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0437] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0438] [Third Embodiment]

[0439] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0440] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0441] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0442] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0443] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0444] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0445] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0446] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0447] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0448] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0449] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0450] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0451] This invention is a system for providing users with a personalized learning experience in educational and corporate training settings. The system consists of a server, terminals, and users.

[0452] The server first collects user profile information. This profile information includes past learning history, interests, and abilities. Based on this data, the server analyzes the user's learning tendencies and generates educational content optimized for the user. The generated content is then sent from the server to the device.

[0453] The device presents educational content received from the server to the user using virtual reality or augmented reality technology. This allows the user to learn in an immersive learning environment. The device also tracks the user's location and movements in real time and adjusts the educational content accordingly.

[0454] The system further monitors the user's emotional state using emotion recognition technology. The server provides support and adaptive instructions tailored to the user's emotions, reducing user stress and improving learning efficiency.

[0455] Furthermore, the server applies gamification techniques to enhance motivation for learning. By providing feedback such as points and badges, it encourages continued learning by giving users a sense of accomplishment.

[0456] As a concrete example, consider a scenario involving new employee training at a certain company. In this case, the server creates a profile of the new employee and generates an individualized learning plan. The terminal uses virtual reality technology to provide virtual training that closely resembles actual work. As the user progresses through the learning process, the server uses emotion recognition technology to understand the new employee's stress level and provides support as needed. Furthermore, the server awards points based on the learning progress and displays them on the terminal to further increase motivation.

[0457] Ultimately, the server controls mechanical devices to physically execute tasks within the virtual scenario, enabling users to apply what they've learned in real-world work environments. This makes it easier to translate learning outcomes into reality.

[0458] The following describes the processing flow.

[0459] Step 1:

[0460] The server collects profile information, learning history, and interest data submitted by the user. This data is stored in a database and used for later analysis.

[0461] Step 2:

[0462] The server runs an algorithm that analyzes the user's learning tendencies based on the collected data. This analysis provides the information necessary to create educational content optimized for the user.

[0463] Step 3:

[0464] The server generates individually optimized educational content based on the analysis results. This content is tailored to each user's abilities and interests and is then sent from the server to their device.

[0465] Step 4:

[0466] The device presents educational content transmitted from the server to the user using virtual reality or augmented reality technology. This allows the user to have an immersive learning experience.

[0467] Step 5:

[0468] The user engages in learning activities within a virtual or augmented reality environment, following interactive instructions presented by the device. During this time, the device tracks the user's location and movements in real time.

[0469] Step 6:

[0470] The server monitors the user's emotional state using emotion recognition technology. If the user is experiencing stress, the server adaptively modifies the learning plan and provides appropriate support.

[0471] Step 7:

[0472] The server applies gamification techniques to enhance learning motivation. Points and badges are awarded based on the user's progress, and the device displays these to provide visual feedback on the user's progress.

[0473] Step 8:

[0474] The server communicates with the machine to handle tasks within the virtual scenario. This allows users to perform tasks relevant to their real-world work environment and apply what they have learned to real-world situations.

[0475] (Example 1)

[0476] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0477] Modern education and corporate training require the provision of individualized learning experiences that cater to each learner's characteristics and interests. However, conventional systems struggle to provide optimal educational information based on individual learner profiles, adjust support according to their emotions, and appropriately motivate them to learn. Furthermore, applying activities in a virtual environment to a real-world work environment is not easy. This invention aims to solve these problems.

[0478] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0479] In this invention, the server includes information processing means for collecting user attribute information and generating educational information based on the user's learning tendencies and interests; means for providing information to the user using extended environment technology to construct a virtual environment; and emotion recognition means for detecting the user's emotions and adjusting support accordingly. This enables the provision of personalized learning information and the optimization of an adaptive learning experience in real time.

[0480] "User attribute information" is a general term for information about an individual user, such as their learning history, interests, and skill level.

[0481] "Educational information" refers to learning materials and content that are customized based on the user's learning needs and goals.

[0482] "Information processing means" refers to methods for collecting, analyzing, and generating data, and specifically, to algorithms and software that operate on computer systems.

[0483] A "virtual environment" refers to an artificial environment created using computer technology, an immersive space that allows users to experience things beyond physical limitations.

[0484] "Augmented environment technology" refers to technology that overlays virtual information onto the real world, allowing users to perceive both reality and the virtual simultaneously.

[0485] "Emotion recognition means" refers to technologies that grasp a user's emotional state from their facial expressions, voice, and actions, and are a means of estimating the user's psychological state.

[0486] "Personalized learning information" refers to learning content tailored to each user's characteristics, and is information provided according to the user's specific needs and goals.

[0487] An "adaptive learning experience" refers to a learning process in which the content and methods are dynamically adjusted according to the learner's real-time situation and needs.

[0488] This invention is a system for providing a user-optimized learning experience, involving three parties: a server, a terminal, and the user. Specific embodiments of this system are described below.

[0489] The server collects user attribute information and analyzes it using information processing tools. A database management system is used to aggregate data on the user's learning history, interests, and skill level. Then, a generative AI model is used to generate personalized educational information for each user. This generation process employs natural language processing algorithms and machine learning techniques to create optimal content tailored to each user's needs.

[0490] The generated educational information is sent from the server to the user's device. The device then presents the content to the user using augmented reality (AR) technology. Specifically, by using AR headsets or VR devices, users can experience a real-time, interactive learning environment. This technology allows users to engage in practical learning in a virtual space without being tied to a physical location.

[0491] Furthermore, the server uses emotion recognition to detect the user's emotions. Based on information obtained from the user's facial expressions and voice, the user's psychological state is analyzed. Based on this data, the server can adjust the content of learning support, reducing the user's burden while providing effective education.

[0492] Furthermore, by employing gamification techniques, the server is designed to enhance user motivation for learning. A point system and badge system are incorporated, providing real-time feedback based on learning progress. This allows users to experience a sense of accomplishment and encourages them to continue learning.

[0493] As a concrete example, in new employee training, the server creates a profile based on the new employee's information and generates an individualized training plan. Based on the generated plan, the terminal uses AR technology to provide hands-on training that closely resembles actual work.

[0494] An example of an input prompt for a generative AI model is: "Design a customized virtual training material for new employees. Incorporate an appropriate point system based on their past learning history and current skills."

[0495] As described above, the entire system works together to provide users with a personalized learning experience.

[0496] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0497] Step 1:

[0498] The server collects user attribute information. This input includes data such as the user's learning history, interests, and skill level. The server stores this data in a database and, as a concrete action to prepare for later analysis, provides an online user profile creation form and stores the information entered by the user with security in mind.

[0499] Step 2:

[0500] The server analyzes learning trends based on the collected user attribute information. In this step, a data analysis algorithm is applied to the profile information as input, and the output is a quantified result of the user's learning style and interests. Specifically, a machine learning model is used to cluster the user dataset and identify patterns of different learning styles.

[0501] Step 3:

[0502] The server generates educational information using a generative AI model. In this step, the AI ​​model uses analyzed learning tendency data as input to design optimal educational content. The output is educational information customized specifically for that user. Specifically, this involves using natural language processing techniques to generate text and interactive learning materials.

[0503] Step 4:

[0504] The server sends the generated educational information to the terminal. The input is content information generated by AI, and the terminal receives that information as output. Specifically, the process involves securely transferring data over the network using encryption technology.

[0505] Step 5:

[0506] The terminal presents the received educational information to the user. In this step, the input is the educational information received from the server, and the output is the information the user receives visually or haptically through the interface. Specifically, an AR device using augmented reality technology displays the content, allowing the user to interact with it.

[0507] Step 6:

[0508] The server collects and analyzes emotional data from the terminal in real time. The input is the user's facial expressions and voice data, and the output is the analyzed emotional state. Specifically, an emotion recognition algorithm evaluates changes in voice tone and facial expressions to determine stress and interest levels.

[0509] Step 7:

[0510] The server provides the user with feedback and support based on emotional data. The input for this step is data obtained through emotion recognition, and the output is learning content and support methods tailored based on that analysis. A concrete example of its operation is when the user is experiencing stress; in such cases, relaxation music might be played.

[0511] (Application Example 1)

[0512] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0513] To promote effective learning in educational and corporate training settings, it is crucial to provide instruction tailored to individual characteristics and to acquire practical skills. However, conventional methods struggle to provide an adaptive learning environment that reflects the tendencies and emotional states of individual learners, and there are also challenges in how to apply the learned content to actual work environments. This invention aims to solve these problems and provide a more efficient and practical learning experience.

[0514] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0515] In this invention, the server comprises information processing means for collecting user characteristic information and generating educational materials based on the user's learning tendencies and interests; means for providing materials to the user using virtual reality or augmented reality technology and constructing an immersive learning environment; emotion recognition processing means for detecting the user's emotional state and adjusting learning support accordingly; means for evaluating the user's actions and state and providing immediate support for tasks in the virtual environment; and means for providing instruction related to the real world in order to practically learn skills in a work environment. This makes it possible to provide each individual learner with an appropriate learning experience and to streamline the acquisition of skills that can be immediately applied in actual workplaces.

[0516] "User characteristic information" refers to information necessary to provide a personalized learning experience, such as each learner's learning history, interests, and abilities.

[0517] "Virtual reality technology" is a technology that presents a computer-generated virtual environment visually and aurally, enabling users to become immersed in it.

[0518] Augmented reality technology is a technology that extends the real environment by overlaying digital information onto real-world visual information.

[0519] "Emotion recognition processing means" refers to a technical means that analyzes the user's emotional state and selects and provides an appropriate response according to that state.

[0520] An "immersive learning environment" is a learning environment that allows users to deeply concentrate on the subject matter and provides an experience that feels as if they are actually present within it.

[0521] "Information processing means" is a general term for methods and technologies used to analyze and interpret collected data and generate desired deliverables.

[0522] "Immediate support" refers to providing real-time feedback and guidance to users regarding problems and questions they encounter during the learning process.

[0523] "Instruction related to the real world" refers to providing specific and practical guidance that enables users to apply what they have learned in the virtual environment to their actual work environment.

[0524] This invention provides a system for personalizing learning experiences in education and corporate training. This system mainly consists of a server, terminals, and users.

[0525] The server first collects user characteristic information, including learning history, interests, and abilities. Using information processing tools, the server analyzes the user's learning tendencies from the collected data and generates optimal educational materials.

[0526] The generated educational materials are delivered to the user via a terminal using virtual reality or augmented reality technology. The user progresses through this immersive learning environment, and their emotional state is continuously monitored in real time by an emotion recognition processing system. Based on this information, the server dynamically adjusts learning support, extracting and providing gamified elements and immediate assistance to enhance the user's motivation and understanding.

[0527] As a concrete example, consider a scenario where a new employee at a manufacturing company undergoes practical training. The user operates a virtual production line via a head-mounted display, learning the safe handling and efficient operation methods of specific machinery. The system provides personalized guidance based on the user's actions and emotions, supporting the learning process.

[0528] Furthermore, when using generative AI models to create real-time scenarios that provide user performance feedback and suggest next steps, the following prompt is used: "Create a scenario in which a new employee simulates a manufacturing line in a VR environment, using sentiment analysis and gamification elements to support learning."

[0529] This system provides a flexible learning environment that takes individual characteristics into account, enabling the practical application of skills.

[0530] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0531] Step 1:

[0532] The server collects user characteristic information. Specifically, it retrieves data from a database based on user input regarding the user's past learning activities, areas of interest, and abilities. This information is processed to analyze the user's past learning trends and used as input data for the next step.

[0533] Step 2:

[0534] The server generates optimal educational materials based on the user's characteristics information obtained in Step 1. Using data processing tools, the server assembles the relevant educational content in digital format and designs an immersive learning experience using virtual reality or augmented reality technology. This generated educational material becomes the output data for the terminal.

[0535] Step 3:

[0536] The device provides the user with educational materials received from the server. The device utilizes virtual reality or augmented reality technology to present the content in an interactive and immersive way. The user begins learning in this environment, and the device senses the user's input and provides immediate feedback.

[0537] Step 4:

[0538] The server monitors the user's emotional state using emotion recognition technology. It analyzes the user's facial expressions and voice as input to identify their stress levels and comprehension level. Based on these analysis results, it adjusts the content and difficulty level of learning support and outputs it to the terminal.

[0539] Step 5:

[0540] The server uses a generative AI model to provide real-time learning feedback and suggestions for the next steps. It processes user performance data as input and generates gamified elements (badges, points, etc.) and specific improvement suggestions. This result is then displayed on the user's device as final feedback.

[0541] Step 6:

[0542] Users apply what they've learned to real-world tasks through their terminals. Practical guidance from terminals and servers is integrated, making it possible to reproduce skills acquired in the virtual environment in real-world situations. Specific operating procedures and safety instructions are output, which users use to perform their tasks.

[0543] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0544] This invention provides a system for providing users with personalized learning experiences in education and corporate training. It incorporates an emotion engine to understand the user's emotional state and utilize that information to optimize the learning process. The system consists of a server, terminals, and the emotion engine.

[0545] The server first collects user profile information and analyzes the user's learning tendencies using learning history, interests, and past performance data. Based on this analysis, it generates individually optimized educational content and sends this content to the user's device.

[0546] The device presents educational content provided by the server to the user using virtual reality or augmented reality technology. Users can gain an immersive learning experience through the device, and the system tracks the user's location and movements in real time. This tracking data helps the system further refine the educational content.

[0547] The emotion engine analyzes the user's facial expressions, voice, and biometric information to recognize the user's emotional state in real time. Based on this emotional state, the server dynamically adjusts the user's learning process. For example, if the server determines that the user is experiencing stress, it can temporarily suspend the learning plan or take measures to alleviate the stress. Furthermore, based on the emotion engine's data, the device provides personalized feedback to the user and evaluates their learning progress.

[0548] As a concrete example, if this system were implemented in a company's new employee training program, the new employees would receive instruction through virtual reality technology. During this time, the emotion engine would continuously monitor each employee's emotional state. For instance, if an employee's attention becomes distracted during learning, the system would detect this and suggest that the employee take a break or present the content in a simplified form. In this way, new employees can learn at their own pace and efficiently acquire the information.

[0549] Ultimately, the server comprehensively analyzes the data obtained upon the user's completion of the learning process and provides the results to the training staff, which can then be used to improve subsequent educational programs. This allows for continuous support in improving learner performance.

[0550] The following describes the processing flow.

[0551] Step 1:

[0552] The server collects profile information from users. This includes learning history, interests, and past performance data, which is stored in a database.

[0553] Step 2:

[0554] The server analyzes the collected profile information and generates educational content based on the user's learning tendencies and interests. This content is customized and optimized for each user.

[0555] Step 3:

[0556] The server sends the generated educational content to the device. The device receives this content and prepares to present it to the user.

[0557] Step 4:

[0558] The device uses virtual reality or augmented reality technology to present educational content to the user, allowing them to begin an immersive learning experience.

[0559] Step 5:

[0560] The emotion engine analyzes the user's facial expressions, voice, and biometric information in real time to determine their emotional state. This data is then sent to the server.

[0561] Step 6:

[0562] The server dynamically adjusts the user's learning process based on emotional data obtained from the emotion engine. For example, if it determines that the user is fatigued, it suggests taking a break.

[0563] Step 7:

[0564] As users engage in learning activities, the device tracks their location and movements, sending this data to the server. This allows for further adjustments to the content.

[0565] Step 8:

[0566] The server provides users with gamified feedback to enhance learning motivation. Points and badges are awarded and displayed on the device based on achievements.

[0567] Step 9:

[0568] Upon completion of the training, the server comprehensively analyzes all training data and evaluates the user's learning performance. The results are then saved as data for further improvement.

[0569] (Example 2)

[0570] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0571] In today's educational and training environments, providing flexible learning plans tailored to the individual characteristics and emotional states of learners, and ensuring their efficient application, presents a significant challenge. In particular, because learners have diverse learning paces and styles, uniform educational content makes effective learning difficult. Furthermore, learning processes that disregard emotional states can lead to decreased learner motivation. Therefore, addressing these challenges is essential to maximizing learning outcomes.

[0572] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0573] In this invention, the server includes information processing means for collecting and generating user characteristic information, means for providing content to the user using virtual representation technology or extended representation technology, and emotion recognition processing means for detecting and adjusting the user's emotional state. This enables the provision of an optimal learning environment for each learner in real time, thereby improving learning effectiveness.

[0574] "Characteristic information" refers to specific information about the user, including learning history, interests, and individual learning style.

[0575] "Information processing means" refers to hardware and software systems for collecting, analyzing, and generating educational content.

[0576] "Virtual representation technology" refers to technology that allows users to immerse themselves in a computer-generated three-dimensional environment, thereby enabling a realistic learning experience.

[0577] "Augmented representation technology" refers to technology that overlays digital information onto the real world, providing an intuitive learning experience in a real-world environment.

[0578] "Emotion recognition processing means" refers to technology that analyzes a user's facial expressions, voice, and biometric data to determine their emotional state in real time.

[0579] A "learner" refers to an individual who utilizes an educational system, and whose learning environment and content are individually optimized for that person.

[0580] This invention provides a specific form for implementing a system that provides individualized learning experiences in education and corporate training. The components of this system and their operation are described below.

[0581] System configuration:

[0582] 1. Server:

[0583] The server collects user characteristic information and performs analysis using information processing tools. This requires various databases and a computing platform to run machine learning algorithms. The content generation engine generates personalized educational content based on the analysis results.

[0584] The server can use emotion recognition processing to analyze biometric data provided by the user, determine the emotional state in real time, and reflect it in the learning content.

[0585] 2. Terminal:

[0586] A terminal is a device used to implement virtual or augmented representation technologies. This includes VR headsets and AR-enabled mobile devices. The terminal presents received educational content to the user in an immersive manner and tracks the user's actions in real time.

[0587] This tracking function enables the analysis of the user's location and movements, allowing for dynamic adjustments to the display of educational content.

[0588] 3. User:

[0589] A user is an individual who experiences the learning content through their device. Users can progress through their learning at their own pace through feedback from the system.

[0590] Users can also receive personalized content from the system based on their emotional state and learning outcomes.

[0591] Specific example:

[0592] For example, if this system is used for new employee training, new employees will receive simulation training in a workplace environment recreated in virtual reality. If a user experiences stress during the learning process, the server will detect this and suggest a break or flexibly adjust the learning content.

[0593] Example of a prompt:

[0594] "Please explain how the system responds when a user's attention becomes sluggish during a training session using virtual reality technology for new employees."

[0595] This format allows us to address the individual learning needs of each user and provide an efficient and effective learning experience.

[0596] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0597] Step 1:

[0598] The server collects user characteristic information. The input is profile data provided by the user during registration. This data includes learning history, areas of interest, and past evaluations. The server stores this input data in a database and manages it as structured data, representing the user profile. Specifically, the server receives and stores data using a secure communication protocol.

[0599] Step 2:

[0600] The server generates educational content based on collected characteristic information. The input is a user profile obtained from a characteristic information database. The server uses machine learning algorithms to analyze this input data and generate individually optimized educational content. The output is user-optimized educational content, consisting of videos, text, and interactive quizzes. Specifically, the server's content generation engine uses an AI model to create learning materials.

[0601] Step 3:

[0602] The server sends the generated educational content to the terminal. The input is optimized educational content generated within the server. The output is content data that can be displayed on the user's terminal. The server uses an end-to-end encryption protocol to securely deliver the content to the terminal. Specifically, the server uses the HTTPS protocol to deliver data to the terminal.

[0603] Step 4:

[0604] The device presents educational content to the user using virtual representation technology. The input is optimized educational content sent from the server. The device displays the received content as a VR or AR device, providing the user with an immersive learning environment. The output is the visual / auditory data experienced by the user. Specifically, the device renders a 3D virtual classroom using a VR headset.

[0605] Step 5:

[0606] The emotion engine monitors the user's emotional state. Inputs include user actions, facial expressions, and voice data. The emotion engine analyzes these inputs in real time to determine the user's emotional state. Outputs are status information related to the user's emotional state. Specifically, the emotion engine collects data through the camera and microphone and processes it using AI analysis tools.

[0607] Step 6:

[0608] The server adjusts the learning experience based on information obtained from the emotion engine. The input is status information regarding the user's emotional state. The server uses this to re-evaluate the learning plan and make changes as needed. The output is the adjusted educational content and learning plan, which may include relaxing music and pace adjustments. Specifically, the server restructures the content to reduce the user's emotional burden.

[0609] Step 7:

[0610] Users progress through customized learning content via their devices. Inputs consist of customized educational content and feedback. Users utilize this for self-study and improve their learning process based on the feedback. Outputs are feedback regarding learning progress and achievement. Specifically, users follow instructions on their devices and solve interactive problems.

[0611] (Application Example 2)

[0612] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0613] Traditional educational programs and skills training systems have struggled to adapt to individual users' emotional states and learning paces by providing uniform content. In particular, training for technicians to improve their machine operation and maintenance skills within factories requires individualized attention, but while technologies exist to grasp and adjust to diverse emotional states in real time, they are far from practical. Furthermore, when conducting training in a virtual environment, the inability to dynamically adjust content according to the user's emotional state hinders effective learning.

[0614] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0615] In this invention, the server includes information processing means for collecting basic user information and generating instructional content based on the user's learning patterns and interests; means for constructing an immersive educational environment by providing content to the user using virtual environment technology or extended environment technology; and emotion recognition processing means for understanding the user's emotional state and adjusting instructional support accordingly. This makes it possible to adjust the educational process to match the user's individual emotional state, enabling efficient and effective learning and training.

[0616] "User basic information" is a general term for information necessary to provide an individualized educational experience, including the user's characteristics and preferences, past learning history and achievements.

[0617] "Information processing means for generating instructional content" refers to processing devices and software that analyze a user's basic information and create and provide optimal educational content.

[0618] "Virtual environment technology or augmented environment technology" refers to technologies that use digital technology to create and provide users with an environment that does not actually exist but can be experienced visually.

[0619] An "immersive educational environment" refers to a learning space that provides an immersive educational experience designed to give users a sense of being in a place equivalent to physical reality.

[0620] "Emotion recognition processing means" refers to a device or algorithm used to analyze and determine a user's emotional state in real time.

[0621] "Adjusting instructional support" refers to the process of dynamically changing the educational process and content based on the user's emotional state and learning progress to ensure that learning progresses smoothly.

[0622] This invention is a system for providing users with an immersive, personalized learning experience. It aggregates user profile information and dynamically generates educational content. The server, as an information processing tool, collects basic user information and generates optimal instructional content based on that information. This process utilizes the user's past learning data and profile information. The server employs a powerful processor and data storage to analyze this data in real time.

[0623] The device provides users with a visual and interactive educational experience using virtual or augmented reality technologies. Specifically, it presents dynamically generated educational content using virtual reality (VR) or augmented reality (AR) devices. This allows users to learn at their own pace and immerse themselves in content tailored to their interests. This can be easily implemented using devices such as Oculus Quest 2 or AR applications on smartphones.

[0624] The user's emotional state is monitored by sensors and cameras on the device and analyzed through emotion recognition software such as the Affectiva SDK. Based on this real-time emotional data, the server adjusts the instruction content; for example, if the user is stressed, the content is made simpler or adjusted to be more relaxing. This maximizes the user's learning effectiveness.

[0625] For example, when a technician is learning to operate a specific machine in a factory, if emotional data detects a lack of concentration, the system will simplify the training content and help the technician regain confidence in their ability to concentrate again.

[0626] Examples of prompt statements in the AI ​​model generated by this system are as follows:

[0627] "For engineers with decreased concentration, how can we dynamically adjust VR training programs to optimize learning efficiency? Please advise."

[0628] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0629] Step 1:

[0630] The server collects basic user information. This includes user personal information, past learning history, and data on interests. This data is integrated to create profiles for generating educational content. The profiles stored in the database form the basis for providing instruction optimized for each user.

[0631] Step 2:

[0632] The server uses the generated profile to create instructional content tailored to the user. The input is the profile data obtained in step 1, and the output is optimized educational content. The generation AI model is used to create customized learning materials based on the user's learning tendencies. Specifically, the AI ​​model determines the difficulty level and theme of the content.

[0633] Step 3:

[0634] The device receives educational content provided by the server and presents it to the user using virtual environment technology. The input is the educational content from the server, and the output is a visual and interactive learning experience for the user. The device uses Oculus Quest 2 and AR applications to provide the user with an immersive educational environment.

[0635] Step 4:

[0636] The device monitors the user's emotional state in real time. Input is biometric information from sensors and cameras installed in the device, and output is the result of emotion recognition analysis. Analysis is performed using Affectiva SDK, etc., to determine the user's emotional state, such as whether they are focused or stressed.

[0637] Step 5:

[0638] The server adjusts the educational content as needed based on the results of the emotion recognition analysis. The input is the emotion data from step 4, and the output is the adjusted educational content. Specifically, if the user is feeling stressed, the server will either simplify the content or suggest a break for relaxation.

[0639] Step 6:

[0640] The user continues with the educational content based on feedback from the device. The input is the adjusted educational content and feedback from the device, and the output is the improvement in the user's learning progress. The device provides feedback visually or audibly to help the user learn more effectively.

[0641] Through the above processing steps, users can efficiently acquire knowledge and skills in a learning environment that best suits their own pace and emotional state.

[0642] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0643] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0644] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0645] [Fourth Embodiment]

[0646] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0647] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0648] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0649] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0650] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0651] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0652] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0653] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0654] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0655] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0656] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0657] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0658] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0659] This invention is a system for providing users with a personalized learning experience in educational and corporate training settings. The system consists of a server, terminals, and users.

[0660] The server first collects user profile information. This profile information includes past learning history, interests, and abilities. Based on this data, the server analyzes the user's learning tendencies and generates educational content optimized for the user. The generated content is then sent from the server to the device.

[0661] The device presents educational content received from the server to the user using virtual reality or augmented reality technology. This allows the user to learn in an immersive learning environment. The device also tracks the user's location and movements in real time and adjusts the educational content accordingly.

[0662] The system further monitors the user's emotional state using emotion recognition technology. The server provides support and adaptive instructions tailored to the user's emotions, reducing user stress and improving learning efficiency.

[0663] Furthermore, the server applies gamification techniques to enhance motivation for learning. By providing feedback such as points and badges, it encourages continued learning by giving users a sense of accomplishment.

[0664] As a concrete example, consider a scenario involving new employee training at a certain company. In this case, the server creates a profile of the new employee and generates an individualized learning plan. The terminal uses virtual reality technology to provide virtual training that closely resembles actual work. As the user progresses through the learning process, the server uses emotion recognition technology to understand the new employee's stress level and provides support as needed. Furthermore, the server awards points based on the learning progress and displays them on the terminal to further increase motivation.

[0665] Ultimately, the server controls mechanical devices to physically execute tasks within the virtual scenario, enabling users to apply what they've learned in real-world work environments. This makes it easier to translate learning outcomes into reality.

[0666] The following describes the processing flow.

[0667] Step 1:

[0668] The server collects profile information, learning history, and interest data submitted by the user. This data is stored in a database and used for later analysis.

[0669] Step 2:

[0670] The server runs an algorithm that analyzes the user's learning tendencies based on the collected data. This analysis provides the information necessary to create educational content optimized for the user.

[0671] Step 3:

[0672] The server generates individually optimized educational content based on the analysis results. This content is tailored to each user's abilities and interests and is then sent from the server to their device.

[0673] Step 4:

[0674] The device presents educational content transmitted from the server to the user using virtual reality or augmented reality technology. This allows the user to have an immersive learning experience.

[0675] Step 5:

[0676] The user engages in learning activities within a virtual or augmented reality environment, following interactive instructions presented by the device. During this time, the device tracks the user's location and movements in real time.

[0677] Step 6:

[0678] The server monitors the user's emotional state using emotion recognition technology. If the user is experiencing stress, the server adaptively modifies the learning plan and provides appropriate support.

[0679] Step 7:

[0680] The server applies gamification techniques to enhance learning motivation. Points and badges are awarded based on the user's progress, and the device displays these to provide visual feedback on the user's progress.

[0681] Step 8:

[0682] The server communicates with the machine to handle tasks within the virtual scenario. This allows users to perform tasks relevant to their real-world work environment and apply what they have learned to real-world situations.

[0683] (Example 1)

[0684] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0685] Modern education and corporate training require the provision of individualized learning experiences that cater to each learner's characteristics and interests. However, conventional systems struggle to provide optimal educational information based on individual learner profiles, adjust support according to their emotions, and appropriately motivate them to learn. Furthermore, applying activities in a virtual environment to a real-world work environment is not easy. This invention aims to solve these problems.

[0686] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0687] In this invention, the server includes information processing means for collecting user attribute information and generating educational information based on the user's learning tendencies and interests; means for providing information to the user using extended environment technology to construct a virtual environment; and emotion recognition means for detecting the user's emotions and adjusting support accordingly. This enables the provision of personalized learning information and the optimization of an adaptive learning experience in real time.

[0688] "User attribute information" is a general term for information about an individual user, such as their learning history, interests, and skill level.

[0689] "Educational information" refers to learning materials and content that are customized based on the user's learning needs and goals.

[0690] "Information processing means" refers to methods for collecting, analyzing, and generating data, and specifically, to algorithms and software that operate on computer systems.

[0691] A "virtual environment" refers to an artificial environment created using computer technology, an immersive space that allows users to experience things beyond physical limitations.

[0692] "Augmented environment technology" refers to technology that overlays virtual information onto the real world, allowing users to perceive both reality and the virtual simultaneously.

[0693] "Emotion recognition means" refers to technologies that grasp a user's emotional state from their facial expressions, voice, and actions, and are a means of estimating the user's psychological state.

[0694] "Personalized learning information" refers to learning content tailored to each user's characteristics, and is information provided according to the user's specific needs and goals.

[0695] An "adaptive learning experience" refers to a learning process in which the content and methods are dynamically adjusted according to the learner's real-time situation and needs.

[0696] This invention is a system for providing a user-optimized learning experience, involving three parties: a server, a terminal, and the user. Specific embodiments of this system are described below.

[0697] The server collects user attribute information and analyzes it using information processing tools. A database management system is used to aggregate data on the user's learning history, interests, and skill level. Then, a generative AI model is used to generate personalized educational information for each user. This generation process employs natural language processing algorithms and machine learning techniques to create optimal content tailored to each user's needs.

[0698] The generated educational information is sent from the server to the user's device. The device then presents the content to the user using augmented reality (AR) technology. Specifically, by using AR headsets or VR devices, users can experience a real-time, interactive learning environment. This technology allows users to engage in practical learning in a virtual space without being tied to a physical location.

[0699] Furthermore, the server uses emotion recognition to detect the user's emotions. Based on information obtained from the user's facial expressions and voice, the user's psychological state is analyzed. Based on this data, the server can adjust the content of learning support, reducing the user's burden while providing effective education.

[0700] Furthermore, by employing gamification techniques, the server is designed to enhance user motivation for learning. A point system and badge system are incorporated, providing real-time feedback based on learning progress. This allows users to experience a sense of accomplishment and encourages them to continue learning.

[0701] As a concrete example, in new employee training, the server creates a profile based on the new employee's information and generates an individualized training plan. Based on the generated plan, the terminal uses AR technology to provide hands-on training that closely resembles actual work.

[0702] An example of an input prompt for a generative AI model is: "Design a customized virtual training material for new employees. Incorporate an appropriate point system based on their past learning history and current skills."

[0703] As described above, the entire system works together to provide users with a personalized learning experience.

[0704] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0705] Step 1:

[0706] The server collects user attribute information. This input includes data such as the user's learning history, interests, and skill level. The server stores this data in a database and, as a concrete action to prepare for later analysis, provides an online user profile creation form and stores the information entered by the user with security in mind.

[0707] Step 2:

[0708] The server analyzes learning trends based on the collected user attribute information. In this step, a data analysis algorithm is applied to the profile information as input, and the output is a quantified result of the user's learning style and interests. Specifically, a machine learning model is used to cluster the user dataset and identify patterns of different learning styles.

[0709] Step 3:

[0710] The server generates educational information using a generative AI model. In this step, the AI ​​model uses analyzed learning tendency data as input to design optimal educational content. The output is educational information customized specifically for that user. Specifically, this involves using natural language processing techniques to generate text and interactive learning materials.

[0711] Step 4:

[0712] The server sends the generated educational information to the terminal. The input is content information generated by AI, and the terminal receives that information as output. Specifically, the process involves securely transferring data over the network using encryption technology.

[0713] Step 5:

[0714] The terminal presents the received educational information to the user. In this step, the input is the educational information received from the server, and the output is the information the user receives visually or haptically through the interface. Specifically, an AR device using augmented reality technology displays the content, allowing the user to interact with it.

[0715] Step 6:

[0716] The server collects and analyzes emotional data from the terminal in real time. The input is the user's facial expressions and voice data, and the output is the analyzed emotional state. Specifically, an emotion recognition algorithm evaluates changes in voice tone and facial expressions to determine stress and interest levels.

[0717] Step 7:

[0718] The server provides the user with feedback and support based on emotional data. The input for this step is data obtained through emotion recognition, and the output is learning content and support methods tailored based on that analysis. A concrete example of its operation is when the user is experiencing stress; in such cases, relaxation music might be played.

[0719] (Application Example 1)

[0720] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0721] To promote effective learning in educational and corporate training settings, it is crucial to provide instruction tailored to individual characteristics and to acquire practical skills. However, conventional methods struggle to provide an adaptive learning environment that reflects the tendencies and emotional states of individual learners, and there are also challenges in how to apply the learned content to actual work environments. This invention aims to solve these problems and provide a more efficient and practical learning experience.

[0722] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0723] In this invention, the server comprises information processing means for collecting user characteristic information and generating educational materials based on the user's learning tendencies and interests; means for providing materials to the user using virtual reality or augmented reality technology and constructing an immersive learning environment; emotion recognition processing means for detecting the user's emotional state and adjusting learning support accordingly; means for evaluating the user's actions and state and providing immediate support for tasks in the virtual environment; and means for providing instruction related to the real world in order to practically learn skills in a work environment. This makes it possible to provide each individual learner with an appropriate learning experience and to streamline the acquisition of skills that can be immediately applied in actual workplaces.

[0724] "User characteristic information" refers to information necessary to provide a personalized learning experience, such as each learner's learning history, interests, and abilities.

[0725] "Virtual reality technology" is a technology that presents a computer-generated virtual environment visually and aurally, enabling users to become immersed in it.

[0726] Augmented reality technology is a technology that extends the real environment by overlaying digital information onto real-world visual information.

[0727] "Emotion recognition processing means" refers to a technical means that analyzes the user's emotional state and selects and provides an appropriate response according to that state.

[0728] An "immersive learning environment" is a learning environment that allows users to deeply concentrate on the subject matter and provides an experience that feels as if they are actually present within it.

[0729] "Information processing means" is a general term for methods and technologies used to analyze and interpret collected data and generate desired deliverables.

[0730] "Immediate support" refers to providing real-time feedback and guidance to users regarding problems and questions they encounter during the learning process.

[0731] "Instruction related to the real world" refers to providing specific and practical guidance that enables users to apply what they have learned in the virtual environment to their actual work environment.

[0732] This invention provides a system for personalizing learning experiences in education and corporate training. This system mainly consists of a server, terminals, and users.

[0733] The server first collects user characteristic information, including learning history, interests, and abilities. Using information processing tools, the server analyzes the user's learning tendencies from the collected data and generates optimal educational materials.

[0734] The generated educational materials are delivered to the user via a terminal using virtual reality or augmented reality technology. The user progresses through this immersive learning environment, and their emotional state is continuously monitored in real time by an emotion recognition processing system. Based on this information, the server dynamically adjusts learning support, extracting and providing gamified elements and immediate assistance to enhance the user's motivation and understanding.

[0735] As a concrete example, consider a scenario where a new employee at a manufacturing company undergoes practical training. The user operates a virtual production line via a head-mounted display, learning the safe handling and efficient operation methods of specific machinery. The system provides personalized guidance based on the user's actions and emotions, supporting the learning process.

[0736] Furthermore, when using generative AI models to create real-time scenarios that provide user performance feedback and suggest next steps, the following prompt is used: "Create a scenario in which a new employee simulates a manufacturing line in a VR environment, using sentiment analysis and gamification elements to support learning."

[0737] This system provides a flexible learning environment that takes individual characteristics into account, enabling the practical application of skills.

[0738] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0739] Step 1:

[0740] The server collects user characteristic information. Specifically, it retrieves data from a database based on user input regarding the user's past learning activities, areas of interest, and abilities. This information is processed to analyze the user's past learning trends and used as input data for the next step.

[0741] Step 2:

[0742] The server generates optimal educational materials based on the user's characteristics information obtained in Step 1. Using data processing tools, the server assembles the relevant educational content in digital format and designs an immersive learning experience using virtual reality or augmented reality technology. This generated educational material becomes the output data for the terminal.

[0743] Step 3:

[0744] The device provides the user with educational materials received from the server. The device utilizes virtual reality or augmented reality technology to present the content in an interactive and immersive way. The user begins learning in this environment, and the device senses the user's input and provides immediate feedback.

[0745] Step 4:

[0746] The server monitors the user's emotional state using emotion recognition technology. It analyzes the user's facial expressions and voice as input to identify their stress levels and comprehension level. Based on these analysis results, it adjusts the content and difficulty level of learning support and outputs it to the terminal.

[0747] Step 5:

[0748] The server uses a generative AI model to provide real-time learning feedback and suggestions for the next steps. It processes user performance data as input and generates gamified elements (badges, points, etc.) and specific improvement suggestions. This result is then displayed on the user's device as final feedback.

[0749] Step 6:

[0750] Users apply what they've learned to real-world tasks through their terminals. Practical guidance from terminals and servers is integrated, making it possible to reproduce skills acquired in the virtual environment in real-world situations. Specific operating procedures and safety instructions are output, which users use to perform their tasks.

[0751] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0752] This invention provides a system for providing users with personalized learning experiences in education and corporate training. It incorporates an emotion engine to understand the user's emotional state and utilize that information to optimize the learning process. The system consists of a server, terminals, and the emotion engine.

[0753] The server first collects user profile information and analyzes the user's learning tendencies using learning history, interests, and past performance data. Based on this analysis, it generates individually optimized educational content and sends this content to the user's device.

[0754] The device presents educational content provided by the server to the user using virtual reality or augmented reality technology. Users can gain an immersive learning experience through the device, and the system tracks the user's location and movements in real time. This tracking data helps the system further refine the educational content.

[0755] The emotion engine analyzes the user's facial expressions, voice, and biometric information to recognize the user's emotional state in real time. Based on this emotional state, the server dynamically adjusts the user's learning process. For example, if the server determines that the user is experiencing stress, it can temporarily suspend the learning plan or take measures to alleviate the stress. Furthermore, based on the emotion engine's data, the device provides personalized feedback to the user and evaluates their learning progress.

[0756] As a concrete example, if this system were implemented in a company's new employee training program, the new employees would receive instruction through virtual reality technology. During this time, the emotion engine would continuously monitor each employee's emotional state. For instance, if an employee's attention becomes distracted during learning, the system would detect this and suggest that the employee take a break or present the content in a simplified form. In this way, new employees can learn at their own pace and efficiently acquire the information.

[0757] Ultimately, the server comprehensively analyzes the data obtained upon the user's completion of the learning process and provides the results to the training staff, which can then be used to improve subsequent educational programs. This allows for continuous support in improving learner performance.

[0758] The following describes the processing flow.

[0759] Step 1:

[0760] The server collects profile information from users. This includes learning history, interests, and past performance data, which is stored in a database.

[0761] Step 2:

[0762] The server analyzes the collected profile information and generates educational content based on the user's learning tendencies and interests. This content is customized and optimized for each user.

[0763] Step 3:

[0764] The server sends the generated educational content to the device. The device receives this content and prepares to present it to the user.

[0765] Step 4:

[0766] The device uses virtual reality or augmented reality technology to present educational content to the user, allowing them to begin an immersive learning experience.

[0767] Step 5:

[0768] The emotion engine analyzes the user's facial expressions, voice, and biometric information in real time to determine their emotional state. This data is then sent to the server.

[0769] Step 6:

[0770] The server dynamically adjusts the user's learning process based on emotional data obtained from the emotion engine. For example, if it determines that the user is fatigued, it suggests taking a break.

[0771] Step 7:

[0772] As users engage in learning activities, the device tracks their location and movements, sending this data to the server. This allows for further adjustments to the content.

[0773] Step 8:

[0774] The server provides users with gamified feedback to enhance learning motivation. Points and badges are awarded and displayed on the device based on achievements.

[0775] Step 9:

[0776] Upon completion of the training, the server comprehensively analyzes all training data and evaluates the user's learning performance. The results are then saved as data for further improvement.

[0777] (Example 2)

[0778] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0779] In today's educational and training environments, providing flexible learning plans tailored to the individual characteristics and emotional states of learners, and ensuring their efficient application, presents a significant challenge. In particular, because learners have diverse learning paces and styles, uniform educational content makes effective learning difficult. Furthermore, learning processes that disregard emotional states can lead to decreased learner motivation. Therefore, addressing these challenges is essential to maximizing learning outcomes.

[0780] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0781] In this invention, the server includes information processing means for collecting and generating user characteristic information, means for providing content to the user using virtual representation technology or extended representation technology, and emotion recognition processing means for detecting and adjusting the user's emotional state. This enables the provision of an optimal learning environment for each learner in real time, thereby improving learning effectiveness.

[0782] "Characteristic information" refers to specific information about the user, including learning history, interests, and individual learning style.

[0783] "Information processing means" refers to hardware and software systems for collecting, analyzing, and generating educational content.

[0784] "Virtual representation technology" refers to technology that allows users to immerse themselves in a computer-generated three-dimensional environment, thereby enabling a realistic learning experience.

[0785] "Augmented representation technology" refers to technology that overlays digital information onto the real world, providing an intuitive learning experience in a real-world environment.

[0786] "Emotion recognition processing means" refers to technology that analyzes a user's facial expressions, voice, and biometric data to determine their emotional state in real time.

[0787] A "learner" refers to an individual who utilizes an educational system, and whose learning environment and content are individually optimized for that person.

[0788] This invention provides a specific form for implementing a system that provides individualized learning experiences in education and corporate training. The components of this system and their operation are described below.

[0789] System configuration:

[0790] 1. Server:

[0791] The server collects user characteristic information and performs analysis using information processing tools. This requires various databases and a computing platform to run machine learning algorithms. The content generation engine generates personalized educational content based on the analysis results.

[0792] The server can use emotion recognition processing to analyze biometric data provided by the user, determine the emotional state in real time, and reflect it in the learning content.

[0793] 2. Terminal:

[0794] A terminal is a device used to implement virtual or augmented representation technologies. This includes VR headsets and AR-enabled mobile devices. The terminal presents received educational content to the user in an immersive manner and tracks the user's actions in real time.

[0795] This tracking function enables the analysis of the user's location and movements, allowing for dynamic adjustments to the display of educational content.

[0796] 3. User:

[0797] A user is an individual who experiences the learning content through their device. Users can progress through their learning at their own pace through feedback from the system.

[0798] Users can also receive personalized content from the system based on their emotional state and learning outcomes.

[0799] Specific example:

[0800] For example, if this system is used for new employee training, new employees will receive simulation training in a workplace environment recreated in virtual reality. If a user experiences stress during the learning process, the server will detect this and suggest a break or flexibly adjust the learning content.

[0801] Example of a prompt:

[0802] "Please explain how the system responds when a user's attention becomes sluggish during a training session using virtual reality technology for new employees."

[0803] This format allows us to address the individual learning needs of each user and provide an efficient and effective learning experience.

[0804] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0805] Step 1:

[0806] The server collects user characteristic information. The input is profile data provided by the user during registration. This data includes learning history, areas of interest, and past evaluations. The server stores this input data in a database and manages it as structured data, representing the user profile. Specifically, the server receives and stores data using a secure communication protocol.

[0807] Step 2:

[0808] The server generates educational content based on collected characteristic information. The input is a user profile obtained from a characteristic information database. The server uses machine learning algorithms to analyze this input data and generate individually optimized educational content. The output is user-optimized educational content, consisting of videos, text, and interactive quizzes. Specifically, the server's content generation engine uses an AI model to create learning materials.

[0809] Step 3:

[0810] The server sends the generated educational content to the terminal. The input is optimized educational content generated within the server. The output is content data that can be displayed on the user's terminal. The server uses an end-to-end encryption protocol to securely deliver the content to the terminal. Specifically, the server uses the HTTPS protocol to deliver data to the terminal.

[0811] Step 4:

[0812] The device presents educational content to the user using virtual representation technology. The input is optimized educational content sent from the server. The device displays the received content as a VR or AR device, providing the user with an immersive learning environment. The output is the visual / auditory data experienced by the user. Specifically, the device renders a 3D virtual classroom using a VR headset.

[0813] Step 5:

[0814] The emotion engine monitors the user's emotional state. Inputs include user actions, facial expressions, and voice data. The emotion engine analyzes these inputs in real time to determine the user's emotional state. Outputs are status information related to the user's emotional state. Specifically, the emotion engine collects data through the camera and microphone and processes it using AI analysis tools.

[0815] Step 6:

[0816] The server adjusts the learning experience based on information obtained from the emotion engine. The input is status information regarding the user's emotional state. The server uses this to re-evaluate the learning plan and make changes as needed. The output is the adjusted educational content and learning plan, which may include relaxing music and pace adjustments. Specifically, the server restructures the content to reduce the user's emotional burden.

[0817] Step 7:

[0818] Users progress through customized learning content via their devices. Inputs consist of customized educational content and feedback. Users utilize this for self-study and improve their learning process based on the feedback. Outputs are feedback regarding learning progress and achievement. Specifically, users follow instructions on their devices and solve interactive problems.

[0819] (Application Example 2)

[0820] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0821] Traditional educational programs and skills training systems have struggled to adapt to individual users' emotional states and learning paces by providing uniform content. In particular, training for technicians to improve their machine operation and maintenance skills within factories requires individualized attention, but while technologies exist to grasp and adjust to diverse emotional states in real time, they are far from practical. Furthermore, when conducting training in a virtual environment, the inability to dynamically adjust content according to the user's emotional state hinders effective learning.

[0822] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0823] In this invention, the server includes information processing means for collecting basic user information and generating instructional content based on the user's learning patterns and interests; means for constructing an immersive educational environment by providing content to the user using virtual environment technology or extended environment technology; and emotion recognition processing means for understanding the user's emotional state and adjusting instructional support accordingly. This makes it possible to adjust the educational process to match the user's individual emotional state, enabling efficient and effective learning and training.

[0824] "User basic information" is a general term for information necessary to provide an individualized educational experience, including the user's characteristics and preferences, past learning history and achievements.

[0825] "Information processing means for generating instructional content" refers to processing devices and software that analyze a user's basic information and create and provide optimal educational content.

[0826] "Virtual environment technology or augmented environment technology" refers to technologies that use digital technology to create and provide users with an environment that does not actually exist but can be experienced visually.

[0827] An "immersive educational environment" refers to a learning space that provides an immersive educational experience designed to give users a sense of being in a place equivalent to physical reality.

[0828] "Emotion recognition processing means" refers to a device or algorithm used to analyze and determine a user's emotional state in real time.

[0829] "Adjusting instructional support" refers to the process of dynamically changing the educational process and content based on the user's emotional state and learning progress to ensure that learning progresses smoothly.

[0830] This invention is a system for providing users with an immersive, personalized learning experience. It aggregates user profile information and dynamically generates educational content. The server, as an information processing tool, collects basic user information and generates optimal instructional content based on that information. This process utilizes the user's past learning data and profile information. The server employs a powerful processor and data storage to analyze this data in real time.

[0831] The device provides users with a visual and interactive educational experience using virtual or augmented reality technologies. Specifically, it presents dynamically generated educational content using virtual reality (VR) or augmented reality (AR) devices. This allows users to learn at their own pace and immerse themselves in content tailored to their interests. This can be easily implemented using devices such as Oculus Quest 2 or AR applications on smartphones.

[0832] The user's emotional state is monitored by sensors and cameras on the device and analyzed through emotion recognition software such as the Affectiva SDK. Based on this real-time emotional data, the server adjusts the instruction content; for example, if the user is stressed, the content is made simpler or adjusted to be more relaxing. This maximizes the user's learning effectiveness.

[0833] For example, when a technician is learning to operate a specific machine in a factory, if emotional data detects a lack of concentration, the system will simplify the training content and help the technician regain confidence in their ability to concentrate again.

[0834] Examples of prompt statements in the AI ​​model generated by this system are as follows:

[0835] "For engineers with decreased concentration, how can we dynamically adjust VR training programs to optimize learning efficiency? Please advise."

[0836] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0837] Step 1:

[0838] The server collects basic user information. This includes user personal information, past learning history, and data on interests. This data is integrated to create profiles for generating educational content. The profiles stored in the database form the basis for providing instruction optimized for each user.

[0839] Step 2:

[0840] The server uses the generated profile to create instructional content tailored to the user. The input is the profile data obtained in step 1, and the output is optimized educational content. The generation AI model is used to create customized learning materials based on the user's learning tendencies. Specifically, the AI ​​model determines the difficulty level and theme of the content.

[0841] Step 3:

[0842] The device receives educational content provided by the server and presents it to the user using virtual environment technology. The input is the educational content from the server, and the output is a visual and interactive learning experience for the user. The device uses Oculus Quest 2 and AR applications to provide the user with an immersive educational environment.

[0843] Step 4:

[0844] The device monitors the user's emotional state in real time. Input is biometric information from sensors and cameras installed in the device, and output is the result of emotion recognition analysis. Analysis is performed using Affectiva SDK, etc., to determine the user's emotional state, such as whether they are focused or stressed.

[0845] Step 5:

[0846] The server adjusts the educational content as needed based on the results of the emotion recognition analysis. The input is the emotion data from step 4, and the output is the adjusted educational content. Specifically, if the user is feeling stressed, the server will either simplify the content or suggest a break for relaxation.

[0847] Step 6:

[0848] The user continues with the educational content based on feedback from the device. The input is the adjusted educational content and feedback from the device, and the output is the improvement in the user's learning progress. The device provides feedback visually or audibly to help the user learn more effectively.

[0849] Through the above processing steps, users can efficiently acquire knowledge and skills in a learning environment that best suits their own pace and emotional state.

[0850] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0851] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0852] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0853] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0854] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0855] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0856] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0857] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0858] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0859] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0860] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0861] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0862] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0863] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0864] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0865] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0866] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0867] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0868] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0869] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0870] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0871] The following is further disclosed regarding the embodiments described above.

[0872] (Claim 1)

[0873] A data processing means for collecting user profile information and generating educational content based on the user's learning tendencies and interests,

[0874] A means of providing content to users using virtual reality or augmented reality technology to create an immersive learning environment,

[0875] An emotion recognition processing means for detecting the user's emotional state and adjusting learning support accordingly,

[0876] A means of gamifying the learning process to motivate learning and managing user feedback,

[0877] A system including a means of communication with a mechanical device for physically performing tasks within a virtual scenario.

[0878] (Claim 2)

[0879] The system according to claim 1, further comprising means for tracking the user's location information and movements in the construction of an immersive learning environment provided by virtual reality technology or augmented reality technology.

[0880] (Claim 3)

[0881] The system according to claim 1, which includes means for analyzing a user's past performance data and formulating an individually customized learning plan in the generation of educational content.

[0882] "Example 1"

[0883] (Claim 1)

[0884] Information processing means for collecting user attribute information and generating educational information based on the user's learning tendencies and interests,

[0885] A means of providing information to users using extended environment technology to build a virtual environment,

[0886] An emotion recognition means for detecting the user's emotions and adjusting support accordingly,

[0887] A means of gamifying the learning process to motivate learning and managing responses to the user,

[0888] A system including a method for providing means of communication with a mechanical system for physically executing activities within a virtual environment.

[0889] (Claim 2)

[0890] The system according to claim 1, further comprising means for tracking the user's location information and actions in the construction of an extended environment provided by virtual environment technology.

[0891] (Claim 3)

[0892] The system according to claim 1, which includes means for analyzing a user's past performance data and formulating an individually customized learning plan in the generation of educational information.

[0893] "Application Example 1"

[0894] (Claim 1)

[0895] Information processing means for collecting user characteristic information and generating educational materials based on the user's learning tendencies and interests,

[0896] A means for providing materials to users using virtual reality or augmented reality technology and constructing an immersive learning environment,

[0897] An emotion recognition processing means for detecting the user's emotional state and adjusting learning support accordingly,

[0898] A means of gamifying the learning process to motivate learning and managing user feedback,

[0899] A means of evaluating user behavior and status and providing immediate support for tasks within a virtual environment,

[0900] To enable practical learning of skills in a work environment, it provides means to offer instruction relevant to the real world.

[0901] A system that includes this.

[0902] (Claim 2)

[0903] The system according to claim 1, further comprising means for tracking the user's location and movements in an immersive learning environment provided by virtual reality technology or augmented reality technology.

[0904] (Claim 3)

[0905] The system according to claim 1, which includes means for analyzing a user's past performance data and formulating an individually customized learning plan in the generation of educational materials.

[0906] "Example 2 of combining an emotion engine"

[0907] (Claim 1)

[0908] Information processing means for collecting user characteristic information and generating educational content based on the user's learning tendencies and interests,

[0909] Means for providing content to users using virtual representation technology or augmented representation technology in order to build an immersive learning environment,

[0910] An emotion recognition processing means for detecting the user's emotional state and adjusting learning support accordingly,

[0911] A tracking means for tracking the user's location and actions in real time while educational content is being displayed,

[0912] means for analysis and generation to provide learning assessment and feedback,

[0913] A system that includes this.

[0914] (Claim 2)

[0915] The system according to claim 1, further comprising means for analyzing the user's actions and biometric information and evaluating their emotional state in the construction of an immersive learning environment provided by virtual representation technology or augmented representation technology.

[0916] (Claim 3)

[0917] The system according to claim 1, which includes means for formulating an individually optimized learning plan based on the user's learning history and interests, and adjusting it in real time, in the generation of educational content.

[0918] "Application example 2 when combining with an emotional engine"

[0919] (Claim 1)

[0920] Information processing means for collecting basic user information and generating instructional content based on the user's learning patterns and interests,

[0921] A means of creating an immersive educational environment by providing content to users using virtual environment technology or augmented environment technology,

[0922] An emotion recognition processing means for understanding the user's emotional state and adjusting guidance and support accordingly,

[0923] To make the educational process more interesting, we have developed a means to entertain the educational process and manage user feedback.

[0924] A system that includes means of communication with mechanical devices for performing virtual tasks in the real world.

[0925] A means for acquiring the user's biometric information and facial expression data, and dynamically changing the virtual content according to the emotional state,

[0926] (Claim 2)

[0927] The system according to claim 1, further comprising means for tracking the user's location information and movements in real time in the construction of an immersive educational environment provided by virtual reality technology or augmented reality technology.

[0928] The system according to claim 1, further comprising means for realistically simulating the user's operating skills and improving real-world operating skills based on the entire environment.

[0929] (Claim 3)

[0930] The system according to claim 1, which includes means for analyzing users' past performance data and planning individually optimized educational plans in the generation of educational content.

[0931] The system according to claim 1, comprising means for adjusting educational scenarios based on user sentiment data to support efficient learning. [Explanation of symbols]

[0932] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A data processing means for collecting user profile information and generating educational content based on the user's learning tendencies and interests, A means of providing content to users using virtual reality or augmented reality technology to create an immersive learning environment, An emotion recognition processing means for detecting the user's emotional state and adjusting learning support accordingly, A means of gamifying the learning process to motivate learning and managing user feedback, A system including a means of communication with a mechanical device for physically performing tasks within a virtual scenario.

2. The system according to claim 1, further comprising means for tracking the user's location information and movements in the construction of an immersive learning environment provided by virtual reality technology or augmented reality technology.

3. The system according to claim 1, which includes means for analyzing a user's past performance data and formulating an individually customized learning plan in the generation of educational content.

Images