system

Abstract

We provide the system. [Solution] A data collection means for collecting employees' past work performance data and learning history, A data analysis method for analyzing the aforementioned data and evaluating the skill level and learning tendencies of each employee, A curriculum generation means that automatically generates training tasks based on the evaluation using generative artificial intelligence technology, A means for constructing a virtual environment that allows users to access virtual reality by designing a virtual space according to the aforementioned training task, A monitoring means that monitors the user's operations within the virtual space and provides real-time feedback, Based on the aforementioned feedback, a progress management means evaluates learning progress and adjusts the curriculum as necessary, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional training methods have problems in that it is difficult to customize according to the skill levels of individual employees, and they are costly and inefficient. Also, in industries involving dangerous operations, there is a lack of effective methods for safely acquiring practical skills. This invention aims to solve these problems and provide an individually adaptable and safe learning environment.

Means for Solving the Problems

[0005] This invention provides a data analysis means for collecting employees' past work performance data and learning history, and evaluating their skill levels and learning tendencies based on this data. It also includes a curriculum generation means that automatically generates optimal training tasks based on the evaluation results using generative artificial intelligence technology. Furthermore, it provides a means for designing a virtual space based on the generated curriculum, enabling users to safely conduct training in a virtual environment. By monitoring the user's operations within the virtual space and providing real-time feedback, skill improvement can be achieved more efficiently.

[0006] "Data collection means" refers to an apparatus or method for collecting employees' past work performance data and learning history.

[0007] "Data analysis means" refers to a device or method for evaluating the skill level and learning tendencies of each employee using collected data.

[0008] A "curriculum generation means" is a device or method for automatically generating optimal training tasks based on evaluated data using generative artificial intelligence technology.

[0009] "Virtual environment construction means" refers to a device or method for designing a virtual space based on generated training tasks and providing an environment in which a user can perform training.

[0010] "Monitoring means" refers to a device or method for monitoring a user's operations within a virtual space and using that data to generate real-time feedback.

[0011] A "progress management device" is a device or method for evaluating a user's learning progress based on real-time feedback and adjusting the curriculum as needed.

[0012] "Feedback generation means" refers to a device or method for generating feedback that points out areas for improvement in real time based on the user's actions.

[0013] A "terminal" refers to a device used by a user to access a virtual environment and receive training; it is a device that provides an interface between the user and the device. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the language used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] In embodiments of the present invention, a server acts as the central control system. The server extracts employees' past work performance data and learning history from a database and analyzes their skill levels and learning tendencies based on this data. The server utilizes machine learning and other statistical methods to generate training curricula optimized for each individual employee.

[0036] Once the curriculum is generated, the server prepares a dataset for the virtual environment. Using this dataset, the terminal renders the virtual reality space that the user will access, preparing it for the user to begin training. The user can access this virtual environment through the terminal and train securely.

[0037] While a user is undergoing training, the device transmits user operation information to the server. The server receives this information in real time and monitors it. Based on the data obtained from monitoring, the server provides immediate feedback to the user. This feedback includes areas for improvement and successes in the operation, allowing the user to acquire skills more efficiently.

[0038] As a concrete example, consider a user learning to operate a new machine tool in the manufacturing industry. The server analyzes the user's past operation history to identify challenges in specific operations and generates a curriculum specifically tailored for improvement. The terminal is configured to allow the user to operate the machine tool in a virtual manufacturing environment via a VR headset. Through repeated training and real-time feedback, the user can acquire skills while reducing risks in the actual work environment.

[0039] This configuration allows for safe and efficient skill development, and is expected to have significant applications, particularly in industries requiring operations that involve risks or complexity.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The server extracts past work performance data and learning history of employees from the database. This data serves as foundational information for understanding each employee's skill level and operational tendencies.

[0043] Step 2:

[0044] The server cleanses the extracted data and analyzes it using machine learning algorithms. This identifies each employee's strengths and skill gaps that need improvement.

[0045] Step 3:

[0046] Based on the analysis results, the server utilizes generative AI technology to automatically generate training curricula optimized for each employee. These curricula include specific training content and virtual environment scenarios.

[0047] Step 4:

[0048] The server prepares the dataset for the virtual environment according to the generated curriculum. It transfers the necessary data to the terminal and sets up a virtual environment that the user can access.

[0049] Step 5:

[0050] The user accesses the VR space using their device and selects their training from the start menu. After selection, they enter the virtual environment and perform each task step by step.

[0051] Step 6:

[0052] The device continuously sends user action data to the server in real time. The server monitors this data and tracks the user's movements.

[0053] Step 7:

[0054] The server analyzes user activity data and generates real-time feedback. It provides feedback to the user via the terminal regarding successful actions and areas for improvement.

[0055] Step 8:

[0056] Users receive feedback and incorporate it into their next actions. Skill improvement is aimed at through repeated training as needed.

[0057] Step 9:

[0058] The server records the overall progress of the training session and evaluates the user's achievements and skill improvement. This data will be used to adjust future training curricula.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] Traditional training methods struggle to efficiently achieve optimal skill improvement for individual employees, and in industries requiring complex operations, skill acquisition is time-consuming and risky. Furthermore, insufficient real-time feedback makes immediate improvement difficult, hindering the maximization of training effectiveness.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes information acquisition means for extracting employees' past work data and learning history, data analysis means for analyzing the information using machine learning technology to derive individual skill levels and learning tendencies, and training content creation means for creating a training curriculum based on the analysis results using a generative artificial intelligence model. This enables the provision of real-time feedback and optimized training curricula to individual employees.

[0064] "Information acquisition means" refers to a function for extracting employees' past work data and learning history.

[0065] "Data analysis tools" refer to functions that use machine learning techniques to analyze individual skill levels and learning tendencies based on acquired information.

[0066] The "training content creation method" is a function that automatically generates a training curriculum using a generative artificial intelligence model based on the analysis results.

[0067] "Digital environment provision means" refers to functions for constructing a virtual environment and generating a simulated reality space that users can experience.

[0068] The "operation evaluation means" is a function that collects operation information within the user's virtual environment, analyzes it in real time, and provides immediate feedback.

[0069] "Educational management tools" are functions for reassessing learning progress based on feedback and updating the training curriculum as needed.

[0070] A "feedback generation mechanism" is a function that provides real-time improvement suggestions based on user behavior information.

[0071] An "electronic device" is a device equipped with functions to provide connectivity between a user and equipment within a virtual environment.

[0072] In this invention, the entire system is centrally managed by a server. The server extracts employees' past work data and learning history from a database using information retrieval means. The extracted data is analyzed by data analysis means using machine learning techniques to derive the skill level and learning tendencies of individual employees. Machine learning libraries such as TENSORFLOW® and PyTorch may be used for this purpose.

[0073] Based on the analysis results, the server uses a generative artificial intelligence model (such as GPT or BERT) to automatically generate an optimal training curriculum through a training content creation mechanism. Following this curriculum, the server executes a digital environment provision mechanism for the virtual environment, constructing a virtual reality space accessible to the user. This virtual reality space is created using rendering software such as Unity or Unreal Engine.

[0074] The terminal displays a provided virtual environment to the user and supports the user's experience using a VR headset or other interaction devices. The user can begin training in this environment, and the terminal uses a motion evaluation system to transmit the user's operation information to the server in real time.

[0075] The server analyzes the received operational information and provides immediate improvement suggestions to the user through a feedback generation mechanism. This allows users to acquire skills efficiently. Furthermore, the education management mechanism allows for reassessment of learning progress and updating the training curriculum as needed.

[0076] As a concrete example, when learning to use a new machine tool in the manufacturing industry, the server identifies challenges from past operation history and generates a curriculum. The terminal is configured to allow the user to operate the machine tool in a virtual factory through a VR headset. The user practices repeatedly in this virtual environment and receives real-time feedback, thereby honing their skills while reducing risks in the actual work environment.

[0077] An example of a prompt message is, "I want to learn how to operate a new machine tool." This prompts the system to suggest an appropriate training program, effectively supporting the user's skill improvement.

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] The server extracts employees' past work data and learning history from the database. Employee IDs and past project information are provided as input. Based on this data, the server outputs a series of pieces of information in an organized format, visualizing their past performance. This step involves using SQL queries to filter the necessary data and organize it into a data frame.

[0081] Step 2:

[0082] The server passes the extracted information to a data analysis tool, which then analyzes it using a machine learning model. Inputs include performance metrics such as past operation time, error rate, and success rate. The server feeds this data into a machine learning algorithm to analyze the skill level and learning tendencies of individual employees. This step involves using TensorFlow or PyTorch to train and predict data.

[0083] Step 3:

[0084] The server utilizes a generative artificial intelligence model to generate training curricula based on the analysis results. Skill assessment results are given as input. The server uses generative AI technology to generate training tasks optimized for each employee and outputs them in text format. This process includes the automatic generation of curricula using technologies such as GPT and BERT.

[0085] Step 4:

[0086] The server constructs digital data for the virtual environment according to the generated curriculum. Inputs include simulation scenario data and 3D model templates. Based on this, the server designs the digital environment and sends it to the terminal in dataset format. This step involves operating Unity or Unreal Engine to build the virtual environment and configuring the virtual space.

[0087] Step 5:

[0088] The terminal uses virtual environment data received from the server to render the visual space accessed by the user. Inputs include 3D model data and simulation scenarios. The terminal uses these to generate a virtual reality space and output it in a format that the user can experience through a VR device. This is a real-time rendering operation utilizing a 3D graphics engine.

[0089] Step 6:

[0090] The user begins training in a virtual space using a terminal. The user performs virtual activities using a VR headset or haptic device. The data input here consists of the user's movements and operation commands, which are transmitted to the server in real time via the terminal.

[0091] Step 7:

[0092] The server receives user operation information and analyzes it in real time through an operation evaluation means. Input includes user operation logs and time data. Based on this, the server provides immediate feedback using a feedback generation means, outputting suggestions for improvement to the user. This step involves high-speed data processing and immediate response generation.

[0093] Step 8:

[0094] The server re-evaluates learning progress using educational management tools based on feedback and updates the curriculum as needed. Feedback data and learning progress information are provided as input. The server uses this to optimize the content for the next training session and outputs the updated curriculum as text. This includes dynamic adjustments to the curriculum.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] Conventional skill acquisition methods lack a suitable training environment for efficiently and safely improving skills in operating industrial automated machinery. In particular, practicing operation while receiving real-time feedback is important, but using actual equipment incurs significant risks and costs. The objective of this invention is to improve this situation and provide an efficient and low-risk skills training method.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes data collection means for collecting past work performance data and learning history of workers; data analysis means for analyzing the data and evaluating each worker's skill level and learning tendencies; and instruction plan generation means for automatically generating training tasks based on the evaluation using artificial intelligence technology. This enables participants to safely and efficiently learn industrial automated machine skills in a virtual reality environment and receive real-time feedback.

[0100] "Worker" refers to an individual receiving training for the purpose of acquiring skills in operating industrial automated machinery.

[0101] "Data collection means" refers to a device or method for automatically collecting past work performance data and learning history of an employee.

[0102] "Data analysis means" refers to methods and devices that evaluate the skill level and learning tendencies of each worker based on collected data.

[0103] A "teaching plan generation means" is a device or system that automatically generates training tasks using artificial intelligence technology based on evaluation results.

[0104] "Virtual space construction means" refers to a method or apparatus for constructing a virtual reality environment according to training tasks and providing a situation that participants can access.

[0105] "Monitoring means" refers to a system or method for monitoring participants' actions in a virtual reality environment in real time and providing feedback.

[0106] "Progress management means" refers to methods or devices that evaluate learning progress based on feedback and adjust instructional plans as needed.

[0107] "Educational tools" refer to functions and devices that enable the safe and efficient learning of industrial automated machine skills in a virtual reality environment.

[0108] "Participant" refers to an individual who participates in training provided in a virtual reality environment.

[0109] A "feedback generation means" is a device or system for indicating areas for improvement in real time based on the participant's actions.

[0110] To implement this invention, a server, a terminal, and a user must be involved in a series of systems. The server is connected to a database system for efficiently collecting the worker's past work performance data and learning history. Specifically, database management systems such as MySQL® or PostgreSQL could be used.

[0111] The server utilizes machine learning libraries such as TensorFlow and PyTorch to analyze the collected data. Based on these analysis results, artificial intelligence technology is used to generate instructional plans tailored to each worker. In this process, a generative AI model is in operation, creating curricula that are customized to each individual's skills and learning tendencies.

[0112] Users access a virtual reality space via a VR headset. This virtual environment is built using virtual reality engines such as Unity and Unreal Engine, making it possible to simulate industrial automated machinery within the virtual space.

[0113] The terminal is responsible for transmitting user operation data to the server in real time. This allows the server to monitor the user's progress and provide immediate feedback. This feedback is used to measure the user's progress in the virtual space and to point out areas for improvement.

[0114] A concrete example would be a scenario where a new employee learns to operate a new machine. Based on the worker's past operation history and learning tendencies, an optimal training plan is devised, allowing the user to practice sufficiently in the VR environment before operating the actual machine. An example of a prompt message would be, "Analyze the following employee data to create an optimal curriculum for welding machine operation: {Employee work history data}."

[0115] This allows participants to efficiently improve their skills while minimizing the risks involved in actual machine operation.

[0116] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0117] Step 1:

[0118] The server uses a database management system (e.g., MySQL) to collect the worker's past work performance data and learning history. The input for this step is the work history data in the database. The server executes SQL queries to extract this data and convert it into the format required for analysis. The output is work data ready for data analysis.

[0119] Step 2:

[0120] The server inputs the collected work data into a machine learning model using TensorFlow. The input consists of performance data from individual workers. The machine learning model processes the data and evaluates the workers' skill levels and learning tendencies. The output provides evaluation information for each worker.

[0121] Step 3:

[0122] The server generates an optimal training plan using a generative AI model based on the acquired evaluation information. The input for this step is the evaluation information obtained through machine learning. The generated training plan provides the most effective curriculum for improving each worker's skills. As output, a customized training plan is obtained for each worker.

[0123] Step 4:

[0124] The server designs a virtual space using the Unity engine and sends it to the terminal. The inputs are the lesson plan and the parameters for the virtual environment design. The server instructs the creation of the virtual environment, and the output is virtual environment data that the user can access.

[0125] Step 5:

[0126] The user enters a virtual reality space through a VR headset. The input in this step is virtual environment data sent from the server. The user works on training tasks and performs operations within the virtual environment. The output is a log of operations performed during training.

[0127] Step 6:

[0128] The terminal transmits user operation data to the server in real time. The input is the user's operation log, which is transferred to the server using a specific protocol. The output is the transmitted operation data, which is received by the server.

[0129] Step 7:

[0130] The server analyzes the received operation data and generates feedback, including areas for improvement and successes. The input for this step is the user's operation data. The server evaluates the data in real time, generates feedback information as output, and sends it to the terminal.

[0131] Step 8:

[0132] The terminal receives feedback from the server and displays it to the user within the virtual environment. The input is the feedback information sent from the server. The terminal presents the feedback appropriately to the user, encouraging understanding and improvement. The output is the feedback received by the user.

[0133] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0134] In embodiments of the present invention, a system incorporating an emotion engine has the ability to recognize the user's emotional state in real time and dynamically adjust the training content and feedback accordingly. The server collects the employee's past work performance data and learning history and performs a skills assessment based on this data. Then, using generative artificial intelligence technology, it generates an optimal training curriculum for each employee.

[0135] The server also processes user emotional data transmitted from the terminal via an emotion engine. This emotional data is obtained from the user's facial expressions, voice, vital signs, etc., using emotion recognition software and sensor technology. The server analyzes the user's emotional state in real time and adjusts training tasks accordingly, thereby reducing stress and fatigue and maximizing learning effectiveness.

[0136] When a user performs training in a virtual reality space, the device sends emotional data along with user operation information to the server. The server uses this data to monitor the user's performance and generate real-time feedback appropriate to their emotional state. This feedback includes not only information on the accuracy of operations and areas for improvement, but also positive information to enhance the user's psychological sense of security.

[0137] As a concrete example, consider a user learning new surgical techniques in a medical setting. The server analyzes data and training information related to past surgeries and generates a curriculum tailored to that user. As the user practices surgery in the VR environment, the device senses their anxiety and level of concentration and sends this information to the server. Through an emotion engine, the server provides corresponding feedback, promoting skill improvement while maintaining the user's sense of security.

[0138] Thus, the present invention provides flexible training tailored to the emotions of employees, realizing a form that enables safer and more effective skill acquisition.

[0139] The following describes the processing flow.

[0140] Step 1:

[0141] The server extracts employees' past work performance data and learning history from the database. This data is used to analyze each employee's skills and identify their learning needs.

[0142] Step 2:

[0143] The server uses an emotion engine to acquire emotional data such as facial expressions, voice, and vital signs transmitted from the terminal in real time. Based on this data, it recognizes the user's emotional state.

[0144] Step 3:

[0145] The server combines skill assessment results and emotional state analysis results to generate a user-optimized training curriculum using generative AI technology. The curriculum incorporates flexible responses based on the user's emotions.

[0146] Step 4:

[0147] The terminal renders the virtual reality environment based on the curriculum provided by the server, preparing it for user access. The user enters the VR space and begins training.

[0148] Step 5:

[0149] The user performs training and operations within the VR space via a device. The device transmits the user's operation data and emotional data to the server in real time.

[0150] Step 6:

[0151] The server monitors the user's performance and emotional state based on the received operational and emotional data, and generates real-time feedback. This feedback is particularly sensitive to the user's emotions.

[0152] Step 7:

[0153] Users continue their training based on feedback received through their devices, improving their skills at their own pace. The server dynamically adjusts the curriculum and training content as needed.

[0154] Step 8:

[0155] The server records the overall progress of the training and evaluates the user's learning outcomes. This information can be used to adjust and improve future training curricula.

[0156] (Example 2)

[0157] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0158] Current training systems struggle to provide flexible training assignments that appropriately reflect each employee's learning tendencies and emotional state, thus failing to contribute to more efficient learning and reduced stress. Furthermore, feedback provided may undermine motivation because it is not based on emotional state.

[0159] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0160] In this invention, the server includes information gathering means, information analysis means, and task generation means. This enables the provision of personalized training tailored to employees' learning history and emotional states, as well as dynamic adjustment of training content in response to real-time circumstances.

[0161] "Information gathering means" refers to devices and technologies for collecting information on employees' past work performance and learning history.

[0162] "Information analysis tools" refer to devices and technologies that evaluate the skill level and learning tendencies of individual employees based on collected information.

[0163] A "task generation method" refers to a device or technology that uses machine learning techniques to automatically generate training tasks based on evaluation results.

[0164] "Virtual environment provisioning means" refers to devices and technologies for constructing a virtual space according to training tasks and making it accessible to users.

[0165] "Monitoring means" are devices or technologies that observe a user's operations within a virtual space and provide real-time feedback.

[0166] "Emotional response methods" refer to devices or technologies that use emotion recognition devices to evaluate the user's emotional state and dynamically adjust training tasks according to that state.

[0167] "Progress management tools" are devices or technologies that evaluate learning progress based on feedback and adjust assignments as needed.

[0168] A "feedback generation means" refers to a device or technology that provides real-time feedback on areas for improvement based on the user's actions and emotional state.

[0169] A "terminal" is a device that provides an interface between the user and the device and is equipped with emotion recognition capabilities.

[0170] In this system, three main entities—the server, the terminal, and the user—work in coordination. The server uses information gathering tools to efficiently collect information on employees' past work performance and learning history. This provides the foundational data necessary to appropriately evaluate each employee's skill level and learning tendencies. This data collection often utilizes software such as database systems and log analysis tools.

[0171] Next, the server uses information analysis tools to process the collected information with advanced algorithms and analyze the learning tendencies of individual employees. For example, it identifies skill areas that employees should particularly strengthen based on past training data and work history. Statistical software and data mining tools are used for this analysis.

[0172] Subsequently, the server uses a task generation mechanism to generate appropriate training tasks based on the analysis results. A generation AI model is used for this generation, and by inputting prompts, an individualized training curriculum is formed. An example of a prompt is, "Create an optimal training plan based on the employee's name, past performance, and current learning needs."

[0173] The terminal uses a virtual environment provisioning mechanism to construct a virtual space according to the generated training tasks. This allows the user to perform training in a virtual reality environment, during which the terminal collects and transmits the user's emotional data through its emotion recognition function. For example, virtual reality headsets and voice / facial recognition technologies are used to measure the user's level of concentration and anxiety in real time.

[0174] When a user performs an action in the virtual space, monitoring devices transmit that action information to a server, which then generates real-time feedback based on that information. This feedback includes suggestions for improvement and positive messages, maximizing the learning effect.

[0175] This system provides a flexible training environment that takes into account the emotional state of employees, promoting more efficient and safer skill acquisition.

[0176] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0177] Step 1:

[0178] The server uses information gathering tools to collect employees' past work performance information and learning history. It takes employee log data and training reports as input and builds a history database for each individual based on this data. Specifically, it extracts relevant information from the database and performs data cleansing to create a highly accurate dataset.

[0179] Step 2:

[0180] The server analyzes the collected data using information analysis tools. It receives historical data acquired in step 1 as input and evaluates employees' skill levels and learning tendencies based on this data. The analysis uses data mining techniques, grouping employees using pattern recognition and clustering methods. The output is a skill evaluation report for each employee.

[0181] Step 3:

[0182] The server generates training tasks based on the evaluation results using a task generation mechanism. As input, prompts are provided to the generating AI model based on the technical evaluation report from Step 2. An example prompt might be, "Design a training scenario tailored to the current skill set of the employee named [Employee Name]." The output is a personalized training plan.

[0183] Step 4:

[0184] The terminal uses a virtual environment provisioning mechanism to construct a virtual space according to the generated training tasks. It receives the training plan created in step 3 as input and designs the virtual environment using visual simulation software. Specific operations include 3D modeling and dynamic scenario generation. The output is a virtual reality space that the user can experience.

[0185] Step 5:

[0186] The user begins training in a virtual space. During this time, the device collects the user's emotional data through its emotion recognition function and sends it to the server. The input includes the user's facial expressions and voice data, and emotion analysis is performed based on this data. Specifically, it acquires raw data from the camera and microphone and analyzes it in real time. The output is a report of the user's emotional state.

[0187] Step 6:

[0188] The server uses monitoring devices to monitor user actions and emotional states in real time and generates feedback. It receives user action data and emotional state reports from step 5 as input and performs a performance evaluation. As output, it generates appropriate feedback for the user and sends it to the terminal. Specific actions may include evaluating the accuracy of actions and including positive messages.

[0189] (Application Example 2)

[0190] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0191] In today's work environment, training to improve employee skills is essential. However, providing individualized instruction tailored to each employee's learning tendencies and emotional state is challenging, and there is a need to facilitate efficient learning while reducing employee stress and anxiety. Traditional training systems are insufficient to address these challenges, hindering efficient skill acquisition.

[0192] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0193] In this invention, the server includes information gathering means for collecting past work performance data and learning history of employees; information analysis means for analyzing the information and evaluating each employee's ability level and learning tendencies; and learning curriculum generation means for automatically generating training tasks based on the evaluation using generative artificial intelligence technology. This enables the provision of optimized training for each employee, reduces stress, and allows for efficient skill acquisition by providing real-time feedback that responds to their emotional state.

[0194] "Information gathering means" is a general term for devices and methods used to collect employees' past work performance data and learning history.

[0195] "Information analysis tools" is a general term for devices and methods used to analyze collected data and evaluate employees' skill levels and learning tendencies.

[0196] "Learning curriculum generation means" is a general term for devices and methods that use artificial intelligence generation technology to automatically generate optimal training tasks for each employee based on evaluation results.

[0197] "Virtual space construction means" is a general term for devices and methods that design a virtual reality environment based on training tasks and allow users to access that environment.

[0198] "Monitoring measures" is a general term for devices and methods that allow users to see in real time what operations they are performing in a virtual space and provide appropriate feedback.

[0199] "Progress management tools" is a general term for devices and methods that use feedback results to evaluate the progress of learning and adjust the curriculum as needed.

[0200] "Emotional analysis tools" is a general term for devices and methods that analyze a user's emotional state in real time and provide feedback that corresponds to those emotions.

[0201] The system that implements this application example first uses a server to collect employees' past work performance data and learning history, recording it as an information gathering tool. The collected data is analyzed by an information analysis tool on the server to evaluate the employees' skill levels and learning tendencies. Based on these analysis results, a learning curriculum generation tool utilizing generative artificial intelligence technology automatically generates training tasks optimized for each employee.

[0202] Next, a virtual reality environment is designed according to the training task using a virtual space construction means, and the user can access this environment via a terminal. While the user is training in the virtual space, their operation information is monitored by a monitoring means. During this process, the user's emotional state is analyzed in real time by an emotion analysis means, and feedback corresponding to that emotion is generated.

[0203] The device functions as smart glasses or a VR headset worn by the user, and its role is to display the real-world training environment in a virtual space. This device is equipped with sensor technology used for emotion recognition (for example, a camera to capture facial expressions and a voice recognition microphone), and transmits emotion data to a server.

[0204] As a concrete example, consider a scenario where a user is receiving training to learn how to operate new machinery in a factory. The server senses the user's anxiety, provides positive feedback tailored to the situation, and adjusts the learning pace, thereby reducing stress and promoting efficient skill acquisition.

[0205] An example of a prompt is, "Analyze the worker's facial expressions and tone of voice, and consider what positive message to display when they feel anxious." This is used as input to a generative AI model and forms part of emotion recognition and feedback generation.

[0206] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0207] Step 1:

[0208] The server collects employees' past work performance data and learning history using information gathering tools. This information is stored in a database. The input is each employee's historical data, and the output is structured data stored in the database. Specifically, the server accesses the company's work record system and extracts data via APIs, etc.

[0209] Step 2:

[0210] The server analyzes data collected by information analysis tools to evaluate employees' skill levels and learning tendencies. The input here is the data collected in step 1, and the output is the skill evaluation score and learning tendency profile calculated for each employee. Specifically, the server applies machine learning algorithms and performs data analysis using Python libraries (e.g., Pandas, scikit-learn).

[0211] Step 3:

[0212] The server uses generative artificial intelligence technology to automatically generate training tasks based on evaluation results using a learning curriculum generation method. The input is the evaluation score and profile obtained in step 2, and the output is an individualized training task. Specifically, a generative AI model is put into practice, and AI inference is performed using prompt sentences to derive the optimal curriculum.

[0213] Step 4:

[0214] The server utilizes virtual space construction methods to design a virtual reality environment based on training tasks, making it accessible to users from their terminals. The input is the training task generated in the previous step, and the output is the constructed virtual environment. Specifically, a simulation environment is generated using a VR development tool (e.g., Unity).

[0215] Step 5:

[0216] The terminal monitors the user's actions within the virtual space and transmits data to the server via the monitoring system. Inputs include user action information and emotional state, while outputs include action logs and emotional data. Specifically, sensors on the terminal record facial expressions and voice, and transmit this data to the server.

[0217] Step 6:

[0218] The server analyzes the user's emotional state in real time using emotion analysis tools and automatically generates feedback using a generative AI model. The input is the emotion data from step 5, and the output is a feedback message tailored to the user's emotions. Specifically, the server utilizes natural language processing technology to generate encouraging messages and other similar messages in real time.

[0219] Step 7:

[0220] The server uses a progress management system to evaluate the user's learning progress based on the generated feedback and adjusts the curriculum as needed. The input is the feedback generated in step 6, and the output is the adjusted new curriculum. Specifically, a progress management algorithm is applied, and the learning content changes dynamically.

[0221] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0222] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0223] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0224] [Second Embodiment]

[0225] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0226] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0227] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0228] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0229] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0230] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0231] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0232] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0233] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0234] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0235] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0236] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0237] In embodiments of the present invention, a server acts as the central control system. The server extracts employees' past work performance data and learning history from a database and analyzes their skill levels and learning tendencies based on this data. The server utilizes machine learning and other statistical methods to generate training curricula optimized for each individual employee.

[0238] Once the curriculum is generated, the server prepares a dataset for the virtual environment. Using this dataset, the terminal renders the virtual reality space that the user will access, preparing it for the user to begin training. The user can access this virtual environment through the terminal and train securely.

[0239] While a user is undergoing training, the device transmits user operation information to the server. The server receives this information in real time and monitors it. Based on the data obtained from monitoring, the server provides immediate feedback to the user. This feedback includes areas for improvement and successes in the operation, allowing the user to acquire skills more efficiently.

[0240] As a concrete example, consider a user learning to operate a new machine tool in the manufacturing industry. The server analyzes the user's past operation history to identify challenges in specific operations and generates a curriculum specifically tailored for improvement. The terminal is configured to allow the user to operate the machine tool in a virtual manufacturing environment via a VR headset. Through repeated training and real-time feedback, the user can acquire skills while reducing risks in the actual work environment.

[0241] This configuration allows for safe and efficient skill development, and is expected to have significant applications, particularly in industries requiring operations that involve risks or complexity.

[0242] The following describes the processing flow.

[0243] Step 1:

[0244] The server extracts past work performance data and learning history of employees from the database. This data serves as foundational information for understanding each employee's skill level and operational tendencies.

[0245] Step 2:

[0246] The server cleanses the extracted data and analyzes it using machine learning algorithms. This identifies each employee's strengths and skill gaps that need improvement.

[0247] Step 3:

[0248] Based on the analysis results, the server utilizes generative AI technology to automatically generate training curricula optimized for each employee. These curricula include specific training content and virtual environment scenarios.

[0249] Step 4:

[0250] The server prepares the dataset for the virtual environment according to the generated curriculum. It transfers the necessary data to the terminal and sets up a virtual environment that the user can access.

[0251] Step 5:

[0252] The user accesses the VR space using their device and selects their training from the start menu. After selection, they enter the virtual environment and perform each task step by step.

[0253] Step 6:

[0254] The device continuously sends user action data to the server in real time. The server monitors this data and tracks the user's movements.

[0255] Step 7:

[0256] The server analyzes user activity data and generates real-time feedback. It provides feedback to the user via the terminal regarding successful actions and areas for improvement.

[0257] Step 8:

[0258] Users receive feedback and incorporate it into their next actions. Skill improvement is aimed at through repeated training as needed.

[0259] Step 9:

[0260] The server records the overall progress of the training session and evaluates the user's achievements and skill improvement. This data will be used to adjust future training curricula.

[0261] (Example 1)

[0262] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0263] Traditional training methods struggle to efficiently achieve optimal skill improvement for individual employees, and in industries requiring complex operations, skill acquisition is time-consuming and risky. Furthermore, insufficient real-time feedback makes immediate improvement difficult, hindering the maximization of training effectiveness.

[0264] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0265] In this invention, the server includes information acquisition means for extracting employees' past work data and learning history, data analysis means for analyzing the information using machine learning technology to derive individual skill levels and learning tendencies, and training content creation means for creating a training curriculum based on the analysis results using a generative artificial intelligence model. This enables the provision of real-time feedback and optimized training curricula to individual employees.

[0266] "Information acquisition means" refers to a function for extracting employees' past work data and learning history.

[0267] "Data analysis tools" refer to functions that use machine learning techniques to analyze individual skill levels and learning tendencies based on acquired information.

[0268] The "training content creation method" is a function that automatically generates a training curriculum using a generative artificial intelligence model based on the analysis results.

[0269] "Digital environment provision means" refers to functions for constructing a virtual environment and generating a simulated reality space that users can experience.

[0270] The "operation evaluation means" is a function that collects operation information within the user's virtual environment, analyzes it in real time, and provides immediate feedback.

[0271] "Educational management tools" are functions for reassessing learning progress based on feedback and updating the training curriculum as needed.

[0272] A "feedback generation mechanism" is a function that provides real-time improvement suggestions based on user behavior information.

[0273] An "electronic device" is a device equipped with functions to provide connectivity between a user and equipment within a virtual environment.

[0274] In this invention, the entire system is centrally managed by a server. The server extracts employees' past work data and learning history from a database using information retrieval means. The extracted data is analyzed by data analysis means using machine learning techniques to derive the skill level and learning tendencies of individual employees. Machine learning libraries such as TensorFlow and PyTorch may be used for this purpose.

[0275] Based on the analysis results, the server uses a generative artificial intelligence model (such as GPT or BERT) to automatically generate an optimal training curriculum through a training content creation mechanism. Following this curriculum, the server executes a digital environment provision mechanism for the virtual environment, constructing a virtual reality space accessible to the user. This virtual reality space is created using rendering software such as Unity or Unreal Engine.

[0276] The terminal displays a provided virtual environment to the user and supports the user's experience using a VR headset or other interaction devices. The user can begin training in this environment, and the terminal uses a motion evaluation system to transmit the user's operation information to the server in real time.

[0277] The server analyzes the received operational information and provides immediate improvement suggestions to the user through a feedback generation mechanism. This allows users to acquire skills efficiently. Furthermore, the education management mechanism allows for reassessment of learning progress and updating the training curriculum as needed.

[0278] As a concrete example, when learning to use a new machine tool in the manufacturing industry, the server identifies challenges from past operation history and generates a curriculum. The terminal is configured to allow the user to operate the machine tool in a virtual factory through a VR headset. The user practices repeatedly in this virtual environment and receives real-time feedback, thereby honing their skills while reducing risks in the actual work environment.

[0279] An example of a prompt message is, "I want to learn how to operate a new machine tool." This prompts the system to suggest an appropriate training program, effectively supporting the user's skill improvement.

[0280] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0281] Step 1:

[0282] The server extracts the past work data and learning history of employees from the database. As inputs, employee IDs and past project information are provided. Based on this data, the server outputs a series of information in an organized format and visualizes their past performance. In this step, SQL queries are used to filter the necessary data and perform operations to summarize it in a data frame format.

[0283] Step 2:

[0284] The server passes the extracted information to data analysis means and analyzes it using a machine learning model. As inputs, performance indicators such as past operation time, error rate, success rate, etc. are included. The server inputs these data into a machine learning algorithm and analyzes the skill levels and learning tendencies of individual employees. This step includes operations such as using TensorFlow or PyTorch for data learning and prediction.

[0285] Step 3:

[0286] The server utilizes a generative artificial intelligence model to generate a training curriculum based on the analysis results. Skill evaluation results are provided as inputs. The server uses generative artificial intelligence technology to generate training tasks optimized for each employee and outputs them in text format. This process includes operations such as automatically generating the curriculum using technologies such as GPT or BERT.

[0287] Step 4:

[0288] The server constructs digital data for a virtual environment according to the generated curriculum. As inputs, scenario data for simulation and templates of 3D models are included. Based on this, the server designs a digital environment and transmits it to the terminal in a dataset format. This step includes operations such as operating Unity or Unreal Engine for virtual environment construction and setting up the virtual space.

[0289] Step 5:

[0290] The terminal uses virtual environment data received from the server to render the visual space accessed by the user. Inputs include 3D model data and simulation scenarios. The terminal uses these to generate a virtual reality space and output it in a format that the user can experience through a VR device. This is a real-time rendering operation utilizing a 3D graphics engine.

[0291] Step 6:

[0292] The user begins training in a virtual space using a terminal. The user performs virtual activities using a VR headset or haptic device. The data input here consists of the user's movements and operation commands, which are transmitted to the server in real time via the terminal.

[0293] Step 7:

[0294] The server receives user operation information and analyzes it in real time through an operation evaluation means. Input includes user operation logs and time data. Based on this, the server provides immediate feedback using a feedback generation means, outputting suggestions for improvement to the user. This step involves high-speed data processing and immediate response generation.

[0295] Step 8:

[0296] The server re-evaluates learning progress using educational management tools based on feedback and updates the curriculum as needed. Feedback data and learning progress information are provided as input. The server uses this to optimize the content for the next training session and outputs the updated curriculum as text. This includes dynamic adjustments to the curriculum.

[0297] (Application Example 1)

[0298] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0299] Conventional skill acquisition methods lack a suitable training environment for efficiently and safely improving skills in operating industrial automated machinery. In particular, practicing operation while receiving real-time feedback is important, but using actual equipment incurs significant risks and costs. The objective of this invention is to improve this situation and provide an efficient and low-risk skills training method.

[0300] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0301] In this invention, the server includes data collection means for collecting past work performance data and learning history of workers; data analysis means for analyzing the data and evaluating each worker's skill level and learning tendencies; and instruction plan generation means for automatically generating training tasks based on the evaluation using artificial intelligence technology. This enables participants to safely and efficiently learn industrial automated machine skills in a virtual reality environment and receive real-time feedback.

[0302] "Worker" refers to an individual receiving training for the purpose of acquiring skills in operating industrial automated machinery.

[0303] "Data collection means" refers to a device or method for automatically collecting past work performance data and learning history of an employee.

[0304] "Data analysis means" refers to methods and devices that evaluate the skill level and learning tendencies of each worker based on collected data.

[0305] A "teaching plan generation means" is a device or system that automatically generates training tasks using artificial intelligence technology based on evaluation results.

[0306] The "virtual space construction means" is a method or device for constructing a virtual reality environment according to training tasks and providing a situation accessible to participants.

[0307] The "monitoring means" is a system or method for monitoring the operations of participants in a virtual reality environment in real time and providing feedback.

[0308] The "progress management means" is a method or device for evaluating the progress of learning based on feedback and adjusting the guidance plan as needed.

[0309] The "education means" is a function or device that enables the skills of industrial automation equipment to be learned safely and efficiently in a virtual reality environment.

[0310] The "participant" refers to an individual who participates in the training provided in a virtual reality environment.

[0311] The "feedback generation means" is a device or system for indicating improvement points in real time based on the operations of participants.

[0312] To implement this invention, a server, a terminal, and a user need to be involved in a series of systems. The server is connected to a database system for efficiently collecting the past work performance data and learning history of workers. Specifically, it is conceivable to use a database management system such as MySQL or PostgreSQL.

[0313] The server uses machine learning libraries such as TensorFlow or PyTorch to analyze the collected data. Based on these analysis results, an artificial intelligence technology is used to generate a guidance plan suitable for each worker. In this process, the generated AI model operates, and a curriculum corresponding to individual skills and learning tendencies is created.

[0314] Users access a virtual reality space via a VR headset. This virtual environment is built using virtual reality engines such as Unity and Unreal Engine, making it possible to simulate industrial automated machinery within the virtual space.

[0315] The terminal is responsible for transmitting user operation data to the server in real time. This allows the server to monitor the user's progress and provide immediate feedback. This feedback is used to measure the user's progress in the virtual space and to point out areas for improvement.

[0316] A concrete example would be a scenario where a new employee learns to operate a new machine. Based on the worker's past operation history and learning tendencies, an optimal training plan is devised, allowing the user to practice sufficiently in the VR environment before operating the actual machine. An example of a prompt message would be, "Analyze the following employee data to create an optimal curriculum for welding machine operation: {Employee work history data}."

[0317] This allows participants to efficiently improve their skills while minimizing the risks involved in actual machine operation.

[0318] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0319] Step 1:

[0320] The server uses a database management system (e.g., MySQL) to collect the worker's past work performance data and learning history. The input for this step is the work history data in the database. The server executes SQL queries to extract this data and convert it into the format required for analysis. The output is work data ready for data analysis.

[0321] Step 2:

[0322] The server inputs the collected work data into a machine learning model using TensorFlow. The input consists of performance data from individual workers. The machine learning model processes the data and evaluates the workers' skill levels and learning tendencies. The output provides evaluation information for each worker.

[0323] Step 3:

[0324] The server generates an optimal training plan using a generative AI model based on the acquired evaluation information. The input for this step is the evaluation information obtained through machine learning. The generated training plan provides the most effective curriculum for improving each worker's skills. As output, a customized training plan is obtained for each worker.

[0325] Step 4:

[0326] The server designs a virtual space using the Unity engine and sends it to the terminal. The inputs are the lesson plan and the parameters for the virtual environment design. The server instructs the creation of the virtual environment, and the output is virtual environment data that the user can access.

[0327] Step 5:

[0328] The user enters a virtual reality space through a VR headset. The input in this step is virtual environment data sent from the server. The user works on training tasks and performs operations within the virtual environment. The output is a log of operations performed during training.

[0329] Step 6:

[0330] The terminal transmits user operation data to the server in real time. The input is the user's operation log, which is transferred to the server using a specific protocol. The output is the transmitted operation data, which is received by the server.

[0331] Step 7:

[0332] The server analyzes the received operation data and generates feedback, including areas for improvement and successes. The input for this step is the user's operation data. The server evaluates the data in real time, generates feedback information as output, and sends it to the terminal.

[0333] Step 8:

[0334] The terminal receives feedback from the server and displays it to the user within the virtual environment. The input is the feedback information sent from the server. The terminal presents the feedback appropriately to the user, encouraging understanding and improvement. The output is the feedback received by the user.

[0335] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0336] In embodiments of the present invention, a system incorporating an emotion engine has the ability to recognize the user's emotional state in real time and dynamically adjust the training content and feedback accordingly. The server collects the employee's past work performance data and learning history and performs a skills assessment based on this data. Then, using generative artificial intelligence technology, it generates an optimal training curriculum for each employee.

[0337] The server also processes user emotional data transmitted from the terminal via an emotion engine. This emotional data is obtained from the user's facial expressions, voice, vital signs, etc., using emotion recognition software and sensor technology. The server analyzes the user's emotional state in real time and adjusts training tasks accordingly, thereby reducing stress and fatigue and maximizing learning effectiveness.

[0338] When a user performs training in a virtual reality space, the device sends emotional data along with user operation information to the server. The server uses this data to monitor the user's performance and generate real-time feedback appropriate to their emotional state. This feedback includes not only information on the accuracy of operations and areas for improvement, but also positive information to enhance the user's psychological sense of security.

[0339] As a concrete example, consider a user learning new surgical techniques in a medical setting. The server analyzes data and training information related to past surgeries and generates a curriculum tailored to that user. As the user practices surgery in the VR environment, the device senses their anxiety and level of concentration and sends this information to the server. Through an emotion engine, the server provides corresponding feedback, promoting skill improvement while maintaining the user's sense of security.

[0340] Thus, the present invention provides flexible training tailored to the emotions of employees, realizing a form that enables safer and more effective skill acquisition.

[0341] The following describes the processing flow.

[0342] Step 1:

[0343] The server extracts employees' past work performance data and learning history from the database. This data is used to analyze each employee's skills and identify their learning needs.

[0344] Step 2:

[0345] The server uses an emotion engine to acquire emotional data such as facial expressions, voice, and vital signs transmitted from the terminal in real time. Based on this data, it recognizes the user's emotional state.

[0346] Step 3:

[0347] The server combines skill assessment results and emotional state analysis results to generate a user-optimized training curriculum using generative AI technology. The curriculum incorporates flexible responses based on the user's emotions.

[0348] Step 4:

[0349] The terminal renders the virtual reality environment based on the curriculum provided by the server, preparing it for user access. The user enters the VR space and begins training.

[0350] Step 5:

[0351] The user performs training and operations within the VR space via a device. The device transmits the user's operation data and emotional data to the server in real time.

[0352] Step 6:

[0353] The server monitors the user's performance and emotional state based on the received operational and emotional data, and generates real-time feedback. This feedback is particularly sensitive to the user's emotions.

[0354] Step 7:

[0355] Users continue their training based on feedback received through their devices, improving their skills at their own pace. The server dynamically adjusts the curriculum and training content as needed.

[0356] Step 8:

[0357] The server records the overall progress of the training and evaluates the user's learning outcomes. This information can be used to adjust and improve future training curricula.

[0358] (Example 2)

[0359] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0360] Current training systems struggle to provide flexible training assignments that appropriately reflect each employee's learning tendencies and emotional state, thus failing to contribute to more efficient learning and reduced stress. Furthermore, feedback provided may undermine motivation because it is not based on emotional state.

[0361] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0362] In this invention, the server includes information gathering means, information analysis means, and task generation means. This enables the provision of personalized training tailored to employees' learning history and emotional states, as well as dynamic adjustment of training content in response to real-time circumstances.

[0363] "Information gathering means" refers to devices and technologies for collecting information on employees' past work performance and learning history.

[0364] "Information analysis tools" refer to devices and technologies that evaluate the skill level and learning tendencies of individual employees based on collected information.

[0365] A "task generation method" refers to a device or technology that uses machine learning techniques to automatically generate training tasks based on evaluation results.

[0366] "Virtual environment provisioning means" refers to devices and technologies for constructing a virtual space according to training tasks and making it accessible to users.

[0367] "Monitoring means" are devices or technologies that observe a user's operations within a virtual space and provide real-time feedback.

[0368] "Emotional response methods" refer to devices or technologies that use emotion recognition devices to evaluate the user's emotional state and dynamically adjust training tasks according to that state.

[0369] "Progress management tools" are devices or technologies that evaluate learning progress based on feedback and adjust assignments as needed.

[0370] A "feedback generation means" refers to a device or technology that provides real-time feedback on areas for improvement based on the user's actions and emotional state.

[0371] A "terminal" is a device that provides an interface between the user and the device and is equipped with emotion recognition capabilities.

[0372] In this system, three main entities—the server, the terminal, and the user—work in coordination. The server uses information gathering tools to efficiently collect information on employees' past work performance and learning history. This provides the foundational data necessary to appropriately evaluate each employee's skill level and learning tendencies. This data collection often utilizes software such as database systems and log analysis tools.

[0373] Next, the server uses information analysis tools to process the collected information with advanced algorithms and analyze the learning tendencies of individual employees. For example, it identifies skill areas that employees should particularly strengthen based on past training data and work history. Statistical software and data mining tools are used for this analysis.

[0374] Subsequently, the server uses a task generation mechanism to generate appropriate training tasks based on the analysis results. A generation AI model is used for this generation, and by inputting prompts, an individualized training curriculum is formed. An example of a prompt is, "Create an optimal training plan based on the employee's name, past performance, and current learning needs."

[0375] The terminal uses a virtual environment provisioning mechanism to construct a virtual space according to the generated training tasks. This allows the user to perform training in a virtual reality environment, during which the terminal collects and transmits the user's emotional data through its emotion recognition function. For example, virtual reality headsets and voice / facial recognition technologies are used to measure the user's level of concentration and anxiety in real time.

[0376] When a user performs an action in the virtual space, monitoring devices transmit that action information to a server, which then generates real-time feedback based on that information. This feedback includes suggestions for improvement and positive messages, maximizing the learning effect.

[0377] This system provides a flexible training environment that takes into account the emotional state of employees, promoting more efficient and safer skill acquisition.

[0378] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0379] Step 1:

[0380] The server uses information gathering tools to collect employees' past work performance information and learning history. It takes employee log data and training reports as input and builds a history database for each individual based on this data. Specifically, it extracts relevant information from the database and performs data cleansing to create a highly accurate dataset.

[0381] Step 2:

[0382] The server analyzes the collected data using information analysis tools. It receives historical data acquired in step 1 as input and evaluates employees' skill levels and learning tendencies based on this data. The analysis uses data mining techniques, grouping employees using pattern recognition and clustering methods. The output is a skill evaluation report for each employee.

[0383] Step 3:

[0384] The server generates training tasks based on the evaluation results using a task generation mechanism. As input, prompts are provided to the generating AI model based on the technical evaluation report from Step 2. An example prompt might be, "Design a training scenario tailored to the current skill set of the employee named [Employee Name]." The output is a personalized training plan.

[0385] Step 4:

[0386] The terminal uses a virtual environment provisioning mechanism to construct a virtual space according to the generated training tasks. It receives the training plan created in step 3 as input and designs the virtual environment using visual simulation software. Specific operations include 3D modeling and dynamic scenario generation. The output is a virtual reality space that the user can experience.

[0387] Step 5:

[0388] The user begins training in a virtual space. During this time, the device collects the user's emotional data through its emotion recognition function and sends it to the server. The input includes the user's facial expressions and voice data, and emotion analysis is performed based on this data. Specifically, it acquires raw data from the camera and microphone and analyzes it in real time. The output is a report of the user's emotional state.

[0389] Step 6:

[0390] The server uses monitoring devices to monitor user actions and emotional states in real time and generates feedback. It receives user action data and emotional state reports from step 5 as input and performs a performance evaluation. As output, it generates appropriate feedback for the user and sends it to the terminal. Specific actions may include evaluating the accuracy of actions and including positive messages.

[0391] (Application Example 2)

[0392] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0393] In today's work environment, training to improve employee skills is essential. However, providing individualized instruction tailored to each employee's learning tendencies and emotional state is challenging, and there is a need to facilitate efficient learning while reducing employee stress and anxiety. Traditional training systems are insufficient to address these challenges, hindering efficient skill acquisition.

[0394] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0395] In this invention, the server includes information gathering means for collecting past work performance data and learning history of employees; information analysis means for analyzing the information and evaluating each employee's ability level and learning tendencies; and learning curriculum generation means for automatically generating training tasks based on the evaluation using generative artificial intelligence technology. This enables the provision of optimized training for each employee, reduces stress, and allows for efficient skill acquisition by providing real-time feedback that responds to their emotional state.

[0396] "Information gathering means" is a general term for devices and methods used to collect employees' past work performance data and learning history.

[0397] "Information analysis tools" is a general term for devices and methods used to analyze collected data and evaluate employees' skill levels and learning tendencies.

[0398] "Learning curriculum generation means" is a general term for devices and methods that use artificial intelligence generation technology to automatically generate optimal training tasks for each employee based on evaluation results.

[0399] "Virtual space construction means" is a general term for devices and methods that design a virtual reality environment based on training tasks and allow users to access that environment.

[0400] "Monitoring measures" is a general term for devices and methods that allow users to see in real time what operations they are performing in a virtual space and provide appropriate feedback.

[0401] "Progress management tools" is a general term for devices and methods that use feedback results to evaluate the progress of learning and adjust the curriculum as needed.

[0402] "Emotional analysis tools" is a general term for devices and methods that analyze a user's emotional state in real time and provide feedback that corresponds to those emotions.

[0403] The system that implements this application example first uses a server to collect employees' past work performance data and learning history, recording it as an information gathering tool. The collected data is analyzed by an information analysis tool on the server to evaluate the employees' skill levels and learning tendencies. Based on these analysis results, a learning curriculum generation tool utilizing generative artificial intelligence technology automatically generates training tasks optimized for each employee.

[0404] Next, a virtual reality environment is designed according to the training task using a virtual space construction means, and the user can access this environment via a terminal. While the user is training in the virtual space, their operation information is monitored by a monitoring means. During this process, the user's emotional state is analyzed in real time by an emotion analysis means, and feedback corresponding to that emotion is generated.

[0405] The device functions as smart glasses or a VR headset worn by the user, and its role is to display the real-world training environment in a virtual space. This device is equipped with sensor technology used for emotion recognition (for example, a camera to capture facial expressions and a voice recognition microphone), and transmits emotion data to a server.

[0406] As a concrete example, consider a scenario where a user is receiving training to learn how to operate new machinery in a factory. The server senses the user's anxiety, provides positive feedback tailored to the situation, and adjusts the learning pace, thereby reducing stress and promoting efficient skill acquisition.

[0407] An example of a prompt is, "Analyze the worker's facial expressions and tone of voice, and consider what positive message to display when they feel anxious." This is used as input to a generative AI model and forms part of emotion recognition and feedback generation.

[0408] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0409] Step 1:

[0410] The server collects employees' past work performance data and learning history using information gathering tools. This information is stored in a database. The input is each employee's historical data, and the output is structured data stored in the database. Specifically, the server accesses the company's work record system and extracts data via APIs, etc.

[0411] Step 2:

[0412] The server analyzes data collected by information analysis tools to evaluate employees' skill levels and learning tendencies. The input here is the data collected in step 1, and the output is the skill evaluation score and learning tendency profile calculated for each employee. Specifically, the server applies machine learning algorithms and performs data analysis using Python libraries (e.g., Pandas, scikit-learn).

[0413] Step 3:

[0414] The server uses generative artificial intelligence technology to automatically generate training tasks based on evaluation results using a learning curriculum generation method. The input is the evaluation score and profile obtained in step 2, and the output is an individualized training task. Specifically, a generative AI model is put into practice, and AI inference is performed using prompt sentences to derive the optimal curriculum.

[0415] Step 4:

[0416] The server utilizes virtual space construction methods to design a virtual reality environment based on training tasks, making it accessible to users from their terminals. The input is the training task generated in the previous step, and the output is the constructed virtual environment. Specifically, a simulation environment is generated using a VR development tool (e.g., Unity).

[0417] Step 5:

[0418] The terminal monitors the user's actions within the virtual space and transmits data to the server via the monitoring system. Inputs include user action information and emotional state, while outputs include action logs and emotional data. Specifically, sensors on the terminal record facial expressions and voice, and transmit this data to the server.

[0419] Step 6:

[0420] The server analyzes the user's emotional state in real time using emotion analysis tools and automatically generates feedback using a generative AI model. The input is the emotion data from step 5, and the output is a feedback message tailored to the user's emotions. Specifically, the server utilizes natural language processing technology to generate encouraging messages and other similar messages in real time.

[0421] Step 7:

[0422] The server uses a progress management system to evaluate the user's learning progress based on the generated feedback and adjusts the curriculum as needed. The input is the feedback generated in step 6, and the output is the adjusted new curriculum. Specifically, a progress management algorithm is applied, and the learning content changes dynamically.

[0423] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0424] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0425] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0426] [Third Embodiment]

[0427] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0428] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0429] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0430] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0431] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0432] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0433] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0434] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0435] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0436] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0437] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0438] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0439] In embodiments of the present invention, a server acts as the central control system. The server extracts employees' past work performance data and learning history from a database and analyzes their skill levels and learning tendencies based on this data. The server utilizes machine learning and other statistical methods to generate training curricula optimized for each individual employee.

[0440] Once the curriculum is generated, the server prepares a dataset for the virtual environment. Using this dataset, the terminal renders the virtual reality space that the user will access, preparing it for the user to begin training. The user can access this virtual environment through the terminal and train securely.

[0441] While a user is undergoing training, the device transmits user operation information to the server. The server receives this information in real time and monitors it. Based on the data obtained from monitoring, the server provides immediate feedback to the user. This feedback includes areas for improvement and successes in the operation, allowing the user to acquire skills more efficiently.

[0442] As a concrete example, consider a user learning to operate a new machine tool in the manufacturing industry. The server analyzes the user's past operation history to identify challenges in specific operations and generates a curriculum specifically tailored for improvement. The terminal is configured to allow the user to operate the machine tool in a virtual manufacturing environment via a VR headset. Through repeated training and real-time feedback, the user can acquire skills while reducing risks in the actual work environment.

[0443] This configuration allows for safe and efficient skill development, and is expected to have significant applications, particularly in industries requiring operations that involve risks or complexity.

[0444] The following describes the processing flow.

[0445] Step 1:

[0446] The server extracts past work performance data and learning history of employees from the database. This data serves as foundational information for understanding each employee's skill level and operational tendencies.

[0447] Step 2:

[0448] The server cleanses the extracted data and analyzes it using machine learning algorithms. This identifies each employee's strengths and skill gaps that need improvement.

[0449] Step 3:

[0450] Based on the analysis results, the server utilizes generative AI technology to automatically generate training curricula optimized for each employee. These curricula include specific training content and virtual environment scenarios.

[0451] Step 4:

[0452] The server prepares the dataset for the virtual environment according to the generated curriculum. It transfers the necessary data to the terminal and sets up a virtual environment that the user can access.

[0453] Step 5:

[0454] The user accesses the VR space using their device and selects their training from the start menu. After selection, they enter the virtual environment and perform each task step by step.

[0455] Step 6:

[0456] The device continuously sends user action data to the server in real time. The server monitors this data and tracks the user's movements.

[0457] Step 7:

[0458] The server analyzes user activity data and generates real-time feedback. It provides feedback to the user via the terminal regarding successful actions and areas for improvement.

[0459] Step 8:

[0460] Users receive feedback and incorporate it into their next actions. Skill improvement is aimed at through repeated training as needed.

[0461] Step 9:

[0462] The server records the overall progress of the training session and evaluates the user's achievements and skill improvement. This data will be used to adjust future training curricula.

[0463] (Example 1)

[0464] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0465] Traditional training methods struggle to efficiently achieve optimal skill improvement for individual employees, and in industries requiring complex operations, skill acquisition is time-consuming and risky. Furthermore, insufficient real-time feedback makes immediate improvement difficult, hindering the maximization of training effectiveness.

[0466] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0467] In this invention, the server includes information acquisition means for extracting employees' past work data and learning history, data analysis means for analyzing the information using machine learning technology to derive individual skill levels and learning tendencies, and training content creation means for creating a training curriculum based on the analysis results using a generative artificial intelligence model. This enables the provision of real-time feedback and optimized training curricula to individual employees.

[0468] "Information acquisition means" refers to a function for extracting employees' past work data and learning history.

[0469] "Data analysis tools" refer to functions that use machine learning techniques to analyze individual skill levels and learning tendencies based on acquired information.

[0470] The "training content creation method" is a function that automatically generates a training curriculum using a generative artificial intelligence model based on the analysis results.

[0471] "Digital environment provision means" refers to functions for constructing a virtual environment and generating a simulated reality space that users can experience.

[0472] The "operation evaluation means" is a function that collects operation information within the user's virtual environment, analyzes it in real time, and provides immediate feedback.

[0473] "Educational management tools" are functions for reassessing learning progress based on feedback and updating the training curriculum as needed.

[0474] A "feedback generation mechanism" is a function that provides real-time improvement suggestions based on user behavior information.

[0475] An "electronic device" is a device equipped with functions to provide connectivity between a user and equipment within a virtual environment.

[0476] In this invention, the entire system is centrally managed by a server. The server extracts employees' past work data and learning history from a database using information retrieval means. The extracted data is analyzed by data analysis means using machine learning techniques to derive the skill level and learning tendencies of individual employees. Machine learning libraries such as TensorFlow and PyTorch may be used for this purpose.

[0477] Based on the analysis results, the server uses a generative artificial intelligence model (such as GPT or BERT) to automatically generate an optimal training curriculum through a training content creation mechanism. Following this curriculum, the server executes a digital environment provision mechanism for the virtual environment, constructing a virtual reality space accessible to the user. This virtual reality space is created using rendering software such as Unity or Unreal Engine.

[0478] The terminal displays a provided virtual environment to the user and supports the user's experience using a VR headset or other interaction devices. The user can begin training in this environment, and the terminal uses a motion evaluation system to transmit the user's operation information to the server in real time.

[0479] The server analyzes the received operational information and provides immediate improvement suggestions to the user through a feedback generation mechanism. This allows users to acquire skills efficiently. Furthermore, the education management mechanism allows for reassessment of learning progress and updating the training curriculum as needed.

[0480] As a concrete example, when learning to use a new machine tool in the manufacturing industry, the server identifies challenges from past operation history and generates a curriculum. The terminal is configured to allow the user to operate the machine tool in a virtual factory through a VR headset. The user practices repeatedly in this virtual environment and receives real-time feedback, thereby honing their skills while reducing risks in the actual work environment.

[0481] An example of a prompt message is, "I want to learn how to operate a new machine tool." This prompts the system to suggest an appropriate training program, effectively supporting the user's skill improvement.

[0482] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0483] Step 1:

[0484] The server extracts employees' past work data and learning history from the database. Employee IDs and past project information are provided as input. Based on this data, the server outputs a series of pieces of information in an organized format, visualizing their past performance. This step involves using SQL queries to filter the necessary data and organize it into a data frame.

[0485] Step 2:

[0486] The server passes the extracted information to a data analysis tool, which then analyzes it using a machine learning model. Inputs include performance metrics such as past operation time, error rate, and success rate. The server feeds this data into a machine learning algorithm to analyze the skill level and learning tendencies of individual employees. This step involves using TensorFlow or PyTorch to train and predict data.

[0487] Step 3:

[0488] The server utilizes a generative artificial intelligence model to generate training curricula based on the analysis results. Skill assessment results are given as input. The server uses generative AI technology to generate training tasks optimized for each employee and outputs them in text format. This process includes the automatic generation of curricula using technologies such as GPT and BERT.

[0489] Step 4:

[0490] The server constructs digital data for the virtual environment according to the generated curriculum. Inputs include simulation scenario data and 3D model templates. Based on this, the server designs the digital environment and sends it to the terminal in dataset format. This step involves operating Unity or Unreal Engine to build the virtual environment and configuring the virtual space.

[0491] Step 5:

[0492] The terminal uses virtual environment data received from the server to render the visual space accessed by the user. Inputs include 3D model data and simulation scenarios. The terminal uses these to generate a virtual reality space and output it in a format that the user can experience through a VR device. This is a real-time rendering operation utilizing a 3D graphics engine.

[0493] Step 6:

[0494] The user begins training in a virtual space using a terminal. The user performs virtual activities using a VR headset or haptic device. The data input here consists of the user's movements and operation commands, which are transmitted to the server in real time via the terminal.

[0495] Step 7:

[0496] The server receives user operation information and analyzes it in real time through an operation evaluation means. Input includes user operation logs and time data. Based on this, the server provides immediate feedback using a feedback generation means, outputting suggestions for improvement to the user. This step involves high-speed data processing and immediate response generation.

[0497] Step 8:

[0498] The server re-evaluates learning progress using educational management tools based on feedback and updates the curriculum as needed. Feedback data and learning progress information are provided as input. The server uses this to optimize the content for the next training session and outputs the updated curriculum as text. This includes dynamic adjustments to the curriculum.

[0499] (Application Example 1)

[0500] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0501] Conventional skill acquisition methods lack a suitable training environment for efficiently and safely improving skills in operating industrial automated machinery. In particular, practicing operation while receiving real-time feedback is important, but using actual equipment incurs significant risks and costs. The objective of this invention is to improve this situation and provide an efficient and low-risk skills training method.

[0502] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0503] In this invention, the server includes data collection means for collecting past work performance data and learning history of workers; data analysis means for analyzing the data and evaluating each worker's skill level and learning tendencies; and instruction plan generation means for automatically generating training tasks based on the evaluation using artificial intelligence technology. This enables participants to safely and efficiently learn industrial automated machine skills in a virtual reality environment and receive real-time feedback.

[0504] "Worker" refers to an individual receiving training for the purpose of acquiring skills in operating industrial automated machinery.

[0505] "Data collection means" refers to a device or method for automatically collecting past work performance data and learning history of an employee.

[0506] "Data analysis means" refers to methods and devices that evaluate the skill level and learning tendencies of each worker based on collected data.

[0507] A "teaching plan generation means" is a device or system that automatically generates training tasks using artificial intelligence technology based on evaluation results.

[0508] "Virtual space construction means" refers to a method or apparatus for constructing a virtual reality environment according to training tasks and providing a situation that participants can access.

[0509] "Monitoring means" refers to a system or method for monitoring participants' actions in a virtual reality environment in real time and providing feedback.

[0510] "Progress management means" refers to methods or devices that evaluate learning progress based on feedback and adjust instructional plans as needed.

[0511] "Educational tools" refer to functions and devices that enable the safe and efficient learning of industrial automated machine skills in a virtual reality environment.

[0512] "Participant" refers to an individual who participates in training provided in a virtual reality environment.

[0513] A "feedback generation means" is a device or system for indicating areas for improvement in real time based on the participant's actions.

[0514] To implement this invention, a server, a terminal, and a user must be involved in a series of systems. The server is connected to a database system for efficiently collecting the worker's past work performance data and learning history. Specifically, database management systems such as MySQL or PostgreSQL could be used.

[0515] The server utilizes machine learning libraries such as TensorFlow and PyTorch to analyze the collected data. Based on these analysis results, artificial intelligence technology is used to generate instructional plans tailored to each worker. In this process, a generative AI model is in operation, creating curricula that are customized to each individual's skills and learning tendencies.

[0516] Users access a virtual reality space via a VR headset. This virtual environment is built using virtual reality engines such as Unity and Unreal Engine, making it possible to simulate industrial automated machinery within the virtual space.

[0517] The terminal is responsible for transmitting user operation data to the server in real time. This allows the server to monitor the user's progress and provide immediate feedback. This feedback is used to measure the user's progress in the virtual space and to point out areas for improvement.

[0518] A concrete example would be a scenario where a new employee learns to operate a new machine. Based on the worker's past operation history and learning tendencies, an optimal training plan is devised, allowing the user to practice sufficiently in the VR environment before operating the actual machine. An example of a prompt message would be, "Analyze the following employee data to create an optimal curriculum for welding machine operation: {Employee work history data}."

[0519] This allows participants to efficiently improve their skills while minimizing the risks involved in actual machine operation.

[0520] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0521] Step 1:

[0522] The server uses a database management system (e.g., MySQL) to collect the worker's past work performance data and learning history. The input for this step is the work history data in the database. The server executes SQL queries to extract this data and convert it into the format required for analysis. The output is work data ready for data analysis.

[0523] Step 2:

[0524] The server inputs the collected work data into a machine learning model using TensorFlow. The input consists of performance data from individual workers. The machine learning model processes the data and evaluates the workers' skill levels and learning tendencies. The output provides evaluation information for each worker.

[0525] Step 3:

[0526] The server generates an optimal training plan using a generative AI model based on the acquired evaluation information. The input for this step is the evaluation information obtained through machine learning. The generated training plan provides the most effective curriculum for improving each worker's skills. As output, a customized training plan is obtained for each worker.

[0527] Step 4:

[0528] The server designs a virtual space using the Unity engine and sends it to the terminal. The inputs are the lesson plan and the parameters for the virtual environment design. The server instructs the creation of the virtual environment, and the output is virtual environment data that the user can access.

[0529] Step 5:

[0530] The user enters a virtual reality space through a VR headset. The input in this step is virtual environment data sent from the server. The user works on training tasks and performs operations within the virtual environment. The output is a log of operations performed during training.

[0531] Step 6:

[0532] The terminal transmits user operation data to the server in real time. The input is the user's operation log, which is transferred to the server using a specific protocol. The output is the transmitted operation data, which is received by the server.

[0533] Step 7:

[0534] The server analyzes the received operation data and generates feedback, including areas for improvement and successes. The input for this step is the user's operation data. The server evaluates the data in real time, generates feedback information as output, and sends it to the terminal.

[0535] Step 8:

[0536] The terminal receives feedback from the server and displays it to the user within the virtual environment. The input is the feedback information sent from the server. The terminal presents the feedback appropriately to the user, encouraging understanding and improvement. The output is the feedback received by the user.

[0537] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0538] In embodiments of the present invention, a system incorporating an emotion engine has the ability to recognize the user's emotional state in real time and dynamically adjust the training content and feedback accordingly. The server collects the employee's past work performance data and learning history and performs a skills assessment based on this data. Then, using generative artificial intelligence technology, it generates an optimal training curriculum for each employee.

[0539] The server also processes user emotional data transmitted from the terminal via an emotion engine. This emotional data is obtained from the user's facial expressions, voice, vital signs, etc., using emotion recognition software and sensor technology. The server analyzes the user's emotional state in real time and adjusts training tasks accordingly, thereby reducing stress and fatigue and maximizing learning effectiveness.

[0540] When a user performs training in a virtual reality space, the device sends emotional data along with user operation information to the server. The server uses this data to monitor the user's performance and generate real-time feedback appropriate to their emotional state. This feedback includes not only information on the accuracy of operations and areas for improvement, but also positive information to enhance the user's psychological sense of security.

[0541] As a concrete example, consider a user learning new surgical techniques in a medical setting. The server analyzes data and training information related to past surgeries and generates a curriculum tailored to that user. As the user practices surgery in the VR environment, the device senses their anxiety and level of concentration and sends this information to the server. Through an emotion engine, the server provides corresponding feedback, promoting skill improvement while maintaining the user's sense of security.

[0542] Thus, the present invention provides flexible training tailored to the emotions of employees, realizing a form that enables safer and more effective skill acquisition.

[0543] The following describes the processing flow.

[0544] Step 1:

[0545] The server extracts employees' past work performance data and learning history from the database. This data is used to analyze each employee's skills and identify their learning needs.

[0546] Step 2:

[0547] The server uses an emotion engine to acquire emotional data such as facial expressions, voice, and vital signs transmitted from the terminal in real time. Based on this data, it recognizes the user's emotional state.

[0548] Step 3:

[0549] The server combines skill assessment results and emotional state analysis results to generate a user-optimized training curriculum using generative AI technology. The curriculum incorporates flexible responses based on the user's emotions.

[0550] Step 4:

[0551] The terminal renders the virtual reality environment based on the curriculum provided by the server, preparing it for user access. The user enters the VR space and begins training.

[0552] Step 5:

[0553] The user performs training and operations within the VR space via a device. The device transmits the user's operation data and emotional data to the server in real time.

[0554] Step 6:

[0555] The server monitors the user's performance and emotional state based on the received operational and emotional data, and generates real-time feedback. This feedback is particularly sensitive to the user's emotions.

[0556] Step 7:

[0557] Users continue their training based on feedback received through their devices, improving their skills at their own pace. The server dynamically adjusts the curriculum and training content as needed.

[0558] Step 8:

[0559] The server records the overall progress of the training and evaluates the user's learning outcomes. This information can be used to adjust and improve future training curricula.

[0560] (Example 2)

[0561] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0562] Current training systems struggle to provide flexible training assignments that appropriately reflect each employee's learning tendencies and emotional state, thus failing to contribute to more efficient learning and reduced stress. Furthermore, feedback provided may undermine motivation because it is not based on emotional state.

[0563] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0564] In this invention, the server includes information gathering means, information analysis means, and task generation means. This enables the provision of personalized training tailored to employees' learning history and emotional states, as well as dynamic adjustment of training content in response to real-time circumstances.

[0565] "Information gathering means" refers to devices and technologies for collecting information on employees' past work performance and learning history.

[0566] "Information analysis tools" refer to devices and technologies that evaluate the skill level and learning tendencies of individual employees based on collected information.

[0567] A "task generation method" refers to a device or technology that uses machine learning techniques to automatically generate training tasks based on evaluation results.

[0568] "Virtual environment provisioning means" refers to devices and technologies for constructing a virtual space according to training tasks and making it accessible to users.

[0569] "Monitoring means" are devices or technologies that observe a user's operations within a virtual space and provide real-time feedback.

[0570] "Emotional response methods" refer to devices or technologies that use emotion recognition devices to evaluate the user's emotional state and dynamically adjust training tasks according to that state.

[0571] "Progress management tools" are devices or technologies that evaluate learning progress based on feedback and adjust assignments as needed.

[0572] A "feedback generation means" refers to a device or technology that provides real-time feedback on areas for improvement based on the user's actions and emotional state.

[0573] A "terminal" is a device that provides an interface between the user and the device and is equipped with emotion recognition capabilities.

[0574] In this system, three main entities—the server, the terminal, and the user—work in coordination. The server uses information gathering tools to efficiently collect information on employees' past work performance and learning history. This provides the foundational data necessary to appropriately evaluate each employee's skill level and learning tendencies. This data collection often utilizes software such as database systems and log analysis tools.

[0575] Next, the server uses information analysis tools to process the collected information with advanced algorithms and analyze the learning tendencies of individual employees. For example, it identifies skill areas that employees should particularly strengthen based on past training data and work history. Statistical software and data mining tools are used for this analysis.

[0576] Subsequently, the server uses a task generation mechanism to generate appropriate training tasks based on the analysis results. A generation AI model is used for this generation, and by inputting prompts, an individualized training curriculum is formed. An example of a prompt is, "Create an optimal training plan based on the employee's name, past performance, and current learning needs."

[0577] The terminal uses a virtual environment provisioning mechanism to construct a virtual space according to the generated training tasks. This allows the user to perform training in a virtual reality environment, during which the terminal collects and transmits the user's emotional data through its emotion recognition function. For example, virtual reality headsets and voice / facial recognition technologies are used to measure the user's level of concentration and anxiety in real time.

[0578] When a user performs an action in the virtual space, monitoring devices transmit that action information to a server, which then generates real-time feedback based on that information. This feedback includes suggestions for improvement and positive messages, maximizing the learning effect.

[0579] This system provides a flexible training environment that takes into account the emotional state of employees, promoting more efficient and safer skill acquisition.

[0580] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0581] Step 1:

[0582] The server uses information gathering tools to collect employees' past work performance information and learning history. It takes employee log data and training reports as input and builds a history database for each individual based on this data. Specifically, it extracts relevant information from the database and performs data cleansing to create a highly accurate dataset.

[0583] Step 2:

[0584] The server analyzes the collected data using information analysis tools. It receives historical data acquired in step 1 as input and evaluates employees' skill levels and learning tendencies based on this data. The analysis uses data mining techniques, grouping employees using pattern recognition and clustering methods. The output is a skill evaluation report for each employee.

[0585] Step 3:

[0586] The server generates training tasks based on the evaluation results using a task generation mechanism. As input, prompts are provided to the generating AI model based on the technical evaluation report from Step 2. An example prompt might be, "Design a training scenario tailored to the current skill set of the employee named [Employee Name]." The output is a personalized training plan.

[0587] Step 4:

[0588] The terminal uses a virtual environment provisioning mechanism to construct a virtual space according to the generated training tasks. It receives the training plan created in step 3 as input and designs the virtual environment using visual simulation software. Specific operations include 3D modeling and dynamic scenario generation. The output is a virtual reality space that the user can experience.

[0589] Step 5:

[0590] The user begins training in a virtual space. During this time, the device collects the user's emotional data through its emotion recognition function and sends it to the server. The input includes the user's facial expressions and voice data, and emotion analysis is performed based on this data. Specifically, it acquires raw data from the camera and microphone and analyzes it in real time. The output is a report of the user's emotional state.

[0591] Step 6:

[0592] The server uses monitoring devices to monitor user actions and emotional states in real time and generates feedback. It receives user action data and emotional state reports from step 5 as input and performs a performance evaluation. As output, it generates appropriate feedback for the user and sends it to the terminal. Specific actions may include evaluating the accuracy of actions and including positive messages.

[0593] (Application Example 2)

[0594] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0595] In today's work environment, training to improve employee skills is essential. However, providing individualized instruction tailored to each employee's learning tendencies and emotional state is challenging, and there is a need to facilitate efficient learning while reducing employee stress and anxiety. Traditional training systems are insufficient to address these challenges, hindering efficient skill acquisition.

[0596] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0597] In this invention, the server includes information gathering means for collecting past work performance data and learning history of employees; information analysis means for analyzing the information and evaluating each employee's ability level and learning tendencies; and learning curriculum generation means for automatically generating training tasks based on the evaluation using generative artificial intelligence technology. This enables the provision of optimized training for each employee, reduces stress, and allows for efficient skill acquisition by providing real-time feedback that responds to their emotional state.

[0598] "Information gathering means" is a general term for devices and methods used to collect employees' past work performance data and learning history.

[0599] "Information analysis tools" is a general term for devices and methods used to analyze collected data and evaluate employees' skill levels and learning tendencies.

[0600] "Learning curriculum generation means" is a general term for devices and methods that use artificial intelligence generation technology to automatically generate optimal training tasks for each employee based on evaluation results.

[0601] "Virtual space construction means" is a general term for devices and methods that design a virtual reality environment based on training tasks and allow users to access that environment.

[0602] "Monitoring measures" is a general term for devices and methods that allow users to see in real time what operations they are performing in a virtual space and provide appropriate feedback.

[0603] "Progress management tools" is a general term for devices and methods that use feedback results to evaluate the progress of learning and adjust the curriculum as needed.

[0604] "Emotional analysis tools" is a general term for devices and methods that analyze a user's emotional state in real time and provide feedback that corresponds to those emotions.

[0605] The system that implements this application example first uses a server to collect employees' past work performance data and learning history, recording it as an information gathering tool. The collected data is analyzed by an information analysis tool on the server to evaluate the employees' skill levels and learning tendencies. Based on these analysis results, a learning curriculum generation tool utilizing generative artificial intelligence technology automatically generates training tasks optimized for each employee.

[0606] Next, a virtual reality environment is designed according to the training task using a virtual space construction means, and the user can access this environment via a terminal. While the user is training in the virtual space, their operation information is monitored by a monitoring means. During this process, the user's emotional state is analyzed in real time by an emotion analysis means, and feedback corresponding to that emotion is generated.

[0607] The device functions as smart glasses or a VR headset worn by the user, and its role is to display the real-world training environment in a virtual space. This device is equipped with sensor technology used for emotion recognition (for example, a camera to capture facial expressions and a voice recognition microphone), and transmits emotion data to a server.

[0608] As a concrete example, consider a scenario where a user is receiving training to learn how to operate new machinery in a factory. The server senses the user's anxiety, provides positive feedback tailored to the situation, and adjusts the learning pace, thereby reducing stress and promoting efficient skill acquisition.

[0609] An example of a prompt is, "Analyze the worker's facial expressions and tone of voice, and consider what positive message to display when they feel anxious." This is used as input to a generative AI model and forms part of emotion recognition and feedback generation.

[0610] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0611] Step 1:

[0612] The server collects employees' past work performance data and learning history using information gathering tools. This information is stored in a database. The input is each employee's historical data, and the output is structured data stored in the database. Specifically, the server accesses the company's work record system and extracts data via APIs, etc.

[0613] Step 2:

[0614] The server analyzes data collected by information analysis tools to evaluate employees' skill levels and learning tendencies. The input here is the data collected in step 1, and the output is the skill evaluation score and learning tendency profile calculated for each employee. Specifically, the server applies machine learning algorithms and performs data analysis using Python libraries (e.g., Pandas, scikit-learn).

[0615] Step 3:

[0616] The server uses generative artificial intelligence technology to automatically generate training tasks based on evaluation results using a learning curriculum generation method. The input is the evaluation score and profile obtained in step 2, and the output is an individualized training task. Specifically, a generative AI model is put into practice, and AI inference is performed using prompt sentences to derive the optimal curriculum.

[0617] Step 4:

[0618] The server utilizes virtual space construction methods to design a virtual reality environment based on training tasks, making it accessible to users from their terminals. The input is the training task generated in the previous step, and the output is the constructed virtual environment. Specifically, a simulation environment is generated using a VR development tool (e.g., Unity).

[0619] Step 5:

[0620] The terminal monitors the user's actions within the virtual space and transmits data to the server via the monitoring system. Inputs include user action information and emotional state, while outputs include action logs and emotional data. Specifically, sensors on the terminal record facial expressions and voice, and transmit this data to the server.

[0621] Step 6:

[0622] The server analyzes the user's emotional state in real time using emotion analysis tools and automatically generates feedback using a generative AI model. The input is the emotion data from step 5, and the output is a feedback message tailored to the user's emotions. Specifically, the server utilizes natural language processing technology to generate encouraging messages and other similar messages in real time.

[0623] Step 7:

[0624] The server uses a progress management system to evaluate the user's learning progress based on the generated feedback and adjusts the curriculum as needed. The input is the feedback generated in step 6, and the output is the adjusted new curriculum. Specifically, a progress management algorithm is applied, and the learning content changes dynamically.

[0625] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0626] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0627] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0628] [Fourth Embodiment]

[0629] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0630] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0631] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0632] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0633] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0634] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0635] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0636] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0637] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0638] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0639] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0640] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0641] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0642] In embodiments of the present invention, a server acts as the central control system. The server extracts employees' past work performance data and learning history from a database and analyzes their skill levels and learning tendencies based on this data. The server utilizes machine learning and other statistical methods to generate training curricula optimized for each individual employee.

[0643] Once the curriculum is generated, the server prepares a dataset for the virtual environment. Using this dataset, the terminal renders the virtual reality space that the user will access, preparing it for the user to begin training. The user can access this virtual environment through the terminal and train securely.

[0644] While a user is undergoing training, the device transmits user operation information to the server. The server receives this information in real time and monitors it. Based on the data obtained from monitoring, the server provides immediate feedback to the user. This feedback includes areas for improvement and successes in the operation, allowing the user to acquire skills more efficiently.

[0645] As a concrete example, consider a user learning to operate a new machine tool in the manufacturing industry. The server analyzes the user's past operation history to identify challenges in specific operations and generates a curriculum specifically tailored for improvement. The terminal is configured to allow the user to operate the machine tool in a virtual manufacturing environment via a VR headset. Through repeated training and real-time feedback, the user can acquire skills while reducing risks in the actual work environment.

[0646] This configuration allows for safe and efficient skill development, and is expected to have significant applications, particularly in industries requiring operations that involve risks or complexity.

[0647] The following describes the processing flow.

[0648] Step 1:

[0649] The server extracts past work performance data and learning history of employees from the database. This data serves as foundational information for understanding each employee's skill level and operational tendencies.

[0650] Step 2:

[0651] The server cleanses the extracted data and analyzes it using machine learning algorithms. This identifies each employee's strengths and skill gaps that need improvement.

[0652] Step 3:

[0653] Based on the analysis results, the server utilizes generative AI technology to automatically generate training curricula optimized for each employee. These curricula include specific training content and virtual environment scenarios.

[0654] Step 4:

[0655] The server prepares the dataset for the virtual environment according to the generated curriculum. It transfers the necessary data to the terminal and sets up a virtual environment that the user can access.

[0656] Step 5:

[0657] The user accesses the VR space using their device and selects their training from the start menu. After selection, they enter the virtual environment and perform each task step by step.

[0658] Step 6:

[0659] The device continuously sends user action data to the server in real time. The server monitors this data and tracks the user's movements.

[0660] Step 7:

[0661] The server analyzes user activity data and generates real-time feedback. It provides feedback to the user via the terminal regarding successful actions and areas for improvement.

[0662] Step 8:

[0663] Users receive feedback and incorporate it into their next actions. Skill improvement is aimed at through repeated training as needed.

[0664] Step 9:

[0665] The server records the overall progress of the training session and evaluates the user's achievements and skill improvement. This data will be used to adjust future training curricula.

[0666] (Example 1)

[0667] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] Traditional training methods struggle to efficiently achieve optimal skill improvement for individual employees, and in industries requiring complex operations, skill acquisition is time-consuming and risky. Furthermore, insufficient real-time feedback makes immediate improvement difficult, hindering the maximization of training effectiveness.

[0669] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0670] In this invention, the server includes information acquisition means for extracting employees' past work data and learning history, data analysis means for analyzing the information using machine learning technology to derive individual skill levels and learning tendencies, and training content creation means for creating a training curriculum based on the analysis results using a generative artificial intelligence model. This enables the provision of real-time feedback and optimized training curricula to individual employees.

[0671] "Information acquisition means" refers to a function for extracting employees' past work data and learning history.

[0672] "Data analysis tools" refer to functions that use machine learning techniques to analyze individual skill levels and learning tendencies based on acquired information.

[0673] The "training content creation method" is a function that automatically generates a training curriculum using a generative artificial intelligence model based on the analysis results.

[0674] "Digital environment provision means" refers to functions for constructing a virtual environment and generating a simulated reality space that users can experience.

[0675] The "operation evaluation means" is a function that collects operation information within the user's virtual environment, analyzes it in real time, and provides immediate feedback.

[0676] "Educational management tools" are functions for reassessing learning progress based on feedback and updating the training curriculum as needed.

[0677] A "feedback generation mechanism" is a function that provides real-time improvement suggestions based on user behavior information.

[0678] An "electronic device" is a device equipped with functions to provide connectivity between a user and equipment within a virtual environment.

[0679] In this invention, the entire system is centrally managed by a server. The server extracts employees' past work data and learning history from a database using information retrieval means. The extracted data is analyzed by data analysis means using machine learning techniques to derive the skill level and learning tendencies of individual employees. Machine learning libraries such as TensorFlow and PyTorch may be used for this purpose.

[0680] Based on the analysis results, the server uses a generative artificial intelligence model (such as GPT or BERT) to automatically generate an optimal training curriculum through a training content creation mechanism. Following this curriculum, the server executes a digital environment provision mechanism for the virtual environment, constructing a virtual reality space accessible to the user. This virtual reality space is created using rendering software such as Unity or Unreal Engine.

[0681] The terminal displays a provided virtual environment to the user and supports the user's experience using a VR headset or other interaction devices. The user can begin training in this environment, and the terminal uses a motion evaluation system to transmit the user's operation information to the server in real time.

[0682] The server analyzes the received operational information and provides immediate improvement suggestions to the user through a feedback generation mechanism. This allows users to acquire skills efficiently. Furthermore, the education management mechanism allows for reassessment of learning progress and updating the training curriculum as needed.

[0683] As a concrete example, when learning to use a new machine tool in the manufacturing industry, the server identifies challenges from past operation history and generates a curriculum. The terminal is configured to allow the user to operate the machine tool in a virtual factory through a VR headset. The user practices repeatedly in this virtual environment and receives real-time feedback, thereby honing their skills while reducing risks in the actual work environment.

[0684] An example of a prompt message is, "I want to learn how to operate a new machine tool." This prompts the system to suggest an appropriate training program, effectively supporting the user's skill improvement.

[0685] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0686] Step 1:

[0687] The server extracts employees' past work data and learning history from the database. Employee IDs and past project information are provided as input. Based on this data, the server outputs a series of pieces of information in an organized format, visualizing their past performance. This step involves using SQL queries to filter the necessary data and organize it into a data frame.

[0688] Step 2:

[0689] The server passes the extracted information to a data analysis tool, which then analyzes it using a machine learning model. Inputs include performance metrics such as past operation time, error rate, and success rate. The server feeds this data into a machine learning algorithm to analyze the skill level and learning tendencies of individual employees. This step involves using TensorFlow or PyTorch to train and predict data.

[0690] Step 3:

[0691] The server utilizes a generative artificial intelligence model to generate training curricula based on the analysis results. Skill assessment results are given as input. The server uses generative AI technology to generate training tasks optimized for each employee and outputs them in text format. This process includes the automatic generation of curricula using technologies such as GPT and BERT.

[0692] Step 4:

[0693] The server constructs digital data for the virtual environment according to the generated curriculum. Inputs include simulation scenario data and 3D model templates. Based on this, the server designs the digital environment and sends it to the terminal in dataset format. This step involves operating Unity or Unreal Engine to build the virtual environment and configuring the virtual space.

[0694] Step 5:

[0695] The terminal uses virtual environment data received from the server to render the visual space accessed by the user. Inputs include 3D model data and simulation scenarios. The terminal uses these to generate a virtual reality space and output it in a format that the user can experience through a VR device. This is a real-time rendering operation utilizing a 3D graphics engine.

[0696] Step 6:

[0697] The user begins training in a virtual space using a terminal. The user performs virtual activities using a VR headset or haptic device. The data input here consists of the user's movements and operation commands, which are transmitted to the server in real time via the terminal.

[0698] Step 7:

[0699] The server receives user operation information and analyzes it in real time through an operation evaluation means. Input includes user operation logs and time data. Based on this, the server provides immediate feedback using a feedback generation means, outputting suggestions for improvement to the user. This step involves high-speed data processing and immediate response generation.

[0700] Step 8:

[0701] The server re-evaluates learning progress using educational management tools based on feedback and updates the curriculum as needed. Feedback data and learning progress information are provided as input. The server uses this to optimize the content for the next training session and outputs the updated curriculum as text. This includes dynamic adjustments to the curriculum.

[0702] (Application Example 1)

[0703] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0704] Conventional skill acquisition methods lack a suitable training environment for efficiently and safely improving skills in operating industrial automated machinery. In particular, practicing operation while receiving real-time feedback is important, but using actual equipment incurs significant risks and costs. The objective of this invention is to improve this situation and provide an efficient and low-risk skills training method.

[0705] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0706] In this invention, the server includes data collection means for collecting past work performance data and learning history of workers; data analysis means for analyzing the data and evaluating each worker's skill level and learning tendencies; and instruction plan generation means for automatically generating training tasks based on the evaluation using artificial intelligence technology. This enables participants to safely and efficiently learn industrial automated machine skills in a virtual reality environment and receive real-time feedback.

[0707] "Worker" refers to an individual receiving training for the purpose of acquiring skills in operating industrial automated machinery.

[0708] "Data collection means" refers to a device or method for automatically collecting past work performance data and learning history of an employee.

[0709] "Data analysis means" refers to methods and devices that evaluate the skill level and learning tendencies of each worker based on collected data.

[0710] A "teaching plan generation means" is a device or system that automatically generates training tasks using artificial intelligence technology based on evaluation results.

[0711] "Virtual space construction means" refers to a method or apparatus for constructing a virtual reality environment according to training tasks and providing a situation that participants can access.

[0712] "Monitoring means" refers to a system or method for monitoring participants' actions in a virtual reality environment in real time and providing feedback.

[0713] "Progress management means" refers to methods or devices that evaluate learning progress based on feedback and adjust instructional plans as needed.

[0714] "Educational tools" refer to functions and devices that enable the safe and efficient learning of industrial automated machine skills in a virtual reality environment.

[0715] "Participant" refers to an individual who participates in training provided in a virtual reality environment.

[0716] A "feedback generation means" is a device or system for indicating areas for improvement in real time based on the participant's actions.

[0717] To implement this invention, a server, a terminal, and a user must be involved in a series of systems. The server is connected to a database system for efficiently collecting the worker's past work performance data and learning history. Specifically, database management systems such as MySQL or PostgreSQL could be used.

[0718] The server utilizes machine learning libraries such as TensorFlow and PyTorch to analyze the collected data. Based on these analysis results, artificial intelligence technology is used to generate instructional plans tailored to each worker. In this process, a generative AI model is in operation, creating curricula that are customized to each individual's skills and learning tendencies.

[0719] Users access a virtual reality space via a VR headset. This virtual environment is built using virtual reality engines such as Unity and Unreal Engine, making it possible to simulate industrial automated machinery within the virtual space.

[0720] The terminal is responsible for transmitting user operation data to the server in real time. This allows the server to monitor the user's progress and provide immediate feedback. This feedback is used to measure the user's progress in the virtual space and to point out areas for improvement.

[0721] A concrete example would be a scenario where a new employee learns to operate a new machine. Based on the worker's past operation history and learning tendencies, an optimal training plan is devised, allowing the user to practice sufficiently in the VR environment before operating the actual machine. An example of a prompt message would be, "Analyze the following employee data to create an optimal curriculum for welding machine operation: {Employee work history data}."

[0722] This allows participants to efficiently improve their skills while minimizing the risks involved in actual machine operation.

[0723] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0724] Step 1:

[0725] The server uses a database management system (e.g., MySQL) to collect the worker's past work performance data and learning history. The input for this step is the work history data in the database. The server executes SQL queries to extract this data and convert it into the format required for analysis. The output is work data ready for data analysis.

[0726] Step 2:

[0727] The server inputs the collected work data into a machine learning model using TensorFlow. The input consists of performance data from individual workers. The machine learning model processes the data and evaluates the workers' skill levels and learning tendencies. The output provides evaluation information for each worker.

[0728] Step 3:

[0729] The server generates an optimal training plan using a generative AI model based on the acquired evaluation information. The input for this step is the evaluation information obtained through machine learning. The generated training plan provides the most effective curriculum for improving each worker's skills. As output, a customized training plan is obtained for each worker.

[0730] Step 4:

[0731] The server designs a virtual space using the Unity engine and sends it to the terminal. The inputs are the lesson plan and the parameters for the virtual environment design. The server instructs the creation of the virtual environment, and the output is virtual environment data that the user can access.

[0732] Step 5:

[0733] The user enters a virtual reality space through a VR headset. The input in this step is virtual environment data sent from the server. The user works on training tasks and performs operations within the virtual environment. The output is a log of operations performed during training.

[0734] Step 6:

[0735] The terminal transmits user operation data to the server in real time. The input is the user's operation log, which is transferred to the server using a specific protocol. The output is the transmitted operation data, which is received by the server.

[0736] Step 7:

[0737] The server analyzes the received operation data and generates feedback, including areas for improvement and successes. The input for this step is the user's operation data. The server evaluates the data in real time, generates feedback information as output, and sends it to the terminal.

[0738] Step 8:

[0739] The terminal receives feedback from the server and displays it to the user within the virtual environment. The input is the feedback information sent from the server. The terminal presents the feedback appropriately to the user, encouraging understanding and improvement. The output is the feedback received by the user.

[0740] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0741] In embodiments of the present invention, a system incorporating an emotion engine has the ability to recognize the user's emotional state in real time and dynamically adjust the training content and feedback accordingly. The server collects the employee's past work performance data and learning history and performs a skills assessment based on this data. Then, using generative artificial intelligence technology, it generates an optimal training curriculum for each employee.

[0742] The server also processes user emotional data transmitted from the terminal via an emotion engine. This emotional data is obtained from the user's facial expressions, voice, vital signs, etc., using emotion recognition software and sensor technology. The server analyzes the user's emotional state in real time and adjusts training tasks accordingly, thereby reducing stress and fatigue and maximizing learning effectiveness.

[0743] When a user performs training in a virtual reality space, the device sends emotional data along with user operation information to the server. The server uses this data to monitor the user's performance and generate real-time feedback appropriate to their emotional state. This feedback includes not only information on the accuracy of operations and areas for improvement, but also positive information to enhance the user's psychological sense of security.

[0744] As a concrete example, consider a user learning new surgical techniques in a medical setting. The server analyzes data and training information related to past surgeries and generates a curriculum tailored to that user. As the user practices surgery in the VR environment, the device senses their anxiety and level of concentration and sends this information to the server. Through an emotion engine, the server provides corresponding feedback, promoting skill improvement while maintaining the user's sense of security.

[0745] Thus, the present invention provides flexible training tailored to the emotions of employees, realizing a form that enables safer and more effective skill acquisition.

[0746] The following describes the processing flow.

[0747] Step 1:

[0748] The server extracts employees' past work performance data and learning history from the database. This data is used to analyze each employee's skills and identify their learning needs.

[0749] Step 2:

[0750] The server uses an emotion engine to acquire emotional data such as facial expressions, voice, and vital signs transmitted from the terminal in real time. Based on this data, it recognizes the user's emotional state.

[0751] Step 3:

[0752] The server combines skill assessment results and emotional state analysis results to generate a user-optimized training curriculum using generative AI technology. The curriculum incorporates flexible responses based on the user's emotions.

[0753] Step 4:

[0754] The terminal renders the virtual reality environment based on the curriculum provided by the server, preparing it for user access. The user enters the VR space and begins training.

[0755] Step 5:

[0756] The user performs training and operations within the VR space via a device. The device transmits the user's operation data and emotional data to the server in real time.

[0757] Step 6:

[0758] The server monitors the user's performance and emotional state based on the received operational and emotional data, and generates real-time feedback. This feedback is particularly sensitive to the user's emotions.

[0759] Step 7:

[0760] Users continue their training based on feedback received through their devices, improving their skills at their own pace. The server dynamically adjusts the curriculum and training content as needed.

[0761] Step 8:

[0762] The server records the overall progress of the training and evaluates the user's learning outcomes. This information can be used to adjust and improve future training curricula.

[0763] (Example 2)

[0764] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0765] Current training systems struggle to provide flexible training assignments that appropriately reflect each employee's learning tendencies and emotional state, thus failing to contribute to more efficient learning and reduced stress. Furthermore, feedback provided may undermine motivation because it is not based on emotional state.

[0766] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0767] In this invention, the server includes information gathering means, information analysis means, and task generation means. This enables the provision of personalized training tailored to employees' learning history and emotional states, as well as dynamic adjustment of training content in response to real-time circumstances.

[0768] "Information gathering means" refers to devices and technologies for collecting information on employees' past work performance and learning history.

[0769] "Information analysis tools" refer to devices and technologies that evaluate the skill level and learning tendencies of individual employees based on collected information.

[0770] A "task generation method" refers to a device or technology that uses machine learning techniques to automatically generate training tasks based on evaluation results.

[0771] "Virtual environment provisioning means" refers to devices and technologies for constructing a virtual space according to training tasks and making it accessible to users.

[0772] "Monitoring means" are devices or technologies that observe a user's operations within a virtual space and provide real-time feedback.

[0773] "Emotional response methods" refer to devices or technologies that use emotion recognition devices to evaluate the user's emotional state and dynamically adjust training tasks according to that state.

[0774] "Progress management tools" are devices or technologies that evaluate learning progress based on feedback and adjust assignments as needed.

[0775] A "feedback generation means" refers to a device or technology that provides real-time feedback on areas for improvement based on the user's actions and emotional state.

[0776] A "terminal" is a device that provides an interface between the user and the device and is equipped with emotion recognition capabilities.

[0777] In this system, three main entities—the server, the terminal, and the user—work in coordination. The server uses information gathering tools to efficiently collect information on employees' past work performance and learning history. This provides the foundational data necessary to appropriately evaluate each employee's skill level and learning tendencies. This data collection often utilizes software such as database systems and log analysis tools.

[0778] Next, the server uses information analysis tools to process the collected information with advanced algorithms and analyze the learning tendencies of individual employees. For example, it identifies skill areas that employees should particularly strengthen based on past training data and work history. Statistical software and data mining tools are used for this analysis.

[0779] Subsequently, the server uses a task generation mechanism to generate appropriate training tasks based on the analysis results. A generation AI model is used for this generation, and by inputting prompts, an individualized training curriculum is formed. An example of a prompt is, "Create an optimal training plan based on the employee's name, past performance, and current learning needs."

[0780] The terminal uses a virtual environment provisioning mechanism to construct a virtual space according to the generated training tasks. This allows the user to perform training in a virtual reality environment, during which the terminal collects and transmits the user's emotional data through its emotion recognition function. For example, virtual reality headsets and voice / facial recognition technologies are used to measure the user's level of concentration and anxiety in real time.

[0781] When a user performs an action in the virtual space, monitoring devices transmit that action information to a server, which then generates real-time feedback based on that information. This feedback includes suggestions for improvement and positive messages, maximizing the learning effect.

[0782] This system provides a flexible training environment that takes into account the emotional state of employees, promoting more efficient and safer skill acquisition.

[0783] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0784] Step 1:

[0785] The server uses information gathering tools to collect employees' past work performance information and learning history. It takes employee log data and training reports as input and builds a history database for each individual based on this data. Specifically, it extracts relevant information from the database and performs data cleansing to create a highly accurate dataset.

[0786] Step 2:

[0787] The server analyzes the collected data using information analysis tools. It receives historical data acquired in step 1 as input and evaluates employees' skill levels and learning tendencies based on this data. The analysis uses data mining techniques, grouping employees using pattern recognition and clustering methods. The output is a skill evaluation report for each employee.

[0788] Step 3:

[0789] The server generates training tasks based on the evaluation results using a task generation mechanism. As input, prompts are provided to the generating AI model based on the technical evaluation report from Step 2. An example prompt might be, "Design a training scenario tailored to the current skill set of the employee named [Employee Name]." The output is a personalized training plan.

[0790] Step 4:

[0791] The terminal uses a virtual environment provisioning mechanism to construct a virtual space according to the generated training tasks. It receives the training plan created in step 3 as input and designs the virtual environment using visual simulation software. Specific operations include 3D modeling and dynamic scenario generation. The output is a virtual reality space that the user can experience.

[0792] Step 5:

[0793] The user begins training in a virtual space. During this time, the device collects the user's emotional data through its emotion recognition function and sends it to the server. The input includes the user's facial expressions and voice data, and emotion analysis is performed based on this data. Specifically, it acquires raw data from the camera and microphone and analyzes it in real time. The output is a report of the user's emotional state.

[0794] Step 6:

[0795] The server uses monitoring devices to monitor user actions and emotional states in real time and generates feedback. It receives user action data and emotional state reports from step 5 as input and performs a performance evaluation. As output, it generates appropriate feedback for the user and sends it to the terminal. Specific actions may include evaluating the accuracy of actions and including positive messages.

[0796] (Application Example 2)

[0797] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0798] In today's work environment, training to improve employee skills is essential. However, providing individualized instruction tailored to each employee's learning tendencies and emotional state is challenging, and there is a need to facilitate efficient learning while reducing employee stress and anxiety. Traditional training systems are insufficient to address these challenges, hindering efficient skill acquisition.

[0799] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0800] In this invention, the server includes information gathering means for collecting past work performance data and learning history of employees; information analysis means for analyzing the information and evaluating each employee's ability level and learning tendencies; and learning curriculum generation means for automatically generating training tasks based on the evaluation using generative artificial intelligence technology. This enables the provision of optimized training for each employee, reduces stress, and allows for efficient skill acquisition by providing real-time feedback that responds to their emotional state.

[0801] "Information gathering means" is a general term for devices and methods used to collect employees' past work performance data and learning history.

[0802] "Information analysis tools" is a general term for devices and methods used to analyze collected data and evaluate employees' skill levels and learning tendencies.

[0803] "Learning curriculum generation means" is a general term for devices and methods that use artificial intelligence generation technology to automatically generate optimal training tasks for each employee based on evaluation results.

[0804] "Virtual space construction means" is a general term for devices and methods that design a virtual reality environment based on training tasks and allow users to access that environment.

[0805] "Monitoring measures" is a general term for devices and methods that allow users to see in real time what operations they are performing in a virtual space and provide appropriate feedback.

[0806] "Progress management tools" is a general term for devices and methods that use feedback results to evaluate the progress of learning and adjust the curriculum as needed.

[0807] "Emotional analysis tools" is a general term for devices and methods that analyze a user's emotional state in real time and provide feedback that corresponds to those emotions.

[0808] The system that implements this application example first uses a server to collect employees' past work performance data and learning history, recording it as an information gathering tool. The collected data is analyzed by an information analysis tool on the server to evaluate the employees' skill levels and learning tendencies. Based on these analysis results, a learning curriculum generation tool utilizing generative artificial intelligence technology automatically generates training tasks optimized for each employee.

[0809] Next, a virtual reality environment is designed according to the training task using a virtual space construction means, and the user can access this environment via a terminal. While the user is training in the virtual space, their operation information is monitored by a monitoring means. During this process, the user's emotional state is analyzed in real time by an emotion analysis means, and feedback corresponding to that emotion is generated.

[0810] The device functions as smart glasses or a VR headset worn by the user, and its role is to display the real-world training environment in a virtual space. This device is equipped with sensor technology used for emotion recognition (for example, a camera to capture facial expressions and a voice recognition microphone), and transmits emotion data to a server.

[0811] As a concrete example, consider a scenario where a user is receiving training to learn how to operate new machinery in a factory. The server senses the user's anxiety, provides positive feedback tailored to the situation, and adjusts the learning pace, thereby reducing stress and promoting efficient skill acquisition.

[0812] An example of a prompt is, "Analyze the worker's facial expressions and tone of voice, and consider what positive message to display when they feel anxious." This is used as input to a generative AI model and forms part of emotion recognition and feedback generation.

[0813] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0814] Step 1:

[0815] The server collects employees' past work performance data and learning history using information gathering tools. This information is stored in a database. The input is each employee's historical data, and the output is structured data stored in the database. Specifically, the server accesses the company's work record system and extracts data via APIs, etc.

[0816] Step 2:

[0817] The server analyzes data collected by information analysis tools to evaluate employees' skill levels and learning tendencies. The input here is the data collected in step 1, and the output is the skill evaluation score and learning tendency profile calculated for each employee. Specifically, the server applies machine learning algorithms and performs data analysis using Python libraries (e.g., Pandas, scikit-learn).

[0818] Step 3:

[0819] The server uses generative artificial intelligence technology to automatically generate training tasks based on evaluation results using a learning curriculum generation method. The input is the evaluation score and profile obtained in step 2, and the output is an individualized training task. Specifically, a generative AI model is put into practice, and AI inference is performed using prompt sentences to derive the optimal curriculum.

[0820] Step 4:

[0821] The server utilizes virtual space construction methods to design a virtual reality environment based on training tasks, making it accessible to users from their terminals. The input is the training task generated in the previous step, and the output is the constructed virtual environment. Specifically, a simulation environment is generated using a VR development tool (e.g., Unity).

[0822] Step 5:

[0823] The terminal monitors the user's actions within the virtual space and transmits data to the server via the monitoring system. Inputs include user action information and emotional state, while outputs include action logs and emotional data. Specifically, sensors on the terminal record facial expressions and voice, and transmit this data to the server.

[0824] Step 6:

[0825] The server analyzes the user's emotional state in real time using emotion analysis tools and automatically generates feedback using a generative AI model. The input is the emotion data from step 5, and the output is a feedback message tailored to the user's emotions. Specifically, the server utilizes natural language processing technology to generate encouraging messages and other similar messages in real time.

[0826] Step 7:

[0827] The server uses a progress management system to evaluate the user's learning progress based on the generated feedback and adjusts the curriculum as needed. The input is the feedback generated in step 6, and the output is the adjusted new curriculum. Specifically, a progress management algorithm is applied, and the learning content changes dynamically.

[0828] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0829] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0830] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0831] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0832] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0833] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0834] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0835] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0836] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0837] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0838] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0839] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0840] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0841] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0842] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0843] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0844] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0845] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0846] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0847] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0848] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0849] The following is further disclosed regarding the embodiments described above.

[0850] (Claim 1)

[0851] A data collection means for collecting employees' past work performance data and learning history,

[0852] A data analysis method for analyzing the aforementioned data and evaluating the skill level and learning tendencies of each employee,

[0853] A curriculum generation means that automatically generates training tasks based on the evaluation using generative artificial intelligence technology,

[0854] A means for constructing a virtual environment that allows users to access virtual reality by designing a virtual space according to the aforementioned training task,

[0855] A monitoring means that monitors the user's operations within the virtual space and provides real-time feedback,

[0856] Based on the aforementioned feedback, a progress management means evaluates learning progress and adjusts the curriculum as necessary,

[0857] A system that includes this.

[0858] (Claim 2)

[0859] The system according to claim 1, further comprising a feedback generation means that points out areas for improvement in real time based on the user's actions.

[0860] (Claim 3)

[0861] The system according to claim 1, further comprising a terminal that provides an interface between a user and a device within the virtual space.

[0862] "Example 1"

[0863] (Claim 1)

[0864] A means of acquiring information to extract past work data and learning history of employees,

[0865] A data analysis method that uses machine learning technology to analyze the aforementioned information and derive individual skill levels and learning tendencies,

[0866] A means for creating training content that utilizes a generative artificial intelligence model to create a training curriculum based on the analysis results,

[0867] A means of providing a digital environment that constructs a virtual environment and generates a simulated reality space that users can experience,

[0868] A means for evaluating user behavior in a virtual environment, which collects real-time information on user behavior and provides immediate feedback based on analysis.

[0869] Based on the aforementioned feedback, an educational management system is provided to re-evaluate the progress of learning and update the curriculum as necessary.

[0870] A system that includes this.

[0871] (Claim 2)

[0872] The system according to claim 1, further comprising a feedback generation means for providing immediate improvement suggestions based on the user's action information.

[0873] (Claim 3)

[0874] The system according to claim 1, comprising an electronic device that provides a connection between a user and equipment within the virtual environment.

[0875] "Application Example 1"

[0876] (Claim 1)

[0877] A data collection means for collecting past work performance data and learning history of workers,

[0878] A data analysis means for analyzing the aforementioned data and evaluating the skill level and learning tendencies of each worker,

[0879] A means for generating instructional plans that automatically generates training tasks based on the evaluation using artificial intelligence technology,

[0880] A means for constructing a virtual space that allows participants to access virtual reality by designing a virtual space according to the aforementioned training task,

[0881] A monitoring system that monitors participants' actions within the virtual space and provides real-time feedback,

[0882] Based on the aforementioned feedback, progress management means evaluates learning progress and adjusts the instruction plan as necessary,

[0883] An educational tool equipped with features that enable learning industrial automated equipment skills in a virtual reality environment,

[0884] A system that includes this.

[0885] (Claim 2)

[0886] The system according to claim 1, further comprising a feedback generation means that points out areas for improvement in real time based on the participant's actions.

[0887] (Claim 3)

[0888] The system according to claim 1, comprising a terminal that provides an interface between participants and devices within the virtual space.

[0889] "Example 2 of combining an emotion engine"

[0890] (Claim 1)

[0891] Information gathering means for collecting employees' past work performance information and learning history,

[0892] An information analysis tool that analyzes the aforementioned information and evaluates the skill level and learning tendencies of individual employees,

[0893] A task generation means that uses machine learning technology to automatically generate training tasks based on the evaluation,

[0894] A means for providing a virtual environment that constructs a virtual space according to the aforementioned training task and allows the user to access virtual reality,

[0895] A monitoring system that monitors the user's actions within the virtual space and provides real-time feedback,

[0896] An emotion response means that estimates the user's emotional state using an emotion recognition device and dynamically adjusts the training task to reduce stress and fatigue,

[0897] A progress management means that evaluates the learning progress based on the aforementioned feedback and adjusts the assignments as necessary,

[0898] A system that includes this.

[0899] (Claim 2)

[0900] The system according to claim 1, further comprising a feedback generation means that points out areas for improvement in real time based on the user's actions and emotional state.

[0901] (Claim 3)

[0902] The system according to claim 1, which provides an interface between a user and a device within the virtual space and includes a terminal equipped with emotion recognition capabilities.

[0903] "Application example 2 when combining with an emotional engine"

[0904] (Claim 1)

[0905] Information gathering means for collecting employees' past work performance data and learning history,

[0906] An information analysis tool that analyzes the aforementioned information and evaluates the skill level and learning tendencies of each employee,

[0907] A learning curriculum generation means that uses generative artificial intelligence technology to automatically generate training tasks based on the evaluation,

[0908] A means for constructing a virtual space that designs a virtual space according to the aforementioned training task and allows the user to access the virtual reality environment,

[0909] A monitoring system that monitors the user's operations within the virtual space and provides real-time feedback,

[0910] A progress management means that evaluates the learning progress based on the aforementioned feedback and adjusts the curriculum as necessary,

[0911] An emotion analysis means that analyzes the user's emotional state in real time and generates corresponding feedback,

[0912] A system that includes this.

[0913] (Claim 2)

[0914] The system according to claim 1, further comprising a feedback generation means that presents information in real time that promotes improvements and a sense of security based on the user's emotional state and past work data.

[0915] (Claim 3)

[0916] The system according to claim 1, comprising a terminal that provides an interface between a user and a device within the virtual space and acquires emotional data. [Explanation of Symbols]

[0917] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A data collection means for collecting employees' past work performance data and learning history, A data analysis method for analyzing the aforementioned data and evaluating the skill level and learning tendencies of each employee, A curriculum generation means that automatically generates training tasks based on the evaluation using generative artificial intelligence technology, A means for constructing a virtual environment that allows users to access virtual reality by designing a virtual space according to the aforementioned training task, A monitoring means that monitors the user's operations within the virtual space and provides real-time feedback, Based on the aforementioned feedback, a progress management means evaluates learning progress and adjusts the curriculum as necessary, A system that includes this.

2. The system according to claim 1, further comprising a feedback generation means that points out areas for improvement in real time based on the user's actions.

3. The system according to claim 1, further comprising a terminal that provides an interface between a user and a device within the virtual space.

Images