System

Abstract

A system is provided.SOLUTION: A system for providing specific skills and occupational training using virtual reality technology, comprising: means for cooperating with a database that manages registration information, training history, and progress information of a user; artificial intelligence means for generating a training scenario in accordance with training progress of the user based on the database; means for transmitting the generated training scenario to a terminal of the user; means for receiving operation data of the user in real time and dynamically updating a virtual environment; and means for analyzing the operation data of the user and generating and transmitting personalized feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional skill and vocational training methods require the securing of physical facilities and resources, which are costly and pose risks when accidents or failures occur during training. Furthermore, because training methods depend on the constraints of the training environment and the ability of the trainer, there are problems with inconsistent effectiveness and efficiency of training. For example, in the fields of medicine, construction, and safety training, it is difficult to train in a real work environment, and there is a need for effective methods for acquiring practical skills while avoiding risks. [Means for solving the problem]

[0005] The present invention provides a system that uses virtual reality technology to provide specific skill or vocational training. Specifically, the system includes an artificial intelligence means that works in conjunction with a database that manages user registration information, training history, and progress information, and generates training scenarios tailored to the user's training progress. This allows the system to provide training content tailored to the user's individual progress.

[0006] The system further includes a means for transmitting the generated training scenario to a user's device and receiving the user's operation data in real time to dynamically update the virtual environment. This allows users to wear virtual reality goggles and safely and efficiently train in an immersive virtual environment without using actual equipment. The system also includes a means for analyzing the user's operation data and generating and transmitting individual feedback, allowing users to check and improve their performance in real time. These means enable effective acquisition of practical skills while reducing unnecessary costs and risks.

[0007] "Virtual reality technology" is a technology that uses computer technology to provide users with virtual environments and experiences that do not exist in reality through their senses of sight, hearing, touch, etc.

[0008] "Skills" are the techniques, knowledge, and abilities required to perform a particular task or activity efficiently and effectively.

[0009] "Vocational training" refers to education and training to learn the skills and knowledge required for a specific occupation and to acquire them through practical experience.

[0010] A "system" is an entire configuration in which multiple elements or devices work together to achieve a specific function or purpose.

[0011] "User" refers to an individual who uses a system that employs virtual reality technology to train or acquire skills.

[0012] "Registration Information" refers to personal information and profile information such as name, contact details, and areas of expertise that a user provides to the system.

[0013] "Training history" refers to a record of the content and results of training that the user has received in the past.

[0014] "Progress information" refers to data or information that indicates how well a user is progressing in a current training session.

[0015] A "database" is a system or software designed to efficiently manage, search, and store collected information and data.

[0016] "Artificial intelligence means" refers to technology that uses computer programs and algorithms to mimic human intellectual processes and perform specific tasks or solve problems within a system.

[0017] A "training scenario" is a series of hypothetical procedures or situations that are executed within the system to allow users to acquire skills or knowledge.

[0018] "Terminal" refers to the device or hardware that a user uses to interact with the system and experience virtual reality technology.

[0019] "Operation data" refers to information related to actions and inputs made by a user within a virtual reality environment.

[0020] A "virtual environment" is a virtual workspace or situation setting, including visual, auditory, and tactile sensations, that is provided to a user through virtual reality technology.

[0021] "Dynamic update" refers to the system changing its state in real time in response to user actions and changes in the environment.

[0022] "Analysis" refers to the detailed examination of collected data and the processing used to derive meaning and trends.

[0023] "Feedback" refers to evaluations and advice provided to users regarding their operations and training results. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0025] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0026] First, the terms used in the following description will be explained.

[0027] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0028] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0029] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0030] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0031] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0032] [First embodiment]

[0033] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0034] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0035] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0036] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0037] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0038] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0039] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0040] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0041] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0042] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0043] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0044] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] This invention relates to a system that provides specific skills and vocational training using virtual reality technology. The system mainly consists of a user, a terminal, and a server, and provides an efficient and safe virtual environment for users to acquire specific skills.

[0046] Server Roles

[0047] User Data Management

[0048] The server connects to a database and manages user registration information, training history, and progress information. Based on each user's progress, it can identify the next skills and training content they should learn. All of this information is stored in the database and is referenced every time a user accesses the system.

[0049] Training scenario generation

[0050] The server uses AI to generate customized training scenarios based on the user's progress. For example, in the medical field, it could generate a scenario for practicing injection techniques, creating specific procedures and a virtual environment.

[0051] Real-time environment updates

[0052] The server receives user operation data in real time and dynamically updates the virtual environment based on that data. Other objects in the environment also change dynamically according to the user's operations in the virtual space.

[0053] Providing Feedback

[0054] It analyzes user operation data and generates personalized feedback based on success rate and error type. This feedback is sent to the device in real time so that the user can reflect it in their next operation.

[0055] Device Role

[0056] Rendering a VR environment

[0057] The device displays a virtual environment to the user based on the training scenario data received from the server. By using VR goggles, the user can receive training in a highly immersive virtual space.

[0058] Operational Data Collection

[0059] The device tracks the user's actions and sends the data to a server in real time. For example, when a user injects a drug in a virtual space, the device collects information about the user's hand movements and location.

[0060] View Feedback

[0061] Feedback data received from the server is displayed in the VR environment, allowing users to check this feedback in real time and improve their operations.

[0062] User Roles

[0063] Putting it on and starting training

[0064] The user puts on VR goggles and enters a virtual environment based on the training scenario sent from the server. Before starting training, the user can check their registration information and training history.

[0065] Operation in the virtual space

[0066] The user follows instructions in the virtual environment to perform a specified task, such as administering an injection to a virtual patient in medical training. All movements during the task are tracked and transmitted to a server via the device.

[0067] Receiving feedback and improving

[0068] Once the training is complete, feedback from the server is displayed on the device. The user is expected to receive the feedback and improve their next operation based on it. The more the user repeats the training, the more efficiently they can learn the skill.

[0069] Specific examples

[0070] For example, in the medical field, a scenario in which a new nurse learns injection techniques in a virtual environment would go something like this: The user (new nurse) puts on VR goggles and enters the virtual environment based on the injection training scenario sent from the server. The device displays a virtual patient and injection equipment, and the user performs the virtual injection. The server receives the user's operation data in real time and analyzes the injection angle, depth, timing, etc. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user can receive the feedback and use it the next time they conduct training.

[0071] This system is a powerful tool for efficiently and safely acquiring skills in fields such as medicine, construction, and safety training without using real resources.

[0072] The processing flow will be explained below.

[0073] Server Processing

[0074] Step 1:

[0075] User Data Collection

[0076] The server connects to the database and collects user registration information, training history, and progress information, which allows the user's profile and past training details to be understood.

[0077] Step 2:

[0078] Training Needs Analysis

[0079] The server analyzes the collected data and identifies the next skill or training content the user should learn. For example, if the user is currently learning injection techniques, it will select a training scenario that matches their progress.

[0080] Step 3:

[0081] AI-based scenario generation

[0082] The server uses artificial intelligence (AI) to generate customized training scenarios based on the user's training progress, including specific procedures and situation settings.

[0083] Step 4:

[0084] Sending scenario data

[0085] The server transmits the generated training scenario data to the user's device, allowing the user to begin practicing based on the training scenario.

[0086] Step 5:

[0087] Receiving real-time operation data

[0088] The server receives user operation data sent from the device in real time, including hand movements and position information performed by the user in the virtual space.

[0089] Step 6:

[0090] Dynamic updates for virtual environments

[0091] Based on the received operation data, the server dynamically updates the state of the virtual environment, for example, updating the virtual patient's reaction and the syringe's position when the user administers an injection.

[0092] Step 7:

[0093] Analyzing operation data and generating feedback

[0094] The server analyzes the user's operation data and generates individual feedback based on the success rate of the operation and the type of error, which specifically indicates areas for improvement for the user.

[0095] Step 8:

[0096] Send Feedback

[0097] The server sends the generated feedback data to the user's device, allowing the user to check their own performance and reflect it in their next training session.

[0098] Terminal handling

[0099] Step 1:

[0100] Receiving scenario data

[0101] The terminal receives the training scenario data sent from the server and prepares the virtual environment to be displayed based on the received data.

[0102] Step 2:

[0103] Rendering a Virtual Environment

[0104] Based on the training scenario, the device displays the virtual environment to the user through the VR goggles, allowing the user to begin training in a highly immersive virtual space.

[0105] Step 3:

[0106] Operational Data Collection

[0107] The device tracks user actions and collects data in real time, including hand movements and location information.

[0108] Step 4:

[0109] Sending operation data

[0110] The collected operation data is sent in real time to the server, which then dynamically updates the virtual environment based on this data.

[0111] Step 5:

[0112] Receiving feedback data

[0113] The terminal receives feedback data from the server, which includes points for improvement and evaluation of the operation.

[0114] Step 6:

[0115] View Feedback

[0116] The device then displays the received feedback in the virtual environment, allowing the user to see the evaluation of their own actions in real time and reflect it in their next training session.

[0117] User Action

[0118] Step 1:

[0119] Wearing VR goggles

[0120] The user puts on the VR goggles and prepares to begin training. The virtual environment is displayed through the goggles.

[0121] Step 2:

[0122] Selection of training scenarios

[0123] Users select a training scenario displayed on their device and enter the virtual environment, setting them up to practice a specific skill.

[0124] Step 3:

[0125] Operation in the virtual space

[0126] The user performs instructed operations in the virtual environment, such as administering an injection to a virtual patient.

[0127] Step 4:

[0128] Check real-time feedback

[0129] Users can view real-time feedback during training to understand areas for improvement in their operations.

[0130] Step 5:

[0131] Reflection on the next training

[0132] Based on the feedback, efforts are made to improve operations in the next training session, which increases the efficiency of skill acquisition.

[0133] Example 1

[0134] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0135] Conventional training systems using virtual reality technology have the problem of delayed feedback to user operations, reducing the effectiveness of real-time training. Furthermore, it is difficult to generate customized scenarios based on the user's training progress, resulting in insufficient individualized support. As a result, it is difficult for users to efficiently acquire skills.

[0136] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0137] In this invention, the server includes means for receiving user operation data in real time and dynamically updating the virtual environment, means for analyzing the user operation data and generating and transmitting individual feedback, means for the terminal to render a virtual reality environment and provide the user with a highly immersive experience, means for the terminal to track the user operation data in real time and transmit it to the server, and means for the server to analyze the user operation data and update the virtual environment in real time, thereby making it possible to provide immediate feedback to the user's operation and dynamically update the virtual environment.

[0138] "User" refers to a person who uses a system that uses virtual reality technology to receive specific skills or vocational training.

[0139] A "server" refers to a computer device that receives operational data from a user, updates the virtual environment, and generates feedback.

[0140] "Terminal" means a device used by a user to interact with a virtual reality environment, including VR goggles and a PC.

[0141] "Database" refers to a system that stores and manages data such as user registration information, training history, and progress information.

[0142] "Artificial intelligence means" refers to an algorithm or program for generating customized training scenarios based on a user's training progress.

[0143] "Training scenario" refers to a simulation program in virtual reality that allows users to learn a particular skill or operation.

[0144] "Operation data" refers to the actions performed by the user within the virtual reality environment and the data generated during those actions.

[0145] "Virtual environment" refers to a three-dimensional space that can be manipulated by a user and is generated using virtual reality technology.

[0146] "Real-time update" refers to the process of instantly changing the state and objects of the virtual environment in response to user operations.

[0147] "Feedback" refers to evaluations and advice obtained by analyzing user operation data, and refers to information provided to the user.

[0148] "Tracking" refers to the process of tracking user actions and collecting location and movement data.

[0149] "High immersion" refers to providing an experience that allows users to be deeply immersed in a virtual reality environment and feels so real.

[0150] This invention relates to a system that uses virtual reality technology to provide specific skills or vocational training. The system mainly consists of a user, a terminal, and a server. The role of each component and the corresponding process are described below.

[0151] Server Roles

[0152] User Data Management

[0153] The server connects to a database (e.g., MySQL or PostgreSQL) and manages user registration information, training history, and progress information. When a user logs into the system, the server retrieves the user's progress information from the database and identifies the next skills and training content to be learned. This information is updated in real time while the system is in use.

[0154] Training scenario generation

[0155] The server uses a generative AI model (e.g., GPT-4) to generate individually optimized training scenarios based on the user's progress. For example, in the medical field, a scenario could be "a new nurse administering an injection to a virtual patient." The training scenario is generated on the server side and sent to the user's device.

[0156] Providing Feedback

[0157] The server receives and analyzes user operation data in real time. Based on the analysis results, it generates specific feedback according to the success rate and type of error. The generated feedback is sent to the user's device and displayed on the device.

[0158] Device Role

[0159] Rendering a VR environment

[0160] The device renders a virtual environment based on the training scenario sent from the server. Specifically, it uses VR goggles (e.g., Oculus Rift or HTC Vive) to provide the user with a virtual space, allowing the user to train with a high level of immersion.

[0161] Operational Data Tracking

[0162] The device tracks the user's actions in real time and sends the data to a server. For example, when a user performs an injection in a virtual space, the device collects information about the user's hand movements and position.

[0163] View Feedback

[0164] The feedback data is sent from the server to the device, which then displays it in the VR environment. The user can check this feedback in real time and improve their next operation.

[0165] User Roles

[0166] Log in and start training

[0167] The user logs in to the system using a terminal (for example, by entering an email address and password) and enters the virtual environment based on the training scenario sent from the server. Before starting training, the user confirms their registration information and training history.

[0168] Operation in a virtual environment

[0169] The user follows instructions in the virtual environment to perform specific operations, such as administering an injection to a virtual patient in medical training. All operation data is sent from the device to a server for analysis.

[0170] Receiving feedback and improving

[0171] After the training is complete, feedback is displayed on the device, and the user is asked to take this feedback and improve their next operation based on it.

[0172] Specific examples

[0173] For example, if a new nurse were to learn injection techniques in a virtual environment, the process could go something like this: First, the user puts on VR goggles and enters the virtual environment based on an "injection training scenario" sent from the server. The device displays a virtual patient and injection equipment, and supports the user in the process of administering the virtual injection. The server receives the user's operation data in real time and analyzes the injection angle, depth, timing, etc. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user can then use that feedback to improve their operation in the next training session.

[0174] This system enables people in many fields, including medicine, construction, and safety training, to acquire skills efficiently and safely without using real resources.

[0175] Prompt Sentence Examples

[0176] Below are some example prompts to input to a generative AI model (e.g., GPT-4):

[0177] "This system allows new nurses to practice injection techniques in a virtual reality environment. Imagine a nurse putting on VR goggles and initiating an injection scenario in the virtual space. Please explain in detail the steps from the first step to providing feedback. Also mention real-time environment updates and feedback generation."

[0178] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0179] Step 1:

[0180] User Login

[0181] A user accesses the system from a terminal and enters authentication information (e.g., username and password) on the login screen.

[0182] Input: Username, Password

[0183] Specific operation: The user enters "example@example.com" and "password123" and clicks the login button.

[0184] Server processing: The server checks the user's authentication information against the database and issues a session ID if authentication is successful.

[0185] Output: Session ID

[0186] Step 2:

[0187] Acquiring and Managing User Data

[0188] The server retrieves the user's training history and progress information from the database based on the session ID.

[0189] Input: Session ID

[0190] Specific operation: The server queries the database to obtain the user's progress information.

[0191] Server processing: The server obtains the progress information described above and identifies the next skills and training to be learned.

[0192] Output: User progress information, next training content

[0193] Step 3:

[0194] Generating training scenarios

[0195] The server uses a generative AI model (e.g., GPT-4) to generate training scenarios based on the user's progress.

[0196] Input: User progress information, next training content

[0197] Specific operation: The server inputs a prompt such as "The user is a new nurse and is learning basic injection techniques" into the generated AI model and generates a scenario.

[0198] Server processing: The generative AI model generates specific training scenarios based on the input information.

[0199] Output: Generated training scenarios

[0200] Step 4:

[0201] Rendering a VR environment

[0202] The terminal renders the virtual environment based on the training scenario data sent from the server.

[0203] Input: Training scenario data

[0204] Specific operation: The device uses VR goggles to display virtual patients, training equipment, etc., and prepares the user to access the virtual space.

[0205] Terminal processing: The terminal analyzes the scenario data and constructs the VR environment.

[0206] Output: Virtual environment

[0207] Step 5:

[0208] Tracking user actions

[0209] The device tracks the user's actions in real time and sends the data to a server.

[0210] Input: User operation data

[0211] Specific operation: The user puts on the VR goggles and performs the action of giving an injection in the virtual space. The device uses the controller sensor and camera to collect hand movement and position information.

[0212] Terminal processing: Collected operation data is sent to the server in real time.

[0213] Output: Operation data sent

[0214] Step 6:

[0215] Real-time environment updates

[0216] The server receives the user's operation data and dynamically updates the virtual environment based on it.

[0217] Input: User operation data

[0218] Specific operation: The server analyzes the operation data and updates other objects in the virtual environment (e.g., the virtual patient's reaction).

[0219] Server processing: Updates information in the virtual environment based on the operation data and sends it to the terminal.

[0220] Output: Updated virtual environment data

[0221] Step 7:

[0222] Generating and Providing Feedback

[0223] The server analyzes the user's operation data and generates and sends individual feedback.

[0224] Input: User operation data

[0225] Specific behavior: The server analyzes the operation data in real time and generates specific feedback such as "The injection angle is too shallow."

[0226] Server processing: Generate feedback based on the analysis results and send it to the device.

[0227] Output: Generated feedback

[0228] Step 8:

[0229] View Feedback

[0230] The device displays the feedback received from the server within the VR environment.

[0231] Input: Feedback data

[0232] Specific behavior: The user sees the feedback in the virtual environment and prepares to incorporate it into their next operation.

[0233] Terminal processing: Feedback is displayed on the VR screen and presented to the user.

[0234] Output: Feedback displayed to the user

[0235] Through the above steps, users can acquire skills efficiently and effectively using virtual reality technology.

[0236] (Application example 1)

[0237] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0238] In today's advanced industrial environments, factory operators and engineers are required to have extremely high levels of robot operation and maintenance skills. However, learning these skills in a real-world environment involves high costs and risks, so there is a need for efficient and safe methods of acquiring these skills. In addition, there is a lack of systems that can provide real-time feedback and analyze operation data. Therefore, it is necessary to provide a system that can effectively acquire such skills using a virtual environment.

[0239] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0240] In this invention, the server includes: means for linking with a database that manages user registration information, training history, and progress information; artificial intelligence means for generating training scenarios tailored to the user's training progress; means for transmitting the generated training scenarios to the user's terminal; means for receiving user operation data in real time and dynamically updating the virtual environment; means for analyzing the user's operation data and generating and transmitting individual feedback; means for generating training scenarios for learning robot operation and maintenance techniques in the virtual factory environment; means for simulating robot arm operation and part installation based on the user's operation data and evaluating the accuracy and timing of the operation in real time; and means for displaying the generated feedback on the user's terminal in real time. This enables factory operators and engineers to efficiently and safely acquire robot operation and maintenance techniques through training in the virtual environment.

[0241] "Virtual reality technology" is a technology that allows users to experience a three-dimensional environment generated by a computer.

[0242] "Specific skills and vocational training" refers to education and training that allows users to acquire specialized knowledge and skills.

[0243] "System" refers to a set of devices and mechanisms consisting of multiple devices and software for providing skills and vocational training using virtual reality technology.

[0244] "User" refers to an individual or group who uses and receives training on this system.

[0245] "Registration information" refers to the user's personal information and data necessary for using the system.

[0246] "Training history" refers to a record of the training a user has done up to now.

[0247] "Progress information" is data that indicates the results that the user has achieved through training and their current level of learning.

[0248] The "database" is a system that manages and stores user registration information, training history, progress information, etc.

[0249] "Artificial intelligence" is a technology that gives computers the ability to learn, judge, and predict in the same way as humans.

[0250] A "training scenario" is a series of virtual reality training programs that a user executes to acquire a particular skill or knowledge.

[0251] A "terminal" is a device such as a computer or VR goggles that allows a user to access the system and experience the virtual reality environment.

[0252] "Operation data" refers to a record of the actions and operations performed by a user within a virtual reality environment.

[0253] A "virtual environment" is a virtual three-dimensional space generated by a computer.

[0254] "Dynamic updating" means changing the environment in real time in response to user operations.

[0255] "Feedback" refers to evaluations and comments on the user's operations and progress.

[0256] A "virtual factory environment" is a three-dimensional virtual reality space that mimics the inside of a factory.

[0257] "Robot operation" refers to the control and operation of robots used in factories.

[0258] "Maintenance technology" refers to technology related to the maintenance, inspection, and repair of factories and machinery equipment.

[0259] "Arm operation" refers to the operation of moving the arm of the robot to perform a specific task.

[0260] "Component installation" refers to the process of accurately placing and fixing components in specific positions.

[0261] A "prompt sentence" is an input sentence that causes a generative AI model to operate.

[0262] This invention provides a training system for factory robot operation and maintenance techniques using virtual reality technology. The system consists of a user, a terminal, and a server.

[0263] Server Roles

[0264] The server has the following functions:

[0265] 1. User data management: The server connects to a database and manages user registration information, training history, and progress information. A common RDBMS (e.g., MySQL) is used as the database.

[0266] 2. Training scenario generation: The server uses artificial intelligence technology to generate customized training scenarios based on the user's progress. It uses a generative AI model to generate specific training scenarios based on prompts. Examples include training scenarios for operating a factory robot's arm or installing parts.

[0267] 3. Real-time environment update: The server receives user operation data in real time and dynamically updates the virtual environment based on it, thereby providing real-time feedback according to the user's operations.

[0268] 4. Feedback provision: Analyzes user operation data and generates personalized feedback based on the accuracy and timing of operations. The generated feedback is sent to the user's device in real time.

[0269] Device Role

[0270] The terminal has the following features:

[0271] 1. Rendering the VR environment: The device displays the virtual environment to the user based on the training scenario data received from the server. The device is a VR headset (e.g., Oculus Quest).

[0272] 2. Collecting operation data: The device tracks the user's operations and transmits the data to the server in real time, using the motion tracking function of the VR headset.

[0273] 3. Feedback display: Feedback data received from the server is displayed in the VR environment, allowing users to check and improve in real time.

[0274] User Roles

[0275] The user uses the system in the following steps:

[0276] 1. Putting on the VR goggles and starting training: The user puts on the VR goggles and enters the virtual factory environment based on the training scenario sent from the server. Before starting the training, the user can check their registration information and training history.

[0277] 2. Operation in virtual space: The user follows instructions in the virtual environment and performs specific operations, such as operating the robot's arm to attach parts. All movements during the operation are tracked and transmitted to the server via the terminal.

[0278] 3. Receiving feedback and improving: After the training is completed, feedback from the server is displayed on the device. The user receives the feedback and is expected to improve their next operation based on it. The more the user repeats the training, the more efficiently they can learn the skill.

[0279] Specific examples

[0280] A specific example is the generation of a VR training scenario that simulates the arm operation of a factory robot based on the following conditions. For example, the prompt sentence "Please generate a VR training scenario that simulates the arm operation of a factory robot based on the following conditions" is input to the generative AI model. This prompt causes the AI ​​model to generate a specific training scenario, and the user can proceed with training according to that scenario.

[0281] As described above, this system is a powerful tool for efficiently and safely learning robot operation and maintenance techniques using a virtual environment.

[0282] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0283] Step 1:

[0284] The server retrieves the user's registration information, training history, and progress information from the database.

[0285] Input: User ID

[0286] Data processing: Performing database queries and extracting the required information.

[0287] Output: User registration information, training history, progress information

[0288] Step 2:

[0289] The server uses artificial intelligence to generate customized training scenarios based on the user's progress.

[0290] Input: User progress information

[0291] Data calculation: The generative AI model analyzes progress information and creates optimal training scenarios for the user.

[0292] Output: Customized training scenario

[0293] Specific operation: The following prompt is input to the generative AI model: "Generate a VR training scenario that simulates the arm operation of a factory robot based on the following conditions."

[0294] Step 3:

[0295] The server transmits the generated training scenario to the user's terminal.

[0296] Input: Training scenario

[0297] Data Transfer: Sending data over a network.

[0298] Output: The training scenario is displayed on the user's device.

[0299] Step 4:

[0300] The terminal renders and displays the virtual reality environment to the user.

[0301] Input: Training scenario data

[0302] Data processing: Pass the data to the virtual environment rendering engine to generate a 3D environment.

[0303] Output: Display of virtual reality environment

[0304] How it works: A training scenario is loaded into the VR headset and the user enters the virtual environment.

[0305] Step 5:

[0306] The user begins training in the virtual environment and inputs operational data into the terminal.

[0307] Input: User action

[0308] Data collection: Collect operation data using the motion tracking function of the VR headset.

[0309] Output: Collecting and sending operational data

[0310] Step 6:

[0311] The terminal transmits the user's operation data to the server in real time.

[0312] Input: Operation data

[0313] Data Transfer: Sending data over a network.

[0314] Output: Operation data is sent to the server

[0315] Step 7:

[0316] The server analyzes the user's operation data and generates feedback.

[0317] Input: Operation data

[0318] Data calculation: A generative AI model analyzes operational data and generates feedback based on the accuracy and timing of operations.

[0319] Output: Feedback data

[0320] Step 8:

[0321] The server transmits the generated feedback to the user's terminal.

[0322] Input: Feedback data

[0323] Data Transfer: Sending data over a network.

[0324] Output: Feedback is displayed on the user's device

[0325] Step 9:

[0326] The device displays feedback within the virtual environment.

[0327] Input: Feedback data

[0328] Data display: The feedback content is displayed in the VR environment.

[0329] Output: User can see the feedback

[0330] Specific operation: The feedback displayed is, "Arm operation accuracy is 85%, but there is room for improvement in part alignment."

[0331] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0332] This invention combines a system that uses virtual reality technology to provide specific skill or vocational training with an emotion engine that recognizes the user's emotions. The system is mainly composed of a user, a terminal, a server, and an emotion engine, and provides an efficient and safe virtual environment for the user to acquire specific skills, while also having the function of adjusting the training content and feedback according to the user's emotional state.

[0333] Server Roles

[0334] User Data Management

[0335] The server connects to a database and manages user registration information, training history, and progress information. Based on each user's progress, it identifies the next skills and training content they should learn. All of this information is stored in the database and is referenced every time a user accesses the system.

[0336] Training scenario generation

[0337] The server uses AI to generate customized training scenarios based on the user's progress. For example, in the medical field, it could generate a scenario for practicing injection techniques, creating specific procedures and a virtual environment.

[0338] Collaboration with emotion engine

[0339] The server works with the emotion engine to receive the user's emotional data. This emotional data is used to adjust the training scenario and generate feedback. For example, if the user is impatient, the difficulty of the scenario can be adjusted.

[0340] Real-time environment updates

[0341] The server receives user interaction data in real time and dynamically updates the virtual environment based on that data. Other objects in the environment also change dynamically in response to user interaction.

[0342] Analysis of operation data and emotion data and feedback generation

[0343] The system analyzes the user's operation data and emotional data, and generates personalized feedback based on the success rate of the operation and the type of error. The generated feedback takes into account the user's emotional state.

[0344] Send Feedback

[0345] The server sends the generated feedback data to the user's device, allowing the user to review the feedback based on their performance and emotions and reflect it in their next training session.

[0346] Device Role

[0347] Rendering a VR environment

[0348] The device displays a virtual environment to the user based on the training scenario data sent from the server. By using VR goggles, users can receive training in a highly immersive virtual space.

[0349] Operational Data Collection

[0350] The device tracks user actions and collects data in real time, including hand movements and location information.

[0351] Collecting Emotional Data

[0352] The device uses an emotion engine to collect user emotion data in real time, using techniques such as facial recognition and voice analysis.

[0353] Transmission of operational and emotional data

[0354] The collected operation data and emotion data are sent in real time to a server, which then dynamically updates the virtual environment based on this data.

[0355] Receiving feedback data

[0356] The device receives feedback data from the server, which includes operational improvements and takes into account the user's emotional state.

[0357] View Feedback

[0358] The device then displays the received feedback in the virtual environment, allowing the user to see the evaluation of their own actions and emotions in real time and reflect this in their next training session.

[0359] User Roles

[0360] Wearing VR goggles

[0361] The user puts on the VR goggles and prepares to begin training. The virtual environment is displayed through the goggles.

[0362] Selection of training scenarios

[0363] Users select a training scenario displayed on their device and enter the virtual environment, setting them up to practice a specific skill.

[0364] Operation in the virtual space

[0365] The user performs instructed operations in the virtual environment, such as administering an injection to a virtual patient.

[0366] Providing emotion data

[0367] Users cooperate by providing emotional data (e.g., facial expressions and voice) that the device collects during training.

[0368] Check real-time feedback

[0369] Users can view real-time feedback during training to understand areas for improvement in their operations and emotions.

[0370] Reflection on the next training

[0371] Based on the feedback, efforts will be made to improve operational and emotional management in the next training session, which will increase the efficiency of skill acquisition and improve the training experience.

[0372] Specific examples

[0373] For example, in the medical field, a scenario in which a new nurse learns injection techniques in a virtual environment would go something like this: The user (new nurse) puts on VR goggles and enters the virtual environment based on the injection training scenario sent from the server. The device displays a virtual patient and injection equipment, and the user performs the virtual injection. The server receives the user's operation data and emotional data (for example, detecting impatience through facial expressions and voice analysis) in real time and analyzes the injection angle, depth, timing, etc. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user is expected to receive feedback and use it to improve their next operation and emotional management.

[0374] This system is a powerful tool for efficiently and safely acquiring skills without using real-world resources in fields such as medicine, construction, and safety training. Furthermore, the introduction of an emotion engine improves the quality of training and makes the user's training experience more personalized.

[0375] The processing flow will be explained below.

[0376] Server Processing

[0377] Step 1:

[0378] User Data Collection

[0379] The server connects to the database and collects user registration information, training history, and progress information, which allows the user's profile and past training details to be understood.

[0380] Step 2:

[0381] Training Needs Analysis

[0382] The server analyzes the collected data and identifies the next skill or training content the user should learn. For example, if the user is currently learning injection techniques, it will select a training scenario that matches their progress.

[0383] Step 3:

[0384] AI-based scenario generation

[0385] The server uses artificial intelligence (AI) to generate customized training scenarios based on the user's training progress, including specific procedures and situation settings.

[0386] Step 4:

[0387] Sending scenario data

[0388] The server transmits the generated training scenario data to the user's device, allowing the user to begin practicing based on the training scenario.

[0389] Step 5:

[0390] Collecting Emotional Data

[0391] The server works in conjunction with the emotion engine to receive the user's emotion data (e.g., the user's facial expressions and voice analysis data) collected from the terminal.

[0392] Step 6:

[0393] Receiving real-time operation data

[0394] The server receives user operation data sent from the device in real time, including hand movements and position information performed by the user in the virtual space.

[0395] Step 7:

[0396] Dynamic updates for virtual environments

[0397] Based on the received operation and emotion data, the server dynamically updates the state of the virtual environment, e.g., when the user administers an injection, the virtual patient's reaction and the syringe's position are updated.

[0398] Step 8:

[0399] Analysis of operation data and emotion data and feedback generation

[0400] The server analyzes the user's operation data and emotional data, and generates individual feedback based on the success rate of the operation, the type of error, and the user's emotional state.

[0401] Step 9:

[0402] Send Feedback

[0403] The server sends the generated feedback data to the user's device, allowing the user to review the feedback based on their performance and emotional state and reflect it in their next training session.

[0404] Terminal handling

[0405] Step 1:

[0406] Receiving scenario data

[0407] The terminal receives the training scenario data sent from the server and prepares the virtual environment to be displayed based on the received data.

[0408] Step 2:

[0409] Rendering a Virtual Environment

[0410] Based on the training scenario, the device displays the virtual environment to the user through the VR goggles, allowing the user to begin training in a highly immersive virtual space.

[0411] Step 3:

[0412] Operational Data Collection

[0413] The device tracks user actions and collects data in real time, including hand movements and location information.

[0414] Step 4:

[0415] Collecting Emotional Data

[0416] The device uses an emotion engine to collect user emotion data in real time, using techniques such as facial recognition and voice analysis.

[0417] Step 5:

[0418] Transmission of operational and emotional data

[0419] The collected operation data and emotion data are sent in real time to a server, which then dynamically updates the virtual environment based on this data.

[0420] Step 6:

[0421] Receiving feedback data

[0422] The device receives feedback data from the server, which includes operational improvements and takes into account the user's emotional state.

[0423] Step 7:

[0424] View Feedback

[0425] The device then displays the received feedback in the virtual environment, allowing the user to see the evaluation of their own actions and emotions in real time and reflect this in their next training session.

[0426] User Action

[0427] Step 1:

[0428] Wearing VR goggles

[0429] The user puts on the VR goggles and prepares to begin training. The virtual environment is displayed through the goggles.

[0430] Step 2:

[0431] Selection of training scenarios

[0432] Users select a training scenario displayed on their device and enter the virtual environment, setting them up to practice a specific skill.

[0433] Step 3:

[0434] Operation in the virtual space

[0435] The user performs instructed operations in the virtual environment, such as administering an injection to a virtual patient.

[0436] Step 4:

[0437] Providing emotion data

[0438] Users cooperate by providing emotional data (e.g., facial expressions and voice) that the device collects during training.

[0439] Step 5:

[0440] Check real-time feedback

[0441] Users can view real-time feedback during training to understand areas for improvement in their operations and emotions.

[0442] Step 6:

[0443] Reflection on the next training

[0444] Based on the feedback, efforts will be made to improve operational and emotional management in the next training session, which will increase the efficiency of skill acquisition and improve the training experience.

[0445] Example 2

[0446] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0447] In conventional skill and vocational training systems using virtual reality technology, training scenarios and feedback were generated based solely on user operation data, making it difficult to provide personalized training content and feedback that took the user's emotional state into account. This limited the effectiveness of training for users, making it difficult to acquire skills efficiently and effectively.

[0448] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for linking with a database that manages user registration information, training history, and progress information; artificial intelligence means for generating a training scenario tailored to the user's training progress based on the database; means for transmitting the generated training scenario to the user's terminal; means for rendering a virtual environment and displaying it to the user; means for receiving user operation data and emotional data in real time and dynamically updating the virtual environment; means for analyzing the user's operation data and emotional data, generating and transmitting individual feedback; and means for transmitting the feedback to the user's terminal and displaying it in the virtual environment. This makes it possible to provide personalized training content that takes the user's emotional state into consideration, thereby achieving efficient and effective skill acquisition.

[0449] "User registration information" refers to personal information and authentication information provided by a user when accessing the system.

[0450] "Training history" refers to information showing records of training that a user has conducted in the past and the results of those training.

[0451] "Progress information" refers to data that indicates the progress and level of achievement of the training that the user has achieved up to now.

[0452] "Database" refers to a system for centrally managing and storing user registration information, training history, progress information, etc.

[0453] "Artificial intelligence means" refers to technology that uses machine learning and advanced data analysis techniques to generate training scenarios based on user progress information.

[0454] A "training scenario" is a program that defines the virtual environment and procedures for conducting specific skill or vocational training.

[0455] A "terminal" refers to a device that allows a user to access and operate a virtual environment. Examples include VR goggles and a personal computer.

[0456] "Rendering a virtual environment" refers to using computer graphics techniques to generate and visually display a virtual three-dimensional environment to a user.

[0457] "Operation data" refers to information about operations performed by a user in a virtual environment, such as hand movements and click history.

[0458] "Emotional data" refers to data that indicates the user's emotional state. Examples include emotional states determined by facial expression recognition or voice analysis.

[0459] "Dynamic updating" refers to changing the virtual environment in real time in response to the user's actions and emotional state.

[0460] "Analyzing" refers to the detailed analysis of collected data to extract meaningful information and patterns.

[0461] "Feedback" refers to a response, including suggestions for improvement and evaluation, generated based on the user's actions and emotional state.

[0462] "Displaying within the virtual environment" refers to visually presenting generated feedback and other information within the virtual three-dimensional space experienced by the user.

[0463] An "emotion engine" refers to the technology and system for collecting and analyzing user emotional data.

[0464] This invention combines a system that uses virtual reality technology to provide specific skill or vocational training with an emotion engine that recognizes the user's emotions. The system is mainly composed of a user, a terminal, a server, and an emotion engine, and provides an efficient and safe virtual environment for the user to acquire specific skills, while also having the function of adjusting the training content and feedback according to the user's emotional state.

[0465] Server Roles

[0466] The server uses multiple pieces of hardware and software to perform the following processes:

[0467] 1. User Data Management:

[0468] The server connects to a database (e.g., PostgreSQL) and manages user registration information, training history, and progress information. This allows it to identify the next skills and training content each user should learn based on their progress. All of this information is stored in the database and is referenced every time a user accesses the system.

[0469] 2. Training scenario generation:

[0470] The server uses AI (e.g., a TensorFlow model) to generate customized training scenarios based on the user's progress. For example, in the medical field, it could generate a scenario for practicing injection techniques, building specific procedures and a virtual environment.

[0471] 3. Collaboration with Emotion Engine:

[0472] The server works with an emotion engine (e.g., Affectiva) to receive real-time user emotion data, which is used to adjust training scenarios and generate feedback.

[0473] 4. Real-time environment updates:

[0474] The server receives user operation data in real time and dynamically updates the virtual environment based on that data, and other objects in the environment also change dynamically in response to the user's operations.

[0475] 5. Analysis of operation and emotion data and feedback generation:

[0476] The server analyzes the user's operation data and emotional data, and generates personalized feedback based on the success rate of the operation and the type of error. The generated feedback takes into account the user's emotional state.

[0477] 6. Submitting Feedback:

[0478] The server sends the generated feedback data to the user's device, allowing the user to review the feedback based on their performance and emotions and reflect it in their next training session.

[0479] Device Role

[0480] The terminal receives data from the server in the following manner.

[0481] 1. Rendering the VR environment:

[0482] The device displays a virtual environment to the user based on the training scenario data sent from the server. By using VR goggles (e.g., Oculus Rift), the user can receive training in a highly immersive virtual space.

[0483] 2. Operational Data Collection:

[0484] The device tracks user actions and collects data in real time, including hand movements and location information.

[0485] 3. Collecting Emotional Data:

[0486] The device uses an emotion engine to collect user emotion data in real time, using techniques such as facial recognition and voice analysis.

[0487] 4. Transmission of Operational and Emotional Data:

[0488] The collected operation data and emotion data are sent to a server in real time.

[0489] 5. Receiving Feedback Data:

[0490] The device receives feedback data from the server and displays it in the virtual environment, allowing the user to check the evaluation of their own actions and emotions in real time and reflect this in their next training session.

[0491] User Roles

[0492] The steps that the user should follow are as follows:

[0493] 1. Put on the VR goggles:

[0494] The user puts on the VR goggles and prepares to begin training.

[0495] 2. Selection of training scenario:

[0496] The user selects a training scenario displayed on the terminal and enters the virtual environment.

[0497] 3. Virtual space operations:

[0498] The user performs the indicated action in the virtual environment, for example, administering an injection to a virtual patient.

[0499] 4. Providing Emotion Data:

[0500] Users cooperate by providing emotional data (e.g., facial expressions and voice) that the device collects during training.

[0501] 5. Check real-time feedback:

[0502] Users can view real-time feedback during training to understand areas for improvement in their operations and emotions.

[0503] 6. Reflection on the next training:

[0504] Based on the feedback, efforts will be made to improve operations and emotional management in the next training session.

[0505] Specific examples

[0506] As a concrete example, we will explain a scenario in which a new nurse in the medical field learns injection techniques in a virtual environment. The user (new nurse) puts on VR goggles and enters the virtual environment based on an injection training scenario sent from the server. The terminal displays a virtual patient and injection equipment, and the user performs the virtual injection. The server receives the user's operation data and emotional data (e.g., detecting impatience through facial expressions and voice analysis) in real time and analyzes the angle, depth, timing, etc. of the injection. Feedback is generated based on the analysis results and displayed to the user in real time via the terminal. The user is expected to receive feedback and use it to improve their next operation and emotional management.

[0507] Examples of prompt statements

[0508] Below are some examples of prompts to input to a generative AI model (e.g., ChatGPT).

[0509] Example 1: "Create a scenario in which a new nurse is learning injection techniques in a virtual environment. Describe how the system adjusts feedback based on the user's emotional state."

[0510] Example 2: "Create a safety training scenario for a construction site. Explain in detail the operation of a system that collects and analyzes emotion data in real time, clearly indicating the roles of the server, device, and user."

[0511] The foregoing is an embodiment of the present invention, which allows for efficient and personalized training and improves user skill acquisition.

[0512] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0513] Step 1: User Registration and Login

[0514] Server: The server receives the new user's registration information (e.g., name, email address, password) as input and processes the data to store it in the database. For existing users, the server compares the entered login information with the information in the database and authenticates the login. The output is the user's dashboard if successful, or an error message if unsuccessful.

[0515] Terminal: The terminal receives registration and login information from the user as input and sends it to the server, which then passes the information as input to the server, which then provides an interface to display the processing results as output.

[0516] User: The user enters personal information such as name, email address, and password into the device to register or log in. The information entered becomes important initial data for the entire system.

[0517] Step 2: Select a training scenario

[0518] Server: The server takes user progress information from the database as input, performs data calculations using an AI model (e.g., TensorFlow) to generate customized training scenarios, and provides the output as recommended training scenarios to the user.

[0519] Terminal: The terminal receives as input the list of training scenarios sent from the server and outputs it as an interface to be displayed to the user.

[0520] User: The user selects the desired scenario from the list of training scenarios displayed on the terminal. The selected scenario becomes the input data used in the next step.

[0521] Step 3: Rendering the VR environment

[0522] Terminal: The terminal receives training scenario data (e.g., virtual patient, injection equipment placement data) sent from the server as input, and uses VR goggles to render a virtual environment based on this data and display it to the user. Specifically, it performs real-time 3D rendering using a GPU. The output is an immersive virtual space that the user can view.

[0523] User: The user wears VR goggles and practices in the displayed virtual environment. The information displayed in the VR environment will be the basis for the user's next step.

[0524] Step 4: Conduct training and collect data

[0525] Device: The device collects user operation data (e.g., hand movements, location information) and emotion data (e.g., facial expressions, voice) as input in real time and sends them to the server. This is done using motion sensors, microphones, and cameras. The output is raw data sent to the server in real time.

[0526] Server: The server receives the operation data and emotion data sent from the device as input, monitors and records the training progress in real time, and uses this data to determine the user's progress and determine the next step.

[0527] User: The user performs the instructed operations in the virtual environment and provides operational and emotional data by performing natural actions.

[0528] Step 5: Data analysis and feedback generation

[0529] Server: The server analyzes the received operation data and emotional data as input. Specifically, it performs data calculations to evaluate the success rate of operations, the type of error, and the user's emotional state (e.g., impatience or concentration). It uses an AI model to analyze behavioral and emotional patterns and generates feedback for the user. The output is individually customized feedback.

[0530] Terminal: Waits for analysis results and feedback data received from the server and prepares them for display to the user.

[0531] Step 6: View real-time feedback

[0532] Terminal: The terminal receives feedback data from the server as input and displays it in real time within the virtual environment. This can include text messages, audio prompts, and visual cues. The output is specific feedback information for the user.

[0533] User: The user reviews the displayed feedback and uses it to improve their own actions and emotional management.

[0534] Step 7: Completing the training and preparing for the next training

[0535] Server: After the training is completed, the server updates the user's training data and prepares new scenarios for the next training. It also provides an overall evaluation of the training if the user so desires. The output is a future training plan and an overall evaluation report.

[0536] Terminal: The terminal displays a message to the user that the training is complete and a list of possible training scenarios for the next training session.

[0537] Users: After completing the training, users can review the points covered based on the feedback and prepare for the next training session, thereby continuously improving their skills.

[0538] (Application example 2)

[0539] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0540] Conventional vocational training systems using virtual reality technology provide feedback based solely on the user's operational data, making it difficult to provide personalized training that takes into account the user's emotional state, such as stress or pressure. Furthermore, if the user falls into an inappropriate emotional state, they are unable to respond appropriately, resulting in a decrease in the efficiency and effectiveness of the training. Therefore, there is a need for a system that can analyze the user's emotional state in real time and appropriately adjust the training scenario and feedback.

[0541] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for linking with a database that manages user registration information, training history, and progress information; artificial intelligence means for generating a training scenario tailored to the user's training progress based on the database; means for transmitting the generated training scenario to the user's terminal; means for receiving the user's operation data and emotional data and adjusting the difficulty level of the virtual training environment based thereon; means for analyzing the collected emotional data using an emotional analysis model and generating feedback according to the user's emotional state; and means for transmitting the generated feedback to the user's terminal and displaying it in the virtual environment. This makes it possible to provide individualized and appropriate feedback that takes the user's emotional state into consideration, allowing the user to train efficiently and comfortably.

[0542] "Virtual reality technology" is a technology that uses computers to generate immersive virtual environments that are different from the real world and allow users to experience them.

[0543] "Specific skills or vocational training" refers to training that enables users to acquire the skills or knowledge required in a specific professional field.

[0544] A "system" is an integrated functional unit that combines multiple means to work toward a set of objectives.

[0545] A "database" is a collection of data organized in a particular way to allow for efficient data management and access.

[0546] "Artificial intelligence means" refers to algorithms or programs for learning, reasoning, and making decisions based on specific data.

[0547] A "training scenario" refers to a virtual environment configuration that includes a series of training procedures or situations designed around specific training objectives.

[0548] A "user terminal" is an electronic device that a user uses to receive training in a virtual reality environment.

[0549] "Operation data" refers to data related to various operations and actions performed by the user during training.

[0550] "Emotion data" is data that represents the user's emotional state and is primarily collected through facial expression recognition and voice analysis.

[0551] "Emotion analysis model" refers to an algorithm or program designed to analyze a user's emotional state.

[0552] "Feedback" refers to information or advice provided based on the user's behavior and emotional state during training.

[0553] A "virtual environment" is an artificial space or situation created using virtual reality technology.

[0554] "Means for adjusting difficulty" refers to a method or device for changing the difficulty of a training scenario depending on the user's performance or emotional state.

[0555] "Dynamic updating means" refers to a method or device for updating and changing the virtual environment as needed based on user operation data that changes in real time.

[0556] To realize the present invention, the following system configuration and program are used.

[0557] The server has a means for linking with a database that manages user registration information, training history, and progress information, and uses artificial intelligence means to generate training scenarios tailored to the user's training progress. The generated training scenarios are sent to the user's terminal. Furthermore, the server has a means for receiving the user's operation data and emotion data and appropriately adjusting the difficulty level of the virtual training environment based on the data.

[0558] The device includes VR goggles for rendering the virtual environment and a camera for facial recognition. The VR goggles are used to provide the user with a highly immersive virtual environment, and data is collected and transmitted by tracking the user's actions within the environment. The device also transmits the collected emotional data to a server in real time using an emotion engine. The device displays the feedback data received from the server in real time within the virtual environment, providing the user with appropriate information.

[0559] The user puts on the VR goggles, selects a training scenario displayed on the device, enters the virtual environment, and performs a specific operation. Emotional data felt during the operation (changes in facial expressions and voice) is collected by the device and sent to the server. The user checks the feedback sent from the server and uses it to improve the next operation.

[0560] Below, a training system for factory robots will be described as a specific example of use.

[0561] A scenario in which a new operator learns to operate a factory robot would go something like this: The user (new operator) puts on VR goggles and enters a virtual environment based on a robot operation training scenario sent from the server. The device displays a robot operation scene from within the factory, and the user operates the virtual robot. The server receives the user's operation data and emotional data (facial expressions and voice analysis detects impatience) in real time and analyzes the accuracy and timing of the operation. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user receives the feedback and uses it to improve their next operation.

[0562] This system is a powerful tool for training new operators of factory robots to acquire skills efficiently and safely without using real-world resources, and the introduction of an emotion engine improves the quality of training and makes the user's training experience more personalized.

[0563] As a concrete example, the following is an example of a prompt sentence to be input to the generative AI model.

[0564] "Create a program that adjusts the difficulty of the training scenario in real time based on the emotions experienced by new nurses while administering injections in the virtual environment. The program should include facial expression recognition, voice analysis, and dynamic updates to the virtual environment."

[0565] In this way, by analyzing the user's emotional state in real time and appropriately adjusting the training scenario and feedback, a system can be realized that allows the user to receive training efficiently and comfortably.

[0566] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0567] Step 1:

[0568] The server retrieves the user's registration information, training history, and progress information from the database. The input data is the user's ID, and the output data is the user's training history and progress information. The server evaluates the user's current skill level based on this information.

[0569] Step 2:

[0570] The server uses artificial intelligence to generate a training scenario tailored to the user's training progress. The input data is the user's training history and progress information acquired in step 1, and the output data is a customized training scenario. The server sends this training scenario to the user's device.

[0571] Step 3:

[0572] The device renders a virtual reality environment based on the training scenario received from the server. The input data is the training scenario sent from the server, and the output data is the virtual environment the user sees through the VR goggles. The device then prompts the user to start training.

[0573] Step 4:

[0574] The user wears VR goggles and performs specific operations in a virtual reality environment. The input is the user's physical movements, and the output is the operation data collected by the device. The user performs operations in the virtual environment according to the instructions.

[0575] Step 5:

[0576] The device collects user operation data and emotional data in real time. Input data includes the user's physical movements, facial expressions, and voice, while output data includes operation data and emotional data. The device analyzes this data and sends it to the server.

[0577] Step 6:

[0578] The server receives the operation data and emotion data sent from the device and adjusts the difficulty of the virtual training environment. The input data are the user's operation data and emotion data, and the output data is the adjusted training scenario. The server then sends instructions to the device to dynamically update the virtual environment based on the input data.

[0579] Step 7:

[0580] The terminal updates the virtual environment based on the adjusted training scenario received from the server. The input data is the adjusted training scenario sent from the server, and the output data is the updated virtual environment. The terminal displays the new training environment to the user.

[0581] Step 8:

[0582] The server analyzes the collected emotional data using an emotion analysis model and generates feedback according to the user's emotional state. The input data is the emotional data, and the output data is the feedback content. The server transmits this feedback data to the terminal.

[0583] Step 9:

[0584] The terminal displays the feedback data received from the server to the user in the virtual environment. The input data is the feedback data sent from the server, and the output data is the feedback information displayed to the user. The user can use this information to improve their next operation.

[0585] The above processing steps enable the user to receive personalized and appropriate feedback that takes into account the user's emotional state, allowing the user to receive training efficiently and comfortably, thereby improving the user's training experience and allowing the user to acquire skills efficiently.

[0586] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0587] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0588] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0589] [Second embodiment]

[0590] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0591] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0592] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0593] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0594] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0595] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0596] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0597] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0598] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0599] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0600] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0601] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0602] This invention relates to a system that provides specific skills and vocational training using virtual reality technology. The system mainly consists of a user, a terminal, and a server, and provides an efficient and safe virtual environment for users to acquire specific skills.

[0603] Server Roles

[0604] User Data Management

[0605] The server connects to a database and manages user registration information, training history, and progress information. Based on each user's progress, it can identify the next skills and training content they should learn. All of this information is stored in the database and is referenced every time a user accesses the system.

[0606] Training scenario generation

[0607] The server uses AI to generate customized training scenarios based on the user's progress. For example, in the medical field, it could generate a scenario for practicing injection techniques, creating specific procedures and a virtual environment.

[0608] Real-time environment updates

[0609] The server receives user operation data in real time and dynamically updates the virtual environment based on that data. Other objects in the environment also change dynamically according to the user's operations in the virtual space.

[0610] Providing Feedback

[0611] It analyzes user operation data and generates personalized feedback based on success rate and error type. This feedback is sent to the device in real time so that the user can reflect it in their next operation.

[0612] Device Role

[0613] Rendering a VR environment

[0614] The device displays a virtual environment to the user based on the training scenario data received from the server. By using VR goggles, the user can receive training in a highly immersive virtual space.

[0615] Operational Data Collection

[0616] The device tracks the user's actions and sends the data to a server in real time. For example, when a user injects a drug in a virtual space, the device collects information about the user's hand movements and location.

[0617] View Feedback

[0618] Feedback data received from the server is displayed in the VR environment, allowing users to check this feedback in real time and improve their operations.

[0619] User Roles

[0620] Putting it on and starting training

[0621] The user puts on VR goggles and enters a virtual environment based on the training scenario sent from the server. Before starting training, the user can check their registration information and training history.

[0622] Operation in the virtual space

[0623] The user follows instructions in the virtual environment to perform a specified task, such as administering an injection to a virtual patient in medical training. All movements during the task are tracked and transmitted to a server via the device.

[0624] Receiving feedback and improving

[0625] Once the training is complete, feedback from the server is displayed on the device. The user is expected to receive the feedback and improve their next operation based on it. The more the user repeats the training, the more efficiently they can learn the skill.

[0626] Specific examples

[0627] For example, in the medical field, a scenario in which a new nurse learns injection techniques in a virtual environment would go something like this: The user (new nurse) puts on VR goggles and enters the virtual environment based on the injection training scenario sent from the server. The device displays a virtual patient and injection equipment, and the user performs the virtual injection. The server receives the user's operation data in real time and analyzes the injection angle, depth, timing, etc. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user can receive the feedback and use it the next time they conduct training.

[0628] This system is a powerful tool for efficiently and safely acquiring skills in fields such as medicine, construction, and safety training without using real resources.

[0629] The processing flow will be explained below.

[0630] Server Processing

[0631] Step 1:

[0632] User Data Collection

[0633] The server connects to the database and collects user registration information, training history, and progress information, which allows the user's profile and past training details to be understood.

[0634] Step 2:

[0635] Training Needs Analysis

[0636] The server analyzes the collected data and identifies the next skill or training content the user should learn. For example, if the user is currently learning injection techniques, it will select a training scenario that matches their progress.

[0637] Step 3:

[0638] AI-based scenario generation

[0639] The server uses artificial intelligence (AI) to generate customized training scenarios based on the user's training progress, including specific procedures and situation settings.

[0640] Step 4:

[0641] Sending scenario data

[0642] The server transmits the generated training scenario data to the user's device, allowing the user to begin practicing based on the training scenario.

[0643] Step 5:

[0644] Receiving real-time operation data

[0645] The server receives user operation data sent from the device in real time, including hand movements and position information performed by the user in the virtual space.

[0646] Step 6:

[0647] Dynamic updates for virtual environments

[0648] Based on the received operation data, the server dynamically updates the state of the virtual environment, for example, updating the virtual patient's reaction and the syringe's position when the user administers an injection.

[0649] Step 7:

[0650] Analyzing operation data and generating feedback

[0651] The server analyzes the user's operation data and generates individual feedback based on the success rate of the operation and the type of error, which specifically indicates areas for improvement for the user.

[0652] Step 8:

[0653] Send Feedback

[0654] The server sends the generated feedback data to the user's device, allowing the user to check their own performance and reflect it in their next training session.

[0655] Terminal handling

[0656] Step 1:

[0657] Receiving scenario data

[0658] The terminal receives the training scenario data sent from the server and prepares the virtual environment to be displayed based on the received data.

[0659] Step 2:

[0660] Rendering a Virtual Environment

[0661] Based on the training scenario, the device displays the virtual environment to the user through the VR goggles, allowing the user to begin training in a highly immersive virtual space.

[0662] Step 3:

[0663] Operational Data Collection

[0664] The device tracks user actions and collects data in real time, including hand movements and location information.

[0665] Step 4:

[0666] Sending operation data

[0667] The collected operation data is sent in real time to the server, which then dynamically updates the virtual environment based on this data.

[0668] Step 5:

[0669] Receiving feedback data

[0670] The terminal receives feedback data from the server, which includes points for improvement and evaluation of the operation.

[0671] Step 6:

[0672] View Feedback

[0673] The device then displays the received feedback in the virtual environment, allowing the user to see the evaluation of their own actions in real time and reflect it in their next training session.

[0674] User Action

[0675] Step 1:

[0676] Wearing VR goggles

[0677] The user puts on the VR goggles and prepares to begin training. The virtual environment is displayed through the goggles.

[0678] Step 2:

[0679] Selection of training scenarios

[0680] Users select a training scenario displayed on their device and enter the virtual environment, setting them up to practice a specific skill.

[0681] Step 3:

[0682] Operation in the virtual space

[0683] The user performs instructed operations in the virtual environment, such as administering an injection to a virtual patient.

[0684] Step 4:

[0685] Check real-time feedback

[0686] Users can view real-time feedback during training to understand areas for improvement in their operations.

[0687] Step 5:

[0688] Reflection on the next training

[0689] Based on the feedback, efforts are made to improve operations in the next training session, which increases the efficiency of skill acquisition.

[0690] Example 1

[0691] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0692] Conventional training systems using virtual reality technology have the problem of delayed feedback to user operations, reducing the effectiveness of real-time training. Furthermore, it is difficult to generate customized scenarios based on the user's training progress, resulting in insufficient individualized support. As a result, it is difficult for users to efficiently acquire skills.

[0693] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0694] In this invention, the server includes means for receiving user operation data in real time and dynamically updating the virtual environment, means for analyzing the user operation data and generating and transmitting individual feedback, means for the terminal to render a virtual reality environment and provide the user with a highly immersive experience, means for the terminal to track the user operation data in real time and transmit it to the server, and means for the server to analyze the user operation data and update the virtual environment in real time, thereby making it possible to provide immediate feedback to the user's operation and dynamically update the virtual environment.

[0695] "User" refers to a person who uses a system that uses virtual reality technology to receive specific skills or vocational training.

[0696] A "server" refers to a computer device that receives operational data from a user, updates the virtual environment, and generates feedback.

[0697] "Terminal" means a device used by a user to interact with a virtual reality environment, including VR goggles and a PC.

[0698] "Database" refers to a system that stores and manages data such as user registration information, training history, and progress information.

[0699] "Artificial intelligence means" refers to an algorithm or program for generating customized training scenarios based on a user's training progress.

[0700] "Training scenario" refers to a simulation program in virtual reality that allows users to learn a particular skill or operation.

[0701] "Operation data" refers to the actions performed by the user within the virtual reality environment and the data generated during those actions.

[0702] "Virtual environment" refers to a three-dimensional space that can be manipulated by a user and is generated using virtual reality technology.

[0703] "Real-time update" refers to the process of instantly changing the state and objects of the virtual environment in response to user operations.

[0704] "Feedback" refers to evaluations and advice obtained by analyzing user operation data, and refers to information provided to the user.

[0705] "Tracking" refers to the process of tracking user actions and collecting location and movement data.

[0706] "High immersion" refers to providing an experience that allows users to be deeply immersed in a virtual reality environment and feels so real.

[0707] This invention relates to a system that uses virtual reality technology to provide specific skills or vocational training. The system mainly consists of a user, a terminal, and a server. The role of each component and the corresponding process are described below.

[0708] Server Roles

[0709] User Data Management

[0710] The server connects to a database (e.g., MySQL or PostgreSQL) and manages user registration information, training history, and progress information. When a user logs into the system, the server retrieves the user's progress information from the database and identifies the next skills and training content to be learned. This information is updated in real time while the system is in use.

[0711] Training scenario generation

[0712] The server uses a generative AI model (e.g., GPT-4) to generate individually optimized training scenarios based on the user's progress. For example, in the medical field, a scenario could be "a new nurse administering an injection to a virtual patient." The training scenario is generated on the server side and sent to the user's device.

[0713] Providing Feedback

[0714] The server receives and analyzes user operation data in real time. Based on the analysis results, it generates specific feedback according to the success rate and type of error. The generated feedback is sent to the user's device and displayed on the device.

[0715] Device Role

[0716] Rendering a VR environment

[0717] The device renders a virtual environment based on the training scenario sent from the server. Specifically, it uses VR goggles (e.g., Oculus Rift or HTC Vive) to provide the user with a virtual space, allowing the user to train with a high level of immersion.

[0718] Operational Data Tracking

[0719] The device tracks the user's actions in real time and sends the data to a server. For example, when a user performs an injection in a virtual space, the device collects information about the user's hand movements and position.

[0720] View Feedback

[0721] The feedback data is sent from the server to the device, which then displays it in the VR environment. The user can check this feedback in real time and improve their next operation.

[0722] User Roles

[0723] Log in and start training

[0724] The user logs in to the system using a terminal (for example, by entering an email address and password) and enters the virtual environment based on the training scenario sent from the server. Before starting training, the user confirms their registration information and training history.

[0725] Operation in a virtual environment

[0726] The user follows instructions in the virtual environment to perform specific operations, such as administering an injection to a virtual patient in medical training. All operation data is sent from the device to a server for analysis.

[0727] Receiving feedback and improving

[0728] After the training is complete, feedback is displayed on the device, and the user is asked to take this feedback and improve their next operation based on it.

[0729] Specific examples

[0730] For example, if a new nurse were to learn injection techniques in a virtual environment, the process could go something like this: First, the user puts on VR goggles and enters the virtual environment based on an "injection training scenario" sent from the server. The device displays a virtual patient and injection equipment, and supports the user in the process of administering the virtual injection. The server receives the user's operation data in real time and analyzes the injection angle, depth, timing, etc. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user can then use that feedback to improve their operation in the next training session.

[0731] This system enables people in many fields, including medicine, construction, and safety training, to acquire skills efficiently and safely without using real resources.

[0732] Prompt Sentence Examples

[0733] Below are some example prompts to input to a generative AI model (e.g., GPT-4):

[0734] "This system allows new nurses to practice injection techniques in a virtual reality environment. Imagine a nurse putting on VR goggles and initiating an injection scenario in the virtual space. Please explain in detail the steps from the first step to providing feedback. Also mention real-time environment updates and feedback generation."

[0735] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0736] Step 1:

[0737] User Login

[0738] A user accesses the system from a terminal and enters authentication information (e.g., username and password) on the login screen.

[0739] Input: Username, Password

[0740] Specific operation: The user enters "example@example.com" and "password123" and clicks the login button.

[0741] Server processing: The server checks the user's authentication information against the database and issues a session ID if authentication is successful.

[0742] Output: Session ID

[0743] Step 2:

[0744] Acquiring and Managing User Data

[0745] The server retrieves the user's training history and progress information from the database based on the session ID.

[0746] Input: Session ID

[0747] Specific operation: The server queries the database to obtain the user's progress information.

[0748] Server processing: The server obtains the progress information described above and identifies the next skills and training to be learned.

[0749] Output: User progress information, next training content

[0750] Step 3:

[0751] Generating training scenarios

[0752] The server uses a generative AI model (e.g., GPT-4) to generate training scenarios based on the user's progress.

[0753] Input: User progress information, next training content

[0754] Specific operation: The server inputs a prompt such as "The user is a new nurse and is learning basic injection techniques" into the generated AI model and generates a scenario.

[0755] Server processing: The generative AI model generates specific training scenarios based on the input information.

[0756] Output: Generated training scenarios

[0757] Step 4:

[0758] Rendering a VR environment

[0759] The terminal renders the virtual environment based on the training scenario data sent from the server.

[0760] Input: Training scenario data

[0761] Specific operation: The device uses VR goggles to display virtual patients, training equipment, etc., and prepares the user to access the virtual space.

[0762] Terminal processing: The terminal analyzes the scenario data and constructs the VR environment.

[0763] Output: Virtual environment

[0764] Step 5:

[0765] Tracking user actions

[0766] The device tracks the user's actions in real time and sends the data to a server.

[0767] Input: User operation data

[0768] Specific operation: The user puts on the VR goggles and performs the action of giving an injection in the virtual space. The device uses the controller sensor and camera to collect hand movement and position information.

[0769] Terminal processing: Collected operation data is sent to the server in real time.

[0770] Output: Operation data sent

[0771] Step 6:

[0772] Real-time environment updates

[0773] The server receives the user's operation data and dynamically updates the virtual environment based on it.

[0774] Input: User operation data

[0775] Specific operation: The server analyzes the operation data and updates other objects in the virtual environment (e.g., the virtual patient's reaction).

[0776] Server processing: Updates information in the virtual environment based on the operation data and sends it to the terminal.

[0777] Output: Updated virtual environment data

[0778] Step 7:

[0779] Generating and Providing Feedback

[0780] The server analyzes the user's operation data and generates and sends individual feedback.

[0781] Input: User operation data

[0782] Specific behavior: The server analyzes the operation data in real time and generates specific feedback such as "The injection angle is too shallow."

[0783] Server processing: Generate feedback based on the analysis results and send it to the device.

[0784] Output: Generated feedback

[0785] Step 8:

[0786] View Feedback

[0787] The device displays the feedback received from the server within the VR environment.

[0788] Input: Feedback data

[0789] Specific behavior: The user sees the feedback in the virtual environment and prepares to incorporate it into their next operation.

[0790] Terminal processing: Feedback is displayed on the VR screen and presented to the user.

[0791] Output: Feedback displayed to the user

[0792] Through the above steps, users can acquire skills efficiently and effectively using virtual reality technology.

[0793] (Application example 1)

[0794] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0795] In today's advanced industrial environments, factory operators and engineers are required to have extremely high levels of robot operation and maintenance skills. However, learning these skills in a real-world environment involves high costs and risks, so there is a need for efficient and safe methods of acquiring these skills. In addition, there is a lack of systems that can provide real-time feedback and analyze operation data. Therefore, it is necessary to provide a system that can effectively acquire such skills using a virtual environment.

[0796] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0797] In this invention, the server includes: means for linking with a database that manages user registration information, training history, and progress information; artificial intelligence means for generating training scenarios tailored to the user's training progress; means for transmitting the generated training scenarios to the user's terminal; means for receiving user operation data in real time and dynamically updating the virtual environment; means for analyzing the user's operation data and generating and transmitting individual feedback; means for generating training scenarios for learning robot operation and maintenance techniques in the virtual factory environment; means for simulating robot arm operation and part installation based on the user's operation data and evaluating the accuracy and timing of the operation in real time; and means for displaying the generated feedback on the user's terminal in real time. This enables factory operators and engineers to efficiently and safely acquire robot operation and maintenance techniques through training in the virtual environment.

[0798] "Virtual reality technology" is a technology that allows users to experience a three-dimensional environment generated by a computer.

[0799] "Specific skills and vocational training" refers to education and training that allows users to acquire specialized knowledge and skills.

[0800] "System" refers to a set of devices and mechanisms consisting of multiple devices and software for providing skills and vocational training using virtual reality technology.

[0801] "User" refers to an individual or group who uses and receives training on this system.

[0802] "Registration information" refers to the user's personal information and data necessary for using the system.

[0803] "Training history" refers to a record of the training a user has done up to now.

[0804] "Progress information" is data that indicates the results that the user has achieved through training and their current level of learning.

[0805] The "database" is a system that manages and stores user registration information, training history, progress information, etc.

[0806] "Artificial intelligence" is a technology that gives computers the ability to learn, judge, and predict in the same way as humans.

[0807] A "training scenario" is a series of virtual reality training programs that a user executes to acquire a particular skill or knowledge.

[0808] A "terminal" is a device such as a computer or VR goggles that allows a user to access the system and experience the virtual reality environment.

[0809] "Operation data" refers to a record of the actions and operations performed by a user within a virtual reality environment.

[0810] A "virtual environment" is a virtual three-dimensional space generated by a computer.

[0811] "Dynamic updating" means changing the environment in real time in response to user operations.

[0812] "Feedback" refers to evaluations and comments on the user's operations and progress.

[0813] A "virtual factory environment" is a three-dimensional virtual reality space that mimics the inside of a factory.

[0814] "Robot operation" refers to the control and operation of robots used in factories.

[0815] "Maintenance technology" refers to technology related to the maintenance, inspection, and repair of factories and machinery equipment.

[0816] "Arm operation" refers to the operation of moving the arm of the robot to perform a specific task.

[0817] "Component installation" refers to the process of accurately placing and fixing components in specific positions.

[0818] A "prompt sentence" is an input sentence that causes a generative AI model to operate.

[0819] This invention provides a training system for factory robot operation and maintenance techniques using virtual reality technology. The system consists of a user, a terminal, and a server.

[0820] Server Roles

[0821] The server has the following functions:

[0822] 1. User data management: The server connects to a database and manages user registration information, training history, and progress information. A common RDBMS (e.g., MySQL) is used as the database.

[0823] 2. Training scenario generation: The server uses artificial intelligence technology to generate customized training scenarios based on the user's progress. It uses a generative AI model to generate specific training scenarios based on prompts. Examples include training scenarios for operating a factory robot's arm or installing parts.

[0824] 3. Real-time environment update: The server receives user operation data in real time and dynamically updates the virtual environment based on it, thereby providing real-time feedback according to the user's operations.

[0825] 4. Feedback provision: Analyzes user operation data and generates personalized feedback based on the accuracy and timing of operations. The generated feedback is sent to the user's device in real time.

[0826] Device Role

[0827] The terminal has the following features:

[0828] 1. Rendering the VR environment: The device displays the virtual environment to the user based on the training scenario data received from the server. The device is a VR headset (e.g., Oculus Quest).

[0829] 2. Collecting operation data: The device tracks the user's operations and transmits the data to the server in real time, using the motion tracking function of the VR headset.

[0830] 3. Feedback display: Feedback data received from the server is displayed in the VR environment, allowing users to check and improve in real time.

[0831] User Roles

[0832] The user uses the system in the following steps:

[0833] 1. Putting on the VR goggles and starting training: The user puts on the VR goggles and enters the virtual factory environment based on the training scenario sent from the server. Before starting the training, the user can check their registration information and training history.

[0834] 2. Operation in virtual space: The user follows instructions in the virtual environment and performs specific operations, such as operating the robot's arm to attach parts. All movements during the operation are tracked and transmitted to the server via the terminal.

[0835] 3. Receiving feedback and improving: After the training is completed, feedback from the server is displayed on the device. The user receives the feedback and is expected to improve their next operation based on it. The more the user repeats the training, the more efficiently they can learn the skill.

[0836] Specific examples

[0837] A specific example is the generation of a VR training scenario that simulates the arm operation of a factory robot based on the following conditions. For example, the prompt sentence "Please generate a VR training scenario that simulates the arm operation of a factory robot based on the following conditions" is input to the generative AI model. This prompt causes the AI ​​model to generate a specific training scenario, and the user can proceed with training according to that scenario.

[0838] As described above, this system is a powerful tool for efficiently and safely learning robot operation and maintenance techniques using a virtual environment.

[0839] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0840] Step 1:

[0841] The server retrieves the user's registration information, training history, and progress information from the database.

[0842] Input: User ID

[0843] Data processing: Performing database queries and extracting the required information.

[0844] Output: User registration information, training history, progress information

[0845] Step 2:

[0846] The server uses artificial intelligence to generate customized training scenarios based on the user's progress.

[0847] Input: User progress information

[0848] Data calculation: The generative AI model analyzes progress information and creates optimal training scenarios for the user.

[0849] Output: Customized training scenario

[0850] Specific operation: The following prompt is input to the generative AI model: "Generate a VR training scenario that simulates the arm operation of a factory robot based on the following conditions."

[0851] Step 3:

[0852] The server transmits the generated training scenario to the user's terminal.

[0853] Input: Training scenario

[0854] Data Transfer: Sending data over a network.

[0855] Output: The training scenario is displayed on the user's device.

[0856] Step 4:

[0857] The terminal renders and displays the virtual reality environment to the user.

[0858] Input: Training scenario data

[0859] Data processing: Pass the data to the virtual environment rendering engine to generate a 3D environment.

[0860] Output: Display of virtual reality environment

[0861] How it works: A training scenario is loaded into the VR headset and the user enters the virtual environment.

[0862] Step 5:

[0863] The user begins training in the virtual environment and inputs operational data into the terminal.

[0864] Input: User action

[0865] Data collection: Collect operation data using the motion tracking function of the VR headset.

[0866] Output: Collecting and sending operational data

[0867] Step 6:

[0868] The terminal transmits the user's operation data to the server in real time.

[0869] Input: Operation data

[0870] Data Transfer: Sending data over a network.

[0871] Output: Operation data is sent to the server

[0872] Step 7:

[0873] The server analyzes the user's operation data and generates feedback.

[0874] Input: Operation data

[0875] Data calculation: A generative AI model analyzes operational data and generates feedback based on the accuracy and timing of operations.

[0876] Output: Feedback data

[0877] Step 8:

[0878] The server transmits the generated feedback to the user's terminal.

[0879] Input: Feedback data

[0880] Data Transfer: Sending data over a network.

[0881] Output: Feedback is displayed on the user's device

[0882] Step 9:

[0883] The device displays feedback within the virtual environment.

[0884] Input: Feedback data

[0885] Data display: The feedback content is displayed in the VR environment.

[0886] Output: User can see the feedback

[0887] Specific operation: The feedback displayed is, "Arm operation accuracy is 85%, but there is room for improvement in part alignment."

[0888] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0889] This invention combines a system that uses virtual reality technology to provide specific skill or vocational training with an emotion engine that recognizes the user's emotions. The system is mainly composed of a user, a terminal, a server, and an emotion engine, and provides an efficient and safe virtual environment for the user to acquire specific skills, while also having the function of adjusting the training content and feedback according to the user's emotional state.

[0890] Server Roles

[0891] User Data Management

[0892] The server connects to a database and manages user registration information, training history, and progress information. Based on each user's progress, it identifies the next skills and training content they should learn. All of this information is stored in the database and is referenced every time a user accesses the system.

[0893] Training scenario generation

[0894] The server uses AI to generate customized training scenarios based on the user's progress. For example, in the medical field, it could generate a scenario for practicing injection techniques, creating specific procedures and a virtual environment.

[0895] Collaboration with emotion engine

[0896] The server works with the emotion engine to receive the user's emotional data. This emotional data is used to adjust the training scenario and generate feedback. For example, if the user is impatient, the difficulty of the scenario can be adjusted.

[0897] Real-time environment updates

[0898] The server receives user interaction data in real time and dynamically updates the virtual environment based on that data. Other objects in the environment also change dynamically in response to user interaction.

[0899] Analysis of operation data and emotion data and feedback generation

[0900] The system analyzes the user's operation data and emotional data, and generates personalized feedback based on the success rate of the operation and the type of error. The generated feedback takes into account the user's emotional state.

[0901] Send Feedback

[0902] The server sends the generated feedback data to the user's device, allowing the user to review the feedback based on their performance and emotions and reflect it in their next training session.

[0903] Device Role

[0904] Rendering a VR environment

[0905] The device displays a virtual environment to the user based on the training scenario data sent from the server. By using VR goggles, users can receive training in a highly immersive virtual space.

[0906] Operational Data Collection

[0907] The device tracks user actions and collects data in real time, including hand movements and location information.

[0908] Collecting Emotional Data

[0909] The device uses an emotion engine to collect user emotion data in real time, using techniques such as facial recognition and voice analysis.

[0910] Transmission of operational and emotional data

[0911] The collected operation data and emotion data are sent in real time to a server, which then dynamically updates the virtual environment based on this data.

[0912] Receiving feedback data

[0913] The device receives feedback data from the server, which includes operational improvements and takes into account the user's emotional state.

[0914] View Feedback

[0915] The device then displays the received feedback in the virtual environment, allowing the user to see the evaluation of their own actions and emotions in real time and reflect this in their next training session.

[0916] User Roles

[0917] Wearing VR goggles

[0918] The user puts on the VR goggles and prepares to begin training. The virtual environment is displayed through the goggles.

[0919] Selection of training scenarios

[0920] Users select a training scenario displayed on their device and enter the virtual environment, setting them up to practice a specific skill.

[0921] Operation in the virtual space

[0922] The user performs instructed operations in the virtual environment, such as administering an injection to a virtual patient.

[0923] Providing emotion data

[0924] Users cooperate by providing emotional data (e.g., facial expressions and voice) that the device collects during training.

[0925] Check real-time feedback

[0926] Users can view real-time feedback during training to understand areas for improvement in their operations and emotions.

[0927] Reflection on the next training

[0928] Based on the feedback, efforts will be made to improve operational and emotional management in the next training session, which will increase the efficiency of skill acquisition and improve the training experience.

[0929] Specific examples

[0930] For example, in the medical field, a scenario in which a new nurse learns injection techniques in a virtual environment would go something like this: The user (new nurse) puts on VR goggles and enters the virtual environment based on the injection training scenario sent from the server. The device displays a virtual patient and injection equipment, and the user performs the virtual injection. The server receives the user's operation data and emotional data (for example, detecting impatience through facial expressions and voice analysis) in real time and analyzes the injection angle, depth, timing, etc. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user is expected to receive feedback and use it to improve their next operation and emotional management.

[0931] This system is a powerful tool for efficiently and safely acquiring skills without using real-world resources in fields such as medicine, construction, and safety training. Furthermore, the introduction of an emotion engine improves the quality of training and makes the user's training experience more personalized.

[0932] The processing flow will be explained below.

[0933] Server Processing

[0934] Step 1:

[0935] User Data Collection

[0936] The server connects to the database and collects user registration information, training history, and progress information, which allows the user's profile and past training details to be understood.

[0937] Step 2:

[0938] Training Needs Analysis

[0939] The server analyzes the collected data and identifies the next skill or training content the user should learn. For example, if the user is currently learning injection techniques, it will select a training scenario that matches their progress.

[0940] Step 3:

[0941] AI-based scenario generation

[0942] The server uses artificial intelligence (AI) to generate customized training scenarios based on the user's training progress, including specific procedures and situation settings.

[0943] Step 4:

[0944] Sending scenario data

[0945] The server transmits the generated training scenario data to the user's device, allowing the user to begin practicing based on the training scenario.

[0946] Step 5:

[0947] Collecting Emotional Data

[0948] The server works in conjunction with the emotion engine to receive the user's emotion data (e.g., the user's facial expressions and voice analysis data) collected from the terminal.

[0949] Step 6:

[0950] Receiving real-time operation data

[0951] The server receives user operation data sent from the device in real time, including hand movements and position information performed by the user in the virtual space.

[0952] Step 7:

[0953] Dynamic updates for virtual environments

[0954] Based on the received operation and emotion data, the server dynamically updates the state of the virtual environment, e.g., when the user administers an injection, the virtual patient's reaction and the syringe's position are updated.

[0955] Step 8:

[0956] Analysis of operation data and emotion data and feedback generation

[0957] The server analyzes the user's operation data and emotional data, and generates individual feedback based on the success rate of the operation, the type of error, and the user's emotional state.

[0958] Step 9:

[0959] Send Feedback

[0960] The server sends the generated feedback data to the user's device, allowing the user to review the feedback based on their performance and emotional state and reflect it in their next training session.

[0961] Terminal handling

[0962] Step 1:

[0963] Receiving scenario data

[0964] The terminal receives the training scenario data sent from the server and prepares the virtual environment to be displayed based on the received data.

[0965] Step 2:

[0966] Rendering a Virtual Environment

[0967] Based on the training scenario, the device displays the virtual environment to the user through the VR goggles, allowing the user to begin training in a highly immersive virtual space.

[0968] Step 3:

[0969] Operational Data Collection

[0970] The device tracks user actions and collects data in real time, including hand movements and location information.

[0971] Step 4:

[0972] Collecting Emotional Data

[0973] The device uses an emotion engine to collect user emotion data in real time, using techniques such as facial recognition and voice analysis.

[0974] Step 5:

[0975] Transmission of operational and emotional data

[0976] The collected operation data and emotion data are sent in real time to a server, which then dynamically updates the virtual environment based on this data.

[0977] Step 6:

[0978] Receiving feedback data

[0979] The device receives feedback data from the server, which includes operational improvements and takes into account the user's emotional state.

[0980] Step 7:

[0981] View Feedback

[0982] The device then displays the received feedback in the virtual environment, allowing the user to see the evaluation of their own actions and emotions in real time and reflect this in their next training session.

[0983] User Action

[0984] Step 1:

[0985] Wearing VR goggles

[0986] The user puts on the VR goggles and prepares to begin training. The virtual environment is displayed through the goggles.

[0987] Step 2:

[0988] Selection of training scenarios

[0989] Users select a training scenario displayed on their device and enter the virtual environment, setting them up to practice a specific skill.

[0990] Step 3:

[0991] Operation in the virtual space

[0992] The user performs instructed operations in the virtual environment, such as administering an injection to a virtual patient.

[0993] Step 4:

[0994] Providing emotion data

[0995] Users cooperate by providing emotional data (e.g., facial expressions and voice) that the device collects during training.

[0996] Step 5:

[0997] Check real-time feedback

[0998] Users can view real-time feedback during training to understand areas for improvement in their operations and emotions.

[0999] Step 6:

[1000] Reflection on the next training

[1001] Based on the feedback, efforts will be made to improve operational and emotional management in the next training session, which will increase the efficiency of skill acquisition and improve the training experience.

[1002] Example 2

[1003] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1004] In conventional skill and vocational training systems using virtual reality technology, training scenarios and feedback were generated based solely on user operation data, making it difficult to provide personalized training content and feedback that took the user's emotional state into account. This limited the effectiveness of training for users, making it difficult to acquire skills efficiently and effectively.

[1005] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for linking with a database that manages user registration information, training history, and progress information; artificial intelligence means for generating a training scenario tailored to the user's training progress based on the database; means for transmitting the generated training scenario to the user's terminal; means for rendering a virtual environment and displaying it to the user; means for receiving user operation data and emotional data in real time and dynamically updating the virtual environment; means for analyzing the user's operation data and emotional data, generating and transmitting individual feedback; and means for transmitting the feedback to the user's terminal and displaying it in the virtual environment. This makes it possible to provide personalized training content that takes the user's emotional state into consideration, thereby achieving efficient and effective skill acquisition.

[1006] "User registration information" refers to personal information and authentication information provided by a user when accessing the system.

[1007] "Training history" refers to information showing records of training that a user has conducted in the past and the results of those training.

[1008] "Progress information" refers to data that indicates the progress and level of achievement of the training that the user has achieved up to now.

[1009] "Database" refers to a system for centrally managing and storing user registration information, training history, progress information, etc.

[1010] "Artificial intelligence means" refers to technology that uses machine learning and advanced data analysis techniques to generate training scenarios based on user progress information.

[1011] A "training scenario" is a program that defines the virtual environment and procedures for conducting specific skill or vocational training.

[1012] A "terminal" refers to a device that allows a user to access and operate a virtual environment. Examples include VR goggles and a personal computer.

[1013] "Rendering a virtual environment" refers to using computer graphics techniques to generate and visually display a virtual three-dimensional environment to a user.

[1014] "Operation data" refers to information about operations performed by a user in a virtual environment, such as hand movements and click history.

[1015] "Emotional data" refers to data that indicates the user's emotional state. Examples include emotional states determined by facial expression recognition or voice analysis.

[1016] "Dynamic updating" refers to changing the virtual environment in real time in response to the user's actions and emotional state.

[1017] "Analyzing" refers to the detailed analysis of collected data to extract meaningful information and patterns.

[1018] "Feedback" refers to a response, including suggestions for improvement and evaluation, generated based on the user's actions and emotional state.

[1019] "Displaying within the virtual environment" refers to visually presenting generated feedback and other information within the virtual three-dimensional space experienced by the user.

[1020] An "emotion engine" refers to the technology and system for collecting and analyzing user emotional data.

[1021] This invention combines a system that uses virtual reality technology to provide specific skill or vocational training with an emotion engine that recognizes the user's emotions. The system is mainly composed of a user, a terminal, a server, and an emotion engine, and provides an efficient and safe virtual environment for the user to acquire specific skills, while also having the function of adjusting the training content and feedback according to the user's emotional state.

[1022] Server Roles

[1023] The server uses multiple pieces of hardware and software to perform the following processes:

[1024] 1. User Data Management:

[1025] The server connects to a database (e.g., PostgreSQL) and manages user registration information, training history, and progress information. This allows it to identify the next skills and training content each user should learn based on their progress. All of this information is stored in the database and is referenced every time a user accesses the system.

[1026] 2. Training scenario generation:

[1027] The server uses AI (e.g., a TensorFlow model) to generate customized training scenarios based on the user's progress. For example, in the medical field, it could generate a scenario for practicing injection techniques, building specific procedures and a virtual environment.

[1028] 3. Collaboration with Emotion Engine:

[1029] The server works with an emotion engine (e.g., Affectiva) to receive real-time user emotion data, which is used to adjust training scenarios and generate feedback.

[1030] 4. Real-time environment updates:

[1031] The server receives user operation data in real time and dynamically updates the virtual environment based on that data, and other objects in the environment also change dynamically in response to the user's operations.

[1032] 5. Analysis of operation and emotion data and feedback generation:

[1033] The server analyzes the user's operation data and emotional data, and generates personalized feedback based on the success rate of the operation and the type of error. The generated feedback takes into account the user's emotional state.

[1034] 6. Submitting Feedback:

[1035] The server sends the generated feedback data to the user's device, allowing the user to review the feedback based on their performance and emotions and reflect it in their next training session.

[1036] Device Role

[1037] The terminal receives data from the server in the following manner.

[1038] 1. Rendering the VR environment:

[1039] The device displays a virtual environment to the user based on the training scenario data sent from the server. By using VR goggles (e.g., Oculus Rift), the user can receive training in a highly immersive virtual space.

[1040] 2. Operational Data Collection:

[1041] The device tracks user actions and collects data in real time, including hand movements and location information.

[1042] 3. Collecting Emotional Data:

[1043] The device uses an emotion engine to collect user emotion data in real time, using techniques such as facial recognition and voice analysis.

[1044] 4. Transmission of Operational and Emotional Data:

[1045] The collected operation data and emotion data are sent to a server in real time.

[1046] 5. Receiving Feedback Data:

[1047] The device receives feedback data from the server and displays it in the virtual environment, allowing the user to check the evaluation of their own actions and emotions in real time and reflect this in their next training session.

[1048] User Roles

[1049] The steps that the user should follow are as follows:

[1050] 1. Put on the VR goggles:

[1051] The user puts on the VR goggles and prepares to begin training.

[1052] 2. Selection of training scenario:

[1053] The user selects a training scenario displayed on the terminal and enters the virtual environment.

[1054] 3. Virtual space operations:

[1055] The user performs the indicated action in the virtual environment, for example, administering an injection to a virtual patient.

[1056] 4. Providing Emotion Data:

[1057] Users cooperate by providing emotional data (e.g., facial expressions and voice) that the device collects during training.

[1058] 5. Check real-time feedback:

[1059] Users can view real-time feedback during training to understand areas for improvement in their operations and emotions.

[1060] 6. Reflection on the next training:

[1061] Based on the feedback, efforts will be made to improve operations and emotional management in the next training session.

[1062] Specific examples

[1063] As a concrete example, we will explain a scenario in which a new nurse in the medical field learns injection techniques in a virtual environment. The user (new nurse) puts on VR goggles and enters the virtual environment based on an injection training scenario sent from the server. The terminal displays a virtual patient and injection equipment, and the user performs the virtual injection. The server receives the user's operation data and emotional data (e.g., detecting impatience through facial expressions and voice analysis) in real time and analyzes the angle, depth, timing, etc. of the injection. Feedback is generated based on the analysis results and displayed to the user in real time via the terminal. The user is expected to receive feedback and use it to improve their next operation and emotional management.

[1064] Examples of prompt statements

[1065] Below are some examples of prompts to input to a generative AI model (e.g., ChatGPT).

[1066] Example 1: "Create a scenario in which a new nurse is learning injection techniques in a virtual environment. Describe how the system adjusts feedback based on the user's emotional state."

[1067] Example 2: "Create a safety training scenario for a construction site. Explain in detail the operation of a system that collects and analyzes emotion data in real time, clearly indicating the roles of the server, device, and user."

[1068] The foregoing is an embodiment of the present invention, which allows for efficient and personalized training and improves user skill acquisition.

[1069] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1070] Step 1: User Registration and Login

[1071] Server: The server receives the new user's registration information (e.g., name, email address, password) as input and processes the data to store it in the database. For existing users, the server compares the entered login information with the information in the database and authenticates the login. The output is the user's dashboard if successful, or an error message if unsuccessful.

[1072] Terminal: The terminal receives registration and login information from the user as input and sends it to the server, which then passes the information as input to the server, which then provides an interface to display the processing results as output.

[1073] User: The user enters personal information such as name, email address, and password into the device to register or log in. The information entered becomes important initial data for the entire system.

[1074] Step 2: Select a training scenario

[1075] Server: The server takes user progress information from the database as input, performs data calculations using an AI model (e.g., TensorFlow) to generate customized training scenarios, and provides the output as recommended training scenarios to the user.

[1076] Terminal: The terminal receives as input the list of training scenarios sent from the server and outputs it as an interface to be displayed to the user.

[1077] User: The user selects the desired scenario from the list of training scenarios displayed on the terminal. The selected scenario becomes the input data used in the next step.

[1078] Step 3: Rendering the VR environment

[1079] Terminal: The terminal receives training scenario data (e.g., virtual patient, injection equipment placement data) sent from the server as input, and uses VR goggles to render a virtual environment based on this data and display it to the user. Specifically, it performs real-time 3D rendering using a GPU. The output is an immersive virtual space that the user can view.

[1080] User: The user wears VR goggles and practices in the displayed virtual environment. The information displayed in the VR environment will be the basis for the user's next step.

[1081] Step 4: Conduct training and collect data

[1082] Device: The device collects user operation data (e.g., hand movements, location information) and emotion data (e.g., facial expressions, voice) as input in real time and sends them to the server. This is done using motion sensors, microphones, and cameras. The output is raw data sent to the server in real time.

[1083] Server: The server receives the operation data and emotion data sent from the device as input, monitors and records the training progress in real time, and uses this data to determine the user's progress and determine the next step.

[1084] User: The user performs the instructed operations in the virtual environment and provides operational and emotional data by performing natural actions.

[1085] Step 5: Data analysis and feedback generation

[1086] Server: The server analyzes the received operation data and emotional data as input. Specifically, it performs data calculations to evaluate the success rate of operations, the type of error, and the user's emotional state (e.g., impatience or concentration). It uses an AI model to analyze behavioral and emotional patterns and generates feedback for the user. The output is individually customized feedback.

[1087] Terminal: Waits for analysis results and feedback data received from the server and prepares them for display to the user.

[1088] Step 6: View real-time feedback

[1089] Terminal: The terminal receives feedback data from the server as input and displays it in real time within the virtual environment. This can include text messages, audio prompts, and visual cues. The output is specific feedback information for the user.

[1090] User: The user reviews the displayed feedback and uses it to improve their own actions and emotional management.

[1091] Step 7: Completing the training and preparing for the next training

[1092] Server: After the training is completed, the server updates the user's training data and prepares new scenarios for the next training. It also provides an overall evaluation of the training if the user so desires. The output is a future training plan and an overall evaluation report.

[1093] Terminal: The terminal displays a message to the user that the training is complete and a list of possible training scenarios for the next training session.

[1094] Users: After completing the training, users can review the points covered based on the feedback and prepare for the next training session, thereby continuously improving their skills.

[1095] (Application example 2)

[1096] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1097] Conventional vocational training systems using virtual reality technology provide feedback based solely on the user's operational data, making it difficult to provide personalized training that takes into account the user's emotional state, such as stress or pressure. Furthermore, if the user falls into an inappropriate emotional state, they are unable to respond appropriately, resulting in a decrease in the efficiency and effectiveness of the training. Therefore, there is a need for a system that can analyze the user's emotional state in real time and appropriately adjust the training scenario and feedback.

[1098] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for linking with a database that manages user registration information, training history, and progress information; artificial intelligence means for generating a training scenario tailored to the user's training progress based on the database; means for transmitting the generated training scenario to the user's terminal; means for receiving the user's operation data and emotional data and adjusting the difficulty level of the virtual training environment based thereon; means for analyzing the collected emotional data using an emotional analysis model and generating feedback according to the user's emotional state; and means for transmitting the generated feedback to the user's terminal and displaying it in the virtual environment. This makes it possible to provide individualized and appropriate feedback that takes the user's emotional state into consideration, allowing the user to train efficiently and comfortably.

[1099] "Virtual reality technology" is a technology that uses computers to generate immersive virtual environments that are different from the real world and allow users to experience them.

[1100] "Specific skills or vocational training" refers to training that enables users to acquire the skills or knowledge required in a specific professional field.

[1101] A "system" is an integrated functional unit that combines multiple means to work toward a set of objectives.

[1102] A "database" is a collection of data organized in a particular way to allow for efficient data management and access.

[1103] "Artificial intelligence means" refers to algorithms or programs for learning, reasoning, and making decisions based on specific data.

[1104] A "training scenario" refers to a virtual environment configuration that includes a series of training procedures or situations designed around specific training objectives.

[1105] A "user terminal" is an electronic device that a user uses to receive training in a virtual reality environment.

[1106] "Operation data" refers to data related to various operations and actions performed by the user during training.

[1107] "Emotion data" is data that represents the user's emotional state and is primarily collected through facial expression recognition and voice analysis.

[1108] "Emotion analysis model" refers to an algorithm or program designed to analyze a user's emotional state.

[1109] "Feedback" refers to information or advice provided based on the user's behavior and emotional state during training.

[1110] A "virtual environment" is an artificial space or situation created using virtual reality technology.

[1111] "Means for adjusting difficulty" refers to a method or device for changing the difficulty of a training scenario depending on the user's performance or emotional state.

[1112] "Dynamic updating means" refers to a method or device for updating and changing the virtual environment as needed based on user operation data that changes in real time.

[1113] To realize the present invention, the following system configuration and program are used.

[1114] The server has a means for linking with a database that manages user registration information, training history, and progress information, and uses artificial intelligence means to generate training scenarios tailored to the user's training progress. The generated training scenarios are sent to the user's terminal. Furthermore, the server has a means for receiving the user's operation data and emotion data and appropriately adjusting the difficulty level of the virtual training environment based on the data.

[1115] The device includes VR goggles for rendering the virtual environment and a camera for facial recognition. The VR goggles are used to provide the user with a highly immersive virtual environment, and data is collected and transmitted by tracking the user's actions within the environment. The device also transmits the collected emotional data to a server in real time using an emotion engine. The device displays the feedback data received from the server in real time within the virtual environment, providing the user with appropriate information.

[1116] The user puts on the VR goggles, selects a training scenario displayed on the device, enters the virtual environment, and performs a specific operation. Emotional data felt during the operation (changes in facial expressions and voice) is collected by the device and sent to the server. The user checks the feedback sent from the server and uses it to improve the next operation.

[1117] Below, a training system for factory robots will be described as a specific example of use.

[1118] A scenario in which a new operator learns to operate a factory robot would go something like this: The user (new operator) puts on VR goggles and enters a virtual environment based on a robot operation training scenario sent from the server. The device displays a robot operation scene from within the factory, and the user operates the virtual robot. The server receives the user's operation data and emotional data (facial expressions and voice analysis detects impatience) in real time and analyzes the accuracy and timing of the operation. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user receives the feedback and uses it to improve their next operation.

[1119] This system is a powerful tool for training new operators of factory robots to acquire skills efficiently and safely without using real-world resources, and the introduction of an emotion engine improves the quality of training and makes the user's training experience more personalized.

[1120] As a concrete example, the following is an example of a prompt sentence to be input to the generative AI model.

[1121] "Create a program that adjusts the difficulty of the training scenario in real time based on the emotions experienced by new nurses while administering injections in the virtual environment. The program should include facial expression recognition, voice analysis, and dynamic updates to the virtual environment."

[1122] In this way, by analyzing the user's emotional state in real time and appropriately adjusting the training scenario and feedback, a system can be realized that allows the user to receive training efficiently and comfortably.

[1123] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1124] Step 1:

[1125] The server retrieves the user's registration information, training history, and progress information from the database. The input data is the user's ID, and the output data is the user's training history and progress information. The server evaluates the user's current skill level based on this information.

[1126] Step 2:

[1127] The server uses artificial intelligence to generate a training scenario tailored to the user's training progress. The input data is the user's training history and progress information acquired in step 1, and the output data is a customized training scenario. The server sends this training scenario to the user's device.

[1128] Step 3:

[1129] The device renders a virtual reality environment based on the training scenario received from the server. The input data is the training scenario sent from the server, and the output data is the virtual environment the user sees through the VR goggles. The device then prompts the user to start training.

[1130] Step 4:

[1131] The user wears VR goggles and performs specific operations in a virtual reality environment. The input is the user's physical movements, and the output is the operation data collected by the device. The user performs operations in the virtual environment according to the instructions.

[1132] Step 5:

[1133] The device collects user operation data and emotional data in real time. Input data includes the user's physical movements, facial expressions, and voice, while output data includes operation data and emotional data. The device analyzes this data and sends it to the server.

[1134] Step 6:

[1135] The server receives the operation data and emotion data sent from the device and adjusts the difficulty of the virtual training environment. The input data are the user's operation data and emotion data, and the output data is the adjusted training scenario. The server then sends instructions to the device to dynamically update the virtual environment based on the input data.

[1136] Step 7:

[1137] The terminal updates the virtual environment based on the adjusted training scenario received from the server. The input data is the adjusted training scenario sent from the server, and the output data is the updated virtual environment. The terminal displays the new training environment to the user.

[1138] Step 8:

[1139] The server analyzes the collected emotional data using an emotion analysis model and generates feedback according to the user's emotional state. The input data is the emotional data, and the output data is the feedback content. The server transmits this feedback data to the terminal.

[1140] Step 9:

[1141] The terminal displays the feedback data received from the server to the user in the virtual environment. The input data is the feedback data sent from the server, and the output data is the feedback information displayed to the user. The user can use this information to improve their next operation.

[1142] The above processing steps enable the user to receive personalized and appropriate feedback that takes into account the user's emotional state, allowing the user to receive training efficiently and comfortably, thereby improving the user's training experience and allowing the user to acquire skills efficiently.

[1143] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1144] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1145] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1146] [Third embodiment]

[1147] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1148] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1149] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1150] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1151] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1152] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1153] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1154] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1155] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1156] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1157] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1158] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1159] This invention relates to a system that provides specific skills and vocational training using virtual reality technology. The system mainly consists of a user, a terminal, and a server, and provides an efficient and safe virtual environment for users to acquire specific skills.

[1160] Server Roles

[1161] User Data Management

[1162] The server connects to a database and manages user registration information, training history, and progress information. Based on each user's progress, it can identify the next skills and training content they should learn. All of this information is stored in the database and is referenced every time a user accesses the system.

[1163] Training scenario generation

[1164] The server uses AI to generate customized training scenarios based on the user's progress. For example, in the medical field, it could generate a scenario for practicing injection techniques, creating specific procedures and a virtual environment.

[1165] Real-time environment updates

[1166] The server receives user operation data in real time and dynamically updates the virtual environment based on that data. Other objects in the environment also change dynamically according to the user's operations in the virtual space.

[1167] Providing Feedback

[1168] It analyzes user operation data and generates personalized feedback based on success rate and error type. This feedback is sent to the device in real time so that the user can reflect it in their next operation.

[1169] Device Role

[1170] Rendering a VR environment

[1171] The device displays a virtual environment to the user based on the training scenario data received from the server. By using VR goggles, the user can receive training in a highly immersive virtual space.

[1172] Operational Data Collection

[1173] The device tracks the user's actions and sends the data to a server in real time. For example, when a user injects a drug in a virtual space, the device collects information about the user's hand movements and location.

[1174] View Feedback

[1175] Feedback data received from the server is displayed in the VR environment, allowing users to check this feedback in real time and improve their operations.

[1176] User Roles

[1177] Putting it on and starting training

[1178] The user puts on VR goggles and enters a virtual environment based on the training scenario sent from the server. Before starting training, the user can check their registration information and training history.

[1179] Operation in the virtual space

[1180] The user follows instructions in the virtual environment to perform a specified task, such as administering an injection to a virtual patient in medical training. All movements during the task are tracked and transmitted to a server via the device.

[1181] Receiving feedback and improving

[1182] Once the training is complete, feedback from the server is displayed on the device. The user is expected to receive the feedback and improve their next operation based on it. The more the user repeats the training, the more efficiently they can learn the skill.

[1183] Specific examples

[1184] For example, in the medical field, a scenario in which a new nurse learns injection techniques in a virtual environment would go something like this: The user (new nurse) puts on VR goggles and enters the virtual environment based on the injection training scenario sent from the server. The device displays a virtual patient and injection equipment, and the user performs the virtual injection. The server receives the user's operation data in real time and analyzes the injection angle, depth, timing, etc. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user can receive the feedback and use it the next time they conduct training.

[1185] This system is a powerful tool for efficiently and safely acquiring skills in fields such as medicine, construction, and safety training without using real resources.

[1186] The processing flow will be explained below.

[1187] Server Processing

[1188] Step 1:

[1189] User Data Collection

[1190] The server connects to the database and collects user registration information, training history, and progress information, which allows the user's profile and past training details to be understood.

[1191] Step 2:

[1192] Training Needs Analysis

[1193] The server analyzes the collected data and identifies the next skill or training content the user should learn. For example, if the user is currently learning injection techniques, it will select a training scenario that matches their progress.

[1194] Step 3:

[1195] AI-based scenario generation

[1196] The server uses artificial intelligence (AI) to generate customized training scenarios based on the user's training progress, including specific procedures and situation settings.

[1197] Step 4:

[1198] Sending scenario data

[1199] The server transmits the generated training scenario data to the user's device, allowing the user to begin practicing based on the training scenario.

[1200] Step 5:

[1201] Receiving real-time operation data

[1202] The server receives user operation data sent from the device in real time, including hand movements and position information performed by the user in the virtual space.

[1203] Step 6:

[1204] Dynamic updates for virtual environments

[1205] Based on the received operation data, the server dynamically updates the state of the virtual environment, for example, updating the virtual patient's reaction and the syringe's position when the user administers an injection.

[1206] Step 7:

[1207] Analyzing operation data and generating feedback

[1208] The server analyzes the user's operation data and generates individual feedback based on the success rate of the operation and the type of error, which specifically indicates areas for improvement for the user.

[1209] Step 8:

[1210] Send Feedback

[1211] The server sends the generated feedback data to the user's device, allowing the user to check their own performance and reflect it in their next training session.

[1212] Terminal handling

[1213] Step 1:

[1214] Receiving scenario data

[1215] The terminal receives the training scenario data sent from the server and prepares the virtual environment to be displayed based on the received data.

[1216] Step 2:

[1217] Rendering a Virtual Environment

[1218] Based on the training scenario, the device displays the virtual environment to the user through the VR goggles, allowing the user to begin training in a highly immersive virtual space.

[1219] Step 3:

[1220] Operational Data Collection

[1221] The device tracks user actions and collects data in real time, including hand movements and location information.

[1222] Step 4:

[1223] Sending operation data

[1224] The collected operation data is sent in real time to the server, which then dynamically updates the virtual environment based on this data.

[1225] Step 5:

[1226] Receiving feedback data

[1227] The terminal receives feedback data from the server, which includes points for improvement and evaluation of the operation.

[1228] Step 6:

[1229] View Feedback

[1230] The device then displays the received feedback in the virtual environment, allowing the user to see the evaluation of their own actions in real time and reflect it in their next training session.

[1231] User Action

[1232] Step 1:

[1233] Wearing VR goggles

[1234] The user puts on the VR goggles and prepares to begin training. The virtual environment is displayed through the goggles.

[1235] Step 2:

[1236] Selection of training scenarios

[1237] Users select a training scenario displayed on their device and enter the virtual environment, setting them up to practice a specific skill.

[1238] Step 3:

[1239] Operation in the virtual space

[1240] The user performs instructed operations in the virtual environment, such as administering an injection to a virtual patient.

[1241] Step 4:

[1242] Check real-time feedback

[1243] Users can view real-time feedback during training to understand areas for improvement in their operations.

[1244] Step 5:

[1245] Reflection on the next training

[1246] Based on the feedback, efforts are made to improve operations in the next training session, which increases the efficiency of skill acquisition.

[1247] Example 1

[1248] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1249] Conventional training systems using virtual reality technology have the problem of delayed feedback to user operations, reducing the effectiveness of real-time training. Furthermore, it is difficult to generate customized scenarios based on the user's training progress, resulting in insufficient individualized support. As a result, it is difficult for users to efficiently acquire skills.

[1250] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1251] In this invention, the server includes means for receiving user operation data in real time and dynamically updating the virtual environment, means for analyzing the user operation data and generating and transmitting individual feedback, means for the terminal to render a virtual reality environment and provide the user with a highly immersive experience, means for the terminal to track the user operation data in real time and transmit it to the server, and means for the server to analyze the user operation data and update the virtual environment in real time, thereby making it possible to provide immediate feedback to the user's operation and dynamically update the virtual environment.

[1252] "User" refers to a person who uses a system that uses virtual reality technology to receive specific skills or vocational training.

[1253] A "server" refers to a computer device that receives operational data from a user, updates the virtual environment, and generates feedback.

[1254] "Terminal" means a device used by a user to interact with a virtual reality environment, including VR goggles and a PC.

[1255] "Database" refers to a system that stores and manages data such as user registration information, training history, and progress information.

[1256] "Artificial intelligence means" refers to an algorithm or program for generating customized training scenarios based on a user's training progress.

[1257] "Training scenario" refers to a simulation program in virtual reality that allows users to learn a particular skill or operation.

[1258] "Operation data" refers to the actions performed by the user within the virtual reality environment and the data generated during those actions.

[1259] "Virtual environment" refers to a three-dimensional space that can be manipulated by a user and is generated using virtual reality technology.

[1260] "Real-time update" refers to the process of instantly changing the state and objects of the virtual environment in response to user operations.

[1261] "Feedback" refers to evaluations and advice obtained by analyzing user operation data, and refers to information provided to the user.

[1262] "Tracking" refers to the process of tracking user actions and collecting location and movement data.

[1263] "High immersion" refers to providing an experience that allows users to be deeply immersed in a virtual reality environment and feels so real.

[1264] This invention relates to a system that uses virtual reality technology to provide specific skills or vocational training. The system mainly consists of a user, a terminal, and a server. The role of each component and the corresponding process are described below.

[1265] Server Roles

[1266] User Data Management

[1267] The server connects to a database (e.g., MySQL or PostgreSQL) and manages user registration information, training history, and progress information. When a user logs into the system, the server retrieves the user's progress information from the database and identifies the next skills and training content to be learned. This information is updated in real time while the system is in use.

[1268] Training scenario generation

[1269] The server uses a generative AI model (e.g., GPT-4) to generate individually optimized training scenarios based on the user's progress. For example, in the medical field, a scenario could be "a new nurse administering an injection to a virtual patient." The training scenario is generated on the server side and sent to the user's device.

[1270] Providing Feedback

[1271] The server receives and analyzes user operation data in real time. Based on the analysis results, it generates specific feedback according to the success rate and type of error. The generated feedback is sent to the user's device and displayed on the device.

[1272] Device Role

[1273] Rendering a VR environment

[1274] The device renders a virtual environment based on the training scenario sent from the server. Specifically, it uses VR goggles (e.g., Oculus Rift or HTC Vive) to provide the user with a virtual space, allowing the user to train with a high level of immersion.

[1275] Operational Data Tracking

[1276] The device tracks the user's actions in real time and sends the data to a server. For example, when a user performs an injection in a virtual space, the device collects information about the user's hand movements and position.

[1277] View Feedback

[1278] The feedback data is sent from the server to the device, which then displays it in the VR environment. The user can check this feedback in real time and improve their next operation.

[1279] User Roles

[1280] Log in and start training

[1281] The user logs in to the system using a terminal (for example, by entering an email address and password) and enters the virtual environment based on the training scenario sent from the server. Before starting training, the user confirms their registration information and training history.

[1282] Operation in a virtual environment

[1283] The user follows instructions in the virtual environment to perform specific operations, such as administering an injection to a virtual patient in medical training. All operation data is sent from the device to a server for analysis.

[1284] Receiving feedback and improving

[1285] After the training is complete, feedback is displayed on the device, and the user is asked to take this feedback and improve their next operation based on it.

[1286] Specific examples

[1287] For example, if a new nurse were to learn injection techniques in a virtual environment, the process could go something like this: First, the user puts on VR goggles and enters the virtual environment based on an "injection training scenario" sent from the server. The device displays a virtual patient and injection equipment, and supports the user in the process of administering the virtual injection. The server receives the user's operation data in real time and analyzes the injection angle, depth, timing, etc. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user can then use that feedback to improve their operation in the next training session.

[1288] This system enables people in many fields, including medicine, construction, and safety training, to acquire skills efficiently and safely without using real resources.

[1289] Prompt Sentence Examples

[1290] Below are some example prompts to input to a generative AI model (e.g., GPT-4):

[1291] "This system allows new nurses to practice injection techniques in a virtual reality environment. Imagine a nurse putting on VR goggles and initiating an injection scenario in the virtual space. Please explain in detail the steps from the first step to providing feedback. Also mention real-time environment updates and feedback generation."

[1292] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1293] Step 1:

[1294] User Login

[1295] A user accesses the system from a terminal and enters authentication information (e.g., username and password) on the login screen.

[1296] Input: Username, Password

[1297] Specific operation: The user enters "example@example.com" and "password123" and clicks the login button.

[1298] Server processing: The server checks the user's authentication information against the database and issues a session ID if authentication is successful.

[1299] Output: Session ID

[1300] Step 2:

[1301] Acquiring and Managing User Data

[1302] The server retrieves the user's training history and progress information from the database based on the session ID.

[1303] Input: Session ID

[1304] Specific operation: The server queries the database to obtain the user's progress information.

[1305] Server processing: The server obtains the progress information described above and identifies the next skills and training to be learned.

[1306] Output: User progress information, next training content

[1307] Step 3:

[1308] Generating training scenarios

[1309] The server uses a generative AI model (e.g., GPT-4) to generate training scenarios based on the user's progress.

[1310] Input: User progress information, next training content

[1311] Specific operation: The server inputs a prompt such as "The user is a new nurse and is learning basic injection techniques" into the generated AI model and generates a scenario.

[1312] Server processing: The generative AI model generates specific training scenarios based on the input information.

[1313] Output: Generated training scenarios

[1314] Step 4:

[1315] Rendering a VR environment

[1316] The terminal renders the virtual environment based on the training scenario data sent from the server.

[1317] Input: Training scenario data

[1318] Specific operation: The device uses VR goggles to display virtual patients, training equipment, etc., and prepares the user to access the virtual space.

[1319] Terminal processing: The terminal analyzes the scenario data and constructs the VR environment.

[1320] Output: Virtual environment

[1321] Step 5:

[1322] Tracking user actions

[1323] The device tracks the user's actions in real time and sends the data to a server.

[1324] Input: User operation data

[1325] Specific operation: The user puts on the VR goggles and performs the action of giving an injection in the virtual space. The device uses the controller sensor and camera to collect hand movement and position information.

[1326] Terminal processing: Collected operation data is sent to the server in real time.

[1327] Output: Operation data sent

[1328] Step 6:

[1329] Real-time environment updates

[1330] The server receives the user's operation data and dynamically updates the virtual environment based on it.

[1331] Input: User operation data

[1332] Specific operation: The server analyzes the operation data and updates other objects in the virtual environment (e.g., the virtual patient's reaction).

[1333] Server processing: Updates information in the virtual environment based on the operation data and sends it to the terminal.

[1334] Output: Updated virtual environment data

[1335] Step 7:

[1336] Generating and Providing Feedback

[1337] The server analyzes the user's operation data and generates and sends individual feedback.

[1338] Input: User operation data

[1339] Specific behavior: The server analyzes the operation data in real time and generates specific feedback such as "The injection angle is too shallow."

[1340] Server processing: Generate feedback based on the analysis results and send it to the device.

[1341] Output: Generated feedback

[1342] Step 8:

[1343] View Feedback

[1344] The device displays the feedback received from the server within the VR environment.

[1345] Input: Feedback data

[1346] Specific behavior: The user sees the feedback in the virtual environment and prepares to incorporate it into their next operation.

[1347] Terminal processing: Feedback is displayed on the VR screen and presented to the user.

[1348] Output: Feedback displayed to the user

[1349] Through the above steps, users can acquire skills efficiently and effectively using virtual reality technology.

[1350] (Application example 1)

[1351] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1352] In today's advanced industrial environments, factory operators and engineers are required to have extremely high levels of robot operation and maintenance skills. However, learning these skills in a real-world environment involves high costs and risks, so there is a need for efficient and safe methods of acquiring these skills. In addition, there is a lack of systems that can provide real-time feedback and analyze operation data. Therefore, it is necessary to provide a system that can effectively acquire such skills using a virtual environment.

[1353] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1354] In this invention, the server includes: means for linking with a database that manages user registration information, training history, and progress information; artificial intelligence means for generating training scenarios tailored to the user's training progress; means for transmitting the generated training scenarios to the user's terminal; means for receiving user operation data in real time and dynamically updating the virtual environment; means for analyzing the user's operation data and generating and transmitting individual feedback; means for generating training scenarios for learning robot operation and maintenance techniques in the virtual factory environment; means for simulating robot arm operation and part installation based on the user's operation data and evaluating the accuracy and timing of the operation in real time; and means for displaying the generated feedback on the user's terminal in real time. This enables factory operators and engineers to efficiently and safely acquire robot operation and maintenance techniques through training in the virtual environment.

[1355] "Virtual reality technology" is a technology that allows users to experience a three-dimensional environment generated by a computer.

[1356] "Specific skills and vocational training" refers to education and training that allows users to acquire specialized knowledge and skills.

[1357] "System" refers to a set of devices and mechanisms consisting of multiple devices and software for providing skills and vocational training using virtual reality technology.

[1358] "User" refers to an individual or group who uses and receives training on this system.

[1359] "Registration information" refers to the user's personal information and data necessary for using the system.

[1360] "Training history" refers to a record of the training a user has done up to now.

[1361] "Progress information" is data that indicates the results that the user has achieved through training and their current level of learning.

[1362] The "database" is a system that manages and stores user registration information, training history, progress information, etc.

[1363] "Artificial intelligence" is a technology that gives computers the ability to learn, judge, and predict in the same way as humans.

[1364] A "training scenario" is a series of virtual reality training programs that a user executes to acquire a particular skill or knowledge.

[1365] A "terminal" is a device such as a computer or VR goggles that allows a user to access the system and experience the virtual reality environment.

[1366] "Operation data" refers to a record of the actions and operations performed by a user within a virtual reality environment.

[1367] A "virtual environment" is a virtual three-dimensional space generated by a computer.

[1368] "Dynamic updating" means changing the environment in real time in response to user operations.

[1369] "Feedback" refers to evaluations and comments on the user's operations and progress.

[1370] A "virtual factory environment" is a three-dimensional virtual reality space that mimics the inside of a factory.

[1371] "Robot operation" refers to the control and operation of robots used in factories.

[1372] "Maintenance technology" refers to technology related to the maintenance, inspection, and repair of factories and machinery equipment.

[1373] "Arm operation" refers to the operation of moving the arm of the robot to perform a specific task.

[1374] "Component installation" refers to the process of accurately placing and fixing components in specific positions.

[1375] A "prompt sentence" is an input sentence that causes a generative AI model to operate.

[1376] This invention provides a training system for factory robot operation and maintenance techniques using virtual reality technology. The system consists of a user, a terminal, and a server.

[1377] Server Roles

[1378] The server has the following functions:

[1379] 1. User data management: The server connects to a database and manages user registration information, training history, and progress information. A common RDBMS (e.g., MySQL) is used as the database.

[1380] 2. Training scenario generation: The server uses artificial intelligence technology to generate customized training scenarios based on the user's progress. It uses a generative AI model to generate specific training scenarios based on prompts. Examples include training scenarios for operating a factory robot's arm or installing parts.

[1381] 3. Real-time environment update: The server receives user operation data in real time and dynamically updates the virtual environment based on it, thereby providing real-time feedback according to the user's operations.

[1382] 4. Feedback provision: Analyzes user operation data and generates personalized feedback based on the accuracy and timing of operations. The generated feedback is sent to the user's device in real time.

[1383] Device Role

[1384] The terminal has the following features:

[1385] 1. Rendering the VR environment: The device displays the virtual environment to the user based on the training scenario data received from the server. The device is a VR headset (e.g., Oculus Quest).

[1386] 2. Collecting operation data: The device tracks the user's operations and transmits the data to the server in real time, using the motion tracking function of the VR headset.

[1387] 3. Feedback display: Feedback data received from the server is displayed in the VR environment, allowing users to check and improve in real time.

[1388] User Roles

[1389] The user uses the system in the following steps:

[1390] 1. Putting on the VR goggles and starting training: The user puts on the VR goggles and enters the virtual factory environment based on the training scenario sent from the server. Before starting the training, the user can check their registration information and training history.

[1391] 2. Operation in virtual space: The user follows instructions in the virtual environment and performs specific operations, such as operating the robot's arm to attach parts. All movements during the operation are tracked and transmitted to the server via the terminal.

[1392] 3. Receiving feedback and improving: After the training is completed, feedback from the server is displayed on the device. The user receives the feedback and is expected to improve their next operation based on it. The more the user repeats the training, the more efficiently they can learn the skill.

[1393] Specific examples

[1394] A specific example is the generation of a VR training scenario that simulates the arm operation of a factory robot based on the following conditions. For example, the prompt sentence "Please generate a VR training scenario that simulates the arm operation of a factory robot based on the following conditions" is input to the generative AI model. This prompt causes the AI ​​model to generate a specific training scenario, and the user can proceed with training according to that scenario.

[1395] As described above, this system is a powerful tool for efficiently and safely learning robot operation and maintenance techniques using a virtual environment.

[1396] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1397] Step 1:

[1398] The server retrieves the user's registration information, training history, and progress information from the database.

[1399] Input: User ID

[1400] Data processing: Performing database queries and extracting the required information.

[1401] Output: User registration information, training history, progress information

[1402] Step 2:

[1403] The server uses artificial intelligence to generate customized training scenarios based on the user's progress.

[1404] Input: User progress information

[1405] Data calculation: The generative AI model analyzes progress information and creates optimal training scenarios for the user.

[1406] Output: Customized training scenario

[1407] Specific operation: The following prompt is input to the generative AI model: "Generate a VR training scenario that simulates the arm operation of a factory robot based on the following conditions."

[1408] Step 3:

[1409] The server transmits the generated training scenario to the user's terminal.

[1410] Input: Training scenario

[1411] Data Transfer: Sending data over a network.

[1412] Output: The training scenario is displayed on the user's device.

[1413] Step 4:

[1414] The terminal renders and displays the virtual reality environment to the user.

[1415] Input: Training scenario data

[1416] Data processing: Pass the data to the virtual environment rendering engine to generate a 3D environment.

[1417] Output: Display of virtual reality environment

[1418] How it works: A training scenario is loaded into the VR headset and the user enters the virtual environment.

[1419] Step 5:

[1420] The user begins training in the virtual environment and inputs operational data into the terminal.

[1421] Input: User action

[1422] Data collection: Collect operation data using the motion tracking function of the VR headset.

[1423] Output: Collecting and sending operational data

[1424] Step 6:

[1425] The terminal transmits the user's operation data to the server in real time.

[1426] Input: Operation data

[1427] Data Transfer: Sending data over a network.

[1428] Output: Operation data is sent to the server

[1429] Step 7:

[1430] The server analyzes the user's operation data and generates feedback.

[1431] Input: Operation data

[1432] Data calculation: A generative AI model analyzes operational data and generates feedback based on the accuracy and timing of operations.

[1433] Output: Feedback data

[1434] Step 8:

[1435] The server transmits the generated feedback to the user's terminal.

[1436] Input: Feedback data

[1437] Data Transfer: Sending data over a network.

[1438] Output: Feedback is displayed on the user's device

[1439] Step 9:

[1440] The device displays feedback within the virtual environment.

[1441] Input: Feedback data

[1442] Data display: The feedback content is displayed in the VR environment.

[1443] Output: User can see the feedback

[1444] Specific operation: The feedback displayed is, "Arm operation accuracy is 85%, but there is room for improvement in part alignment."

[1445] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1446] This invention combines a system that uses virtual reality technology to provide specific skill or vocational training with an emotion engine that recognizes the user's emotions. The system is mainly composed of a user, a terminal, a server, and an emotion engine, and provides an efficient and safe virtual environment for the user to acquire specific skills, while also having the function of adjusting the training content and feedback according to the user's emotional state.

[1447] Server Roles

[1448] User Data Management

[1449] The server connects to a database and manages user registration information, training history, and progress information. Based on each user's progress, it identifies the next skills and training content they should learn. All of this information is stored in the database and is referenced every time a user accesses the system.

[1450] Training scenario generation

[1451] The server uses AI to generate customized training scenarios based on the user's progress. For example, in the medical field, it could generate a scenario for practicing injection techniques, creating specific procedures and a virtual environment.

[1452] Collaboration with emotion engine

[1453] The server works with the emotion engine to receive the user's emotional data. This emotional data is used to adjust the training scenario and generate feedback. For example, if the user is impatient, the difficulty of the scenario can be adjusted.

[1454] Real-time environment updates

[1455] The server receives user interaction data in real time and dynamically updates the virtual environment based on that data. Other objects in the environment also change dynamically in response to user interaction.

[1456] Analysis of operation data and emotion data and feedback generation

[1457] The system analyzes the user's operation data and emotional data, and generates personalized feedback based on the success rate of the operation and the type of error. The generated feedback takes into account the user's emotional state.

[1458] Send Feedback

[1459] The server sends the generated feedback data to the user's device, allowing the user to review the feedback based on their performance and emotions and reflect it in their next training session.

[1460] Device Role

[1461] Rendering a VR environment

[1462] The device displays a virtual environment to the user based on the training scenario data sent from the server. By using VR goggles, users can receive training in a highly immersive virtual space.

[1463] Operational Data Collection

[1464] The device tracks user actions and collects data in real time, including hand movements and location information.

[1465] Collecting Emotional Data

[1466] The device uses an emotion engine to collect user emotion data in real time, using techniques such as facial recognition and voice analysis.

[1467] Transmission of operational and emotional data

[1468] The collected operation data and emotion data are sent in real time to a server, which then dynamically updates the virtual environment based on this data.

[1469] Receiving feedback data

[1470] The device receives feedback data from the server, which includes operational improvements and takes into account the user's emotional state.

[1471] View Feedback

[1472] The device then displays the received feedback in the virtual environment, allowing the user to see the evaluation of their own actions and emotions in real time and reflect this in their next training session.

[1473] User Roles

[1474] Wearing VR goggles

[1475] The user puts on the VR goggles and prepares to begin training. The virtual environment is displayed through the goggles.

[1476] Selection of training scenarios

[1477] Users select a training scenario displayed on their device and enter the virtual environment, setting them up to practice a specific skill.

[1478] Operation in the virtual space

[1479] The user performs instructed operations in the virtual environment, such as administering an injection to a virtual patient.

[1480] Providing emotion data

[1481] Users cooperate by providing emotional data (e.g., facial expressions and voice) that the device collects during training.

[1482] Check real-time feedback

[1483] Users can view real-time feedback during training to understand areas for improvement in their operations and emotions.

[1484] Reflection on the next training

[1485] Based on the feedback, efforts will be made to improve operational and emotional management in the next training session, which will increase the efficiency of skill acquisition and improve the training experience.

[1486] Specific examples

[1487] For example, in the medical field, a scenario in which a new nurse learns injection techniques in a virtual environment would go something like this: The user (new nurse) puts on VR goggles and enters the virtual environment based on the injection training scenario sent from the server. The device displays a virtual patient and injection equipment, and the user performs the virtual injection. The server receives the user's operation data and emotional data (for example, detecting impatience through facial expressions and voice analysis) in real time and analyzes the injection angle, depth, timing, etc. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user is expected to receive feedback and use it to improve their next operation and emotional management.

[1488] This system is a powerful tool for efficiently and safely acquiring skills without using real-world resources in fields such as medicine, construction, and safety training. Furthermore, the introduction of an emotion engine improves the quality of training and makes the user's training experience more personalized.

[1489] The processing flow will be explained below.

[1490] Server Processing

[1491] Step 1:

[1492] User Data Collection

[1493] The server connects to the database and collects user registration information, training history, and progress information, which allows the user's profile and past training details to be understood.

[1494] Step 2:

[1495] Training Needs Analysis

[1496] The server analyzes the collected data and identifies the next skill or training content the user should learn. For example, if the user is currently learning injection techniques, it will select a training scenario that matches their progress.

[1497] Step 3:

[1498] AI-based scenario generation

[1499] The server uses artificial intelligence (AI) to generate customized training scenarios based on the user's training progress, including specific procedures and situation settings.

[1500] Step 4:

[1501] Sending scenario data

[1502] The server transmits the generated training scenario data to the user's device, allowing the user to begin practicing based on the training scenario.

[1503] Step 5:

[1504] Collecting Emotional Data

[1505] The server works in conjunction with the emotion engine to receive the user's emotion data (e.g., the user's facial expressions and voice analysis data) collected from the terminal.

[1506] Step 6:

[1507] Receiving real-time operation data

[1508] The server receives user operation data sent from the device in real time, including hand movements and position information performed by the user in the virtual space.

[1509] Step 7:

[1510] Dynamic updates for virtual environments

[1511] Based on the received operation and emotion data, the server dynamically updates the state of the virtual environment, e.g., when the user administers an injection, the virtual patient's reaction and the syringe's position are updated.

[1512] Step 8:

[1513] Analysis of operation data and emotion data and feedback generation

[1514] The server analyzes the user's operation data and emotional data, and generates individual feedback based on the success rate of the operation, the type of error, and the user's emotional state.

[1515] Step 9:

[1516] Send Feedback

[1517] The server sends the generated feedback data to the user's device, allowing the user to review the feedback based on their performance and emotional state and reflect it in their next training session.

[1518] Terminal handling

[1519] Step 1:

[1520] Receiving scenario data

[1521] The terminal receives the training scenario data sent from the server and prepares the virtual environment to be displayed based on the received data.

[1522] Step 2:

[1523] Rendering a Virtual Environment

[1524] Based on the training scenario, the device displays the virtual environment to the user through the VR goggles, allowing the user to begin training in a highly immersive virtual space.

[1525] Step 3:

[1526] Operational Data Collection

[1527] The device tracks user actions and collects data in real time, including hand movements and location information.

[1528] Step 4:

[1529] Collecting Emotional Data

[1530] The device uses an emotion engine to collect user emotion data in real time, using techniques such as facial recognition and voice analysis.

[1531] Step 5:

[1532] Transmission of operational and emotional data

[1533] The collected operation data and emotion data are sent in real time to a server, which then dynamically updates the virtual environment based on this data.

[1534] Step 6:

[1535] Receiving feedback data

[1536] The device receives feedback data from the server, which includes operational improvements and takes into account the user's emotional state.

[1537] Step 7:

[1538] View Feedback

[1539] The device then displays the received feedback in the virtual environment, allowing the user to see the evaluation of their own actions and emotions in real time and reflect this in their next training session.

[1540] User Action

[1541] Step 1:

[1542] Wearing VR goggles

[1543] The user puts on the VR goggles and prepares to begin training. The virtual environment is displayed through the goggles.

[1544] Step 2:

[1545] Selection of training scenarios

[1546] Users select a training scenario displayed on their device and enter the virtual environment, setting them up to practice a specific skill.

[1547] Step 3:

[1548] Operation in the virtual space

[1549] The user performs instructed operations in the virtual environment, such as administering an injection to a virtual patient.

[1550] Step 4:

[1551] Providing emotion data

[1552] Users cooperate by providing emotional data (e.g., facial expressions and voice) that the device collects during training.

[1553] Step 5:

[1554] Check real-time feedback

[1555] Users can view real-time feedback during training to understand areas for improvement in their operations and emotions.

[1556] Step 6:

[1557] Reflection on the next training

[1558] Based on the feedback, efforts will be made to improve operational and emotional management in the next training session, which will increase the efficiency of skill acquisition and improve the training experience.

[1559] Example 2

[1560] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1561] In conventional skill and vocational training systems using virtual reality technology, training scenarios and feedback were generated based solely on user operation data, making it difficult to provide personalized training content and feedback that took the user's emotional state into account. This limited the effectiveness of training for users, making it difficult to acquire skills efficiently and effectively.

[1562] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for linking with a database that manages user registration information, training history, and progress information; artificial intelligence means for generating a training scenario tailored to the user's training progress based on the database; means for transmitting the generated training scenario to the user's terminal; means for rendering a virtual environment and displaying it to the user; means for receiving user operation data and emotional data in real time and dynamically updating the virtual environment; means for analyzing the user's operation data and emotional data, generating and transmitting individual feedback; and means for transmitting the feedback to the user's terminal and displaying it in the virtual environment. This makes it possible to provide personalized training content that takes the user's emotional state into consideration, thereby achieving efficient and effective skill acquisition.

[1563] "User registration information" refers to personal information and authentication information provided by a user when accessing the system.

[1564] "Training history" refers to information showing records of training that a user has conducted in the past and the results of those training.

[1565] "Progress information" refers to data that indicates the progress and level of achievement of the training that the user has achieved up to now.

[1566] "Database" refers to a system for centrally managing and storing user registration information, training history, progress information, etc.

[1567] "Artificial intelligence means" refers to technology that uses machine learning and advanced data analysis techniques to generate training scenarios based on user progress information.

[1568] A "training scenario" is a program that defines the virtual environment and procedures for conducting specific skill or vocational training.

[1569] A "terminal" refers to a device that allows a user to access and operate a virtual environment. Examples include VR goggles and a personal computer.

[1570] "Rendering a virtual environment" refers to using computer graphics techniques to generate and visually display a virtual three-dimensional environment to a user.

[1571] "Operation data" refers to information about operations performed by a user in a virtual environment, such as hand movements and click history.

[1572] "Emotional data" refers to data that indicates the user's emotional state. Examples include emotional states determined by facial expression recognition or voice analysis.

[1573] "Dynamic updating" refers to changing the virtual environment in real time in response to the user's actions and emotional state.

[1574] "Analyzing" refers to the detailed analysis of collected data to extract meaningful information and patterns.

[1575] "Feedback" refers to a response, including suggestions for improvement and evaluation, generated based on the user's actions and emotional state.

[1576] "Displaying within the virtual environment" refers to visually presenting generated feedback and other information within the virtual three-dimensional space experienced by the user.

[1577] An "emotion engine" refers to the technology and system for collecting and analyzing user emotional data.

[1578] This invention combines a system that uses virtual reality technology to provide specific skill or vocational training with an emotion engine that recognizes the user's emotions. The system is mainly composed of a user, a terminal, a server, and an emotion engine, and provides an efficient and safe virtual environment for the user to acquire specific skills, while also having the function of adjusting the training content and feedback according to the user's emotional state.

[1579] Server Roles

[1580] The server uses multiple pieces of hardware and software to perform the following processes:

[1581] 1. User Data Management:

[1582] The server connects to a database (e.g., PostgreSQL) and manages user registration information, training history, and progress information. This allows it to identify the next skills and training content each user should learn based on their progress. All of this information is stored in the database and is referenced every time a user accesses the system.

[1583] 2. Training scenario generation:

[1584] The server uses AI (e.g., a TensorFlow model) to generate customized training scenarios based on the user's progress. For example, in the medical field, it could generate a scenario for practicing injection techniques, building specific procedures and a virtual environment.

[1585] 3. Collaboration with Emotion Engine:

[1586] The server works with an emotion engine (e.g., Affectiva) to receive real-time user emotion data, which is used to adjust training scenarios and generate feedback.

[1587] 4. Real-time environment updates:

[1588] The server receives user operation data in real time and dynamically updates the virtual environment based on that data, and other objects in the environment also change dynamically in response to the user's operations.

[1589] 5. Analysis of operation and emotion data and feedback generation:

[1590] The server analyzes the user's operation data and emotional data, and generates personalized feedback based on the success rate of the operation and the type of error. The generated feedback takes into account the user's emotional state.

[1591] 6. Submitting Feedback:

[1592] The server sends the generated feedback data to the user's device, allowing the user to review the feedback based on their performance and emotions and reflect it in their next training session.

[1593] Device Role

[1594] The terminal receives data from the server in the following manner.

[1595] 1. Rendering the VR environment:

[1596] The device displays a virtual environment to the user based on the training scenario data sent from the server. By using VR goggles (e.g., Oculus Rift), the user can receive training in a highly immersive virtual space.

[1597] 2. Operational Data Collection:

[1598] The device tracks user actions and collects data in real time, including hand movements and location information.

[1599] 3. Collecting Emotional Data:

[1600] The device uses an emotion engine to collect user emotion data in real time, using techniques such as facial recognition and voice analysis.

[1601] 4. Transmission of Operational and Emotional Data:

[1602] The collected operation data and emotion data are sent to a server in real time.

[1603] 5. Receiving Feedback Data:

[1604] The device receives feedback data from the server and displays it in the virtual environment, allowing the user to check the evaluation of their own actions and emotions in real time and reflect this in their next training session.

[1605] User Roles

[1606] The steps that the user should follow are as follows:

[1607] 1. Put on the VR goggles:

[1608] The user puts on the VR goggles and prepares to begin training.

[1609] 2. Selection of training scenario:

[1610] The user selects a training scenario displayed on the terminal and enters the virtual environment.

[1611] 3. Virtual space operations:

[1612] The user performs the indicated action in the virtual environment, for example, administering an injection to a virtual patient.

[1613] 4. Providing Emotion Data:

[1614] Users cooperate by providing emotional data (e.g., facial expressions and voice) that the device collects during training.

[1615] 5. Check real-time feedback:

[1616] Users can view real-time feedback during training to understand areas for improvement in their operations and emotions.

[1617] 6. Reflection on the next training:

[1618] Based on the feedback, efforts will be made to improve operations and emotional management in the next training session.

[1619] Specific examples

[1620] As a concrete example, we will explain a scenario in which a new nurse in the medical field learns injection techniques in a virtual environment. The user (new nurse) puts on VR goggles and enters the virtual environment based on an injection training scenario sent from the server. The terminal displays a virtual patient and injection equipment, and the user performs the virtual injection. The server receives the user's operation data and emotional data (e.g., detecting impatience through facial expressions and voice analysis) in real time and analyzes the angle, depth, timing, etc. of the injection. Feedback is generated based on the analysis results and displayed to the user in real time via the terminal. The user is expected to receive feedback and use it to improve their next operation and emotional management.

[1621] Examples of prompt statements

[1622] Below are some examples of prompts to input to a generative AI model (e.g., ChatGPT).

[1623] Example 1: "Create a scenario in which a new nurse is learning injection techniques in a virtual environment. Describe how the system adjusts feedback based on the user's emotional state."

[1624] Example 2: "Create a safety training scenario for a construction site. Explain in detail the operation of a system that collects and analyzes emotion data in real time, clearly indicating the roles of the server, device, and user."

[1625] The foregoing is an embodiment of the present invention, which allows for efficient and personalized training and improves user skill acquisition.

[1626] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1627] Step 1: User Registration and Login

[1628] Server: The server receives the new user's registration information (e.g., name, email address, password) as input and processes the data to store it in the database. For existing users, the server compares the entered login information with the information in the database and authenticates the login. The output is the user's dashboard if successful, or an error message if unsuccessful.

[1629] Terminal: The terminal receives registration and login information from the user as input and sends it to the server, which then passes the information as input to the server, which then provides an interface to display the processing results as output.

[1630] User: The user enters personal information such as name, email address, and password into the device to register or log in. The information entered becomes important initial data for the entire system.

[1631] Step 2: Select a training scenario

[1632] Server: The server takes user progress information from the database as input, performs data calculations using an AI model (e.g., TensorFlow) to generate customized training scenarios, and provides the output as recommended training scenarios to the user.

[1633] Terminal: The terminal receives as input the list of training scenarios sent from the server and outputs it as an interface to be displayed to the user.

[1634] User: The user selects the desired scenario from the list of training scenarios displayed on the terminal. The selected scenario becomes the input data used in the next step.

[1635] Step 3: Rendering the VR environment

[1636] Terminal: The terminal receives training scenario data (e.g., virtual patient, injection equipment placement data) sent from the server as input, and uses VR goggles to render a virtual environment based on this data and display it to the user. Specifically, it performs real-time 3D rendering using a GPU. The output is an immersive virtual space that the user can view.

[1637] User: The user wears VR goggles and practices in the displayed virtual environment. The information displayed in the VR environment will be the basis for the user's next step.

[1638] Step 4: Conduct training and collect data

[1639] Device: The device collects user operation data (e.g., hand movements, location information) and emotion data (e.g., facial expressions, voice) as input in real time and sends them to the server. This is done using motion sensors, microphones, and cameras. The output is raw data sent to the server in real time.

[1640] Server: The server receives the operation data and emotion data sent from the device as input, monitors and records the training progress in real time, and uses this data to determine the user's progress and determine the next step.

[1641] User: The user performs the instructed operations in the virtual environment and provides operational and emotional data by performing natural actions.

[1642] Step 5: Data analysis and feedback generation

[1643] Server: The server analyzes the received operation data and emotional data as input. Specifically, it performs data calculations to evaluate the success rate of operations, the type of error, and the user's emotional state (e.g., impatience or concentration). It uses an AI model to analyze behavioral and emotional patterns and generates feedback for the user. The output is individually customized feedback.

[1644] Terminal: Waits for analysis results and feedback data received from the server and prepares them for display to the user.

[1645] Step 6: View real-time feedback

[1646] Terminal: The terminal receives feedback data from the server as input and displays it in real time within the virtual environment. This can include text messages, audio prompts, and visual cues. The output is specific feedback information for the user.

[1647] User: The user reviews the displayed feedback and uses it to improve their own actions and emotional management.

[1648] Step 7: Completing the training and preparing for the next training

[1649] Server: After the training is completed, the server updates the user's training data and prepares new scenarios for the next training. It also provides an overall evaluation of the training if the user so desires. The output is a future training plan and an overall evaluation report.

[1650] Terminal: The terminal displays a message to the user that the training is complete and a list of possible training scenarios for the next training session.

[1651] Users: After completing the training, users can review the points covered based on the feedback and prepare for the next training session, thereby continuously improving their skills.

[1652] (Application example 2)

[1653] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1654] Conventional vocational training systems using virtual reality technology provide feedback based solely on the user's operational data, making it difficult to provide personalized training that takes into account the user's emotional state, such as stress or pressure. Furthermore, if the user falls into an inappropriate emotional state, they are unable to respond appropriately, resulting in a decrease in the efficiency and effectiveness of the training. Therefore, there is a need for a system that can analyze the user's emotional state in real time and appropriately adjust the training scenario and feedback.

[1655] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for linking with a database that manages user registration information, training history, and progress information; artificial intelligence means for generating a training scenario tailored to the user's training progress based on the database; means for transmitting the generated training scenario to the user's terminal; means for receiving the user's operation data and emotional data and adjusting the difficulty level of the virtual training environment based thereon; means for analyzing the collected emotional data using an emotional analysis model and generating feedback according to the user's emotional state; and means for transmitting the generated feedback to the user's terminal and displaying it in the virtual environment. This makes it possible to provide individualized and appropriate feedback that takes the user's emotional state into consideration, allowing the user to train efficiently and comfortably.

[1656] "Virtual reality technology" is a technology that uses computers to generate immersive virtual environments that are different from the real world and allow users to experience them.

[1657] "Specific skills or vocational training" refers to training that enables users to acquire the skills or knowledge required in a specific professional field.

[1658] A "system" is an integrated functional unit that combines multiple means to work toward a set of objectives.

[1659] A "database" is a collection of data organized in a particular way to allow for efficient data management and access.

[1660] "Artificial intelligence means" refers to algorithms or programs for learning, reasoning, and making decisions based on specific data.

[1661] A "training scenario" refers to a virtual environment configuration that includes a series of training procedures or situations designed around specific training objectives.

[1662] A "user terminal" is an electronic device that a user uses to receive training in a virtual reality environment.

[1663] "Operation data" refers to data related to various operations and actions performed by the user during training.

[1664] "Emotion data" is data that represents the user's emotional state and is primarily collected through facial expression recognition and voice analysis.

[1665] "Emotion analysis model" refers to an algorithm or program designed to analyze a user's emotional state.

[1666] "Feedback" refers to information or advice provided based on the user's behavior and emotional state during training.

[1667] A "virtual environment" is an artificial space or situation created using virtual reality technology.

[1668] "Means for adjusting difficulty" refers to a method or device for changing the difficulty of a training scenario depending on the user's performance or emotional state.

[1669] "Dynamic updating means" refers to a method or device for updating and changing the virtual environment as needed based on user operation data that changes in real time.

[1670] To realize the present invention, the following system configuration and program are used.

[1671] The server has a means for linking with a database that manages user registration information, training history, and progress information, and uses artificial intelligence means to generate training scenarios tailored to the user's training progress. The generated training scenarios are sent to the user's terminal. Furthermore, the server has a means for receiving the user's operation data and emotion data and appropriately adjusting the difficulty level of the virtual training environment based on the data.

[1672] The device includes VR goggles for rendering the virtual environment and a camera for facial recognition. The VR goggles are used to provide the user with a highly immersive virtual environment, and data is collected and transmitted by tracking the user's actions within the environment. The device also transmits the collected emotional data to a server in real time using an emotion engine. The device displays the feedback data received from the server in real time within the virtual environment, providing the user with appropriate information.

[1673] The user puts on the VR goggles, selects a training scenario displayed on the device, enters the virtual environment, and performs a specific operation. Emotional data felt during the operation (changes in facial expressions and voice) is collected by the device and sent to the server. The user checks the feedback sent from the server and uses it to improve the next operation.

[1674] Below, a training system for factory robots will be described as a specific example of use.

[1675] A scenario in which a new operator learns to operate a factory robot would go something like this: The user (new operator) puts on VR goggles and enters a virtual environment based on a robot operation training scenario sent from the server. The device displays a robot operation scene from within the factory, and the user operates the virtual robot. The server receives the user's operation data and emotional data (facial expressions and voice analysis detects impatience) in real time and analyzes the accuracy and timing of the operation. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user receives the feedback and uses it to improve their next operation.

[1676] This system is a powerful tool for training new operators of factory robots to acquire skills efficiently and safely without using real-world resources, and the introduction of an emotion engine improves the quality of training and makes the user's training experience more personalized.

[1677] As a concrete example, the following is an example of a prompt sentence to be input to the generative AI model.

[1678] "Create a program that adjusts the difficulty of the training scenario in real time based on the emotions experienced by new nurses while administering injections in the virtual environment. The program should include facial expression recognition, voice analysis, and dynamic updates to the virtual environment."

[1679] In this way, by analyzing the user's emotional state in real time and appropriately adjusting the training scenario and feedback, a system can be realized that allows the user to receive training efficiently and comfortably.

[1680] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1681] Step 1:

[1682] The server retrieves the user's registration information, training history, and progress information from the database. The input data is the user's ID, and the output data is the user's training history and progress information. The server evaluates the user's current skill level based on this information.

[1683] Step 2:

[1684] The server uses artificial intelligence to generate a training scenario tailored to the user's training progress. The input data is the user's training history and progress information acquired in step 1, and the output data is a customized training scenario. The server sends this training scenario to the user's device.

[1685] Step 3:

[1686] The device renders a virtual reality environment based on the training scenario received from the server. The input data is the training scenario sent from the server, and the output data is the virtual environment the user sees through the VR goggles. The device then prompts the user to start training.

[1687] Step 4:

[1688] The user wears VR goggles and performs specific operations in a virtual reality environment. The input is the user's physical movements, and the output is the operation data collected by the device. The user performs operations in the virtual environment according to the instructions.

[1689] Step 5:

[1690] The device collects user operation data and emotional data in real time. Input data includes the user's physical movements, facial expressions, and voice, while output data includes operation data and emotional data. The device analyzes this data and sends it to the server.

[1691] Step 6:

[1692] The server receives the operation data and emotion data sent from the device and adjusts the difficulty of the virtual training environment. The input data are the user's operation data and emotion data, and the output data is the adjusted training scenario. The server then sends instructions to the device to dynamically update the virtual environment based on the input data.

[1693] Step 7:

[1694] The terminal updates the virtual environment based on the adjusted training scenario received from the server. The input data is the adjusted training scenario sent from the server, and the output data is the updated virtual environment. The terminal displays the new training environment to the user.

[1695] Step 8:

[1696] The server analyzes the collected emotional data using an emotion analysis model and generates feedback according to the user's emotional state. The input data is the emotional data, and the output data is the feedback content. The server transmits this feedback data to the terminal.

[1697] Step 9:

[1698] The terminal displays the feedback data received from the server to the user in the virtual environment. The input data is the feedback data sent from the server, and the output data is the feedback information displayed to the user. The user can use this information to improve their next operation.

[1699] The above processing steps enable the user to receive personalized and appropriate feedback that takes into account the user's emotional state, allowing the user to receive training efficiently and comfortably, thereby improving the user's training experience and allowing the user to acquire skills efficiently.

[1700] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1701] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1702] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1703] [Fourth embodiment]

[1704] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1705] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1706] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1707] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1708] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1709] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1710] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1711] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1712] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1713] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1714] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1715] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1716] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1717] This invention relates to a system that provides specific skills and vocational training using virtual reality technology. The system mainly consists of a user, a terminal, and a server, and provides an efficient and safe virtual environment for users to acquire specific skills.

[1718] Server Roles

[1719] User Data Management

[1720] The server connects to a database and manages user registration information, training history, and progress information. Based on each user's progress, it can identify the next skills and training content they should learn. All of this information is stored in the database and is referenced every time a user accesses the system.

[1721] Training scenario generation

[1722] The server uses AI to generate customized training scenarios based on the user's progress. For example, in the medical field, it could generate a scenario for practicing injection techniques, creating specific procedures and a virtual environment.

[1723] Real-time environment updates

[1724] The server receives user operation data in real time and dynamically updates the virtual environment based on that data. Other objects in the environment also change dynamically according to the user's operations in the virtual space.

[1725] Providing Feedback

[1726] It analyzes user operation data and generates personalized feedback based on success rate and error type. This feedback is sent to the device in real time so that the user can reflect it in their next operation.

[1727] Device Role

[1728] Rendering a VR environment

[1729] The device displays a virtual environment to the user based on the training scenario data received from the server. By using VR goggles, the user can receive training in a highly immersive virtual space.

[1730] Operational Data Collection

[1731] The device tracks the user's actions and sends the data to a server in real time. For example, when a user injects a drug in a virtual space, the device collects information about the user's hand movements and location.

[1732] View Feedback

[1733] Feedback data received from the server is displayed in the VR environment, allowing users to check this feedback in real time and improve their operations.

[1734] User Roles

[1735] Putting it on and starting training

[1736] The user puts on VR goggles and enters a virtual environment based on the training scenario sent from the server. Before starting training, the user can check their registration information and training history.

[1737] Operation in the virtual space

[1738] The user follows instructions in the virtual environment to perform a specified task, such as administering an injection to a virtual patient in medical training. All movements during the task are tracked and transmitted to a server via the device.

[1739] Receiving feedback and improving

[1740] Once the training is complete, feedback from the server is displayed on the device. The user is expected to receive the feedback and improve their next operation based on it. The more the user repeats the training, the more efficiently they can learn the skill.

[1741] Specific examples

[1742] For example, in the medical field, a scenario in which a new nurse learns injection techniques in a virtual environment would go something like this: The user (new nurse) puts on VR goggles and enters the virtual environment based on the injection training scenario sent from the server. The device displays a virtual patient and injection equipment, and the user performs the virtual injection. The server receives the user's operation data in real time and analyzes the injection angle, depth, timing, etc. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user can receive the feedback and use it the next time they conduct training.

[1743] This system is a powerful tool for efficiently and safely acquiring skills in fields such as medicine, construction, and safety training without using real resources.

[1744] The processing flow will be explained below.

[1745] Server Processing

[1746] Step 1:

[1747] User Data Collection

[1748] The server connects to the database and collects user registration information, training history, and progress information, which allows the user's profile and past training details to be understood.

[1749] Step 2:

[1750] Training Needs Analysis

[1751] The server analyzes the collected data and identifies the next skill or training content the user should learn. For example, if the user is currently learning injection techniques, it will select a training scenario that matches their progress.

[1752] Step 3:

[1753] AI-based scenario generation

[1754] The server uses artificial intelligence (AI) to generate customized training scenarios based on the user's training progress, including specific procedures and situation settings.

[1755] Step 4:

[1756] Sending scenario data

[1757] The server transmits the generated training scenario data to the user's device, allowing the user to begin practicing based on the training scenario.

[1758] Step 5:

[1759] Receiving real-time operation data

[1760] The server receives user operation data sent from the device in real time, including hand movements and position information performed by the user in the virtual space.

[1761] Step 6:

[1762] Dynamic updates for virtual environments

[1763] Based on the received operation data, the server dynamically updates the state of the virtual environment, for example, updating the virtual patient's reaction and the syringe's position when the user administers an injection.

[1764] Step 7:

[1765] Analyzing operation data and generating feedback

[1766] The server analyzes the user's operation data and generates individual feedback based on the success rate of the operation and the type of error, which specifically indicates areas for improvement for the user.

[1767] Step 8:

[1768] Send Feedback

[1769] The server sends the generated feedback data to the user's device, allowing the user to check their own performance and reflect it in their next training session.

[1770] Terminal handling

[1771] Step 1:

[1772] Receiving scenario data

[1773] The terminal receives the training scenario data sent from the server and prepares the virtual environment to be displayed based on the received data.

[1774] Step 2:

[1775] Rendering a Virtual Environment

[1776] Based on the training scenario, the device displays the virtual environment to the user through the VR goggles, allowing the user to begin training in a highly immersive virtual space.

[1777] Step 3:

[1778] Operational Data Collection

[1779] The device tracks user actions and collects data in real time, including hand movements and location information.

[1780] Step 4:

[1781] Sending operation data

[1782] The collected operation data is sent in real time to the server, which then dynamically updates the virtual environment based on this data.

[1783] Step 5:

[1784] Receiving feedback data

[1785] The terminal receives feedback data from the server, which includes points for improvement and evaluation of the operation.

[1786] Step 6:

[1787] View Feedback

[1788] The device then displays the received feedback in the virtual environment, allowing the user to see the evaluation of their own actions in real time and reflect it in their next training session.

[1789] User Action

[1790] Step 1:

[1791] Wearing VR goggles

[1792] The user puts on the VR goggles and prepares to begin training. The virtual environment is displayed through the goggles.

[1793] Step 2:

[1794] Selection of training scenarios

[1795] Users select a training scenario displayed on their device and enter the virtual environment, setting them up to practice a specific skill.

[1796] Step 3:

[1797] Operation in the virtual space

[1798] The user performs instructed operations in the virtual environment, such as administering an injection to a virtual patient.

[1799] Step 4:

[1800] Check real-time feedback

[1801] Users can view real-time feedback during training to understand areas for improvement in their operations.

[1802] Step 5:

[1803] Reflection on the next training

[1804] Based on the feedback, efforts are made to improve operations in the next training session, which increases the efficiency of skill acquisition.

[1805] Example 1

[1806] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1807] Conventional training systems using virtual reality technology have the problem of delayed feedback to user operations, reducing the effectiveness of real-time training. Furthermore, it is difficult to generate customized scenarios based on the user's training progress, resulting in insufficient individualized support. As a result, it is difficult for users to efficiently acquire skills.

[1808] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1809] In this invention, the server includes means for receiving user operation data in real time and dynamically updating the virtual environment, means for analyzing the user operation data and generating and transmitting individual feedback, means for the terminal to render a virtual reality environment and provide the user with a highly immersive experience, means for the terminal to track the user operation data in real time and transmit it to the server, and means for the server to analyze the user operation data and update the virtual environment in real time, thereby making it possible to provide immediate feedback to the user's operation and dynamically update the virtual environment.

[1810] "User" refers to a person who uses a system that uses virtual reality technology to receive specific skills or vocational training.

[1811] A "server" refers to a computer device that receives operational data from a user, updates the virtual environment, and generates feedback.

[1812] "Terminal" means a device used by a user to interact with a virtual reality environment, including VR goggles and a PC.

[1813] "Database" refers to a system that stores and manages data such as user registration information, training history, and progress information.

[1814] "Artificial intelligence means" refers to an algorithm or program for generating customized training scenarios based on a user's training progress.

[1815] "Training scenario" refers to a simulation program in virtual reality that allows users to learn a particular skill or operation.

[1816] "Operation data" refers to the actions performed by the user within the virtual reality environment and the data generated during those actions.

[1817] "Virtual environment" refers to a three-dimensional space that can be manipulated by a user and is generated using virtual reality technology.

[1818] "Real-time update" refers to the process of instantly changing the state and objects of the virtual environment in response to user operations.

[1819] "Feedback" refers to evaluations and advice obtained by analyzing user operation data, and refers to information provided to the user.

[1820] "Tracking" refers to the process of tracking user actions and collecting location and movement data.

[1821] "High immersion" refers to providing an experience that allows users to be deeply immersed in a virtual reality environment and feels so real.

[1822] This invention relates to a system that uses virtual reality technology to provide specific skills or vocational training. The system mainly consists of a user, a terminal, and a server. The role of each component and the corresponding process are described below.

[1823] Server Roles

[1824] User Data Management

[1825] The server connects to a database (e.g., MySQL or PostgreSQL) and manages user registration information, training history, and progress information. When a user logs into the system, the server retrieves the user's progress information from the database and identifies the next skills and training content to be learned. This information is updated in real time while the system is in use.

[1826] Training scenario generation

[1827] The server uses a generative AI model (e.g., GPT-4) to generate individually optimized training scenarios based on the user's progress. For example, in the medical field, a scenario could be "a new nurse administering an injection to a virtual patient." The training scenario is generated on the server side and sent to the user's device.

[1828] Providing Feedback

[1829] The server receives and analyzes user operation data in real time. Based on the analysis results, it generates specific feedback according to the success rate and type of error. The generated feedback is sent to the user's device and displayed on the device.

[1830] Device Role

[1831] Rendering a VR environment

[1832] The device renders a virtual environment based on the training scenario sent from the server. Specifically, it uses VR goggles (e.g., Oculus Rift or HTC Vive) to provide the user with a virtual space, allowing the user to train with a high level of immersion.

[1833] Operational Data Tracking

[1834] The device tracks the user's actions in real time and sends the data to a server. For example, when a user performs an injection in a virtual space, the device collects information about the user's hand movements and position.

[1835] View Feedback

[1836] The feedback data is sent from the server to the device, which then displays it in the VR environment. The user can check this feedback in real time and improve their next operation.

[1837] User Roles

[1838] Log in and start training

[1839] The user logs in to the system using a terminal (for example, by entering an email address and password) and enters the virtual environment based on the training scenario sent from the server. Before starting training, the user confirms their registration information and training history.

[1840] Operation in a virtual environment

[1841] The user follows instructions in the virtual environment to perform specific operations, such as administering an injection to a virtual patient in medical training. All operation data is sent from the device to a server for analysis.

[1842] Receiving feedback and improving

[1843] After the training is complete, feedback is displayed on the device, and the user is asked to take this feedback and improve their next operation based on it.

[1844] Specific examples

[1845] For example, if a new nurse were to learn injection techniques in a virtual environment, the process could go something like this: First, the user puts on VR goggles and enters the virtual environment based on an "injection training scenario" sent from the server. The device displays a virtual patient and injection equipment, and supports the user in the process of administering the virtual injection. The server receives the user's operation data in real time and analyzes the injection angle, depth, timing, etc. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user can then use that feedback to improve their operation in the next training session.

[1846] This system enables people in many fields, including medicine, construction, and safety training, to acquire skills efficiently and safely without using real resources.

[1847] Prompt Sentence Examples

[1848] Below are some example prompts to input to a generative AI model (e.g., GPT-4):

[1849] "This system allows new nurses to practice injection techniques in a virtual reality environment. Imagine a nurse putting on VR goggles and initiating an injection scenario in the virtual space. Please explain in detail the steps from the first step to providing feedback. Also mention real-time environment updates and feedback generation."

[1850] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1851] Step 1:

[1852] User Login

[1853] A user accesses the system from a terminal and enters authentication information (e.g., username and password) on the login screen.

[1854] Input: Username, Password

[1855] Specific operation: The user enters "example@example.com" and "password123" and clicks the login button.

[1856] Server processing: The server checks the user's authentication information against the database and issues a session ID if authentication is successful.

[1857] Output: Session ID

[1858] Step 2:

[1859] Acquiring and Managing User Data

[1860] The server retrieves the user's training history and progress information from the database based on the session ID.

[1861] Input: Session ID

[1862] Specific operation: The server queries the database to obtain the user's progress information.

[1863] Server processing: The server obtains the progress information described above and identifies the next skills and training to be learned.

[1864] Output: User progress information, next training content

[1865] Step 3:

[1866] Generating training scenarios

[1867] The server uses a generative AI model (e.g., GPT-4) to generate training scenarios based on the user's progress.

[1868] Input: User progress information, next training content

[1869] Specific operation: The server inputs a prompt such as "The user is a new nurse and is learning basic injection techniques" into the generated AI model and generates a scenario.

[1870] Server processing: The generative AI model generates specific training scenarios based on the input information.

[1871] Output: Generated training scenarios

[1872] Step 4:

[1873] Rendering a VR environment

[1874] The terminal renders the virtual environment based on the training scenario data sent from the server.

[1875] Input: Training scenario data

[1876] Specific operation: The device uses VR goggles to display virtual patients, training equipment, etc., and prepares the user to access the virtual space.

[1877] Terminal processing: The terminal analyzes the scenario data and constructs the VR environment.

[1878] Output: Virtual environment

[1879] Step 5:

[1880] Tracking user actions

[1881] The device tracks the user's actions in real time and sends the data to a server.

[1882] Input: User operation data

[1883] Specific operation: The user puts on the VR goggles and performs the action of giving an injection in the virtual space. The device uses the controller sensor and camera to collect hand movement and position information.

[1884] Terminal processing: Collected operation data is sent to the server in real time.

[1885] Output: Operation data sent

[1886] Step 6:

[1887] Real-time environment updates

[1888] The server receives the user's operation data and dynamically updates the virtual environment based on it.

[1889] Input: User operation data

[1890] Specific operation: The server analyzes the operation data and updates other objects in the virtual environment (e.g., the virtual patient's reaction).

[1891] Server processing: Updates information in the virtual environment based on the operation data and sends it to the terminal.

[1892] Output: Updated virtual environment data

[1893] Step 7:

[1894] Generating and Providing Feedback

[1895] The server analyzes the user's operation data and generates and sends individual feedback.

[1896] Input: User operation data

[1897] Specific behavior: The server analyzes the operation data in real time and generates specific feedback such as "The injection angle is too shallow."

[1898] Server processing: Generate feedback based on the analysis results and send it to the device.

[1899] Output: Generated feedback

[1900] Step 8:

[1901] View Feedback

[1902] The device displays the feedback received from the server within the VR environment.

[1903] Input: Feedback data

[1904] Specific behavior: The user sees the feedback in the virtual environment and prepares to incorporate it into their next operation.

[1905] Terminal processing: Feedback is displayed on the VR screen and presented to the user.

[1906] Output: Feedback displayed to the user

[1907] Through the above steps, users can acquire skills efficiently and effectively using virtual reality technology.

[1908] (Application example 1)

[1909] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1910] In today's advanced industrial environments, factory operators and engineers are required to have extremely high levels of robot operation and maintenance skills. However, learning these skills in a real-world environment involves high costs and risks, so there is a need for efficient and safe methods of acquiring these skills. In addition, there is a lack of systems that can provide real-time feedback and analyze operation data. Therefore, it is necessary to provide a system that can effectively acquire such skills using a virtual environment.

[1911] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1912] In this invention, the server includes: means for linking with a database that manages user registration information, training history, and progress information; artificial intelligence means for generating training scenarios tailored to the user's training progress; means for transmitting the generated training scenarios to the user's terminal; means for receiving user operation data in real time and dynamically updating the virtual environment; means for analyzing the user's operation data and generating and transmitting individual feedback; means for generating training scenarios for learning robot operation and maintenance techniques in the virtual factory environment; means for simulating robot arm operation and part installation based on the user's operation data and evaluating the accuracy and timing of the operation in real time; and means for displaying the generated feedback on the user's terminal in real time. This enables factory operators and engineers to efficiently and safely acquire robot operation and maintenance techniques through training in the virtual environment.

[1913] "Virtual reality technology" is a technology that allows users to experience a three-dimensional environment generated by a computer.

[1914] "Specific skills and vocational training" refers to education and training that allows users to acquire specialized knowledge and skills.

[1915] "System" refers to a set of devices and mechanisms consisting of multiple devices and software for providing skills and vocational training using virtual reality technology.

[1916] "User" refers to an individual or group who uses and receives training on this system.

[1917] "Registration information" refers to the user's personal information and data necessary for using the system.

[1918] "Training history" refers to a record of the training a user has done up to now.

[1919] "Progress information" is data that indicates the results that the user has achieved through training and their current level of learning.

[1920] The "database" is a system that manages and stores user registration information, training history, progress information, etc.

[1921] "Artificial intelligence" is a technology that gives computers the ability to learn, judge, and predict in the same way as humans.

[1922] A "training scenario" is a series of virtual reality training programs that a user executes to acquire a particular skill or knowledge.

[1923] A "terminal" is a device such as a computer or VR goggles that allows a user to access the system and experience the virtual reality environment.

[1924] "Operation data" refers to a record of the actions and operations performed by a user within a virtual reality environment.

[1925] A "virtual environment" is a virtual three-dimensional space generated by a computer.

[1926] "Dynamic updating" means changing the environment in real time in response to user operations.

[1927] "Feedback" refers to evaluations and comments on the user's operations and progress.

[1928] A "virtual factory environment" is a three-dimensional virtual reality space that mimics the inside of a factory.

[1929] "Robot operation" refers to the control and operation of robots used in factories.

[1930] "Maintenance technology" refers to technology related to the maintenance, inspection, and repair of factories and machinery equipment.

[1931] "Arm operation" refers to the operation of moving the arm of the robot to perform a specific task.

[1932] "Component installation" refers to the process of accurately placing and fixing components in specific positions.

[1933] A "prompt sentence" is an input sentence that causes a generative AI model to operate.

[1934] This invention provides a training system for factory robot operation and maintenance techniques using virtual reality technology. The system consists of a user, a terminal, and a server.

[1935] Server Roles

[1936] The server has the following functions:

[1937] 1. User data management: The server connects to a database and manages user registration information, training history, and progress information. A common RDBMS (e.g., MySQL) is used as the database.

[1938] 2. Training scenario generation: The server uses artificial intelligence technology to generate customized training scenarios based on the user's progress. It uses a generative AI model to generate specific training scenarios based on prompts. Examples include training scenarios for operating a factory robot's arm or installing parts.

[1939] 3. Real-time environment update: The server receives user operation data in real time and dynamically updates the virtual environment based on it, thereby providing real-time feedback according to the user's operations.

[1940] 4. Feedback provision: Analyzes user operation data and generates personalized feedback based on the accuracy and timing of operations. The generated feedback is sent to the user's device in real time.

[1941] Device Role

[1942] The terminal has the following features:

[1943] 1. Rendering the VR environment: The device displays the virtual environment to the user based on the training scenario data received from the server. The device is a VR headset (e.g., Oculus Quest).

[1944] 2. Collecting operation data: The device tracks the user's operations and transmits the data to the server in real time, using the motion tracking function of the VR headset.

[1945] 3. Feedback display: Feedback data received from the server is displayed in the VR environment, allowing users to check and improve in real time.

[1946] User Roles

[1947] The user uses the system in the following steps:

[1948] 1. Putting on the VR goggles and starting training: The user puts on the VR goggles and enters the virtual factory environment based on the training scenario sent from the server. Before starting the training, the user can check their registration information and training history.

[1949] 2. Operation in virtual space: The user follows instructions in the virtual environment and performs specific operations, such as operating the robot's arm to attach parts. All movements during the operation are tracked and transmitted to the server via the terminal.

[1950] 3. Receiving feedback and improving: After the training is completed, feedback from the server is displayed on the device. The user receives the feedback and is expected to improve their next operation based on it. The more the user repeats the training, the more efficiently they can learn the skill.

[1951] Specific examples

[1952] A specific example is the generation of a VR training scenario that simulates the arm operation of a factory robot based on the following conditions. For example, the prompt sentence "Please generate a VR training scenario that simulates the arm operation of a factory robot based on the following conditions" is input to the generative AI model. This prompt causes the AI ​​model to generate a specific training scenario, and the user can proceed with training according to that scenario.

[1953] As described above, this system is a powerful tool for efficiently and safely learning robot operation and maintenance techniques using a virtual environment.

[1954] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1955] Step 1:

[1956] The server retrieves the user's registration information, training history, and progress information from the database.

[1957] Input: User ID

[1958] Data processing: Performing database queries and extracting the required information.

[1959] Output: User registration information, training history, progress information

[1960] Step 2:

[1961] The server uses artificial intelligence to generate customized training scenarios based on the user's progress.

[1962] Input: User progress information

[1963] Data calculation: The generative AI model analyzes progress information and creates optimal training scenarios for the user.

[1964] Output: Customized training scenario

[1965] Specific operation: The following prompt is input to the generative AI model: "Generate a VR training scenario that simulates the arm operation of a factory robot based on the following conditions."

[1966] Step 3:

[1967] The server transmits the generated training scenario to the user's terminal.

[1968] Input: Training scenario

[1969] Data Transfer: Sending data over a network.

[1970] Output: The training scenario is displayed on the user's device.

[1971] Step 4:

[1972] The terminal renders and displays the virtual reality environment to the user.

[1973] Input: Training scenario data

[1974] Data processing: Pass the data to the virtual environment rendering engine to generate a 3D environment.

[1975] Output: Display of virtual reality environment

[1976] How it works: A training scenario is loaded into the VR headset and the user enters the virtual environment.

[1977] Step 5:

[1978] The user begins training in the virtual environment and inputs operational data into the terminal.

[1979] Input: User action

[1980] Data collection: Collect operation data using the motion tracking function of the VR headset.

[1981] Output: Collecting and sending operational data

[1982] Step 6:

[1983] The terminal transmits the user's operation data to the server in real time.

[1984] Input: Operation data

[1985] Data Transfer: Sending data over a network.

[1986] Output: Operation data is sent to the server

[1987] Step 7:

[1988] The server analyzes the user's operation data and generates feedback.

[1989] Input: Operation data

[1990] Data calculation: A generative AI model analyzes operational data and generates feedback based on the accuracy and timing of operations.

[1991] Output: Feedback data

[1992] Step 8:

[1993] The server transmits the generated feedback to the user's terminal.

[1994] Input: Feedback data

[1995] Data Transfer: Sending data over a network.

[1996] Output: Feedback is displayed on the user's device

[1997] Step 9:

[1998] The device displays feedback within the virtual environment.

[1999] Input: Feedback data

[2000] Data display: The feedback content is displayed in the VR environment.

[2001] Output: User can see the feedback

[2002] Specific operation: The feedback displayed is, "Arm operation accuracy is 85%, but there is room for improvement in part alignment."

[2003] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2004] This invention combines a system that uses virtual reality technology to provide specific skill or vocational training with an emotion engine that recognizes the user's emotions. The system is mainly composed of a user, a terminal, a server, and an emotion engine, and provides an efficient and safe virtual environment for the user to acquire specific skills, while also having the function of adjusting the training content and feedback according to the user's emotional state.

[2005] Server Roles

[2006] User Data Management

[2007] The server connects to a database and manages user registration information, training history, and progress information. Based on each user's progress, it identifies the next skills and training content they should learn. All of this information is stored in the database and is referenced every time a user accesses the system.

[2008] Training scenario generation

[2009] The server uses AI to generate customized training scenarios based on the user's progress. For example, in the medical field, it could generate a scenario for practicing injection techniques, creating specific procedures and a virtual environment.

[2010] Collaboration with emotion engine

[2011] The server works with the emotion engine to receive the user's emotional data. This emotional data is used to adjust the training scenario and generate feedback. For example, if the user is impatient, the difficulty of the scenario can be adjusted.

[2012] Real-time environment updates

[2013] The server receives user interaction data in real time and dynamically updates the virtual environment based on that data. Other objects in the environment also change dynamically in response to user interaction.

[2014] Analysis of operation data and emotion data and feedback generation

[2015] The system analyzes the user's operation data and emotional data, and generates personalized feedback based on the success rate of the operation and the type of error. The generated feedback takes into account the user's emotional state.

[2016] Send Feedback

[2017] The server sends the generated feedback data to the user's device, allowing the user to review the feedback based on their performance and emotions and reflect it in their next training session.

[2018] Device Role

[2019] Rendering a VR environment

[2020] The device displays a virtual environment to the user based on the training scenario data sent from the server. By using VR goggles, users can receive training in a highly immersive virtual space.

[2021] Operational Data Collection

[2022] The device tracks user actions and collects data in real time, including hand movements and location information.

[2023] Collecting Emotional Data

[2024] The device uses an emotion engine to collect user emotion data in real time, using techniques such as facial recognition and voice analysis.

[2025] Transmission of operational and emotional data

[2026] The collected operation data and emotion data are sent in real time to a server, which then dynamically updates the virtual environment based on this data.

[2027] Receiving feedback data

[2028] The device receives feedback data from the server, which includes operational improvements and takes into account the user's emotional state.

[2029] View Feedback

[2030] The device then displays the received feedback in the virtual environment, allowing the user to see the evaluation of their own actions and emotions in real time and reflect this in their next training session.

[2031] User Roles

[2032] Wearing VR goggles

[2033] The user puts on the VR goggles and prepares to begin training. The virtual environment is displayed through the goggles.

[2034] Selection of training scenarios

[2035] Users select a training scenario displayed on their device and enter the virtual environment, setting them up to practice a specific skill.

[2036] Operation in the virtual space

[2037] The user performs instructed operations in the virtual environment, such as administering an injection to a virtual patient.

[2038] Providing emotion data

[2039] Users cooperate by providing emotional data (e.g., facial expressions and voice) that the device collects during training.

[2040] Check real-time feedback

[2041] Users can view real-time feedback during training to understand areas for improvement in their operations and emotions.

[2042] Reflection on the next training

[2043] Based on the feedback, efforts will be made to improve operational and emotional management in the next training session, which will increase the efficiency of skill acquisition and improve the training experience.

[2044] Specific examples

[2045] For example, in the medical field, a scenario in which a new nurse learns injection techniques in a virtual environment would go something like this: The user (new nurse) puts on VR goggles and enters the virtual environment based on the injection training scenario sent from the server. The device displays a virtual patient and injection equipment, and the user performs the virtual injection. The server receives the user's operation data and emotional data (for example, detecting impatience through facial expressions and voice analysis) in real time and analyzes the injection angle, depth, timing, etc. Feedback is generated based on the analysis results and displayed to the user in real time via the device. The user is expected to receive feedback and use it to improve their next operation and emotional management.

[2046] This system is a powerful tool for efficiently and safely acquiring skills without using real-world resources in fields such as medicine, construction, and safety training. Furthermore, the introduction of an emotion engine improves the quality of training and makes the user's training experience more personalized.

[2047] The processing flow will be explained below.

[2048] Server Processing

[2049] Step 1:

[2050] User Data Collection

[2051] The server connects to the database and collects user registration information, training history, and progress information, which allows the user's profile and past training details to be understood.

[2052] Step 2:

[2053] Training Needs Analysis

[2054] The server analyzes the collected data and identifies the next skill or training content the user should learn. For example, if the user is currently learning injection techniques, it will select a training scenario that matches their progress.

[2055] Step 3:

[2056] AI-based scenario generation

[2057] The server uses artificial intelligence (AI) to generate customized training scenarios based on the user's training progress, including specific procedures and situation settings.

[2058] Step 4:

[2059] Sending scenario data

[2060] The server transmits the generated training scenario data to the user's device, allowing the user to begin practicing based on the training scenario.

[2061] Step 5:

[2062] Collecting Emotional Data

[2063] The server works in conjunction with the emotion engine to receive the user's emotion data (e.g., the user's facial expressions and voice analysis data) collected from the terminal.

[2064] Step 6:

[2065] Receiving real-time operation data

[2066] The server receives user operation data sent from the device in real time, including hand movements and position information performed by the user in the virtual space.

[2067] Step 7:

[2068] Dynamic updates for virtual environments

[2069] Based on the received operation and emotion data, the server dynamically updates the state of the virtual environment, e.g., when the user administers an injection, the virtual patient's reaction and the syringe's position are updated.

[2070] Step 8:

[2071] Analysis of operation data and emotion data and feedback generation

[2072] The server analyzes the user's operation data and emotional data, and generates individual feedback based on the success rate of the operation, the type of error, and the user's emotional state.

[2073] Step 9:

[2074] Send Feedback

[2075] The server sends the generated feedback data to the user's device, allowing the user to review the feedback based on their performance and emotional state and reflect it in their next training session.

[2076] Terminal handling

[2077] Step 1:

[2078] Receiving scenario data

[2079] The terminal receives the training scenario data sent from the server and prepares the virtual environment to be displayed based on the received data.

[2080] Step 2:

[2081] Rendering a Virtual Environment

[2082] Based on the training scenario, the device displays the virtual environment to the user through the VR goggles, allowing the user to begin training in a highly immersive virtual space.

[2083] Step 3:

[2084] Operational Data Collection

[2085] The device tracks user actions and collects data in real time, including hand movements and location information.

[2086] Step 4:

[2087] Collecting Emotional Data

[2088] The device uses an emotion engine to collect user emotion data in real time, using techniques such as facial recognition and voice analysis.

[2089] Step 5:

[2090] Transmission of operational and emotional data

[2091] The collected operation data and emotion data are sent in real time to a server, which then dynamically updates the virtual environment based on this data.

[2092] Step 6:

[2093] Receiving feedback data

[2094] The device receives feedback data from the server, which includes operational improvements and takes into account the user's emotional state.

[2095] Step 7:

[2096] View Feedback

[2097] The device then displays the received feedback in the virtual environment, allowing the user to see the evaluation of their own actions and emotions in real time and reflect this in their next training session.

[2098] User Action

[2099] Step 1:

[2100] Wearing VR goggles

[2101] The user puts on the VR goggles and prepares to begin training. The virtual environment is displayed through the goggles.

[2102] Step 2:

[2103] Selection of training scenarios

[2104] Users select a training scenario displayed on their device and enter the virtual environment, setting them up to practice a specific skill.

[2105] Step 3:

[2106] Operation in the virtual space

[2107] The user performs instructed operations in the virtual environment, such as administering an injection to a virtual patient.

[2108] Step 4:

[2109] Providing emotion data

[2110] Users cooperate by providing emotional data (e.g., facial expressions and voice) that the device collects during training.

[2111] Step 5:

[2112] Check real-time feedback

[2113] Users can view real-time feedback during training to understand areas for improvement in their operations and emotions.

[2114] Step 6:

[2115] Reflection on the next training

[2116] Based on the feedback, efforts will be made to improve operational and emotional management in the next training session, which will increase the efficiency of skill acquisition and improve the training experience.

[2117] Example 2

[2118] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2119] In conventional skill and vocational training systems using virtual reality technology, training scenarios and feedback were generated based solely on user operation data, making it difficult to provide personalized training content and feedback that took the user's emotional state into account. This limited the effectiveness of training for users, making it difficult to acquire skills efficiently and effectively.

[2120] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for linking with a database that manages user registration information, training history, and progress information; artificial intelligence means for generating a training scenario tailored to the user's training progress based on the database; means for transmitting the generated training scenario to the user's terminal; means for rendering a virtual environment and displaying it to the user; means for receiving user operation data and emotional data in real time and dynamically updating the virtual environment; means for analyzing the user's operation data and emotional data, generating and transmitting individual feedback; and means for transmitting the feedback to the user's terminal and displaying it in the virtual environment. This makes it possible to provide personalized training content that takes the user's emotional state into consideration, thereby achieving efficient and effective skill acquisition.

[2121] "User registration information" refers to personal information and authentication information provided by a user when accessing the system.

[2122] "Training history" refers to information showing records of training that a user has conducted in the past and the results of those training.

[2123] "Progress information" refers to data that indicates the progress and level of achievement of the training that the user has achieved up to now.

[2124] "Database" refers to a system for centrally managing and storing user registration information, training history, progress information, etc.

[2125] "Artificial intelligence means" refers to technology that uses machine learning and advanced data analysis techniques to generate training scenarios based on user progress information.

[2126] A "training scenario" is a program that defines the virtual environment and procedures for conducting specific skill or vocational training.

[2127] A "terminal" refers to a device that allows a user to access and operate a virtual environment. Examples include VR goggles and a personal computer.

[2128] "Rendering a virtual environment" refers to using computer graphics techniques to generate and visually display a virtual three-dimensional environment to a user.

[2129] "Operation data" refers to information about operations performed by a user in a virtual environment, such as hand movements and click history.

[2130] "Emotional data" refers to data that indicates the user's emotional state. Examples include emotional states determined by facial expression recognition or voice analysis.

[2131] "Dynamic updating" refers to changing the virtual environment in real time in response to the user's actions and emotional state.

[2132] "Analyzing" refers to the detailed analysis of collected data to extract meaningful information and patterns.

[2133] "Feedback" refers to a response, including suggestions for improvement and evaluation, generated based on the user's actions and emotional state.

[2134] "Displaying within the virtual environment" refers to visually presenting generated feedback and other information within the virtual three-dimensional space experienced by the user.

[2135] An "emotion engine" refers to the technology and system for collecting and analyzing user emotional data.

[2136] This invention combines a system that uses virtual reality technology to provide specific skill or vocational training with an emotion engine that recognizes the user's emotions. The system is mainly composed of a user, a terminal, a server, and an emotion engine, and provides an efficient and safe virtual environment for the user to acquire specific skills, while also having the function of adjusting the training content and feedback according to the user's emotional state.

[2137] Server Roles

[2138] The server uses multiple pieces of hardware and software to perform the following processes:

[2139] 1. User Data Management:

[2140] The server connects to a database (e.g., PostgreSQL) and manages user registration information, training history, and progress information. This allows it to identify the next skills and training content each user should learn based on their progress. All of this information is stored in the database and is referenced every time a user accesses the system.

[2141] 2. Training scen...

Claims

1. A system for providing specific skills or vocational training using virtual reality technology, A means for linking with a database that manages user registration information, training history, and progress information; an artificial intelligence means for generating a training scenario based on the database and adapted to the user's training progress; means for transmitting the generated training scenario to a user's terminal; means for receiving user operation data in real time and dynamically updating the virtual environment; A means for analyzing user operation data and generating and transmitting individual feedback; A system including:

2. means for identifying training scenarios based on a user's area of ​​expertise and generating a virtual environment; means for tracking user operations within the virtual environment and transmitting the information to a server in real time; means for displaying the feedback sent from the server within the virtual environment; The system of claim 1 , comprising:

3. A means for a user to wear virtual reality goggles and perform specific operations in a virtual environment based on a training scenario; A means for receiving real-time feedback on operations within the virtual environment and reflecting it in future operations; The system of claim 1 , comprising:

Images