system

Abstract

We provide the system. [Solution] Sensors for monitoring user behavior, An analysis device that analyzes data from the aforementioned sensor and evaluates the accuracy of the operation, A display device that provides information to the user in augmented reality based on the evaluation by the aforementioned analysis device, A storage device for accumulating user progress information and evaluating skill levels, A generation device that generates a customized training plan based on the evaluation of the storage device, A system including a training support device for providing the generated training plan to the user.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional training system, there is a problem that it is difficult for a user to reduce mistakes during actual work and efficiently improve skills. Also, in the conventional method, it is impossible to automatically generate a training plan suitable for an individual user and provide real-time feedback. Furthermore, there is also a lack of a function to respond to questions that occur during work in natural language.

Means for Solving the Problems

[0005] The present invention includes an analysis device equipped with sensors for monitoring user movements, analyzing the obtained data, and evaluating the accuracy of those movements. Furthermore, it enables efficient training through a system that includes a display device that provides information in augmented reality in response to this evaluation, an accumulation device that stores progress information and evaluates individual skill levels, and a generation device that generates a customized training plan based on the evaluation results. The system also further includes a language understanding device that interprets voice input from the user using natural language processing and generates an appropriate response, and an immediate feedback device that provides immediate corrective instructions when an incorrect operation is detected.

[0006] A "sensor" is a device that detects a user's movements and records them as digital data.

[0007] An "analysis device" is a device that analyzes data obtained from sensors and evaluates whether the user's actions are accurate.

[0008] A "display device" is a device that provides users with visual information as augmented reality based on evaluation results from an analysis device.

[0009] A "data storage device" is a device that collects user progress data and evaluates and records the user's skill level based on this data.

[0010] A "generation device" is a device that creates a training plan optimized for each individual user based on the evaluation results of the storage device.

[0011] A "training support device" is a device that provides users with training plans created by a generation device, thereby supporting their learning.

[0012] A "language understanding device" is a device that receives voice input from a user and generates an appropriate response using natural language processing technology.

[0013] An "instant feedback device" is a device that monitors user actions in real time and provides immediate corrective instructions when an error is detected. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a processor with a reference number (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a RAM (Random Access Memory) with a reference number is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a storage with a reference number is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The system of this invention consists of multiple devices mounted on smart glasses. Users can receive training assistance by wearing the smart glasses and performing tasks. At the core of the system are sensors for real-time monitoring of the user's movements. These sensors capture the user's movements and transmit the data to an analysis device.

[0036] The terminal evaluates the data received by the analysis device and determines whether the user's actions are correct. This evaluation result is sent to a display device that provides information to the user. The terminal's display device projects auxiliary information in augmented reality into the user's field of view. For example, when the user is assembling parts, the next steps and points to note are displayed as graphics such as arrows.

[0037] The server stores user activity data and progress information in a storage device and uses this data to evaluate the user's skill level. Based on this evaluation, a training plan tailored to the user is created by a generation device. The plan created by the server is then presented to the user through the terminal's training support device during the next training session.

[0038] Furthermore, the system includes a language understanding device that allows users to input questions via voice while working, and obtain appropriate responses using natural language processing. This process enables users to instantly obtain the necessary information without interrupting their work.

[0039] As a concrete example, this system is highly effective when factory workers are learning to operate new machinery. The terminal uses an immediate feedback device to warn the user when they make a mistake and provides detailed instructions on the correct operation. This allows users to improve their skills while performing actual work, maximizing the effectiveness of the training.

[0040] In this way, the system is optimized to improve training efficiency and reduce errors during operation.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The device activates sensors to detect when the user starts moving and begins capturing the user's movements in real time. The captured data is converted into a digital signal.

[0044] Step 2:

[0045] The terminal transmits the acquired motion data to the analysis device. The analysis device processes the received data using an AI algorithm and compares the current motion pattern with a predefined reference pattern.

[0046] Step 3:

[0047] The analysis device evaluates the accuracy of the operation based on the analysis results. The evaluation results are then transmitted to a display device that shows the information in the user's field of view.

[0048] Step 4:

[0049] Based on the evaluation results obtained, the device displays the user the next actions to take and important notes using augmented reality. For example, the next parts to use and the assembly order are visually presented.

[0050] Step 5:

[0051] The server stores user activity data and evaluation results in a storage device. The server then evaluates the user's progress and skill level based on the collected data.

[0052] Step 6:

[0053] Based on the evaluated skill information, the server generates an optimized training plan for each user using a generator.

[0054] Step 7:

[0055] The terminal provides the user with the generated training plan through a training support device, assisting with learning. It also plans the next training session as needed.

[0056] Step 8:

[0057] If a user has a question during the process, they input it by voice. The voice signal is sent to a language understanding device.

[0058] Step 9:

[0059] The server's language understanding device analyzes the speech input and generates an appropriate response using natural language processing. This response is then provided to the user via the terminal.

[0060] Step 10:

[0061] If the terminal detects an error in the user's actions, it uses an immediate feedback device to provide the user with the necessary corrective instructions right away. This allows the user to correct their mistakes in real time.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] In today's industrial environment, efficient training and skill development of the workforce are crucial, but traditional methods have made it difficult to provide appropriate guidance tailored to the individual abilities and progress of each worker. Furthermore, the inability to provide immediate and appropriate instructions and information in response to incorrect operations or questions during the process has led to decreased efficiency and increased errors.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes detection means for monitoring user actions, evaluation means for evaluating the accuracy of actions, and display means. This makes it possible to evaluate user actions in real time and provide appropriate feedback and information. Furthermore, by using recording means to grasp the skill level and generating means to formulate individual training plans, optimal training can be provided to each worker. In addition, by providing an environment in which questions during work can be answered in real time using natural language understanding means and incorrect actions can be quickly corrected using immediate feedback means, training efficiency is improved and errors are reduced.

[0067] "Detection means for monitoring user behavior" refers to devices and technologies for tracking a user's physical condition and movements in real time and acquiring them as digital data.

[0068] "Evaluation means for evaluating the accuracy of actions" refers to a device or technology that analyzes data obtained from detection means and has the function of determining how close the user's actions are to a predetermined standard.

[0069] "Display means" refers to devices or technologies that visually present necessary information or instructions to the user, such as those that provide information using augmented reality technology.

[0070] "Recording means" refers to devices or technologies for saving user action data and progress, and enables the evaluation of user capabilities using the accumulated data.

[0071] "Generation means" refers to devices or technologies for creating an optimal training plan for a user based on data stored in recording means.

[0072] "Natural language understanding means" refers to technologies that analyze voice input from users, understand their intent, and generate appropriate responses, using speech recognition and natural language processing.

[0073] An "immediate feedback mechanism" is a device or technology that provides quick corrective instructions to users when they perform an incorrect operation, thereby encouraging them to take the correct action.

[0074] This invention is a system that provides efficient training in situations where a user is working with a visual device such as smart glasses. The system includes detection means, evaluation means, display means, recording means, generation means, natural language understanding means, and immediate feedback means.

[0075] When a user wears smart glasses, the device's detection system monitors the user's movements in real time and acquires motion data. This data is analyzed by an evaluation system to assess whether the user's actions conform to the established criteria. For example, image processing software and motion capture technology are used for motion analysis. Based on the analysis results, the display system, which displays advertisements, provides information to the user's field of view using augmented reality (AR). This allows the user to check the next steps and points to note in real time.

[0076] Furthermore, the server's recording mechanism accumulates user activity data to understand the user's skill level. The generation mechanism then creates individually customized training plans based on this information. By using this plan in subsequent sessions, users can expect efficient skill improvement.

[0077] If a user has a question during their work, they can ask it via voice input using natural language understanding. This system uses a generative AI model to analyze the user's question and provide an appropriate answer in voice or text. This allows the user to easily obtain the necessary information without interrupting their workflow.

[0078] For example, in a manufacturing plant where workers need to learn how to operate a new machine in a short period of time, introducing this system allows them to intuitively learn how to operate the machine and rapidly improve their skills through practice. Useful examples of prompts include phrases like, "Generate the optimal training plan for learning the assembly procedure of the new machine," or "Analyze user behavior data and suggest an efficient learning method."

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] The user puts on smart glasses and begins working. The device's detection means senses the user's movements in real time. Sensors capture the user's body movements and hand positions and transmit this data to the evaluation means. The input data is user movement information, and the output is raw movement data.

[0082] Step 2:

[0083] The terminal evaluation method analyzes detected motion data. Motion analysis software is used to evaluate whether the user's actions conform to predetermined criteria. In this process, the data is quantified and its accuracy is determined. The input is raw motion data, and the output is numerical values ​​of motion accuracy and evaluation metrics as evaluation results.

[0084] Step 3:

[0085] The device's display mechanism activates to provide the user with feedback based on the evaluation results. This display mechanism presents auxiliary information in augmented reality within the user's field of view. The user's next actions and points to note are visually displayed. The input is the evaluation result, and the output is the visual feedback.

[0086] Step 4:

[0087] The server stores user behavior data and evaluation results using recording devices. This data is used for later analysis and training plan generation. The input is evaluation results and behavior data, and the output is the stored dataset.

[0088] Step 5:

[0089] Based on accumulated data, the server generates an optimal training plan for the user. Utilizing a generation AI model, it develops a customized plan tailored to the user's skill level. The input is accumulated data, and the output is a customized training plan.

[0090] Step 6:

[0091] The training plan will be presented to the user in the next session. The device's support features will display this plan, allowing the user to follow along and complete the training. The input is the training plan, and the output is a visual presentation to the user.

[0092] Step 7:

[0093] When a user has a question while working, the device's natural language understanding (NLP) system receives the voice input, and a generative AI model generates an appropriate response. This process allows the user to receive an immediate answer. The input is the user's question (voice data), and the output is a response in natural language.

[0094] Step 8:

[0095] If the user performs an incorrect action, the device's immediate feedback mechanism provides instant corrective instructions. This allows the user to quickly learn the correct procedure. The input is the user's action data and evaluation results, and the output is immediate corrective feedback.

[0096] (Application Example 1)

[0097] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0098] There is a need for a support system that enables efficient and accurate skill acquisition among engineers and workers in factories. In particular, proper training is essential for operating machinery and performing maintenance tasks. However, traditional training methods have made it difficult to monitor individual user progress in real time and provide instruction tailored to their abilities. A system is needed that solves this problem while providing rapid, flexible, and situation-appropriate instruction.

[0099] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0100] In this invention, the server includes a sensor device for monitoring user actions, an analysis means, and a display means. This makes it possible to evaluate user actions in real time and provide appropriate information in augmented reality.

[0101] A "user" refers to an individual worker or technician who uses this system for training or performing tasks.

[0102] "Actions" refer to the physical actions and work procedures that a user actually performs, and this system evaluates the accuracy of these actions.

[0103] A "sensor device" is a device that captures a user's movements in real time and transmits the data to an analysis system.

[0104] "Analysis means" refers to a process that processes data acquired by sensor devices and evaluates the accuracy and progress of the user's actions.

[0105] "Display means" refers to devices and technologies for providing auxiliary information as augmented reality within the user's field of vision.

[0106] "Storage means" refers to databases or recording devices that store user action data and progress information and use them to evaluate skill levels.

[0107] "Generation means" refers to a process or apparatus that creates a customized training plan tailored to the user's skill level based on data stored in the storage means.

[0108] "Training support means" refers to the overall system components that provide users with generated training plans and real-time feedback to support their training.

[0109] A system for implementing this invention comprises a sensor device, an analysis means, a display means, a storage means, a generation means, and a training support means.

[0110] The sensor device is used to capture user movements in real time and transmit the data to the analysis system. This device includes cameras and motion sensors for accurately tracking movement. The analysis system processes the received data to evaluate the accuracy of the user's movements. Specifically, it uses an AI model to analyze data patterns and determine how well the user's movements conform to defined criteria.

[0111] The display means provides information to the user in augmented reality format based on evaluation results from the analysis means. Specifically, smart glasses are used to display graphics and text indicating operating procedures and next actions within the user's field of view. This allows the user to receive guidance on the spot while concentrating on their work.

[0112] The data storage system records user behavior data and progress information in a database and uses it to evaluate the skill level of individual users. The data generation system uses the stored data to create a customized training plan that matches the individual's progress and skill level, and presents it in the next training session.

[0113] The training support system provides real-time feedback based on the generated training plan to assist users in acquiring skills. It also features a voice input function, allowing users to input questions by voice when they have doubts, and receives instant responses through natural language processing.

[0114] For example, if a user is learning how to operate a new robot in a factory, the system will immediately provide feedback such as, "That action is incorrect. Please perform this step next," if the user makes a mistake.

[0115] Examples of prompt messages include the following:

[0116] "Please tell me how to install this part."

[0117] "What will you do next?"

[0118] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0119] Step 1:

[0120] The server receives data acquired by sensor devices to capture user movements. The input data consists of coordinate information of the user's hand and body movements. Based on this data, the server normalizes the data using a generative AI model and extracts movement patterns in order to analyze the movements in real time.

[0121] Step 2:

[0122] The server analyzes the motion patterns extracted using a generative AI model and evaluates their accuracy by comparing them to a baseline motion. The input is normalized motion data, and the output is an evaluation value of the motion's accuracy and deviation. This evaluation result is used to determine how accurately a particular motion was performed.

[0123] Step 3:

[0124] The server sends data to the terminal to provide real-time feedback based on the analysis results. The input is the evaluation result, and the output is feedback information that is easy for the user to understand visually. For example, this could be procedural instructions such as "This is the next step" or visual guides such as arrows.

[0125] Step 4:

[0126] The device displays the received feedback data to the user in augmented reality. The input is feedback data from the server, and the output is augmented reality information fused into the user's field of view. The user confirms the steps through the smart glasses screen and then proceeds to the next action.

[0127] Step 5:

[0128] The user inputs a question via a voice interface and sends it to the server. The input is the user's voice data, which is then converted by a natural language processing engine. The server converts the input voice into text and searches for the appropriate information.

[0129] Step 6:

[0130] The server generates appropriate responses to user questions based on natural language processing. The input is the user's question converted into text format, and the output is the answer information. For example, if the prompt is "Please tell me how to install this part," a guide with specific steps will be provided.

[0131] Step 7:

[0132] The data storage method involves recording user progress data and performance evaluation results in a database. Input consists of performance evaluation and feedback data from each session. This allows for the accumulation of information for improving and customizing future training plans.

[0133] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0134] This invention enhances user motivation and learning efficiency by adding an emotion engine to a system that utilizes multiple devices mounted on smart glasses to support user training. When a user puts on the smart glasses and begins working, the system immediately starts monitoring and analyzing their actions.

[0135] The terminal uses sensors to capture user motion data, and an analysis device evaluates the accuracy of that data. Based on this evaluation, a display device provides the necessary information in augmented reality within the user's field of view. This includes the next step in the task, points to pay attention to, and encouraging messages if successful.

[0136] The server collects user activity data and progress in a storage device. Based on the stored information, the skill level is evaluated, and a training plan optimized for each user is created by a generation device. The training plan is transferred to the terminal via a training support device and presented to the user.

[0137] Furthermore, the system incorporates an emotion engine, and the terminal collects emotional data from the user's voice tone and facial expressions. The server works in conjunction with the language understanding device and the emotion engine to analyze the user's emotional state. Training feedback and plans are dynamically adjusted according to the emotional state. For example, if the user shows frustration, the system will suggest easier tasks or send encouraging messages.

[0138] As a concrete example, when factory workers are learning to operate new machinery, the terminal provides immediate feedback when the user makes a mistake. Meanwhile, if the emotion engine detects user stress, the server displays advice to help the user relax and adjusts the pace of work. This allows the user to continue training in a comfortable environment, thereby facilitating learning and adaptation.

[0139] Such a system will overcome the limitations of traditional training support and enable a more personalized educational experience.

[0140] The following describes the processing flow.

[0141] Step 1:

[0142] The terminal activates its built-in sensors to detect when the user starts working and captures the user's movement data in real time. The captured data is then transmitted to an analysis device.

[0143] Step 2:

[0144] The analysis device processes the received motion data using an AI algorithm to evaluate the accuracy of the motion. This evaluation result includes the presence or absence of errors and areas for performance improvement.

[0145] Step 3:

[0146] The device's display provides the user with augmented reality information based on the analysis results. It visually displays the next action to take and how to correct any mistakes.

[0147] Step 4:

[0148] The server stores user activity data and evaluation results in a storage device, forming a dataset for evaluating the user's skill level and progress.

[0149] Step 5:

[0150] The server uses a generator to create a customized training plan based on the assessed skill level. This plan will then be provided to the user in future training sessions.

[0151] Step 6:

[0152] The device captures the user's facial expressions and voice tone using an emotion engine and generates data to evaluate their emotional state.

[0153] Step 7:

[0154] The server's emotion engine analyzes captured emotion data to determine the user's emotional state. Based on this emotional state, it dynamically adjusts the training content.

[0155] Step 8:

[0156] The device adjusts the user's learning speed and feedback based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, it will offer hints to simplify tasks or suggestions for relaxation.

[0157] Step 9:

[0158] If a user encounters any difficulties during the process, they can ask questions using voice. This voice information is sent to the server's language understanding device.

[0159] Step 10:

[0160] The server's language understanding device interprets speech input using natural language processing and generates an appropriate response. This response is provided to the user via the terminal to help them continue their work.

[0161] (Example 2)

[0162] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0163] Providing personalized training effectively and creating a flexible learning environment that takes into account users' emotional states has been difficult with conventional technologies. In particular, there was a lack of means to provide immediate training adjustments in response to changes in users' emotions and to provide specific feedback. It is necessary to solve this problem and improve users' learning efficiency and motivation.

[0164] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0165] In this invention, the server includes sensor means for detecting user movements, analysis means for analyzing data from the sensor means and evaluating the accuracy of the movements, and visual presentation means for providing information to the user in augmented reality based on the evaluation by the analysis means. This makes it possible to provide individualized and adaptive training content.

[0166] A "sensor" is a device used to detect user actions and collect data related to those actions.

[0167] An "analysis means" is a device that analyzes data acquired from sensor means and evaluates the accuracy of the user's actions.

[0168] A "visual presentation means" is a device that provides information to the user using augmented reality based on the evaluation results of the analysis means.

[0169] A "data collection device" is a device that collects and stores user progress information and evaluates the user's skill level based on this information.

[0170] The "plan generation means" is a device that creates an individualized training plan optimized for the user based on evaluations performed by the aggregation means.

[0171] A "training support device" is a device that provides the user with a generated training plan and effectively supports the training.

[0172] An "emotion analysis device" is a device that analyzes a user's voice and facial expressions to evaluate their emotional state.

[0173] An "adaptive measure" is a device that dynamically adjusts training content and feedback based on evaluations obtained through emotion analysis.

[0174] The system for implementing this invention supports user training through smart glasses. The terminal is equipped with multiple sensor means that capture the user's movements in real time. Sensors include posture sensors, accelerometers, and cameras. This data is immediately sent to an analysis means, where the accuracy of the movements is evaluated by a dedicated algorithm.

[0175] For example, when a factory worker is learning to operate a new machine, the system can determine if their hand position and movements are correct and immediately point out any incorrect actions. Based on this, the visual presentation system uses augmented reality to display the correct next steps and points to pay attention to within the user's field of vision.

[0176] The server stores user activity data and progress information transmitted from the terminal using an aggregation means. The server is equipped with analysis software for evaluating competency levels, which continuously assesses the user's skill improvement. The evaluation results are used by a plan generation means to create a training plan optimized for the user. This training plan is then transferred to the terminal via a training support means and incorporated into the ongoing training.

[0177] In addition, the device is equipped with emotion analysis capabilities that analyze the user's voice tone and facial expressions, and this data is sent to the server. The server works in conjunction with the emotion analysis capabilities to evaluate the user's emotional state and dynamically adjusts feedback and training content using adaptive mechanisms. If the analysis indicates that the user is experiencing stress, the server takes measures to reduce the load, such as lowering the difficulty of the task.

[0178] As a concrete example, by inputting the prompt "Please explain in detail the steps to safely operate the new machine" into the AI ​​model, appropriate instruction content is generated and provided to the user. In this way, a personalized educational experience that goes beyond the limitations of conventional training can be achieved.

[0179] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0180] Step 1:

[0181] The user puts on smart glasses and begins training. The device uses sensors to capture the user's movements in real time. The input data from the sensors includes the user's hand movements and body movements. This data is immediately sent to a motion analysis system.

[0182] Step 2:

[0183] The server receives motion data sent from the terminal and uses analysis tools to evaluate the accuracy of the motion. It receives motion coordinates and velocity information as input data and processes this by comparing it with reference values. As output, it generates an evaluation result that shows how well the motion conforms to the specified training plan.

[0184] Step 3:

[0185] The device receives the analysis results and provides feedback to the user using visual means. Based on the evaluation results, instructions and points to note in augmented reality are displayed in the user's field of view. For example, arrows indicating the correct position for placing machine parts are visually displayed.

[0186] Step 4:

[0187] The server collects user activity data and progress information using aggregation methods. This collected data tracks the user's past performance and serves as material for optimizing future training plans. It is stored as input data in a database and later output for analysis.

[0188] Step 5:

[0189] The server's plan generation mechanism creates individual training plans using a generated AI model based on accumulated user data. It takes past training history and motion analysis results as input data and outputs the plan best suited to the user.

[0190] Step 6:

[0191] The device analyzes the user's voice tone and facial expressions using emotion analysis tools and collects emotional data. This data is sent from the device to the server. The server receives emotional characteristic data as input and outputs the user's current emotional state.

[0192] Step 7:

[0193] The server receives the emotion analysis results and dynamically adjusts the feedback and training content using adaptive mechanisms. If a specific emotional state is detected, it automatically generates prompt messages using a generative AI model and outputs appropriate feedback to the user. For example, if the user is feeling frustrated, a suggestion such as "Take a breath and calm down, then start with these steps" might be displayed.

[0194] (Application Example 2)

[0195] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0196] In today's world, personalized training and educational support are becoming increasingly important. However, traditional systems have struggled to adequately consider user emotions, making it difficult to optimize learning effectiveness. Furthermore, it has been challenging to create systems that provide immediate feedback on user errors and dynamic adjustments based on emotions.

[0197] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0198] In this invention, the server includes emotion analysis means for analyzing the user's emotional state and dynamically adjusting training feedback, language understanding means for generating responses by natural language processing, and control means for adjusting the difficulty level of training based on the user's emotions. This makes it possible to comprehensively consider the user's actions and emotions and provide individually optimized training.

[0199] A "detection device" is a device that observes a user's actions in real time and acquires action data.

[0200] "Analysis means" refers to functions or devices used to analyze data obtained from a detection device and evaluate the accuracy of its operation.

[0201] A "display means" refers to a function or device that provides information to the user visually as augmented reality based on the analyzed information.

[0202] A "memory device" is a device that stores user progress information and operation data for later analysis.

[0203] "Generation means" refers to functions or devices for creating personalized training plans based on data stored in a memory device.

[0204] A "training support means" refers to a function or device that presents a generated training plan to the user and supports the progress of the training.

[0205] "Emotional analysis means" refers to functions or devices that analyze a user's emotional state based on factors such as the user's voice tone and facial expressions.

[0206] A "language understanding means" refers to a function or device that receives voice input from a user and generates an appropriate response using natural language processing technology.

[0207] "Control means" refers to functions or devices that dynamically adjust the content and difficulty level of training based on the results of user emotion analysis.

[0208] To implement this invention, the user first puts on smart glasses. The smart glasses, which act as the terminal, are equipped with a detection device that monitors movement in real time and acquires user behavior data. This allows the user's movements to be captured in detail. The data detected includes, for example, hand movements and posture.

[0209] Next, the device analyzes the acquired motion data using analysis tools and evaluates the accuracy of the movements. For this analysis, programs primarily written in Python and machine learning libraries such as TENSORFLOW® are used. The analysis results are made available to the user through a display tool. For example, if the user is exercising, real-time feedback is provided on whether their movements are correct.

[0210] Furthermore, the server stores user activity data and progress information in its storage device, managing the training history. This ensures that the necessary foundational data is available to generate training plans tailored to individual users. Based on the accumulated data, a personalized training plan is created by the generation mechanism and provided to the user through the training support mechanism. This process is managed using the Django framework.

[0211] Furthermore, the system uses emotion analysis to evaluate the user's emotional state. Voice tone and facial expression data are processed by the emotion analysis engine. This allows the server to dynamically adjust training feedback to match the user's emotions. For example, if the system determines that the user is emotionally exhausted, it will slow down the training pace and suggest a break.

[0212] As a concrete example, if a user performing a fitness activity uses smart glasses to perform an exercise with incorrect form, the smart glasses will immediately inform them of this, demonstrate the correct form, and send encouraging messages to prevent a drop in motivation.

[0213] An example of a prompt for a generative AI model is: "Please tell me the steps required to implement an algorithm that analyzes the emotional state of users and applies appropriate feedback and training content based on that data."

[0214] In this way, users are supported in both behavioral and emotional aspects, allowing them to receive more effective training.

[0215] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0216] Step 1:

[0217] The device uses sensors to acquire user motion data. This motion data includes the user's hand position and movement speed. The input from the sensors is transmitted to the device as a digital signal and stored in a database in real time.

[0218] Step 2:

[0219] An analysis device on the terminal analyzes the acquired motion data. The input data is processed into various features, and the accuracy of the motion is evaluated by a machine learning model. As a result of the analysis, information is generated indicating whether the motion is appropriate or whether improvement is needed.

[0220] Step 3:

[0221] The device displays the analysis results to the user via an augmented reality display. The user receives visual feedback on the next steps and points to note. Additionally, a message of praise is displayed when the correct actions are taken.

[0222] Step 4:

[0223] The server stores user activity data and progress information in its storage device. This preserves training history and serves as foundational data for later analysis. This data is organized into different folders for each user.

[0224] Step 5:

[0225] The server generates a personalized training plan based on data stored in its memory. Input data includes past performance history and skill level information, and the output is a set of training steps optimized for the user. A generation AI model is used, and the plan generated by the AI ​​is sent to the terminal via a training support system.

[0226] Step 6:

[0227] The device uses an emotion analysis engine to collect emotional data from the user's voice tone and facial expressions. This data is processed as input indicating the user's fatigue level and stress level.

[0228] Step 7:

[0229] The server uses emotion analysis tools to analyze the user's emotional state. Based on the processed data, the server quantifies the user's emotional state and calculates how that state affects training.

[0230] Step 8:

[0231] The server dynamically adjusts training feedback based on the user's emotional state. Based on the analysis results, it generates suggestions to help the user relax and re-evaluates the plan to suit the user's emotional state. The prompt used for this is "Analyze the user's emotional state and apply feedback and training content based on that data."

[0232] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0233] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0234] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0235] [Second Embodiment]

[0236] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0237] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0238] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0239] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0240] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0241] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0242] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0243] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0244] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0245] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0246] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0247] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0248] The system of this invention consists of multiple devices mounted on smart glasses. Users can receive training assistance by wearing the smart glasses and performing tasks. At the core of the system are sensors for real-time monitoring of the user's movements. These sensors capture the user's movements and transmit the data to an analysis device.

[0249] The terminal evaluates the data received by the analysis device and determines whether the user's actions are correct. This evaluation result is sent to a display device that provides information to the user. The terminal's display device projects auxiliary information in augmented reality into the user's field of view. For example, when the user is assembling parts, the next steps and points to note are displayed as graphics such as arrows.

[0250] The server stores user activity data and progress information in a storage device and uses this data to evaluate the user's skill level. Based on this evaluation, a training plan tailored to the user is created by a generation device. The plan created by the server is then presented to the user through the terminal's training support device during the next training session.

[0251] Furthermore, the system includes a language understanding device that allows users to input questions via voice while working, and obtain appropriate responses using natural language processing. This process enables users to instantly obtain the necessary information without interrupting their work.

[0252] As a concrete example, this system is highly effective when factory workers are learning to operate new machinery. The terminal uses an immediate feedback device to warn the user when they make a mistake and provides detailed instructions on the correct operation. This allows users to improve their skills while performing actual work, maximizing the effectiveness of the training.

[0253] In this way, the system is optimized to improve training efficiency and reduce errors during operation.

[0254] The following describes the processing flow.

[0255] Step 1:

[0256] The device activates sensors to detect when the user starts moving and begins capturing the user's movements in real time. The captured data is converted into a digital signal.

[0257] Step 2:

[0258] The terminal transmits the acquired motion data to the analysis device. The analysis device processes the received data using an AI algorithm and compares the current motion pattern with a predefined reference pattern.

[0259] Step 3:

[0260] The analysis device evaluates the accuracy of the operation based on the analysis results. The evaluation results are then transmitted to a display device that shows the information in the user's field of view.

[0261] Step 4:

[0262] Based on the evaluation results obtained, the device displays the user the next actions to take and important notes using augmented reality. For example, the next parts to use and the assembly order are visually presented.

[0263] Step 5:

[0264] The server stores user activity data and evaluation results in a storage device. The server then evaluates the user's progress and skill level based on the collected data.

[0265] Step 6:

[0266] Based on the evaluated skill information, the server generates an optimized training plan for each user using a generator.

[0267] Step 7:

[0268] The terminal provides the user with the generated training plan through a training support device, assisting with learning. It also plans the next training session as needed.

[0269] Step 8:

[0270] If a user has a question during the process, they input it by voice. The voice signal is sent to a language understanding device.

[0271] Step 9:

[0272] The server's language understanding device analyzes the speech input and generates an appropriate response using natural language processing. This response is then provided to the user via the terminal.

[0273] Step 10:

[0274] If the terminal detects an error in the user's actions, it uses an immediate feedback device to provide the user with the necessary corrective instructions right away. This allows the user to correct their mistakes in real time.

[0275] (Example 1)

[0276] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0277] In today's industrial environment, efficient training and skill development of the workforce are crucial, but traditional methods have made it difficult to provide appropriate guidance tailored to the individual abilities and progress of each worker. Furthermore, the inability to provide immediate and appropriate instructions and information in response to incorrect operations or questions during the process has led to decreased efficiency and increased errors.

[0278] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0279] In this invention, the server includes detection means for monitoring the user's actions, evaluation means for evaluating the accuracy of the actions, and display means. As a result, it becomes possible to evaluate the user's actions in real time and provide appropriate feedback and information. Also, by using recording means to grasp the ability level and formulating an individual training plan by generation means, it is possible to provide optimal training for each worker. Furthermore, by establishing an environment where the natural language understanding means can respond to questions during work in real time and the incorrect actions can be quickly corrected by the immediate feedback means, the improvement of training efficiency and the reduction of errors are realized.

[0280] The "detection means for monitoring the user's actions" is a device or technology for tracking the user's physical condition and actions in real time and obtaining them as digital data.

[0281] The "evaluation means for evaluating the accuracy of the actions" is a device or technology having a function of analyzing the data obtained from the detection means and determining how close the user's actions are to a predetermined standard.

[0282] The "display means" is a device or technology for visually presenting necessary information and instructions to the user, and for example, it uses augmented reality technology to provide information.

[0283] The "recording means" is a device or technology for storing the user's action data and progress status, and enables the evaluation of the user's ability by using the accumulated data.

[0284] The "generation means" is a device or technology for creating an optimal training plan for the user based on the data stored in the recording means.

[0285] The "natural language understanding means" is a technology for analyzing the voice input from the user, understanding its intention, and generating an appropriate response, which uses voice recognition and natural language processing.

[0286] "Real-time feedback means" refers to a device or technology that provides a quick correction instruction to prompt correct actions when a user makes a wrong operation.

[0287] This invention is a system that provides efficient training in a scenario where a user performs work using a visual device such as smart glasses. This system includes detection means, evaluation means, display means, recording means, generation means, natural language understanding means, and real-time feedback means.

[0288] When the user wears smart glasses, the detection means of the terminal monitors the user's actions in real time and acquires action data. This data is analyzed by the evaluation means to evaluate whether the user's operations comply with the standards. For motion analysis, for example, image processing software or motion capture technology is used. The display means for advertising provides information in augmented reality (AR) within the user's field of vision based on the analysis results. Thereby, the user can confirm the next operation and points to note in real time.

[0289] Furthermore, the recording means of the server accumulates the user's action data to grasp the user's skill level. The generation means creates an individually customized training plan based on this information. By using this plan in the next session, the user can expect efficient skill improvement.

[0290] When the user has a question during work, they can use the natural language understanding means to ask questions by voice input. This means uses a generative AI model to analyze the user's questions and provide appropriate answers in voice or text. Thereby, the user can easily obtain the necessary information without interrupting the work flow.

[0291] For example, in a manufacturing plant where workers need to learn how to operate a new machine in a short period of time, introducing this system allows them to intuitively learn how to operate the machine and rapidly improve their skills through practice. Useful examples of prompts include phrases like, "Generate the optimal training plan for learning the assembly procedure of the new machine," or "Analyze user behavior data and suggest an efficient learning method."

[0292] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0293] Step 1:

[0294] The user puts on smart glasses and begins working. The device's detection means senses the user's movements in real time. Sensors capture the user's body movements and hand positions and transmit this data to the evaluation means. The input data is user movement information, and the output is raw movement data.

[0295] Step 2:

[0296] The terminal evaluation method analyzes detected motion data. Motion analysis software is used to evaluate whether the user's actions conform to predetermined criteria. In this process, the data is quantified and its accuracy is determined. The input is raw motion data, and the output is numerical values ​​of motion accuracy and evaluation metrics as evaluation results.

[0297] Step 3:

[0298] The device's display mechanism activates to provide the user with feedback based on the evaluation results. This display mechanism presents auxiliary information in augmented reality within the user's field of view. The user's next actions and points to note are visually displayed. The input is the evaluation result, and the output is the visual feedback.

[0299] Step 4:

[0300] The server accumulates the user's operation data and evaluation results by means of a recording means. This data is used for subsequent analysis and training plan generation. The input is the evaluation result and operation data, and the output is the accumulated dataset.

[0301] Step 5:

[0302] Based on the accumulated data, the server's generation means creates an optimal training plan for the user. Utilizing a generation AI model, a customized plan according to the user's skill level is formulated. The input is the accumulated data, and the output is the customized training plan.

[0303] Step 6:

[0304] The training plan is presented to the user in the next session. The terminal's assistance means displays this plan, and the user can perform training according to it. The input is the training plan, and the output is the visual presentation to the user.

[0305] Step 7:

[0306] When the user has a question during work, the terminal's natural language understanding means receives the voice input and generates an appropriate response by the generation AI model. Through this process, the user obtains an immediate answer. The input is the user's question (voice data), and the output is the response in natural language.

[0307] Step 8:

[0308] When the user performs an incorrect operation, the terminal's immediate feedback means immediately provides a correction instruction. Thereby, the user can learn the correct procedure immediately. The input is the user's operation data and evaluation result, and the output is the immediate corrective feedback.

[0309] (Application Example 1)

[0310] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0311] There is a need for a support system that enables efficient and accurate skill acquisition among engineers and workers in factories. In particular, proper training is essential for operating machinery and performing maintenance tasks. However, traditional training methods have made it difficult to monitor individual user progress in real time and provide instruction tailored to their abilities. A system is needed that solves this problem while providing rapid, flexible, and situation-appropriate instruction.

[0312] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0313] In this invention, the server includes a sensor device for monitoring user actions, an analysis means, and a display means. This makes it possible to evaluate user actions in real time and provide appropriate information in augmented reality.

[0314] A "user" refers to an individual worker or technician who uses this system for training or performing tasks.

[0315] "Actions" refer to the physical actions and work procedures that a user actually performs, and this system evaluates the accuracy of these actions.

[0316] A "sensor device" is a device that captures a user's movements in real time and transmits the data to an analysis system.

[0317] "Analysis means" refers to a process that processes data acquired by sensor devices and evaluates the accuracy and progress of the user's actions.

[0318] "Display means" refers to devices and technologies for providing auxiliary information as augmented reality within the user's field of vision.

[0319] "Storage means" refers to databases or recording devices that store user action data and progress information and use them to evaluate skill levels.

[0320] "Generation means" refers to a process or apparatus that creates a customized training plan tailored to the user's skill level based on data stored in the storage means.

[0321] "Training support means" refers to the overall system components that provide users with generated training plans and real-time feedback to support their training.

[0322] A system for implementing this invention comprises a sensor device, an analysis means, a display means, a storage means, a generation means, and a training support means.

[0323] The sensor device is used to capture user movements in real time and transmit the data to the analysis system. This device includes cameras and motion sensors for accurately tracking movement. The analysis system processes the received data to evaluate the accuracy of the user's movements. Specifically, it uses an AI model to analyze data patterns and determine how well the user's movements conform to defined criteria.

[0324] The display means provides information to the user in augmented reality format based on evaluation results from the analysis means. Specifically, smart glasses are used to display graphics and text indicating operating procedures and next actions within the user's field of view. This allows the user to receive guidance on the spot while concentrating on their work.

[0325] The data storage system records user behavior data and progress information in a database and uses it to evaluate the skill level of individual users. The data generation system uses the stored data to create a customized training plan that matches the individual's progress and skill level, and presents it in the next training session.

[0326] The training support system provides real-time feedback based on the generated training plan to assist users in acquiring skills. It also features a voice input function, allowing users to input questions by voice when they have doubts, and receives instant responses through natural language processing.

[0327] For example, if a user is learning how to operate a new robot in a factory, the system will immediately provide feedback such as, "That action is incorrect. Please perform this step next," if the user makes a mistake.

[0328] Examples of prompt messages include the following:

[0329] "Please tell me how to install this part."

[0330] "What will you do next?"

[0331] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0332] Step 1:

[0333] The server receives data acquired by sensor devices to capture user movements. The input data consists of coordinate information of the user's hand and body movements. Based on this data, the server normalizes the data using a generative AI model and extracts movement patterns in order to analyze the movements in real time.

[0334] Step 2:

[0335] The server analyzes the motion patterns extracted using a generative AI model and evaluates their accuracy by comparing them to a baseline motion. The input is normalized motion data, and the output is an evaluation value of the motion's accuracy and deviation. This evaluation result is used to determine how accurately a particular motion was performed.

[0336] Step 3:

[0337] The server sends data to the terminal to provide real-time feedback based on the analysis results. The input is the evaluation result, and the output is feedback information that is easy for the user to understand visually. For example, this could be procedural instructions such as "This is the next step" or visual guides such as arrows.

[0338] Step 4:

[0339] The device displays the received feedback data to the user in augmented reality. The input is feedback data from the server, and the output is augmented reality information fused into the user's field of view. The user confirms the steps through the smart glasses screen and then proceeds to the next action.

[0340] Step 5:

[0341] The user inputs a question via a voice interface and sends it to the server. The input is the user's voice data, which is then converted by a natural language processing engine. The server converts the input voice into text and searches for the appropriate information.

[0342] Step 6:

[0343] The server generates appropriate responses to user questions based on natural language processing. The input is the user's question converted into text format, and the output is the answer information. For example, if the prompt is "Please tell me how to install this part," a guide with specific steps will be provided.

[0344] Step 7:

[0345] The data storage method involves recording user progress data and performance evaluation results in a database. Input consists of performance evaluation and feedback data from each session. This allows for the accumulation of information for improving and customizing future training plans.

[0346] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0347] This invention enhances user motivation and learning efficiency by adding an emotion engine to a system that utilizes multiple devices mounted on smart glasses to support user training. When a user puts on the smart glasses and begins working, the system immediately starts monitoring and analyzing their actions.

[0348] The terminal uses sensors to capture user motion data, and an analysis device evaluates the accuracy of that data. Based on this evaluation, a display device provides the necessary information in augmented reality within the user's field of view. This includes the next step in the task, points to pay attention to, and encouraging messages if successful.

[0349] The server collects user activity data and progress in a storage device. Based on the stored information, the skill level is evaluated, and a training plan optimized for each user is created by a generation device. The training plan is transferred to the terminal via a training support device and presented to the user.

[0350] Furthermore, the system incorporates an emotion engine, and the terminal collects emotional data from the user's voice tone and facial expressions. The server works in conjunction with the language understanding device and the emotion engine to analyze the user's emotional state. Training feedback and plans are dynamically adjusted according to the emotional state. For example, if the user shows frustration, the system will suggest easier tasks or send encouraging messages.

[0351] As a concrete example, when factory workers are learning to operate new machinery, the terminal provides immediate feedback when the user makes a mistake. Meanwhile, if the emotion engine detects user stress, the server displays advice to help the user relax and adjusts the pace of work. This allows the user to continue training in a comfortable environment, thereby facilitating learning and adaptation.

[0352] Such a system will overcome the limitations of traditional training support and enable a more personalized educational experience.

[0353] The following describes the processing flow.

[0354] Step 1:

[0355] The terminal activates its built-in sensors to detect when the user starts working and captures the user's movement data in real time. The captured data is then transmitted to an analysis device.

[0356] Step 2:

[0357] The analysis device processes the received motion data using an AI algorithm to evaluate the accuracy of the motion. This evaluation result includes the presence or absence of errors and areas for performance improvement.

[0358] Step 3:

[0359] The device's display provides the user with augmented reality information based on the analysis results. It visually displays the next action to take and how to correct any mistakes.

[0360] Step 4:

[0361] The server stores user activity data and evaluation results in a storage device, forming a dataset for evaluating the user's skill level and progress.

[0362] Step 5:

[0363] The server uses a generator to create a customized training plan based on the assessed skill level. This plan will then be provided to the user in future training sessions.

[0364] Step 6:

[0365] The device captures the user's facial expressions and voice tone using an emotion engine and generates data to evaluate their emotional state.

[0366] Step 7:

[0367] The server's emotion engine analyzes captured emotion data to determine the user's emotional state. Based on this emotional state, it dynamically adjusts the training content.

[0368] Step 8:

[0369] The device adjusts the user's learning speed and feedback based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, it will offer hints to simplify tasks or suggestions for relaxation.

[0370] Step 9:

[0371] If a user encounters any difficulties during the process, they can ask questions using voice. This voice information is sent to the server's language understanding device.

[0372] Step 10:

[0373] The server's language understanding device interprets speech input using natural language processing and generates an appropriate response. This response is provided to the user via the terminal to help them continue their work.

[0374] (Example 2)

[0375] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0376] Providing personalized training effectively and creating a flexible learning environment that takes into account users' emotional states has been difficult with conventional technologies. In particular, there was a lack of means to provide immediate training adjustments in response to changes in users' emotions and to provide specific feedback. It is necessary to solve this problem and improve users' learning efficiency and motivation.

[0377] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0378] In this invention, the server includes sensor means for detecting user movements, analysis means for analyzing data from the sensor means and evaluating the accuracy of the movements, and visual presentation means for providing information to the user in augmented reality based on the evaluation by the analysis means. This makes it possible to provide individualized and adaptive training content.

[0379] A "sensor" is a device used to detect user actions and collect data related to those actions.

[0380] An "analysis means" is a device that analyzes data acquired from sensor means and evaluates the accuracy of the user's actions.

[0381] A "visual presentation means" is a device that provides information to the user using augmented reality based on the evaluation results of the analysis means.

[0382] A "data collection device" is a device that collects and stores user progress information and evaluates the user's skill level based on this information.

[0383] The "plan generation means" is a device that creates an individualized training plan optimized for the user based on evaluations performed by the aggregation means.

[0384] A "training support device" is a device that provides the user with a generated training plan and effectively supports the training.

[0385] An "emotion analysis device" is a device that analyzes a user's voice and facial expressions to evaluate their emotional state.

[0386] An "adaptive measure" is a device that dynamically adjusts training content and feedback based on evaluations obtained through emotion analysis.

[0387] The system for implementing this invention supports user training through smart glasses. The terminal is equipped with multiple sensor means that capture the user's movements in real time. Sensors include posture sensors, accelerometers, and cameras. This data is immediately sent to an analysis means, where the accuracy of the movements is evaluated by a dedicated algorithm.

[0388] For example, when a factory worker is learning to operate a new machine, the system can determine if their hand position and movements are correct and immediately point out any incorrect actions. Based on this, the visual presentation system uses augmented reality to display the correct next steps and points to pay attention to within the user's field of vision.

[0389] The server stores user activity data and progress information transmitted from the terminal using an aggregation means. The server is equipped with analysis software for evaluating competency levels, which continuously assesses the user's skill improvement. The evaluation results are used by a plan generation means to create a training plan optimized for the user. This training plan is then transferred to the terminal via a training support means and incorporated into the ongoing training.

[0390] In addition, the device is equipped with emotion analysis capabilities that analyze the user's voice tone and facial expressions, and this data is sent to the server. The server works in conjunction with the emotion analysis capabilities to evaluate the user's emotional state and dynamically adjusts feedback and training content using adaptive mechanisms. If the analysis indicates that the user is experiencing stress, the server takes measures to reduce the load, such as lowering the difficulty of the task.

[0391] As a concrete example, by inputting the prompt "Please explain in detail the steps to safely operate the new machine" into the AI ​​model, appropriate instruction content is generated and provided to the user. In this way, a personalized educational experience that goes beyond the limitations of conventional training can be achieved.

[0392] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0393] Step 1:

[0394] The user puts on smart glasses and begins training. The device uses sensors to capture the user's movements in real time. The input data from the sensors includes the user's hand movements and body movements. This data is immediately sent to a motion analysis system.

[0395] Step 2:

[0396] The server receives motion data sent from the terminal and uses analysis tools to evaluate the accuracy of the motion. It receives motion coordinates and velocity information as input data and processes this by comparing it with reference values. As output, it generates an evaluation result that shows how well the motion conforms to the specified training plan.

[0397] Step 3:

[0398] The device receives the analysis results and provides feedback to the user using visual means. Based on the evaluation results, instructions and points to note in augmented reality are displayed in the user's field of view. For example, arrows indicating the correct position for placing machine parts are visually displayed.

[0399] Step 4:

[0400] The server collects user activity data and progress information using aggregation methods. This collected data tracks the user's past performance and serves as material for optimizing future training plans. It is stored as input data in a database and later output for analysis.

[0401] Step 5:

[0402] The server's plan generation mechanism creates individual training plans using a generated AI model based on accumulated user data. It takes past training history and motion analysis results as input data and outputs the plan best suited to the user.

[0403] Step 6:

[0404] The device analyzes the user's voice tone and facial expressions using emotion analysis tools and collects emotional data. This data is sent from the device to the server. The server receives emotional characteristic data as input and outputs the user's current emotional state.

[0405] Step 7:

[0406] The server receives the emotion analysis results and dynamically adjusts the feedback and training content using adaptive mechanisms. If a specific emotional state is detected, it automatically generates prompt messages using a generative AI model and outputs appropriate feedback to the user. For example, if the user is feeling frustrated, a suggestion such as "Take a breath and calm down, then start with these steps" might be displayed.

[0407] (Application Example 2)

[0408] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0409] In today's world, personalized training and educational support are becoming increasingly important. However, traditional systems have struggled to adequately consider user emotions, making it difficult to optimize learning effectiveness. Furthermore, it has been challenging to create systems that provide immediate feedback on user errors and dynamic adjustments based on emotions.

[0410] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0411] In this invention, the server includes emotion analysis means for analyzing the user's emotional state and dynamically adjusting training feedback, language understanding means for generating responses by natural language processing, and control means for adjusting the difficulty level of training based on the user's emotions. This makes it possible to comprehensively consider the user's actions and emotions and provide individually optimized training.

[0412] A "detection device" is a device that observes a user's actions in real time and acquires action data.

[0413] "Analysis means" refers to functions or devices used to analyze data obtained from a detection device and evaluate the accuracy of its operation.

[0414] A "display means" refers to a function or device that provides information to the user visually as augmented reality based on the analyzed information.

[0415] A "memory device" is a device that stores user progress information and operation data for later analysis.

[0416] "Generation means" refers to functions or devices for creating personalized training plans based on data stored in a memory device.

[0417] A "training support means" refers to a function or device that presents a generated training plan to the user and supports the progress of the training.

[0418] "Emotional analysis means" refers to functions or devices that analyze a user's emotional state based on factors such as the user's voice tone and facial expressions.

[0419] A "language understanding means" refers to a function or device that receives voice input from a user and generates an appropriate response using natural language processing technology.

[0420] "Control means" refers to functions or devices that dynamically adjust the content and difficulty level of training based on the results of user emotion analysis.

[0421] To implement this invention, the user first puts on smart glasses. The smart glasses, which act as the terminal, are equipped with a detection device that monitors movement in real time and acquires user behavior data. This allows the user's movements to be captured in detail. The data detected includes, for example, hand movements and posture.

[0422] Next, the device analyzes the acquired motion data using an analysis tool and evaluates the accuracy of the movements. For this analysis, programs primarily written in Python and machine learning libraries such as TensorFlow are used. The analysis results are made available to the user through a display tool. For example, if the user is exercising, real-time feedback is provided on whether their movements are correct.

[0423] Furthermore, the server stores user activity data and progress information in its storage device, managing the training history. This ensures that the necessary foundational data is available to generate training plans tailored to individual users. Based on the accumulated data, a personalized training plan is created by the generation mechanism and provided to the user through the training support mechanism. This process is managed using the Django framework.

[0424] Furthermore, the system uses emotion analysis to evaluate the user's emotional state. Voice tone and facial expression data are processed by the emotion analysis engine. This allows the server to dynamically adjust training feedback to match the user's emotions. For example, if the system determines that the user is emotionally exhausted, it will slow down the training pace and suggest a break.

[0425] As a concrete example, if a user performing a fitness activity uses smart glasses to perform an exercise with incorrect form, the smart glasses will immediately inform them of this, demonstrate the correct form, and send encouraging messages to prevent a drop in motivation.

[0426] An example of a prompt for a generative AI model is: "Please tell me the steps required to implement an algorithm that analyzes the emotional state of users and applies appropriate feedback and training content based on that data."

[0427] In this way, users are supported in both behavioral and emotional aspects, allowing them to receive more effective training.

[0428] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0429] Step 1:

[0430] The device uses sensors to acquire user motion data. This motion data includes the user's hand position and movement speed. The input from the sensors is transmitted to the device as a digital signal and stored in a database in real time.

[0431] Step 2:

[0432] An analysis device on the terminal analyzes the acquired motion data. The input data is processed into various features, and the accuracy of the motion is evaluated by a machine learning model. As a result of the analysis, information is generated indicating whether the motion is appropriate or whether improvement is needed.

[0433] Step 3:

[0434] The device displays the analysis results to the user via an augmented reality display. The user receives visual feedback on the next steps and points to note. Additionally, a message of praise is displayed when the correct actions are taken.

[0435] Step 4:

[0436] The server stores user activity data and progress information in its storage device. This preserves training history and serves as foundational data for later analysis. This data is organized into different folders for each user.

[0437] Step 5:

[0438] The server generates a personalized training plan based on data stored in its memory. Input data includes past performance history and skill level information, and the output is a set of training steps optimized for the user. A generation AI model is used, and the plan generated by the AI ​​is sent to the terminal via a training support system.

[0439] Step 6:

[0440] The device uses an emotion analysis engine to collect emotional data from the user's voice tone and facial expressions. This data is processed as input indicating the user's fatigue level and stress level.

[0441] Step 7:

[0442] The server uses emotion analysis tools to analyze the user's emotional state. Based on the processed data, the server quantifies the user's emotional state and calculates how that state affects training.

[0443] Step 8:

[0444] The server dynamically adjusts training feedback based on the user's emotional state. Based on the analysis results, it generates suggestions to help the user relax and re-evaluates the plan to suit the user's emotional state. The prompt used for this is "Analyze the user's emotional state and apply feedback and training content based on that data."

[0445] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0446] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0447] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0448] [Third Embodiment]

[0449] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0450] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0451] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0452] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0453] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0454] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0455] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0456] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0457] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0458] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0459] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0460] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0461] The system of this invention consists of multiple devices mounted on smart glasses. Users can receive training assistance by wearing the smart glasses and performing tasks. At the core of the system are sensors for real-time monitoring of the user's movements. These sensors capture the user's movements and transmit the data to an analysis device.

[0462] The terminal evaluates the data received by the analysis device and determines whether the user's actions are correct. This evaluation result is sent to a display device that provides information to the user. The terminal's display device projects auxiliary information in augmented reality into the user's field of view. For example, when the user is assembling parts, the next steps and points to note are displayed as graphics such as arrows.

[0463] The server stores user activity data and progress information in a storage device and uses this data to evaluate the user's skill level. Based on this evaluation, a training plan tailored to the user is created by a generation device. The plan created by the server is then presented to the user through the terminal's training support device during the next training session.

[0464] Furthermore, the system includes a language understanding device that allows users to input questions via voice while working, and obtain appropriate responses using natural language processing. This process enables users to instantly obtain the necessary information without interrupting their work.

[0465] As a concrete example, this system is highly effective when factory workers are learning to operate new machinery. The terminal uses an immediate feedback device to warn the user when they make a mistake and provides detailed instructions on the correct operation. This allows users to improve their skills while performing actual work, maximizing the effectiveness of the training.

[0466] In this way, the system is optimized to improve training efficiency and reduce errors during operation.

[0467] The following describes the processing flow.

[0468] Step 1:

[0469] The device activates sensors to detect when the user starts moving and begins capturing the user's movements in real time. The captured data is converted into a digital signal.

[0470] Step 2:

[0471] The terminal transmits the acquired motion data to the analysis device. The analysis device processes the received data using an AI algorithm and compares the current motion pattern with a predefined reference pattern.

[0472] Step 3:

[0473] The analysis device evaluates the accuracy of the operation based on the analysis results. The evaluation results are then transmitted to a display device that shows the information in the user's field of view.

[0474] Step 4:

[0475] Based on the evaluation results obtained, the device displays the user the next actions to take and important notes using augmented reality. For example, the next parts to use and the assembly order are visually presented.

[0476] Step 5:

[0477] The server stores user activity data and evaluation results in a storage device. The server then evaluates the user's progress and skill level based on the collected data.

[0478] Step 6:

[0479] Based on the evaluated skill information, the server generates an optimized training plan for each user using a generator.

[0480] Step 7:

[0481] The terminal provides the user with the generated training plan through a training support device, assisting with learning. It also plans the next training session as needed.

[0482] Step 8:

[0483] If a user has a question during the process, they input it by voice. The voice signal is sent to a language understanding device.

[0484] Step 9:

[0485] The server's language understanding device analyzes the speech input and generates an appropriate response using natural language processing. This response is then provided to the user via the terminal.

[0486] Step 10:

[0487] If the terminal detects an error in the user's actions, it uses an immediate feedback device to provide the user with the necessary corrective instructions right away. This allows the user to correct their mistakes in real time.

[0488] (Example 1)

[0489] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0490] In today's industrial environment, efficient training and skill development of the workforce are crucial, but traditional methods have made it difficult to provide appropriate guidance tailored to the individual abilities and progress of each worker. Furthermore, the inability to provide immediate and appropriate instructions and information in response to incorrect operations or questions during the process has led to decreased efficiency and increased errors.

[0491] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0492] In this invention, the server includes detection means for monitoring user actions, evaluation means for evaluating the accuracy of actions, and display means. This makes it possible to evaluate user actions in real time and provide appropriate feedback and information. Furthermore, by using recording means to grasp the skill level and generating means to formulate individual training plans, optimal training can be provided to each worker. In addition, by providing an environment in which questions during work can be answered in real time using natural language understanding means and incorrect actions can be quickly corrected using immediate feedback means, training efficiency is improved and errors are reduced.

[0493] "Detection means for monitoring user behavior" refers to devices and technologies for tracking a user's physical condition and movements in real time and acquiring them as digital data.

[0494] "Evaluation means for evaluating the accuracy of actions" refers to a device or technology that analyzes data obtained from detection means and has the function of determining how close the user's actions are to a predetermined standard.

[0495] "Display means" refers to devices or technologies that visually present necessary information or instructions to the user, such as those that provide information using augmented reality technology.

[0496] "Recording means" refers to devices or technologies for saving user action data and progress, and enables the evaluation of user capabilities using the accumulated data.

[0497] "Generation means" refers to devices or technologies for creating an optimal training plan for a user based on data stored in recording means.

[0498] "Natural language understanding means" refers to technologies that analyze voice input from users, understand their intent, and generate appropriate responses, using speech recognition and natural language processing.

[0499] An "immediate feedback mechanism" is a device or technology that provides quick corrective instructions to users when they perform an incorrect operation, thereby encouraging them to take the correct action.

[0500] This invention is a system that provides efficient training in situations where a user is working with a visual device such as smart glasses. The system includes detection means, evaluation means, display means, recording means, generation means, natural language understanding means, and immediate feedback means.

[0501] When a user wears smart glasses, the device's detection system monitors the user's movements in real time and acquires motion data. This data is analyzed by an evaluation system to assess whether the user's actions conform to the established criteria. For example, image processing software and motion capture technology are used for motion analysis. Based on the analysis results, the display system, which displays advertisements, provides information to the user's field of view using augmented reality (AR). This allows the user to check the next steps and points to note in real time.

[0502] Furthermore, the server's recording mechanism accumulates user activity data to understand the user's skill level. The generation mechanism then creates individually customized training plans based on this information. By using this plan in subsequent sessions, users can expect efficient skill improvement.

[0503] If a user has a question during their work, they can ask it via voice input using natural language understanding. This system uses a generative AI model to analyze the user's question and provide an appropriate answer in voice or text. This allows the user to easily obtain the necessary information without interrupting their workflow.

[0504] For example, in a manufacturing plant where workers need to learn how to operate a new machine in a short period of time, introducing this system allows them to intuitively learn how to operate the machine and rapidly improve their skills through practice. Useful examples of prompts include phrases like, "Generate the optimal training plan for learning the assembly procedure of the new machine," or "Analyze user behavior data and suggest an efficient learning method."

[0505] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0506] Step 1:

[0507] The user puts on smart glasses and begins working. The device's detection means senses the user's movements in real time. Sensors capture the user's body movements and hand positions and transmit this data to the evaluation means. The input data is user movement information, and the output is raw movement data.

[0508] Step 2:

[0509] The terminal evaluation method analyzes detected motion data. Motion analysis software is used to evaluate whether the user's actions conform to predetermined criteria. In this process, the data is quantified and its accuracy is determined. The input is raw motion data, and the output is numerical values ​​of motion accuracy and evaluation metrics as evaluation results.

[0510] Step 3:

[0511] The device's display mechanism activates to provide the user with feedback based on the evaluation results. This display mechanism presents auxiliary information in augmented reality within the user's field of view. The user's next actions and points to note are visually displayed. The input is the evaluation result, and the output is the visual feedback.

[0512] Step 4:

[0513] The server stores user behavior data and evaluation results using recording devices. This data is used for later analysis and training plan generation. The input is evaluation results and behavior data, and the output is the stored dataset.

[0514] Step 5:

[0515] Based on accumulated data, the server generates an optimal training plan for the user. Utilizing a generation AI model, it develops a customized plan tailored to the user's skill level. The input is accumulated data, and the output is a customized training plan.

[0516] Step 6:

[0517] The training plan will be presented to the user in the next session. The device's support features will display this plan, allowing the user to follow along and complete the training. The input is the training plan, and the output is a visual presentation to the user.

[0518] Step 7:

[0519] When a user has a question while working, the device's natural language understanding (NLP) system receives the voice input, and a generative AI model generates an appropriate response. This process allows the user to receive an immediate answer. The input is the user's question (voice data), and the output is a response in natural language.

[0520] Step 8:

[0521] If the user performs an incorrect action, the device's immediate feedback mechanism provides instant corrective instructions. This allows the user to quickly learn the correct procedure. The input is the user's action data and evaluation results, and the output is immediate corrective feedback.

[0522] (Application Example 1)

[0523] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0524] There is a need for a support system that enables efficient and accurate skill acquisition among engineers and workers in factories. In particular, proper training is essential for operating machinery and performing maintenance tasks. However, traditional training methods have made it difficult to monitor individual user progress in real time and provide instruction tailored to their abilities. A system is needed that solves this problem while providing rapid, flexible, and situation-appropriate instruction.

[0525] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0526] In this invention, the server includes a sensor device for monitoring user actions, an analysis means, and a display means. This makes it possible to evaluate user actions in real time and provide appropriate information in augmented reality.

[0527] A "user" refers to an individual worker or technician who uses this system for training or performing tasks.

[0528] "Actions" refer to the physical actions and work procedures that a user actually performs, and this system evaluates the accuracy of these actions.

[0529] A "sensor device" is a device that captures a user's movements in real time and transmits the data to an analysis system.

[0530] "Analysis means" refers to a process that processes data acquired by sensor devices and evaluates the accuracy and progress of the user's actions.

[0531] "Display means" refers to devices and technologies for providing auxiliary information as augmented reality within the user's field of vision.

[0532] "Storage means" refers to databases or recording devices that store user action data and progress information and use them to evaluate skill levels.

[0533] "Generation means" refers to a process or apparatus that creates a customized training plan tailored to the user's skill level based on data stored in the storage means.

[0534] "Training support means" refers to the overall system components that provide users with generated training plans and real-time feedback to support their training.

[0535] A system for implementing this invention comprises a sensor device, an analysis means, a display means, a storage means, a generation means, and a training support means.

[0536] The sensor device is used to capture user movements in real time and transmit the data to the analysis system. This device includes cameras and motion sensors for accurately tracking movement. The analysis system processes the received data to evaluate the accuracy of the user's movements. Specifically, it uses an AI model to analyze data patterns and determine how well the user's movements conform to defined criteria.

[0537] The display means provides information to the user in augmented reality format based on evaluation results from the analysis means. Specifically, smart glasses are used to display graphics and text indicating operating procedures and next actions within the user's field of view. This allows the user to receive guidance on the spot while concentrating on their work.

[0538] The data storage system records user behavior data and progress information in a database and uses it to evaluate the skill level of individual users. The data generation system uses the stored data to create a customized training plan that matches the individual's progress and skill level, and presents it in the next training session.

[0539] The training support system provides real-time feedback based on the generated training plan to assist users in acquiring skills. It also features a voice input function, allowing users to input questions by voice when they have doubts, and receives instant responses through natural language processing.

[0540] For example, if a user is learning how to operate a new robot in a factory, the system will immediately provide feedback such as, "That action is incorrect. Please perform this step next," if the user makes a mistake.

[0541] Examples of prompt messages include the following:

[0542] "Please tell me how to install this part."

[0543] "What will you do next?"

[0544] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0545] Step 1:

[0546] The server receives data acquired by sensor devices to capture user movements. The input data consists of coordinate information of the user's hand and body movements. Based on this data, the server normalizes the data using a generative AI model and extracts movement patterns in order to analyze the movements in real time.

[0547] Step 2:

[0548] The server analyzes the motion patterns extracted using a generative AI model and evaluates their accuracy by comparing them to a baseline motion. The input is normalized motion data, and the output is an evaluation value of the motion's accuracy and deviation. This evaluation result is used to determine how accurately a particular motion was performed.

[0549] Step 3:

[0550] The server sends data to the terminal to provide real-time feedback based on the analysis results. The input is the evaluation result, and the output is feedback information that is easy for the user to understand visually. For example, this could be procedural instructions such as "This is the next step" or visual guides such as arrows.

[0551] Step 4:

[0552] The device displays the received feedback data to the user in augmented reality. The input is feedback data from the server, and the output is augmented reality information fused into the user's field of view. The user confirms the steps through the smart glasses screen and then proceeds to the next action.

[0553] Step 5:

[0554] The user inputs a question via a voice interface and sends it to the server. The input is the user's voice data, which is then converted by a natural language processing engine. The server converts the input voice into text and searches for the appropriate information.

[0555] Step 6:

[0556] The server generates appropriate responses to user questions based on natural language processing. The input is the user's question converted into text format, and the output is the answer information. For example, if the prompt is "Please tell me how to install this part," a guide with specific steps will be provided.

[0557] Step 7:

[0558] The data storage method involves recording user progress data and performance evaluation results in a database. Input consists of performance evaluation and feedback data from each session. This allows for the accumulation of information for improving and customizing future training plans.

[0559] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0560] This invention enhances user motivation and learning efficiency by adding an emotion engine to a system that utilizes multiple devices mounted on smart glasses to support user training. When a user puts on the smart glasses and begins working, the system immediately starts monitoring and analyzing their actions.

[0561] The terminal uses sensors to capture user motion data, and an analysis device evaluates the accuracy of that data. Based on this evaluation, a display device provides the necessary information in augmented reality within the user's field of view. This includes the next step in the task, points to pay attention to, and encouraging messages if successful.

[0562] The server collects user activity data and progress in a storage device. Based on the stored information, the skill level is evaluated, and a training plan optimized for each user is created by a generation device. The training plan is transferred to the terminal via a training support device and presented to the user.

[0563] Furthermore, the system incorporates an emotion engine, and the terminal collects emotional data from the user's voice tone and facial expressions. The server works in conjunction with the language understanding device and the emotion engine to analyze the user's emotional state. Training feedback and plans are dynamically adjusted according to the emotional state. For example, if the user shows frustration, the system will suggest easier tasks or send encouraging messages.

[0564] As a concrete example, when factory workers are learning to operate new machinery, the terminal provides immediate feedback when the user makes a mistake. Meanwhile, if the emotion engine detects user stress, the server displays advice to help the user relax and adjusts the pace of work. This allows the user to continue training in a comfortable environment, thereby facilitating learning and adaptation.

[0565] Such a system will overcome the limitations of traditional training support and enable a more personalized educational experience.

[0566] The following describes the processing flow.

[0567] Step 1:

[0568] The terminal activates its built-in sensors to detect when the user starts working and captures the user's movement data in real time. The captured data is then transmitted to an analysis device.

[0569] Step 2:

[0570] The analysis device processes the received motion data using an AI algorithm to evaluate the accuracy of the motion. This evaluation result includes the presence or absence of errors and areas for performance improvement.

[0571] Step 3:

[0572] The device's display provides the user with augmented reality information based on the analysis results. It visually displays the next action to take and how to correct any mistakes.

[0573] Step 4:

[0574] The server stores user activity data and evaluation results in a storage device, forming a dataset for evaluating the user's skill level and progress.

[0575] Step 5:

[0576] The server uses a generator to create a customized training plan based on the assessed skill level. This plan will then be provided to the user in future training sessions.

[0577] Step 6:

[0578] The device captures the user's facial expressions and voice tone using an emotion engine and generates data to evaluate their emotional state.

[0579] Step 7:

[0580] The server's emotion engine analyzes captured emotion data to determine the user's emotional state. Based on this emotional state, it dynamically adjusts the training content.

[0581] Step 8:

[0582] The device adjusts the user's learning speed and feedback based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, it will offer hints to simplify tasks or suggestions for relaxation.

[0583] Step 9:

[0584] If a user encounters any difficulties during the process, they can ask questions using voice. This voice information is sent to the server's language understanding device.

[0585] Step 10:

[0586] The server's language understanding device interprets speech input using natural language processing and generates an appropriate response. This response is provided to the user via the terminal to help them continue their work.

[0587] (Example 2)

[0588] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0589] Providing personalized training effectively and creating a flexible learning environment that takes into account users' emotional states has been difficult with conventional technologies. In particular, there was a lack of means to provide immediate training adjustments in response to changes in users' emotions and to provide specific feedback. It is necessary to solve this problem and improve users' learning efficiency and motivation.

[0590] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0591] In this invention, the server includes sensor means for detecting user movements, analysis means for analyzing data from the sensor means and evaluating the accuracy of the movements, and visual presentation means for providing information to the user in augmented reality based on the evaluation by the analysis means. This makes it possible to provide individualized and adaptive training content.

[0592] A "sensor" is a device used to detect user actions and collect data related to those actions.

[0593] An "analysis means" is a device that analyzes data acquired from sensor means and evaluates the accuracy of the user's actions.

[0594] A "visual presentation means" is a device that provides information to the user using augmented reality based on the evaluation results of the analysis means.

[0595] A "data collection device" is a device that collects and stores user progress information and evaluates the user's skill level based on this information.

[0596] The "plan generation means" is a device that creates an individualized training plan optimized for the user based on evaluations performed by the aggregation means.

[0597] A "training support device" is a device that provides the user with a generated training plan and effectively supports the training.

[0598] An "emotion analysis device" is a device that analyzes a user's voice and facial expressions to evaluate their emotional state.

[0599] An "adaptive measure" is a device that dynamically adjusts training content and feedback based on evaluations obtained through emotion analysis.

[0600] The system for implementing this invention supports user training through smart glasses. The terminal is equipped with multiple sensor means that capture the user's movements in real time. Sensors include posture sensors, accelerometers, and cameras. This data is immediately sent to an analysis means, where the accuracy of the movements is evaluated by a dedicated algorithm.

[0601] For example, when a factory worker is learning to operate a new machine, the system can determine if their hand position and movements are correct and immediately point out any incorrect actions. Based on this, the visual presentation system uses augmented reality to display the correct next steps and points to pay attention to within the user's field of vision.

[0602] The server stores user activity data and progress information transmitted from the terminal using an aggregation means. The server is equipped with analysis software for evaluating competency levels, which continuously assesses the user's skill improvement. The evaluation results are used by a plan generation means to create a training plan optimized for the user. This training plan is then transferred to the terminal via a training support means and incorporated into the ongoing training.

[0603] In addition, the device is equipped with emotion analysis capabilities that analyze the user's voice tone and facial expressions, and this data is sent to the server. The server works in conjunction with the emotion analysis capabilities to evaluate the user's emotional state and dynamically adjusts feedback and training content using adaptive mechanisms. If the analysis indicates that the user is experiencing stress, the server takes measures to reduce the load, such as lowering the difficulty of the task.

[0604] As a concrete example, by inputting the prompt "Please explain in detail the steps to safely operate the new machine" into the AI ​​model, appropriate instruction content is generated and provided to the user. In this way, a personalized educational experience that goes beyond the limitations of conventional training can be achieved.

[0605] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0606] Step 1:

[0607] The user puts on smart glasses and begins training. The device uses sensors to capture the user's movements in real time. The input data from the sensors includes the user's hand movements and body movements. This data is immediately sent to a motion analysis system.

[0608] Step 2:

[0609] The server receives motion data sent from the terminal and uses analysis tools to evaluate the accuracy of the motion. It receives motion coordinates and velocity information as input data and processes this by comparing it with reference values. As output, it generates an evaluation result that shows how well the motion conforms to the specified training plan.

[0610] Step 3:

[0611] The device receives the analysis results and provides feedback to the user using visual means. Based on the evaluation results, instructions and points to note in augmented reality are displayed in the user's field of view. For example, arrows indicating the correct position for placing machine parts are visually displayed.

[0612] Step 4:

[0613] The server collects user activity data and progress information using aggregation methods. This collected data tracks the user's past performance and serves as material for optimizing future training plans. It is stored as input data in a database and later output for analysis.

[0614] Step 5:

[0615] The server's plan generation mechanism creates individual training plans using a generated AI model based on accumulated user data. It takes past training history and motion analysis results as input data and outputs the plan best suited to the user.

[0616] Step 6:

[0617] The device analyzes the user's voice tone and facial expressions using emotion analysis tools and collects emotional data. This data is sent from the device to the server. The server receives emotional characteristic data as input and outputs the user's current emotional state.

[0618] Step 7:

[0619] The server receives the emotion analysis results and dynamically adjusts the feedback and training content using adaptive mechanisms. If a specific emotional state is detected, it automatically generates prompt messages using a generative AI model and outputs appropriate feedback to the user. For example, if the user is feeling frustrated, a suggestion such as "Take a breath and calm down, then start with these steps" might be displayed.

[0620] (Application Example 2)

[0621] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0622] In today's world, personalized training and educational support are becoming increasingly important. However, traditional systems have struggled to adequately consider user emotions, making it difficult to optimize learning effectiveness. Furthermore, it has been challenging to create systems that provide immediate feedback on user errors and dynamic adjustments based on emotions.

[0623] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0624] In this invention, the server includes emotion analysis means for analyzing the user's emotional state and dynamically adjusting training feedback, language understanding means for generating responses by natural language processing, and control means for adjusting the difficulty level of training based on the user's emotions. This makes it possible to comprehensively consider the user's actions and emotions and provide individually optimized training.

[0625] A "detection device" is a device that observes a user's actions in real time and acquires action data.

[0626] "Analysis means" refers to functions or devices used to analyze data obtained from a detection device and evaluate the accuracy of its operation.

[0627] A "display means" refers to a function or device that provides information to the user visually as augmented reality based on the analyzed information.

[0628] A "memory device" is a device that stores user progress information and operation data for later analysis.

[0629] "Generation means" refers to functions or devices for creating personalized training plans based on data stored in a memory device.

[0630] A "training support means" refers to a function or device that presents a generated training plan to the user and supports the progress of the training.

[0631] "Emotional analysis means" refers to functions or devices that analyze a user's emotional state based on factors such as the user's voice tone and facial expressions.

[0632] A "language understanding means" refers to a function or device that receives voice input from a user and generates an appropriate response using natural language processing technology.

[0633] "Control means" refers to functions or devices that dynamically adjust the content and difficulty level of training based on the results of user emotion analysis.

[0634] To implement this invention, the user first puts on smart glasses. The smart glasses, which act as the terminal, are equipped with a detection device that monitors movement in real time and acquires user behavior data. This allows the user's movements to be captured in detail. The data detected includes, for example, hand movements and posture.

[0635] Next, the device analyzes the acquired motion data using an analysis tool and evaluates the accuracy of the movements. For this analysis, programs primarily written in Python and machine learning libraries such as TensorFlow are used. The analysis results are made available to the user through a display tool. For example, if the user is exercising, real-time feedback is provided on whether their movements are correct.

[0636] Furthermore, the server stores user activity data and progress information in its storage device, managing the training history. This ensures that the necessary foundational data is available to generate training plans tailored to individual users. Based on the accumulated data, a personalized training plan is created by the generation mechanism and provided to the user through the training support mechanism. This process is managed using the Django framework.

[0637] Furthermore, the system uses emotion analysis to evaluate the user's emotional state. Voice tone and facial expression data are processed by the emotion analysis engine. This allows the server to dynamically adjust training feedback to match the user's emotions. For example, if the system determines that the user is emotionally exhausted, it will slow down the training pace and suggest a break.

[0638] As a concrete example, if a user performing a fitness activity uses smart glasses to perform an exercise with incorrect form, the smart glasses will immediately inform them of this, demonstrate the correct form, and send encouraging messages to prevent a drop in motivation.

[0639] An example of a prompt for a generative AI model is: "Please tell me the steps required to implement an algorithm that analyzes the emotional state of users and applies appropriate feedback and training content based on that data."

[0640] In this way, users are supported in both behavioral and emotional aspects, allowing them to receive more effective training.

[0641] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0642] Step 1:

[0643] The device uses sensors to acquire user motion data. This motion data includes the user's hand position and movement speed. The input from the sensors is transmitted to the device as a digital signal and stored in a database in real time.

[0644] Step 2:

[0645] An analysis device on the terminal analyzes the acquired motion data. The input data is processed into various features, and the accuracy of the motion is evaluated by a machine learning model. As a result of the analysis, information is generated indicating whether the motion is appropriate or whether improvement is needed.

[0646] Step 3:

[0647] The device displays the analysis results to the user via an augmented reality display. The user receives visual feedback on the next steps and points to note. Additionally, a message of praise is displayed when the correct actions are taken.

[0648] Step 4:

[0649] The server stores user activity data and progress information in its storage device. This preserves training history and serves as foundational data for later analysis. This data is organized into different folders for each user.

[0650] Step 5:

[0651] The server generates a personalized training plan based on data stored in its memory. Input data includes past performance history and skill level information, and the output is a set of training steps optimized for the user. A generation AI model is used, and the plan generated by the AI ​​is sent to the terminal via a training support system.

[0652] Step 6:

[0653] The device uses an emotion analysis engine to collect emotional data from the user's voice tone and facial expressions. This data is processed as input indicating the user's fatigue level and stress level.

[0654] Step 7:

[0655] The server uses emotion analysis tools to analyze the user's emotional state. Based on the processed data, the server quantifies the user's emotional state and calculates how that state affects training.

[0656] Step 8:

[0657] The server dynamically adjusts training feedback based on the user's emotional state. Based on the analysis results, it generates suggestions to help the user relax and re-evaluates the plan to suit the user's emotional state. The prompt used for this is "Analyze the user's emotional state and apply feedback and training content based on that data."

[0658] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0659] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0660] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0661] [Fourth Embodiment]

[0662] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0663] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0664] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0665] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0666] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0667] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0668] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0669] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0670] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0671] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0672] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0673] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0674] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0675] The system of this invention consists of multiple devices mounted on smart glasses. Users can receive training assistance by wearing the smart glasses and performing tasks. At the core of the system are sensors for real-time monitoring of the user's movements. These sensors capture the user's movements and transmit the data to an analysis device.

[0676] The terminal evaluates the data received by the analysis device and determines whether the user's actions are correct. This evaluation result is sent to a display device that provides information to the user. The terminal's display device projects auxiliary information in augmented reality into the user's field of view. For example, when the user is assembling parts, the next steps and points to note are displayed as graphics such as arrows.

[0677] The server stores user activity data and progress information in a storage device and uses this data to evaluate the user's skill level. Based on this evaluation, a training plan tailored to the user is created by a generation device. The plan created by the server is then presented to the user through the terminal's training support device during the next training session.

[0678] Furthermore, the system includes a language understanding device that allows users to input questions via voice while working, and obtain appropriate responses using natural language processing. This process enables users to instantly obtain the necessary information without interrupting their work.

[0679] As a concrete example, this system is highly effective when factory workers are learning to operate new machinery. The terminal uses an immediate feedback device to warn the user when they make a mistake and provides detailed instructions on the correct operation. This allows users to improve their skills while performing actual work, maximizing the effectiveness of the training.

[0680] In this way, the system is optimized to improve training efficiency and reduce errors during operation.

[0681] The following describes the processing flow.

[0682] Step 1:

[0683] The device activates sensors to detect when the user starts moving and begins capturing the user's movements in real time. The captured data is converted into a digital signal.

[0684] Step 2:

[0685] The terminal transmits the acquired motion data to the analysis device. The analysis device processes the received data using an AI algorithm and compares the current motion pattern with a predefined reference pattern.

[0686] Step 3:

[0687] The analysis device evaluates the accuracy of the operation based on the analysis results. The evaluation results are then transmitted to a display device that shows the information in the user's field of view.

[0688] Step 4:

[0689] Based on the evaluation results obtained, the device displays the user the next actions to take and important notes using augmented reality. For example, the next parts to use and the assembly order are visually presented.

[0690] Step 5:

[0691] The server stores user activity data and evaluation results in a storage device. The server then evaluates the user's progress and skill level based on the collected data.

[0692] Step 6:

[0693] Based on the evaluated skill information, the server generates an optimized training plan for each user using a generator.

[0694] Step 7:

[0695] The terminal provides the user with the generated training plan through a training support device, assisting with learning. It also plans the next training session as needed.

[0696] Step 8:

[0697] If a user has a question during the process, they input it by voice. The voice signal is sent to a language understanding device.

[0698] Step 9:

[0699] The server's language understanding device analyzes the speech input and generates an appropriate response using natural language processing. This response is then provided to the user via the terminal.

[0700] Step 10:

[0701] If the terminal detects an error in the user's actions, it uses an immediate feedback device to provide the user with the necessary corrective instructions right away. This allows the user to correct their mistakes in real time.

[0702] (Example 1)

[0703] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0704] In today's industrial environment, efficient training and skill development of the workforce are crucial, but traditional methods have made it difficult to provide appropriate guidance tailored to the individual abilities and progress of each worker. Furthermore, the inability to provide immediate and appropriate instructions and information in response to incorrect operations or questions during the process has led to decreased efficiency and increased errors.

[0705] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0706] In this invention, the server includes detection means for monitoring user actions, evaluation means for evaluating the accuracy of actions, and display means. This makes it possible to evaluate user actions in real time and provide appropriate feedback and information. Furthermore, by using recording means to grasp the skill level and generating means to formulate individual training plans, optimal training can be provided to each worker. In addition, by providing an environment in which questions during work can be answered in real time using natural language understanding means and incorrect actions can be quickly corrected using immediate feedback means, training efficiency is improved and errors are reduced.

[0707] "Detection means for monitoring user behavior" refers to devices and technologies for tracking a user's physical condition and movements in real time and acquiring them as digital data.

[0708] "Evaluation means for evaluating the accuracy of actions" refers to a device or technology that analyzes data obtained from detection means and has the function of determining how close the user's actions are to a predetermined standard.

[0709] "Display means" refers to devices or technologies that visually present necessary information or instructions to the user, such as those that provide information using augmented reality technology.

[0710] "Recording means" refers to devices or technologies for saving user action data and progress, and enables the evaluation of user capabilities using the accumulated data.

[0711] "Generation means" refers to devices or technologies for creating an optimal training plan for a user based on data stored in recording means.

[0712] "Natural language understanding means" refers to technologies that analyze voice input from users, understand their intent, and generate appropriate responses, using speech recognition and natural language processing.

[0713] An "immediate feedback mechanism" is a device or technology that provides quick corrective instructions to users when they perform an incorrect operation, thereby encouraging them to take the correct action.

[0714] This invention is a system that provides efficient training in situations where a user is working with a visual device such as smart glasses. The system includes detection means, evaluation means, display means, recording means, generation means, natural language understanding means, and immediate feedback means.

[0715] When a user wears smart glasses, the device's detection system monitors the user's movements in real time and acquires motion data. This data is analyzed by an evaluation system to assess whether the user's actions conform to the established criteria. For example, image processing software and motion capture technology are used for motion analysis. Based on the analysis results, the display system, which displays advertisements, provides information to the user's field of view using augmented reality (AR). This allows the user to check the next steps and points to note in real time.

[0716] Furthermore, the server's recording mechanism accumulates user activity data to understand the user's skill level. The generation mechanism then creates individually customized training plans based on this information. By using this plan in subsequent sessions, users can expect efficient skill improvement.

[0717] If a user has a question during their work, they can ask it via voice input using natural language understanding. This system uses a generative AI model to analyze the user's question and provide an appropriate answer in voice or text. This allows the user to easily obtain the necessary information without interrupting their workflow.

[0718] For example, in a manufacturing plant where workers need to learn how to operate a new machine in a short period of time, introducing this system allows them to intuitively learn how to operate the machine and rapidly improve their skills through practice. Useful examples of prompts include phrases like, "Generate the optimal training plan for learning the assembly procedure of the new machine," or "Analyze user behavior data and suggest an efficient learning method."

[0719] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0720] Step 1:

[0721] The user puts on smart glasses and begins working. The device's detection means senses the user's movements in real time. Sensors capture the user's body movements and hand positions and transmit this data to the evaluation means. The input data is user movement information, and the output is raw movement data.

[0722] Step 2:

[0723] The terminal evaluation method analyzes detected motion data. Motion analysis software is used to evaluate whether the user's actions conform to predetermined criteria. In this process, the data is quantified and its accuracy is determined. The input is raw motion data, and the output is numerical values ​​of motion accuracy and evaluation metrics as evaluation results.

[0724] Step 3:

[0725] The device's display mechanism activates to provide the user with feedback based on the evaluation results. This display mechanism presents auxiliary information in augmented reality within the user's field of view. The user's next actions and points to note are visually displayed. The input is the evaluation result, and the output is the visual feedback.

[0726] Step 4:

[0727] The server stores user behavior data and evaluation results using recording devices. This data is used for later analysis and training plan generation. The input is evaluation results and behavior data, and the output is the stored dataset.

[0728] Step 5:

[0729] Based on accumulated data, the server generates an optimal training plan for the user. Utilizing a generation AI model, it develops a customized plan tailored to the user's skill level. The input is accumulated data, and the output is a customized training plan.

[0730] Step 6:

[0731] The training plan will be presented to the user in the next session. The device's support features will display this plan, allowing the user to follow along and complete the training. The input is the training plan, and the output is a visual presentation to the user.

[0732] Step 7:

[0733] When a user has a question while working, the device's natural language understanding (NLP) system receives the voice input, and a generative AI model generates an appropriate response. This process allows the user to receive an immediate answer. The input is the user's question (voice data), and the output is a response in natural language.

[0734] Step 8:

[0735] If the user performs an incorrect action, the device's immediate feedback mechanism provides instant corrective instructions. This allows the user to quickly learn the correct procedure. The input is the user's action data and evaluation results, and the output is immediate corrective feedback.

[0736] (Application Example 1)

[0737] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0738] There is a need for a support system that enables efficient and accurate skill acquisition among engineers and workers in factories. In particular, proper training is essential for operating machinery and performing maintenance tasks. However, traditional training methods have made it difficult to monitor individual user progress in real time and provide instruction tailored to their abilities. A system is needed that solves this problem while providing rapid, flexible, and situation-appropriate instruction.

[0739] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0740] In this invention, the server includes a sensor device for monitoring user actions, an analysis means, and a display means. This makes it possible to evaluate user actions in real time and provide appropriate information in augmented reality.

[0741] A "user" refers to an individual worker or technician who uses this system for training or performing tasks.

[0742] "Actions" refer to the physical actions and work procedures that a user actually performs, and this system evaluates the accuracy of these actions.

[0743] A "sensor device" is a device that captures a user's movements in real time and transmits the data to an analysis system.

[0744] "Analysis means" refers to a process that processes data acquired by sensor devices and evaluates the accuracy and progress of the user's actions.

[0745] "Display means" refers to devices and technologies for providing auxiliary information as augmented reality within the user's field of vision.

[0746] "Storage means" refers to databases or recording devices that store user action data and progress information and use them to evaluate skill levels.

[0747] "Generation means" refers to a process or apparatus that creates a customized training plan tailored to the user's skill level based on data stored in the storage means.

[0748] "Training support means" refers to the overall system components that provide users with generated training plans and real-time feedback to support their training.

[0749] A system for implementing this invention comprises a sensor device, an analysis means, a display means, a storage means, a generation means, and a training support means.

[0750] The sensor device is used to capture user movements in real time and transmit the data to the analysis system. This device includes cameras and motion sensors for accurately tracking movement. The analysis system processes the received data to evaluate the accuracy of the user's movements. Specifically, it uses an AI model to analyze data patterns and determine how well the user's movements conform to defined criteria.

[0751] The display means provides information to the user in augmented reality format based on evaluation results from the analysis means. Specifically, smart glasses are used to display graphics and text indicating operating procedures and next actions within the user's field of view. This allows the user to receive guidance on the spot while concentrating on their work.

[0752] The data storage system records user behavior data and progress information in a database and uses it to evaluate the skill level of individual users. The data generation system uses the stored data to create a customized training plan that matches the individual's progress and skill level, and presents it in the next training session.

[0753] The training support system provides real-time feedback based on the generated training plan to assist users in acquiring skills. It also features a voice input function, allowing users to input questions by voice when they have doubts, and receives instant responses through natural language processing.

[0754] For example, if a user is learning how to operate a new robot in a factory, the system will immediately provide feedback such as, "That action is incorrect. Please perform this step next," if the user makes a mistake.

[0755] Examples of prompt messages include the following:

[0756] "Please tell me how to install this part."

[0757] "What will you do next?"

[0758] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0759] Step 1:

[0760] The server receives data acquired by sensor devices to capture user movements. The input data consists of coordinate information of the user's hand and body movements. Based on this data, the server normalizes the data using a generative AI model and extracts movement patterns in order to analyze the movements in real time.

[0761] Step 2:

[0762] The server analyzes the motion patterns extracted using a generative AI model and evaluates their accuracy by comparing them to a baseline motion. The input is normalized motion data, and the output is an evaluation value of the motion's accuracy and deviation. This evaluation result is used to determine how accurately a particular motion was performed.

[0763] Step 3:

[0764] The server sends data to the terminal to provide real-time feedback based on the analysis results. The input is the evaluation result, and the output is feedback information that is easy for the user to understand visually. For example, this could be procedural instructions such as "This is the next step" or visual guides such as arrows.

[0765] Step 4:

[0766] The device displays the received feedback data to the user in augmented reality. The input is feedback data from the server, and the output is augmented reality information fused into the user's field of view. The user confirms the steps through the smart glasses screen and then proceeds to the next action.

[0767] Step 5:

[0768] The user inputs a question via a voice interface and sends it to the server. The input is the user's voice data, which is then converted by a natural language processing engine. The server converts the input voice into text and searches for the appropriate information.

[0769] Step 6:

[0770] The server generates appropriate responses to user questions based on natural language processing. The input is the user's question converted into text format, and the output is the answer information. For example, if the prompt is "Please tell me how to install this part," a guide with specific steps will be provided.

[0771] Step 7:

[0772] The data storage method involves recording user progress data and performance evaluation results in a database. Input consists of performance evaluation and feedback data from each session. This allows for the accumulation of information for improving and customizing future training plans.

[0773] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0774] This invention enhances user motivation and learning efficiency by adding an emotion engine to a system that utilizes multiple devices mounted on smart glasses to support user training. When a user puts on the smart glasses and begins working, the system immediately starts monitoring and analyzing their actions.

[0775] The terminal uses sensors to capture user motion data, and an analysis device evaluates the accuracy of that data. Based on this evaluation, a display device provides the necessary information in augmented reality within the user's field of view. This includes the next step in the task, points to pay attention to, and encouraging messages if successful.

[0776] The server collects user activity data and progress in a storage device. Based on the stored information, the skill level is evaluated, and a training plan optimized for each user is created by a generation device. The training plan is transferred to the terminal via a training support device and presented to the user.

[0777] Furthermore, the system incorporates an emotion engine, and the terminal collects emotional data from the user's voice tone and facial expressions. The server works in conjunction with the language understanding device and the emotion engine to analyze the user's emotional state. Training feedback and plans are dynamically adjusted according to the emotional state. For example, if the user shows frustration, the system will suggest easier tasks or send encouraging messages.

[0778] As a concrete example, when factory workers are learning to operate new machinery, the terminal provides immediate feedback when the user makes a mistake. Meanwhile, if the emotion engine detects user stress, the server displays advice to help the user relax and adjusts the pace of work. This allows the user to continue training in a comfortable environment, thereby facilitating learning and adaptation.

[0779] Such a system will overcome the limitations of traditional training support and enable a more personalized educational experience.

[0780] The following describes the processing flow.

[0781] Step 1:

[0782] The terminal activates its built-in sensors to detect when the user starts working and captures the user's movement data in real time. The captured data is then transmitted to an analysis device.

[0783] Step 2:

[0784] The analysis device processes the received motion data using an AI algorithm to evaluate the accuracy of the motion. This evaluation result includes the presence or absence of errors and areas for performance improvement.

[0785] Step 3:

[0786] The device's display provides the user with augmented reality information based on the analysis results. It visually displays the next action to take and how to correct any mistakes.

[0787] Step 4:

[0788] The server stores user activity data and evaluation results in a storage device, forming a dataset for evaluating the user's skill level and progress.

[0789] Step 5:

[0790] The server uses a generator to create a customized training plan based on the assessed skill level. This plan will then be provided to the user in future training sessions.

[0791] Step 6:

[0792] The device captures the user's facial expressions and voice tone using an emotion engine and generates data to evaluate their emotional state.

[0793] Step 7:

[0794] The server's emotion engine analyzes captured emotion data to determine the user's emotional state. Based on this emotional state, it dynamically adjusts the training content.

[0795] Step 8:

[0796] The device adjusts the user's learning speed and feedback based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, it will offer hints to simplify tasks or suggestions for relaxation.

[0797] Step 9:

[0798] If a user encounters any difficulties during the process, they can ask questions using voice. This voice information is sent to the server's language understanding device.

[0799] Step 10:

[0800] The server's language understanding device interprets speech input using natural language processing and generates an appropriate response. This response is provided to the user via the terminal to help them continue their work.

[0801] (Example 2)

[0802] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0803] Providing personalized training effectively and creating a flexible learning environment that takes into account users' emotional states has been difficult with conventional technologies. In particular, there was a lack of means to provide immediate training adjustments in response to changes in users' emotions and to provide specific feedback. It is necessary to solve this problem and improve users' learning efficiency and motivation.

[0804] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0805] In this invention, the server includes sensor means for detecting user movements, analysis means for analyzing data from the sensor means and evaluating the accuracy of the movements, and visual presentation means for providing information to the user in augmented reality based on the evaluation by the analysis means. This makes it possible to provide individualized and adaptive training content.

[0806] A "sensor" is a device used to detect user actions and collect data related to those actions.

[0807] An "analysis means" is a device that analyzes data acquired from sensor means and evaluates the accuracy of the user's actions.

[0808] A "visual presentation means" is a device that provides information to the user using augmented reality based on the evaluation results of the analysis means.

[0809] A "data collection device" is a device that collects and stores user progress information and evaluates the user's skill level based on this information.

[0810] The "plan generation means" is a device that creates an individualized training plan optimized for the user based on evaluations performed by the aggregation means.

[0811] A "training support device" is a device that provides the user with a generated training plan and effectively supports the training.

[0812] An "emotion analysis device" is a device that analyzes a user's voice and facial expressions to evaluate their emotional state.

[0813] An "adaptive measure" is a device that dynamically adjusts training content and feedback based on evaluations obtained through emotion analysis.

[0814] The system for implementing this invention supports user training through smart glasses. The terminal is equipped with multiple sensor means that capture the user's movements in real time. Sensors include posture sensors, accelerometers, and cameras. This data is immediately sent to an analysis means, where the accuracy of the movements is evaluated by a dedicated algorithm.

[0815] For example, when a factory worker is learning to operate a new machine, the system can determine if their hand position and movements are correct and immediately point out any incorrect actions. Based on this, the visual presentation system uses augmented reality to display the correct next steps and points to pay attention to within the user's field of vision.

[0816] The server stores user activity data and progress information transmitted from the terminal using an aggregation means. The server is equipped with analysis software for evaluating competency levels, which continuously assesses the user's skill improvement. The evaluation results are used by a plan generation means to create a training plan optimized for the user. This training plan is then transferred to the terminal via a training support means and incorporated into the ongoing training.

[0817] In addition, the device is equipped with emotion analysis capabilities that analyze the user's voice tone and facial expressions, and this data is sent to the server. The server works in conjunction with the emotion analysis capabilities to evaluate the user's emotional state and dynamically adjusts feedback and training content using adaptive mechanisms. If the analysis indicates that the user is experiencing stress, the server takes measures to reduce the load, such as lowering the difficulty of the task.

[0818] As a concrete example, by inputting the prompt "Please explain in detail the steps to safely operate the new machine" into the AI ​​model, appropriate instruction content is generated and provided to the user. In this way, a personalized educational experience that goes beyond the limitations of conventional training can be achieved.

[0819] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0820] Step 1:

[0821] The user puts on smart glasses and begins training. The device uses sensors to capture the user's movements in real time. The input data from the sensors includes the user's hand movements and body movements. This data is immediately sent to a motion analysis system.

[0822] Step 2:

[0823] The server receives motion data sent from the terminal and uses analysis tools to evaluate the accuracy of the motion. It receives motion coordinates and velocity information as input data and processes this by comparing it with reference values. As output, it generates an evaluation result that shows how well the motion conforms to the specified training plan.

[0824] Step 3:

[0825] The device receives the analysis results and provides feedback to the user using visual means. Based on the evaluation results, instructions and points to note in augmented reality are displayed in the user's field of view. For example, arrows indicating the correct position for placing machine parts are visually displayed.

[0826] Step 4:

[0827] The server collects user activity data and progress information using aggregation methods. This collected data tracks the user's past performance and serves as material for optimizing future training plans. It is stored as input data in a database and later output for analysis.

[0828] Step 5:

[0829] The server's plan generation mechanism creates individual training plans using a generated AI model based on accumulated user data. It takes past training history and motion analysis results as input data and outputs the plan best suited to the user.

[0830] Step 6:

[0831] The device analyzes the user's voice tone and facial expressions using emotion analysis tools and collects emotional data. This data is sent from the device to the server. The server receives emotional characteristic data as input and outputs the user's current emotional state.

[0832] Step 7:

[0833] The server receives the emotion analysis results and dynamically adjusts the feedback and training content using adaptive mechanisms. If a specific emotional state is detected, it automatically generates prompt messages using a generative AI model and outputs appropriate feedback to the user. For example, if the user is feeling frustrated, a suggestion such as "Take a breath and calm down, then start with these steps" might be displayed.

[0834] (Application Example 2)

[0835] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0836] In today's world, personalized training and educational support are becoming increasingly important. However, traditional systems have struggled to adequately consider user emotions, making it difficult to optimize learning effectiveness. Furthermore, it has been challenging to create systems that provide immediate feedback on user errors and dynamic adjustments based on emotions.

[0837] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0838] In this invention, the server includes emotion analysis means for analyzing the user's emotional state and dynamically adjusting training feedback, language understanding means for generating responses by natural language processing, and control means for adjusting the difficulty level of training based on the user's emotions. This makes it possible to comprehensively consider the user's actions and emotions and provide individually optimized training.

[0839] A "detection device" is a device that observes a user's actions in real time and acquires action data.

[0840] "Analysis means" refers to functions or devices used to analyze data obtained from a detection device and evaluate the accuracy of its operation.

[0841] A "display means" refers to a function or device that provides information to the user visually as augmented reality based on the analyzed information.

[0842] A "memory device" is a device that stores user progress information and operation data for later analysis.

[0843] "Generation means" refers to functions or devices for creating personalized training plans based on data stored in a memory device.

[0844] A "training support means" refers to a function or device that presents a generated training plan to the user and supports the progress of the training.

[0845] "Emotional analysis means" refers to functions or devices that analyze a user's emotional state based on factors such as the user's voice tone and facial expressions.

[0846] A "language understanding means" refers to a function or device that receives voice input from a user and generates an appropriate response using natural language processing technology.

[0847] "Control means" refers to functions or devices that dynamically adjust the content and difficulty level of training based on the results of user emotion analysis.

[0848] To implement this invention, the user first puts on smart glasses. The smart glasses, which act as the terminal, are equipped with a detection device that monitors movement in real time and acquires user behavior data. This allows the user's movements to be captured in detail. The data detected includes, for example, hand movements and posture.

[0849] Next, the device analyzes the acquired motion data using an analysis tool and evaluates the accuracy of the movements. For this analysis, programs primarily written in Python and machine learning libraries such as TensorFlow are used. The analysis results are made available to the user through a display tool. For example, if the user is exercising, real-time feedback is provided on whether their movements are correct.

[0850] Furthermore, the server stores user activity data and progress information in its storage device, managing the training history. This ensures that the necessary foundational data is available to generate training plans tailored to individual users. Based on the accumulated data, a personalized training plan is created by the generation mechanism and provided to the user through the training support mechanism. This process is managed using the Django framework.

[0851] Furthermore, the system uses emotion analysis to evaluate the user's emotional state. Voice tone and facial expression data are processed by the emotion analysis engine. This allows the server to dynamically adjust training feedback to match the user's emotions. For example, if the system determines that the user is emotionally exhausted, it will slow down the training pace and suggest a break.

[0852] As a concrete example, if a user performing a fitness activity uses smart glasses to perform an exercise with incorrect form, the smart glasses will immediately inform them of this, demonstrate the correct form, and send encouraging messages to prevent a drop in motivation.

[0853] An example of a prompt for a generative AI model is: "Please tell me the steps required to implement an algorithm that analyzes the emotional state of users and applies appropriate feedback and training content based on that data."

[0854] In this way, users are supported in both behavioral and emotional aspects, allowing them to receive more effective training.

[0855] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0856] Step 1:

[0857] The device uses sensors to acquire user motion data. This motion data includes the user's hand position and movement speed. The input from the sensors is transmitted to the device as a digital signal and stored in a database in real time.

[0858] Step 2:

[0859] An analysis device on the terminal analyzes the acquired motion data. The input data is processed into various features, and the accuracy of the motion is evaluated by a machine learning model. As a result of the analysis, information is generated indicating whether the motion is appropriate or whether improvement is needed.

[0860] Step 3:

[0861] The device displays the analysis results to the user via an augmented reality display. The user receives visual feedback on the next steps and points to note. Additionally, a message of praise is displayed when the correct actions are taken.

[0862] Step 4:

[0863] The server stores user activity data and progress information in its storage device. This preserves training history and serves as foundational data for later analysis. This data is organized into different folders for each user.

[0864] Step 5:

[0865] The server generates a personalized training plan based on data stored in its memory. Input data includes past performance history and skill level information, and the output is a set of training steps optimized for the user. A generation AI model is used, and the plan generated by the AI ​​is sent to the terminal via a training support system.

[0866] Step 6:

[0867] The device uses an emotion analysis engine to collect emotional data from the user's voice tone and facial expressions. This data is processed as input indicating the user's fatigue level and stress level.

[0868] Step 7:

[0869] The server uses emotion analysis tools to analyze the user's emotional state. Based on the processed data, the server quantifies the user's emotional state and calculates how that state affects training.

[0870] Step 8:

[0871] The server dynamically adjusts training feedback based on the user's emotional state. Based on the analysis results, it generates suggestions to help the user relax and re-evaluates the plan to suit the user's emotional state. The prompt used for this is "Analyze the user's emotional state and apply feedback and training content based on that data."

[0872] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0873] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0874] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0875] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0876] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0877] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0878] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0879] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0880] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0881] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0882] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0883] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0884] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0885] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0886] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0887] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0888] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0889] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0890] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0891] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0892] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0893] The following is further disclosed regarding the embodiments described above.

[0894] (Claim 1)

[0895] Sensors for monitoring user behavior,

[0896] An analysis device that analyzes data from the aforementioned sensor and evaluates the accuracy of the operation,

[0897] A display device that provides information to the user in augmented reality based on the evaluation by the aforementioned analysis device,

[0898] A storage device for accumulating user progress information and evaluating skill levels,

[0899] A generation device that generates a customized training plan based on the evaluation of the storage device,

[0900] A system including a training support device for providing the generated training plan to the user.

[0901] (Claim 2)

[0902] The system according to claim 1, further comprising a language understanding device that receives voice input from a user and generates a response using natural language processing.

[0903] (Claim 3)

[0904] The system according to claim 1, further comprising an immediate feedback device for providing immediate corrective instructions when a user error is detected.

[0905] "Example 1"

[0906] (Claim 1)

[0907] Detection means for monitoring user behavior,

[0908] An evaluation means that analyzes information from the detection means to evaluate the accuracy of the operation,

[0909] A display means that provides information to the user in augmented reality based on the evaluation by the aforementioned evaluation means,

[0910] A means of recording user progress information and evaluating competence levels,

[0911] A generation means for generating a customized training plan based on the evaluation of the recording means,

[0912] Support means for providing the generated training plan to the user,

[0913] A natural language understanding means that allows users to input questions via voice while working, and obtain responses using natural language processing.

[0914] An immediate feedback mechanism that provides corrective instructions immediately when a user operation error is detected,

[0915] A system that includes this.

[0916] (Claim 2)

[0917] The system according to claim 1, further comprising language understanding means for receiving voice input and generating a response using natural language processing.

[0918] (Claim 3)

[0919] The system according to claim 1, further comprising auxiliary means for analyzing user movements and providing visual or audible instructions for the accuracy of the movements.

[0920] "Application Example 1"

[0921] (Claim 1)

[0922] A sensor device for monitoring user movements,

[0923] An analysis means for analyzing data from the aforementioned sensor device and evaluating the accuracy of its operation,

[0924] A display means that provides information to the user in augmented reality based on the evaluation by the aforementioned analysis means,

[0925] A means of accumulating user progress information and evaluating skill levels,

[0926] A generation means that generates a customized training plan based on the evaluation of the storage means,

[0927] Training support means for providing the generated training plan to the user,

[0928] When learning to operate machinery and equipment within a factory, a means of displaying procedures and feedback in real time is needed.

[0929] A system that includes this.

[0930] (Claim 2)

[0931] The system according to claim 1, further comprising a language understanding means for receiving voice input from a user and generating a response using natural language processing.

[0932] (Claim 3)

[0933] The system according to claim 1, further comprising immediate feedback means for providing immediate corrective instructions when a user error is detected.

[0934] "Example 2 of combining an emotion engine"

[0935] (Claim 1)

[0936] Sensor means for detecting user movements,

[0937] An analysis means that analyzes data from the aforementioned sensor means to evaluate the accuracy of the operation,

[0938] A visual presentation means that provides information to the user in augmented reality based on the evaluation by the aforementioned analysis means,

[0939] A means for collecting user progress information and evaluating competence levels,

[0940] A plan generation means that creates an individualized training plan based on the evaluation of the aforementioned accumulation means,

[0941] Training support means for providing the generated training plan to the user,

[0942] An emotion analysis method that analyzes the user's voice tone and facial expressions to evaluate their emotional state,

[0943] A system including adaptive means for dynamically adjusting feedback and training plans based on evaluations by the aforementioned emotion analysis means.

[0944] (Claim 2)

[0945] The system according to claim 1, further comprising a language understanding means for receiving voice input from a user and generating a response using natural language processing.

[0946] (Claim 3)

[0947] The system according to claim 1, further comprising immediate feedback means for providing immediate corrective instructions when a user error is detected.

[0948] "Application example 2 when combining with an emotional engine"

[0949] (Claim 1)

[0950] A detection device for monitoring user behavior,

[0951] An analysis means for analyzing data from the aforementioned detection device and evaluating the accuracy of its operation,

[0952] A display means that provides information to the user in augmented reality based on the evaluation by the aforementioned analysis means,

[0953] A storage device for accumulating user progress information and evaluating competence levels,

[0954] A generation means for generating an individualized training plan based on the evaluation of the storage device,

[0955] Training support means for providing the generated training plan to the user,

[0956] A system that includes emotion analysis means for analyzing the user's emotional state and dynamically adjusting training feedback.

[0957] (Claim 2)

[0958] The system according to claim 1, further comprising a language understanding means for receiving voice input from a user and generating a response using natural language processing, and an emotion analysis engine for analyzing the user's voice tone and facial expressions.

[0959] (Claim 3)

[0960] The system according to claim 1, further comprising: an immediate feedback means for providing immediate corrective instructions when a user error is detected; and a control means for adjusting the difficulty level of training based on the user's emotions. [Explanation of Symbols]

[0961] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Sensors for monitoring user behavior, An analysis device that analyzes data from the aforementioned sensor and evaluates the accuracy of the operation, A display device that provides information to the user in augmented reality based on the evaluation by the aforementioned analysis device, A storage device for accumulating user progress information and evaluating skill levels, A generation device that generates a customized training plan based on the evaluation of the storage device, A system including a training support device for providing the generated training plan to the user.

2. The system according to claim 1, further comprising a language understanding device that receives voice input from a user and generates a response using natural language processing.

3. The system according to claim 1, further comprising an immediate feedback device for providing immediate corrective instructions when a user error is detected.

Images