System

Abstract

A system is provided.SOLUTION: Means for automatically generating a moving image of a basic form of a aikido on the basis of a selection of a user by using a generation AI, means for displaying virtual instructor in a field of view of the user on the basis of the generated moving image by using an AR technology, means for providing sensory feedback of the user by using a sensory technology, and means for tracking a motion of the user in real time by using a motion sensor or a camera, AI, A system comprising: means for generating and transmitting assessment results; and means for providing assessment results received from a server at a terminal to a user to prompt modification based on feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In Aikido training, especially for advanced practitioners, the forms can become more complex, so it is important to practice while checking the correct form. However, with conventional methods, simply referring to a video can leave practitioners unsure of their accuracy and lack the bodily feedback they need during actual practice. Furthermore, self-evaluation is often difficult, and proper feedback from others is often unavailable. This makes it difficult to learn forms efficiently and accurately. [Means for solving the problem]

[0005] This invention provides a system that includes: a means for automatically generating a video of a basic Aikido kata based on a user's selection using a generation AI; a means for displaying a virtual instructor in the user's field of view based on the video generated using AR technology; a means for providing the user with tactile feedback using sensory technology; a means for tracking the user's movements in real time using a motion sensor or camera; a means for receiving the user's movement data on a server and evaluating the accuracy of the movements using an AI analysis engine, generating and transmitting evaluation results; and a means for providing the user with the evaluation results received from the server on a terminal and encouraging corrections based on the feedback. This allows for realistic practice that is similar to actual training, even from the comfort of your own home, and enables accurate evaluation of movements and continuous feedback for corrections. Furthermore, by providing a means for selecting the user's Aikido level and the type of kata they wish to learn, effective training tailored to individual needs is realized.

[0006] "Generative AI" is an artificial intelligence technology that automatically creates specific content or data based on user input.

[0007] "AR technology" is a technology that displays virtual information overlaid on the real environment, providing users with an experience that blends reality and virtuality.

[0008] "Sensory technology" is a technology that provides users with physical sensations and feedback, giving them a sense of realism through touch and vibration.

[0009] A "motion sensor" is a sensor device that detects human movement and captures it as data in real time.

[0010] A "camera" is a device that captures optical data and outputs that data as an image.

[0011] A "server" is a computer system that processes and manages data over a network and provides services to other devices and systems.

[0012] "Terminal" refers to a device or system that can be directly operated by a user, and is a device that inputs and outputs data.

[0013] An "AI analytics engine" is software or a system that uses artificial intelligence to analyze data and derive results based on a specific purpose.

[0014] A "virtual instructor" is a computer-generated person or character whose role is to provide instruction and guidance to the user.

[0015] A "haptic device" is a device used to provide tactile feedback, recreating physical sensations to a user.

[0016] "Tracking" is a technology that tracks the movements of objects and people and records their positions and actions in real time.

[0017] "Evaluation results" are conclusions or feedback information obtained after analysis or review based on specific criteria.

[0018] "Feedback" refers to providing a response or corrective instructions to a user's actions or behavior, and is a means of encouraging improvement and enhancement.

[0019] "Kata" is a term that refers to the patterns of basic movements and techniques in Aikido.

[0020] Aikido is a traditional Japanese martial art that includes techniques and a philosophy aimed at self-defense and training the mind and body. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] ---

[0043] The present invention is a system that combines generative AI, AR technology, sensory technology, and other technologies to allow users to practice Aikido at home. Specific embodiments for implementing this system will be described below.

[0044] First, the user starts the application and selects their Aikido level and the type of kata they want to learn. At this time, the terminal accepts the user's input and sends the selection information to the server.

[0045] The server uses generative AI to automatically generate a video of a basic Aikido kata based on the kata selected by the user. The generated video is stored on the server and sent to the user's device.

[0046] The device then uses the received video to display a virtual instructor in the user's field of view using AR technology, allowing the user to practice with the virtual instructor in a real practice space. Additionally, a haptic device is provided using sensory technology to provide feedback on the user's movements, allowing the user to continue practicing while experiencing the accuracy of their movements.

[0047] When a user begins practicing, the device's motion sensor or camera tracks the user's movements in real time. This data is sent from the device to the server. The server then analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The evaluation results are then sent back to the device.

[0048] Based on the evaluation results received from the server, the device provides the user with visual and tactile feedback on the accuracy of their movements and areas for improvement. By correcting their movements and practicing again, the user can master the kata accurately.

[0049] For example, when a user practices the advanced kata of Katatori Ikkyo, the server generates a detailed video of the kata and sends it to the device. The user visually watches the virtual instructor's movements through the goggles and performs the movements in sync with them. If there is a misalignment in the movements, the device's haptic device vibrates specific parts of the body, providing feedback on what needs to be corrected. The server's AI analysis engine then analyzes the user's movements in detail and suggests specific corrections.

[0050] This system allows users to have an experience similar to that of a real classroom, even from the comfort of their own home, and aims to accurately master techniques. It also provides custom feedback based on the user's level and the form they select, enabling efficient practice.

[0051] As described above, the present invention provides a system for efficiently and accurately practicing Aikido, and provides the user with a realistic training environment and accurate feedback.

[0052] The processing flow will be explained below.

[0053] ---

[0054] Step 1:

[0055] The user launches the application and fills in the interface to select their Aikido level (beginner, intermediate, advanced) and the type of kata they wish to learn.

[0056] Step 2:

[0057] The terminal receives the user's input information and transmits the selected data to the server.

[0058] Step 3:

[0059] Based on the information received by the server, a generation AI is used to automatically generate a video of the basic Aikido form specified by the user. The generated video is then saved on the server.

[0060] Step 4:

[0061] The server transmits the generated video to the user's terminal.

[0062] Step 5:

[0063] The device receives the video and starts the AR engine to prepare to display the virtual instructor in the user's field of view.

[0064] Step 6:

[0065] The device initializes the haptic device and configures it to provide tactile feedback in response to the user's movements.

[0066] Step 7:

[0067] The user puts on the goggles and sensory device and confirms they are ready.

[0068] Step 8:

[0069] When the user presses the start button, the terminal receives the start signal and the virtual instructor appears in the user's field of vision.

[0070] Step 9:

[0071] The virtual instructor initiates the selected form of movement and the user replicates the same movement.

[0072] Step 10:

[0073] The device uses motion sensors or a camera to track the user's movements in real time.

[0074] Step 11:

[0075] The device collects data on the user's movements and sends it to the server.

[0076] Step 12:

[0077] The tracking data received by the server is analyzed using an AI analysis engine.

[0078] Step 13:

[0079] The server evaluates the accuracy of the user's movements based on the analysis results and identifies areas for improvement. The evaluation results are stored on the server.

[0080] Step 14:

[0081] The server sends the evaluation results and correction instructions to the user's terminal.

[0082] Step 15:

[0083] The terminal receives the evaluation results from the server and provides visual and tactile feedback to the user.

[0084] Step 16:

[0085] The user practices the form again while correcting their movements based on the feedback.

[0086] Step 17:

[0087] The device again tracks the user's movements and sends them to the server, and this process is repeated until the user's movements are accurate.

[0088] ---

[0089] These are the specific processing steps of the program. This process allows the user to receive feedback on their movements and accurately correct them, allowing them to master Aikido forms.

[0090] Example 1

[0091] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0092] Traditional Aikido training required a dedicated dojo and instructor, making it difficult to practice efficiently and accurately at home. Furthermore, there were limited ways to check whether one's movements were accurate, making self-improvement difficult. This created a major hurdle for beginners and intermediates to master accurate techniques.

[0093] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0094] In this invention, the server includes means for automatically generating videos of basic Aikido forms based on user selections using a generative AI model, means for displaying a virtual instructor in the user's field of view based on the generated videos using AR technology, and means for providing the user with tactile feedback using sensory technology, thereby enabling users to efficiently practice Aikido from the comfort of their own homes while receiving accurate feedback in real time.

[0095] A "generative AI model" is an artificial intelligence algorithm that automatically generates information based on data for a specific task.

[0096] A "prompt sentence" is text-based input data used to give instructions to a generative AI model.

[0097] "AR technology" is a technology that displays digital information superimposed on the real world.

[0098] "Sensory technology" is technology that provides users with visual, tactile, or other feedback.

[0099] A "haptic device" is a device that conveys tactile feedback to a user.

[0100] A "server" is a computer system that processes and stores data on a network.

[0101] A "terminal" is a device that a user operates to input and output data.

[0102] "Means for tracking user movements in real time" refers to a function that uses motion sensors and cameras to capture user movements sequentially.

[0103] An "AI analysis engine" is a system that uses AI technology to analyze input data and provide specific insights and evaluations.

[0104] "Means for assessing the accuracy of movements" refers to a method for measuring the accuracy of a user's movements using an AI analysis engine.

[0105] "Means for providing feedback" refers to a function that provides visual and tactile information to the user, informing them of areas for improvement in their actions.

[0106] The present invention relates to a system that combines generative AI models, AR technology, sensory technology, and other technologies to enable users to practice Aikido at home. Specific embodiments for implementing this system are described below.

[0107] The user launches the smartphone application and selects their Aikido level and the type of kata they wish to learn. For example, an intermediate practitioner selects "Intermediate" and the kata "Shouldering Icchi." At this point, the device accepts the selection information and sends it to the server. The information is sent in JSON format via the REST API.

[0108] The server automatically generates a video of the basic Aikido kata based on the kata selected by the user using a generative AI model (e.g., OpenAI's GPT-4). Specifically, the following prompt sentence is input to the generative AI model:

[0109] Example prompt sentence:

[0110] "Create an instructional video with detailed movements for the intermediate Katatori Ikkyo technique. Include a complete explanation of the steps, grips, and throws."

[0111] The server saves the generated video in cloud storage (e.g., AWS S3) and sends the URL of the video to the device.

[0112] The device uses the received video URL to stream the video and uses AR technology (e.g., Unity or ARKit) to display the virtual instructor in the user's field of view, allowing the user to practice with the virtual instructor using their smartphone or AR goggles.

[0113] When a user begins practicing, the device's motion sensors and camera track the user's movements in real time. The device then sends the data obtained from these sensors to a server. The server then analyzes the received tracking data using an AI analysis engine (e.g., TensorFlow or PyTorch) and evaluates the user's movements. The evaluation results are fed back as specific corrections and sent back to the device.

[0114] The device provides the user with visual and tactile feedback based on the evaluation results received from the server. For example, the device may display feedback such as "your right hand is positioned too low" on the screen and vibrate the user's right hand using a haptic device. In this way, the user can correct their movements as they practice.

[0115] As described above, the present invention allows users to experience a similar experience to practicing in a real classroom from the comfort of their own home. It provides an efficient training method aimed at mastering accurate techniques, and provides users with a realistic practice environment and accurate feedback.

[0116] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0117] Step 1:

[0118] The user launches the application on their smartphone and is presented with a home screen to begin Aikido practice. The user taps the "Start Practice" button and selects their Aikido level and the kata they wish to learn. The user's level and kata selection information are obtained as input. The selection information is sent from the device to the server as output.

[0119] Step 2:

[0120] The device sends the user's selected level and type information to the server. During this process, it receives the user's selection information as input and sends it to the server in JSON format via the REST API. The output returns a successful status.

[0121] Step 3:

[0122] Based on the received data, the server uses a generative AI model to automatically generate a video of the basic Aikido forms. As input, the user's selection information and a prompt sentence are provided to the generative AI model. Specifically, the following prompt sentence is used:

[0123] "Create an instructional video with detailed movements for the intermediate Katatori Ikkyo technique. Include a complete explanation of the steps, grips, and throws."

[0124] The output is a video generated by the generative AI model, which the server stores in cloud storage (e.g., AWS S3).

[0125] Step 4:

[0126] The server sends the URL of the stored video to the device. As input, the video URL is retrieved from the cloud storage and sent to the user's device. As output, the video URL is provided to the device.

[0127] Step 5:

[0128] The device streams the video using the received video URL. As input, it receives the video URL and starts the streaming process. It uses AR technology (e.g., Unity or ARKit) to display the virtual instructor in the user's field of view. As output, it displays the virtual instructor in the user's field of view.

[0129] Step 6:

[0130] The user begins practicing while watching the virtual instructor's movements displayed in AR. As input, the user sees the virtual instructor's movements and synchronizes their own movements. As output, the user's movements are tracked in real time.

[0131] Step 7:

[0132] The device's motion sensors and camera track the user's movements in real time. As input, user movement data is obtained from the motion sensors and camera. As output, tracking data is generated.

[0133] Step 8:

[0134] The device sends the captured tracking data to the server. As input, it receives the acquired tracking data and sends it to the server. As output, it returns a status indicating that the transmission was successful.

[0135] Step 9:

[0136] The server analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The tracking data is provided to the AI ​​analysis engine as input. The movement evaluation results are obtained as output. The evaluation results include the accuracy of the user's movements and areas for improvement.

[0137] Step 10:

[0138] The server sends the motion evaluation results to the terminal. As input, the evaluation results are generated and sent to the terminal. As output, the evaluation results are provided to the terminal.

[0139] Step 11:

[0140] The device provides visual and tactile feedback to the user based on the evaluation results received from the server. As input, it receives the evaluation results and presents them to the user. For example, feedback such as "your right hand is positioned low" is displayed on the screen, and the haptic device vibrates the right hand. This allows the user to correct their own movements. As output, feedback information is generated and provided to the user.

[0141] Step 12:

[0142] The user corrects their movements based on the provided feedback and continues practicing again. As input, they receive the feedback and correct their movements. As output, the correct movements are gradually learned.

[0143] (Application example 1)

[0144] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0145] It is difficult for new employees and inexperienced personnel to quickly learn how to operate robots in factories or on production lines. Conventional training methods require training using actual machines, which is costly and time-consuming. Furthermore, it requires a lot of effort from the instructor, making efficient training difficult. The present invention solves these problems by providing a training system that allows employees to efficiently and effectively learn how to operate robots.

[0146] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0147] In this invention, the server includes: means for automatically generating a video of a basic form of a work procedure based on a user's selection using a generation AI; means for displaying a virtual instructor in the user's field of view based on the generated video using AR technology; means for providing the user with tactile feedback using haptic technology; means for tracking the user's movements in real time using a motion sensor or a camera; means for receiving data on the user's movements at the server, evaluating the accuracy of the movements using an AI analysis engine, and generating and transmitting the evaluation results; means for providing the user with the evaluation results received from the server at a terminal and encouraging corrections based on the feedback; and means for supporting training for new employees using virtual instructor display technology for work training, thereby enabling new employees and inexperienced people to quickly learn robot operation.

[0148] "Generative AI" is an artificial intelligence technology that automatically generates videos of actions and work procedures based on user input.

[0149] "AR technology" is an abbreviation for augmented reality technology, which is a technology that displays virtual information overlaid on the real field of view.

[0150] "Haptic technology" refers to technology for providing tactile feedback to users, particularly technology that uses haptic devices.

[0151] A "motion sensor" is a sensor that detects the user's physical movements and acquires them as digital data in real time.

[0152] A "camera" is a device that captures a user's actions as video data and analyzes it.

[0153] The "AI analysis engine" is an artificial intelligence technology that analyzes acquired user movement data and evaluates the accuracy of the movements.

[0154] A "virtual instructor" is a virtual teacher who is displayed in the user's field of view using AR technology.

[0155] "Terminal" refers to the device used by the user for operation and viewing, and is the device that communicates with the server and provides feedback.

[0156] The "training support system" is an integrated system that enables new employees and inexperienced personnel to efficiently learn how to operate robots and other work procedures.

[0157] A "work procedure" is a series of steps or operating methods for performing a specific task.

[0158] MODE FOR CARRYING OUT THE INVENTION

[0159] This invention relates to a robot operation training system for new factory employees. Specifically, it provides an efficient training method using generative AI, AR technology, and haptic technology.

[0160] First, the user starts the training. The user puts on a device (e.g., a smart headset) and launches the application. First, the user is shown a screen where they can select their skill level and the type of work procedure they want to learn. The device accepts the user's input and sends the selection information to the server.

[0161] Next, the server uses a generative AI to automatically generate a basic video based on the work steps selected by the user. The generated video is stored on the server and sent to the user's device. The server uses an AI analysis engine to do this.

[0162] The received video is played on the device, and a virtual instructor is displayed in the user's field of view using AR technology. The user follows the virtual instructor's instructions and begins training to operate the robot. The virtual instructor explains the user's movements in real time and provides visual guidance.

[0163] When a user actually performs an action, a motion sensor or camera tracks the user's movement in real time. This tracking data is sent from the device to a server. The server analyzes the received tracking data using an AI analysis engine and evaluates the user's movement.

[0164] The evaluation results are sent back to the device from the server, and the device then provides feedback to the user. Feedback can be visual or tactile, with tactile feedback provided using a haptic device. Specifically, a haptic glove worn on the user's hand vibrates when the user has performed an accurate movement or when a correction is required.

[0165] This allows users to efficiently train together with the virtual instructor while experiencing the accuracy of their movements in real time. For example, if a user is learning how to "install a part," the server generates a detailed video of the procedure and sends it to the device. The user can visually confirm the virtual instructor's movements through the headset and perform the appropriate movements. If there is any deviation in the movements, the haptic device will vibrate specific parts of the body, providing feedback on what needs to be corrected. The server's AI analysis engine then analyzes the user's movements in detail and suggests specific corrections.

[0166] This system allows new employees and those with no experience to quickly learn how to operate robots, ensuring efficient training.

[0167] Generative AI model prompt example:

[0168] "We are building a training program for new employees to learn the operating procedures for a robotic assembly task. The operating procedures need to be generated as a series of videos."

[0169] The above is an embodiment of the present invention.

[0170] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0171] Step 1:

[0172] The user launches the application on the device. A screen appears in which the user can select their skill level and the type of work procedure they wish to learn. The user's selection (input data) is sent from the device to the server. The input is the user's level and work procedure, and the output is the selection information sent to the server.

[0173] Step 2:

[0174] The server generates a prompt for the generative AI model based on the received selection information. Based on this prompt, the generative AI generates a basic video corresponding to the user's selection. The input is the user's selection information, and the output is a video of the generated work procedure.

[0175] Step 3:

[0176] The server sends the generated video to the terminal. The input is the generated video, and the output is the transmission of video data to the terminal.

[0177] Step 4:

[0178] The device plays the received video and uses AR technology to display the virtual instructor in the user's field of view. The user receives visual guidance through the device's display. The input is the video data, and the output is the display of the virtual instructor.

[0179] Step 5:

[0180] The user follows the instructions of the virtual instructor and begins training. The user's movements are tracked in real time by a motion sensor or camera. The input is the user's movements, and the output is the tracking data.

[0181] Step 6:

[0182] The terminal sends tracking data to the server. The input is tracking data, and the output is data transmission to the server.

[0183] Step 7:

[0184] The server analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The result is an evaluation of the accuracy of the movements and points to be corrected, which is then sent to the device. The input is the tracking data, and the output is the evaluation result.

[0185] Step 8:

[0186] The terminal provides the user with the evaluation results received from the server. The terminal displays visual feedback and simultaneously sends instructions to the haptic device to provide tactile feedback. The input is the evaluation results, and the output is visual and tactile feedback.

[0187] Step 9:

[0188] The user then corrects their movements based on the feedback and repeats the training. Through this feedback cycle, the user gradually acquires accurate movements. The input is visual and tactile feedback, and the output is the user's corrected movements.

[0189] The above is a specific explanation of each processing step of the program.

[0190] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0191] ---

[0192] The present invention is a system that combines generative AI, AR technology, sensory technology, an emotion engine, and other technologies to enable users to practice Aikido at home. A specific embodiment for implementing this system will be described.

[0193] First, the user starts the application and selects their Aikido level and the type of kata they want to learn. At this time, the terminal accepts the user's input and sends the selection information to the server.

[0194] The server uses generative AI to automatically generate a video of a basic Aikido kata based on the kata selected by the user. The generated video is stored on the server and sent to the user's device.

[0195] The device then uses the received video to display a virtual instructor in the user's field of view using AR technology, allowing the user to practice with the virtual instructor in a real practice space. Additionally, a haptic device is provided using sensory technology to provide feedback on the user's movements, allowing the user to continue practicing while experiencing the accuracy of their movements.

[0196] When a user begins practicing, the device's motion sensor or camera tracks the user's movements in real time. This data is sent from the device to the server. The server then analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The evaluation results are then sent back to the device.

[0197] Based on the evaluation results received from the server, the device provides the user with visual and tactile feedback on the accuracy of their movements and areas for improvement. By correcting their movements and practicing again, the user can master the kata accurately.

[0198] Furthermore, the present invention incorporates an emotion engine that recognizes emotions from the user's facial expressions and voice. The device uses the emotion engine to determine the user's emotions during training in real time and adjust the content and difficulty of the training accordingly. For example, if the user is feeling frustrated, the device sends that information to the server, and the server generates a training plan based on the user's emotional state and sends it to the device.

[0199] For example, when a user practices the advanced kata, Katatori Ikkyo, the server generates a detailed video of the kata and sends it to the device. The user watches the virtual instructor's movements and replicates them. A motion sensor tracks the user's movements and sends the data to the server. The server's AI analysis engine analyzes the user's movements and generates feedback. At the same time, the emotion engine recognizes the user's face and analyzes their voice to determine their emotional state. If the user is feeling frustrated or stressed, the server adjusts the practice and provides advice on how to relax.

[0200] In this way, the present invention provides a system for efficient and accurate Aikido training, not only providing the user with a realistic training environment and accurate feedback, but also providing support according to the user's emotional state.

[0201] The processing flow will be explained below.

[0202] ---

[0203] Step 1:

[0204] The user starts the application and selects their Aikido level (beginner, intermediate, advanced) and the type of kata they want to learn. By entering their input into the interface, the selection information is sent to the terminal.

[0205] Step 2:

[0206] The terminal receives the user's input information and transmits this information to the server.

[0207] Step 3:

[0208] Based on the information received by the server, a generation AI is used to automatically generate a video of the basic Aikido form specified by the user. The generated video is saved on the server.

[0209] Step 4:

[0210] The server transmits the generated video to the user's terminal.

[0211] Step 5:

[0212] The device receives the video, starts the AR engine, and prepares to display the virtual instructor in the user's field of view. At the same time, it initializes the haptics device and configures it to provide tactile feedback according to the user's movements.

[0213] Step 6:

[0214] The device displays a message prompting the user to put on the goggles and sensory device, and waits until they are ready.

[0215] Step 7:

[0216] The user puts on the goggles and sensory device and confirms that they are ready. The user presses the start button.

[0217] Step 8:

[0218] The terminal receives the start signal, and the virtual instructor appears in the user's field of view. The virtual instructor begins the selected form of movement, and the user performs the movement accordingly.

[0219] Step 9:

[0220] The device uses motion sensors or a camera to track the user's movements in real time.

[0221] Step 10:

[0222] The device collects tracking data and sends it to the server. At the same time, the device's emotion engine analyzes the user's facial expressions and voice and sends the results to the server.

[0223] Step 11:

[0224] The tracking data received by the server is analyzed using an AI analysis engine to evaluate the accuracy of the user's movements, and the evaluation results are stored on the server.

[0225] Step 12:

[0226] The server analyzes the received emotional data to determine the user's emotional state. If the user is feeling frustrated or stressed, the content and difficulty of the training will be adjusted based on that information.

[0227] Step 13:

[0228] The server generates feedback and correction instructions appropriate for the user based on the evaluation results and emotional state, and sends them to the terminal.

[0229] Step 14:

[0230] The device receives feedback and correction instructions from the server and provides them to the user visually and tactilely, automatically adjusting the difficulty of the training as needed and suggesting relaxation techniques.

[0231] Step 15:

[0232] Based on the feedback provided by the user, the user practices the form again, correcting the movements.

[0233] Step 16:

[0234] The device again tracks the user's movements and sends the data to the server, and this process is repeated until the user's movements are accurate and emotionally stable.

[0235] ---

[0236] These are the specific processing steps of the program. This process allows the user to receive feedback on their movements and accurately correct them, allowing them to master Aikido forms. Furthermore, the real-time emotional feedback provided by the emotion engine can further improve the quality of training.

[0237] Example 2

[0238] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0239] In recent years, there has been a growing demand for methods to efficiently learn Aikido at home. Conventional methods require training at a dojo or rely on books and video materials, which require the guidance of an instructor. Furthermore, self-practice is difficult because it is difficult to understand and experience the correct movements, making it difficult to practice efficiently. Furthermore, there is also the issue of difficulty in maintaining motivation due to insufficient feedback and emotional awareness during self-practice. There is an urgent need to solve these problems and provide a system that allows users to effectively and accurately learn Aikido at home.

[0240] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0241] In this invention, the server includes means for automatically generating videos of basic Aikido forms based on a user's selection using a generation AI, means for displaying a virtual instructor in the user's field of view based on the generated videos using augmented reality technology, means for providing the user with tactile feedback using sensory technology, means for tracking the user's movements in real time using a motion detector or camera, means for evaluating the accuracy of the movements using an AI analysis engine and generating and transmitting the evaluation results, and means for analyzing the user's facial expressions and voice in real time and adjusting the training content and difficulty based on the user's emotional state. This allows users to receive a realistic training environment and accurate feedback even at home, enabling them to accurately master the movements.

[0242] "Generative AI" refers to a system that uses artificial intelligence technology to automatically generate the necessary content based on user selections.

[0243] "Augmented reality technology" is a technology that overlays computer-generated information onto real-world images and videos.

[0244] "Sensory technology" refers to technology that allows users to obtain information through their senses, such as sight, hearing, and touch.

[0245] A "motion detector" is a device that detects a user's movements in real time and acquires them as data.

[0246] An "artificial intelligence analysis engine" is an artificial intelligence system that analyzes collected data and generates appropriate feedback and actions based on the results.

[0247] "Feedback" refers to information provided to the user by the system regarding corrections and areas for improvement.

[0248] "Real-time tracking" refers to a technology that tracks a user's movements instantly without any time delay and processes the data.

[0249] A "virtual instructor" is a virtual instructor who appears in the user's field of view through AR technology.

[0250] "Adjusting training content" refers to changing the difficulty and content of the training menu based on the user's movements and emotional state.

[0251] "User's emotional state" refers to the emotions (joy, frustration, etc.) that the user is feeling during training.

[0252] The present invention is a system that combines generative AI, augmented reality technology, sensory technology, an artificial intelligence analysis engine, and emotion recognition technology to enable users to effectively practice Aikido at home. Specific embodiments of this system are described below.

[0253] First, the user launches the application on a device such as a smartphone or tablet and selects their Aikido level and the kata they wish to learn. The device then sends this selection information to the server, using the device's input device and communication module.

[0254] Based on the received information, the server uses a generative AI model (e.g., OpenAI's GPT-4) to automatically generate an instructional video for a basic Aikido form. An example of a specific prompt used in the generation process is, "Please generate a detailed video about the basic Aikido form of Shoulder Drill Ikkyo." The generated video is stored on the server and transmitted to the terminal using data streaming technology (e.g., RTMP protocol).

[0255] The device plays the received video and uses augmented reality technology (e.g., ARKit) to display the virtual instructor in the user's field of view, allowing the user to practice with the virtual instructor in the real practice space.

[0256] When a user starts practicing, the device's motion sensors and camera track the user's movements in real time, and this data is sent from the device to the server. The server analyzes the received tracking data using an artificial intelligence analysis engine (e.g., TensorFlow) to accurately evaluate the user's movements. This evaluation result is then sent back to the device.

[0257] Based on the evaluation results, the device provides the user with visual and tactile feedback on the accuracy of their movements and areas for improvement. A haptic device (for example, a wearable device with vibration) is provided that uses sensory technology to provide feedback on the user's movements. Specific feedback might include, "It would be better if you moved your right arm a little more to the left." The user can then correct their movements and continue practicing.

[0258] Furthermore, the device uses an emotion engine (e.g., Affectiva's emotion recognition API) to analyze the user's emotional state in real time. It analyzes the user's facial expressions and voice through the camera and microphone to determine emotions such as joy or frustration. If the user is feeling frustrated, the device sends this information to the server, which then generates a training plan based on the user's emotional state and sends it to the device. For example, it might display a guide message such as "Relax and take a deep breath" and temporarily reduce the difficulty of the exercises.

[0259] This system allows users to practice at home in a realistic environment and receive accurate feedback, enabling them to accurately master movements.

[0260] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0261] Step 1:

[0262] The user launches the application and selects their Aikido level and the kata they wish to practice.

[0263] Input: Aikido level and kata type selected by user operation.

[0264] Output: JSON data containing the selection information.

[0265] Specific operation: The user selects "Advanced Form" or "Shoulder Training" from the menu on the device screen and presses the OK button. The selected information is converted into JSON format and passed to the next process.

[0266] Step 2:

[0267] The terminal transmits the selection information to the server.

[0268] Input: JSON data containing the user's selections.

[0269] Output: HTTPS request sent to the server.

[0270] Specific operation: The device generates JSON data of the selected information and sends it to the server using the HTTPS protocol. Specifically, it sends a POST request to the endpoint URL.

[0271] Step 3:

[0272] The server uses the generative AI model to generate videos of basic Aikido forms and sends them to the device.

[0273] Input: JSON data containing the user's selections.

[0274] Output: Generated video data of Aikido basic forms.

[0275] Specific operation: Based on the received JSON data, the server inputs the prompt "Please generate a detailed video about the basic form of Aikido's Shoulder Pulling Ikkyo" into the generative AI model. The AI ​​model generates the video and stores it on the server. The generated video data is sent to the terminal via the RTMP protocol.

[0276] Step 4:

[0277] The device displays the received video and displays the virtual instructor using AR technology.

[0278] Input: Generated video data of Aikido basic forms.

[0279] Output: Virtual instructor display.

[0280] Specific operation: The device plays the received video data and uses ARKit to display the virtual instructor in the user's field of view. Specifically, a 3D model of the virtual instructor is overlaid on the device's camera image.

[0281] Step 5:

[0282] The user begins practicing and the device tracks the user's movements.

[0283] Input: User movement.

[0284] Output: Tracking data.

[0285] Specific movements: The user watches the movements of the virtual instructor and then reproduces them. The device's motion sensors and camera capture the user's movements in real time, obtaining coordinate data and movement speed data.

[0286] Step 6:

[0287] The device sends the tracking data to the server.

[0288] Input: Tracking data.

[0289] Output: Sends a WebSocket request to the server.

[0290] What it does: The device converts the acquired tracking data into JSON format and sends it to the server using the secure WebSocket protocol.

[0291] Step 7:

[0292] The server analyzes the tracking data, generates feedback, and sends it to the device.

[0293] Input: Tracking data.

[0294] Output: Feedback data.

[0295] Specific operation: The server analyzes the received tracking data using an artificial intelligence analysis engine (e.g., TensorFlow). As a result of the analysis, it evaluates the accuracy of the user's movements and areas for improvement, and generates specific feedback such as "It would be better if you moved your right arm a little more to the left." This feedback data is sent to the device in JSON format.

[0296] Step 8:

[0297] The device provides feedback to the user and instructs them to correct their movements.

[0298] Input: Feedback data.

[0299] Output: Visual and tactile feedback.

[0300] Specific actions: The device reflects the received feedback data on the display screen, provides specific correction instructions to the user, and controls the haptic device to provide feedback to the user's physical sensations.

[0301] Step 9:

[0302] The user corrects the movement and repeats the practice again.

[0303] Input: Feedback information.

[0304] Output: Corrected movement.

[0305] Specific movements: The user corrects their movements based on the feedback and repeats the practice while watching the virtual instructor's movements again.

[0306] Step 10:

[0307] The device uses an emotion engine to analyze the user's emotions.

[0308] Input: User's facial and voice data.

[0309] Output: Emotional state data.

[0310] How it works: The device uses a camera and microphone to capture the user's facial expressions and voice, then analyzes them using an emotion engine to determine a specific emotional state (e.g., joy, frustration) in real time.

[0311] Step 11:

[0312] The server adjusts the training content based on the emotional data and sends it to the device.

[0313] Input: Emotional state data.

[0314] Output: A tailored training plan.

[0315] Specific operation: The server analyzes the emotional state data sent from the device and generates a training plan according to the user's emotions. Specifically, if the user feels frustrated, the server generates a plan that lowers the difficulty of the training or includes a guide message such as "Relax and take a deep breath." This plan is sent to the device and applied to the user.

[0316] (Application example 2)

[0317] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0318] Conventional online training and shopping systems have the problem that it is difficult for users to get the same experience as a real instructor or salesperson at home. In particular, the user experience is poor due to the lack of accurate motion feedback, reproduction of the product touch feeling, and appropriate responses based on the user's emotional state. This leads to issues such as a decrease in user learning and purchasing motivation, and a low overall user satisfaction.

[0319] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for automatically generating a video of a basic Aikido form based on a user's selection using a generation AI, means for displaying a virtual instructor in the user's field of view using AR technology, means for providing the user with haptic feedback using sensory technology, means for tracking the user's movements in real time using a motion sensor or a camera, means for receiving data on the user's movements in the server, evaluating the accuracy of the movements using an AI analysis engine, and generating and transmitting the evaluation results, means for providing the user with the evaluation results received from the server at the terminal and prompting the user to make corrections based on the feedback, means for generating a product explanation video using a generation AI, means for displaying a virtual assistant in the field of view using AR technology, means for tracking the user's line of sight and movements using a motion sensor, analyzing the data, and providing feedback, means for evaluating the user's emotional state using an emotion engine and dynamically adjusting the training content, and means for providing tactile feedback of the product's texture using a haptic device. This allows users to have a realistic practice and shopping experience at home, making it easier to accurately learn movements and experience the texture of products, and by allowing the system to respond appropriately based on the user's emotional state, it is possible to improve overall user satisfaction.

[0320] "Generative AI" is a type of artificial intelligence technology that automatically generates necessary content based on user specifications.

[0321] "AR technology" stands for augmented reality technology, which is a technology that displays digital content overlaid on the real world.

[0322] "Sensory technology" refers to technology that provides feedback to the user's senses, and specifically includes feedback to the sense of touch and vision.

[0323] A "motion sensor" is a device that detects and tracks a user's movements.

[0324] A "camera" is a device that captures video and analyzes the user's movements based on that video.

[0325] A "server" is a computer system connected to a network that processes and stores various types of data.

[0326] An "AI analysis engine" is a system that uses artificial intelligence technology to analyze collected data and make specific evaluations and predictions.

[0327] A "virtual instructor" is a digital instructor that appears in the user's field of vision using AR technology and is responsible for providing training guidance.

[0328] "Feedback" is information for guidance and correction that the system provides based on the user's movements and state.

[0329] A "haptic device" is a device for providing tactile feedback to a user.

[0330] The "emotion engine" is an artificial intelligence technology that detects the user's emotional state from their facial expressions and voice, and adjusts their behavior based on that.

[0331] A "product description video" is a video containing detailed product explanations generated using AI.

[0332] A "virtual assistant" is a digital character that appears in the user's field of vision using AR technology and serves to support and guide the user.

[0333] The present invention is a system that fully utilizes generative AI, AR technology, sensory technology, emotion engine, and haptics technology to enable users to have an online learning or shopping experience at home. Specific embodiments for implementing this system are described below.

[0334] Hardware and Software

[0335] Smart glasses: Uses AR technology to display virtual instructors and virtual assistants.

[0336] Generative AI models (e.g., GPT-4): Generate video or text content based on user selections.

[0337] Motion sensors (e.g. Kinect): Track user movements in real time.

[0338] Haptic devices (e.g., SenseGlove): Provide tactile feedback to the user.

[0339] Server: Responsible for data processing and storage, and equipped with an AI analysis engine and emotion engine.

[0340] Emotion engine (e.g. Microsoft Azure Emotion API): Analyzes the user's emotional state.

[0341] System Operation

[0342] First, the user launches the application and selects information about Aikido training or products. By entering user information, the form they want to learn, and categories of interest, the following steps are carried out.

[0343] Video generation using generative AI

[0344] The server uses a generative AI model (e.g., GPT-4) to automatically generate a basic Aikido form or a product instruction video based on the user's selection. For example, input a prompt such as, "Please explain the features and functions of a new smartphone in a video of no more than 3 minutes."

[0345] AR display

[0346] Through smart glasses, a virtual instructor or virtual assistant is displayed in the user's field of view based on videos generated by AR technology, providing an immersive training or shopping experience.

[0347] Motion sensor tracking

[0348] When a user starts practicing, a motion sensor (e.g., Kinect) tracks the user's movements in real time. This data is sent to a server, where an AI analysis engine evaluates the accuracy of the movements.

[0349] feedback

[0350] The evaluation results are sent from the server to the device and provided to the user. The user can then modify their movements based on this feedback. Tactile feedback is also provided using a haptic device (e.g., SenseGlove).

[0351] Emotion analysis using an emotion engine

[0352] An emotion engine (e.g., Microsoft Azure Emotion API) analyzes the user's emotional state from their facial expressions and voice and dynamically adjusts the training or shopping experience, for example, using prompts such as, "Please rate the user's emotional state (interested, confused, etc.) from their facial expressions and voice while they are watching the smartphone introductory video."

[0353] Specific examples

[0354] 1. If a user wants to practice advanced shoulder-grab techniques, the server generates a detailed video and sends it to the device. The user can then visualize the virtual instructor's movements and replicate them.

[0355] 2. In the case of shopping, when a user is looking for a new smartphone, generative AI will generate a detailed product video, and AR will bring a virtual shopping assistant into view. An emotion engine will read the user's interests and confusions and make appropriate suggestions.

[0356] In this way, the present invention provides a user with a high sense of realism and appropriate feedback, and responds to individual emotional states, thereby providing a more satisfying experience.

[0357] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0358] Step 1:

[0359] The user starts the application and selects information about Aikido training or products. They enter user information, the forms they want to learn, and categories they are interested in. The entered information is sent from the device to the server.

[0360] Step 2:

[0361] The server uses a generative AI model to generate a basic Aikido kata or a product instruction video based on the user's selection. For example, the prompt might be, "Please explain the features and functions of your new smartphone in a video of no more than three minutes." The video generated based on this data processing is then stored on the server.

[0362] Step 3:

[0363] The device receives the generated video from the server and displays a virtual instructor or virtual assistant through the smart glasses using AR technology, allowing users to watch the video and enjoy an immersive training or shopping experience.

[0364] Step 4:

[0365] When a user starts practicing, the motion sensor tracks the user's movements in real time. This movement data is sent from the device to the server. The server analyzes this data using an AI analysis engine and evaluates the accuracy of the movements. This evaluation result is generated.

[0366] Step 5:

[0367] The evaluation results are sent from the server to the device, which then provides visual and tactile feedback to the user. The feedback indicates areas for improvement in the user's movements, and haptic feedback is also provided using a haptic device. This data processing allows for more accurate movement correction.

[0368] Step 6:

[0369] The emotion engine analyzes the user's emotional state from their facial expressions and voice. This analysis data is sent to the server, which then dynamically adjusts the training content based on the user's emotional state. For example, the prompt might be, "Please rate the user's emotional state (interest, confusion, etc.) from their facial expressions and voice while they are watching the smartphone introductory video." Based on this data processing, the training content is adjusted appropriately.

[0370] Step 7:

[0371] The user adjusts their movements based on the feedback and continues practicing or shopping. The device continues to collect data using motion sensors and an emotion engine, and sends it to the server. The server continues to analyze the data in real time and provide feedback, allowing the user to achieve their goals efficiently and accurately.

[0372] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0373] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0374] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0375] [Second embodiment]

[0376] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0377] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0378] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0379] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0380] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0381] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0382] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0383] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0384] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0385] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0386] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0387] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0388] ---

[0389] The present invention is a system that combines generative AI, AR technology, sensory technology, and other technologies to allow users to practice Aikido at home. Specific embodiments for implementing this system will be described below.

[0390] First, the user starts the application and selects their Aikido level and the type of kata they want to learn. At this time, the terminal accepts the user's input and sends the selection information to the server.

[0391] The server uses generative AI to automatically generate a video of a basic Aikido kata based on the kata selected by the user. The generated video is stored on the server and sent to the user's device.

[0392] The device then uses the received video to display a virtual instructor in the user's field of view using AR technology, allowing the user to practice with the virtual instructor in a real practice space. Additionally, a haptic device is provided using sensory technology to provide feedback on the user's movements, allowing the user to continue practicing while experiencing the accuracy of their movements.

[0393] When a user begins practicing, the device's motion sensor or camera tracks the user's movements in real time. This data is sent from the device to the server. The server then analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The evaluation results are then sent back to the device.

[0394] Based on the evaluation results received from the server, the device provides the user with visual and tactile feedback on the accuracy of their movements and areas for improvement. By correcting their movements and practicing again, the user can master the kata accurately.

[0395] For example, when a user practices the advanced kata of Katatori Ikkyo, the server generates a detailed video of the kata and sends it to the device. The user visually watches the virtual instructor's movements through the goggles and performs the movements in sync with them. If there is a misalignment in the movements, the device's haptic device vibrates specific parts of the body, providing feedback on what needs to be corrected. The server's AI analysis engine then analyzes the user's movements in detail and suggests specific corrections.

[0396] This system allows users to have an experience similar to that of a real classroom, even from the comfort of their own home, and aims to accurately master techniques. It also provides custom feedback based on the user's level and the form they select, enabling efficient practice.

[0397] As described above, the present invention provides a system for efficiently and accurately practicing Aikido, and provides the user with a realistic training environment and accurate feedback.

[0398] The processing flow will be explained below.

[0399] ---

[0400] Step 1:

[0401] The user launches the application and fills in the interface to select their Aikido level (beginner, intermediate, advanced) and the type of kata they wish to learn.

[0402] Step 2:

[0403] The terminal receives the user's input information and transmits the selected data to the server.

[0404] Step 3:

[0405] Based on the information received by the server, a generation AI is used to automatically generate a video of the basic Aikido form specified by the user. The generated video is then saved on the server.

[0406] Step 4:

[0407] The server transmits the generated video to the user's terminal.

[0408] Step 5:

[0409] The device receives the video and starts the AR engine to prepare to display the virtual instructor in the user's field of view.

[0410] Step 6:

[0411] The device initializes the haptic device and configures it to provide tactile feedback in response to the user's movements.

[0412] Step 7:

[0413] The user puts on the goggles and sensory device and confirms they are ready.

[0414] Step 8:

[0415] When the user presses the start button, the terminal receives the start signal and the virtual instructor appears in the user's field of vision.

[0416] Step 9:

[0417] The virtual instructor initiates the selected form of movement and the user replicates the same movement.

[0418] Step 10:

[0419] The device uses motion sensors or a camera to track the user's movements in real time.

[0420] Step 11:

[0421] The device collects data on the user's movements and sends it to the server.

[0422] Step 12:

[0423] The tracking data received by the server is analyzed using an AI analysis engine.

[0424] Step 13:

[0425] The server evaluates the accuracy of the user's movements based on the analysis results and identifies areas for improvement. The evaluation results are stored on the server.

[0426] Step 14:

[0427] The server sends the evaluation results and correction instructions to the user's terminal.

[0428] Step 15:

[0429] The terminal receives the evaluation results from the server and provides visual and tactile feedback to the user.

[0430] Step 16:

[0431] The user practices the form again while correcting their movements based on the feedback.

[0432] Step 17:

[0433] The device again tracks the user's movements and sends them to the server, and this process is repeated until the user's movements are accurate.

[0434] ---

[0435] These are the specific processing steps of the program. This process allows the user to receive feedback on their movements and accurately correct them, allowing them to master Aikido forms.

[0436] Example 1

[0437] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0438] Traditional Aikido training required a dedicated dojo and instructor, making it difficult to practice efficiently and accurately at home. Furthermore, there were limited ways to check whether one's movements were accurate, making self-improvement difficult. This created a major hurdle for beginners and intermediates to master accurate techniques.

[0439] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0440] In this invention, the server includes means for automatically generating videos of basic Aikido forms based on user selections using a generative AI model, means for displaying a virtual instructor in the user's field of view based on the generated videos using AR technology, and means for providing the user with tactile feedback using sensory technology, thereby enabling users to efficiently practice Aikido from the comfort of their own homes while receiving accurate feedback in real time.

[0441] A "generative AI model" is an artificial intelligence algorithm that automatically generates information based on data for a specific task.

[0442] A "prompt sentence" is text-based input data used to give instructions to a generative AI model.

[0443] "AR technology" is a technology that displays digital information superimposed on the real world.

[0444] "Sensory technology" is technology that provides users with visual, tactile, or other feedback.

[0445] A "haptic device" is a device that conveys tactile feedback to a user.

[0446] A "server" is a computer system that processes and stores data on a network.

[0447] A "terminal" is a device that a user operates to input and output data.

[0448] "Means for tracking user movements in real time" refers to a function that uses motion sensors and cameras to capture user movements sequentially.

[0449] An "AI analysis engine" is a system that uses AI technology to analyze input data and provide specific insights and evaluations.

[0450] "Means for assessing the accuracy of movements" refers to a method for measuring the accuracy of a user's movements using an AI analysis engine.

[0451] "Means for providing feedback" refers to a function that provides visual and tactile information to the user, informing them of areas for improvement in their actions.

[0452] The present invention relates to a system that combines generative AI models, AR technology, sensory technology, and other technologies to enable users to practice Aikido at home. Specific embodiments for implementing this system are described below.

[0453] The user launches the smartphone application and selects their Aikido level and the type of kata they wish to learn. For example, an intermediate practitioner selects "Intermediate" and the kata "Shouldering Icchi." At this point, the device accepts the selection information and sends it to the server. The information is sent in JSON format via the REST API.

[0454] The server automatically generates a video of the basic Aikido kata based on the kata selected by the user using a generative AI model (e.g., OpenAI's GPT-4). Specifically, the following prompt sentence is input to the generative AI model:

[0455] Example prompt sentence:

[0456] "Create an instructional video with detailed movements for the intermediate Katatori Ikkyo technique. Include a complete explanation of the steps, grips, and throws."

[0457] The server saves the generated video in cloud storage (e.g., AWS S3) and sends the URL of the video to the device.

[0458] The device uses the received video URL to stream the video and uses AR technology (e.g., Unity or ARKit) to display the virtual instructor in the user's field of view, allowing the user to practice with the virtual instructor using their smartphone or AR goggles.

[0459] When a user begins practicing, the device's motion sensors and camera track the user's movements in real time. The device then sends the data obtained from these sensors to a server. The server then analyzes the received tracking data using an AI analysis engine (e.g., TensorFlow or PyTorch) and evaluates the user's movements. The evaluation results are fed back as specific corrections and sent back to the device.

[0460] The device provides the user with visual and tactile feedback based on the evaluation results received from the server. For example, the device may display feedback such as "your right hand is positioned too low" on the screen and vibrate the user's right hand using a haptic device. In this way, the user can correct their movements as they practice.

[0461] As described above, the present invention allows users to experience a similar experience to practicing in a real classroom from the comfort of their own home. It provides an efficient training method aimed at mastering accurate techniques, and provides users with a realistic practice environment and accurate feedback.

[0462] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0463] Step 1:

[0464] The user launches the application on their smartphone and is presented with a home screen to begin Aikido practice. The user taps the "Start Practice" button and selects their Aikido level and the kata they wish to learn. The user's level and kata selection information are obtained as input. The selection information is sent from the device to the server as output.

[0465] Step 2:

[0466] The device sends the user's selected level and type information to the server. During this process, it receives the user's selection information as input and sends it to the server in JSON format via the REST API. The output returns a successful status.

[0467] Step 3:

[0468] Based on the received data, the server uses a generative AI model to automatically generate a video of the basic Aikido forms. As input, the user's selection information and a prompt sentence are provided to the generative AI model. Specifically, the following prompt sentence is used:

[0469] "Create an instructional video with detailed movements for the intermediate Katatori Ikkyo technique. Include a complete explanation of the steps, grips, and throws."

[0470] The output is a video generated by the generative AI model, which the server stores in cloud storage (e.g., AWS S3).

[0471] Step 4:

[0472] The server sends the URL of the stored video to the device. As input, the video URL is retrieved from the cloud storage and sent to the user's device. As output, the video URL is provided to the device.

[0473] Step 5:

[0474] The device streams the video using the received video URL. As input, it receives the video URL and starts the streaming process. It uses AR technology (e.g., Unity or ARKit) to display the virtual instructor in the user's field of view. As output, it displays the virtual instructor in the user's field of view.

[0475] Step 6:

[0476] The user begins practicing while watching the virtual instructor's movements displayed in AR. As input, the user sees the virtual instructor's movements and synchronizes their own movements. As output, the user's movements are tracked in real time.

[0477] Step 7:

[0478] The device's motion sensors and camera track the user's movements in real time. As input, user movement data is obtained from the motion sensors and camera. As output, tracking data is generated.

[0479] Step 8:

[0480] The device sends the captured tracking data to the server. As input, it receives the acquired tracking data and sends it to the server. As output, it returns a status indicating that the transmission was successful.

[0481] Step 9:

[0482] The server analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The tracking data is provided to the AI ​​analysis engine as input. The movement evaluation results are obtained as output. The evaluation results include the accuracy of the user's movements and areas for improvement.

[0483] Step 10:

[0484] The server sends the motion evaluation results to the terminal. As input, the evaluation results are generated and sent to the terminal. As output, the evaluation results are provided to the terminal.

[0485] Step 11:

[0486] The device provides visual and tactile feedback to the user based on the evaluation results received from the server. As input, it receives the evaluation results and presents them to the user. For example, feedback such as "your right hand is positioned low" is displayed on the screen, and the haptic device vibrates the right hand. This allows the user to correct their own movements. As output, feedback information is generated and provided to the user.

[0487] Step 12:

[0488] The user corrects their movements based on the provided feedback and continues practicing again. As input, they receive the feedback and correct their movements. As output, the correct movements are gradually learned.

[0489] (Application example 1)

[0490] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0491] It is difficult for new employees and inexperienced personnel to quickly learn how to operate robots in factories or on production lines. Conventional training methods require training using actual machines, which is costly and time-consuming. Furthermore, it requires a lot of effort from the instructor, making efficient training difficult. The present invention solves these problems by providing a training system that allows employees to efficiently and effectively learn how to operate robots.

[0492] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0493] In this invention, the server includes: means for automatically generating a video of a basic form of a work procedure based on a user's selection using a generation AI; means for displaying a virtual instructor in the user's field of view based on the generated video using AR technology; means for providing the user with tactile feedback using haptic technology; means for tracking the user's movements in real time using a motion sensor or a camera; means for receiving data on the user's movements at the server, evaluating the accuracy of the movements using an AI analysis engine, and generating and transmitting the evaluation results; means for providing the user with the evaluation results received from the server at a terminal and encouraging corrections based on the feedback; and means for supporting training for new employees using virtual instructor display technology for work training, thereby enabling new employees and inexperienced people to quickly learn robot operation.

[0494] "Generative AI" is an artificial intelligence technology that automatically generates videos of actions and work procedures based on user input.

[0495] "AR technology" is an abbreviation for augmented reality technology, which is a technology that displays virtual information overlaid on the real field of view.

[0496] "Haptic technology" refers to technology for providing tactile feedback to users, particularly technology that uses haptic devices.

[0497] A "motion sensor" is a sensor that detects the user's physical movements and acquires them as digital data in real time.

[0498] A "camera" is a device that captures a user's actions as video data and analyzes it.

[0499] The "AI analysis engine" is an artificial intelligence technology that analyzes acquired user movement data and evaluates the accuracy of the movements.

[0500] A "virtual instructor" is a virtual teacher who is displayed in the user's field of view using AR technology.

[0501] "Terminal" refers to the device used by the user for operation and viewing, and is the device that communicates with the server and provides feedback.

[0502] The "training support system" is an integrated system that enables new employees and inexperienced personnel to efficiently learn how to operate robots and other work procedures.

[0503] A "work procedure" is a series of steps or operating methods for performing a specific task.

[0504] MODE FOR CARRYING OUT THE INVENTION

[0505] This invention relates to a robot operation training system for new factory employees. Specifically, it provides an efficient training method using generative AI, AR technology, and haptic technology.

[0506] First, the user starts the training. The user puts on a device (e.g., a smart headset) and launches the application. First, the user is shown a screen where they can select their skill level and the type of work procedure they want to learn. The device accepts the user's input and sends the selection information to the server.

[0507] Next, the server uses a generative AI to automatically generate a basic video based on the work steps selected by the user. The generated video is stored on the server and sent to the user's device. The server uses an AI analysis engine to do this.

[0508] The received video is played on the device, and a virtual instructor is displayed in the user's field of view using AR technology. The user follows the virtual instructor's instructions and begins training to operate the robot. The virtual instructor explains the user's movements in real time and provides visual guidance.

[0509] When a user actually performs an action, a motion sensor or camera tracks the user's movement in real time. This tracking data is sent from the device to a server. The server analyzes the received tracking data using an AI analysis engine and evaluates the user's movement.

[0510] The evaluation results are sent back to the device from the server, and the device then provides feedback to the user. Feedback can be visual or tactile, with tactile feedback provided using a haptic device. Specifically, a haptic glove worn on the user's hand vibrates when the user has performed an accurate movement or when a correction is required.

[0511] This allows users to efficiently train together with the virtual instructor while experiencing the accuracy of their movements in real time. For example, if a user is learning how to "install a part," the server generates a detailed video of the procedure and sends it to the device. The user can visually confirm the virtual instructor's movements through the headset and perform the appropriate movements. If there is any deviation in the movements, the haptic device will vibrate specific parts of the body, providing feedback on what needs to be corrected. The server's AI analysis engine then analyzes the user's movements in detail and suggests specific corrections.

[0512] This system allows new employees and those with no experience to quickly learn how to operate robots, ensuring efficient training.

[0513] Generative AI model prompt example:

[0514] "We are building a training program for new employees to learn the operating procedures for a robotic assembly task. The operating procedures need to be generated as a series of videos."

[0515] The above is an embodiment of the present invention.

[0516] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0517] Step 1:

[0518] The user launches the application on the device. A screen appears in which the user can select their skill level and the type of work procedure they wish to learn. The user's selection (input data) is sent from the device to the server. The input is the user's level and work procedure, and the output is the selection information sent to the server.

[0519] Step 2:

[0520] The server generates a prompt for the generative AI model based on the received selection information. Based on this prompt, the generative AI generates a basic video corresponding to the user's selection. The input is the user's selection information, and the output is a video of the generated work procedure.

[0521] Step 3:

[0522] The server sends the generated video to the terminal. The input is the generated video, and the output is the transmission of video data to the terminal.

[0523] Step 4:

[0524] The device plays the received video and uses AR technology to display the virtual instructor in the user's field of view. The user receives visual guidance through the device's display. The input is the video data, and the output is the display of the virtual instructor.

[0525] Step 5:

[0526] The user follows the instructions of the virtual instructor and begins training. The user's movements are tracked in real time by a motion sensor or camera. The input is the user's movements, and the output is the tracking data.

[0527] Step 6:

[0528] The terminal sends tracking data to the server. The input is tracking data, and the output is data transmission to the server.

[0529] Step 7:

[0530] The server analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The result is an evaluation of the accuracy of the movements and points to be corrected, which is then sent to the device. The input is the tracking data, and the output is the evaluation result.

[0531] Step 8:

[0532] The terminal provides the user with the evaluation results received from the server. The terminal displays visual feedback and simultaneously sends instructions to the haptic device to provide tactile feedback. The input is the evaluation results, and the output is visual and tactile feedback.

[0533] Step 9:

[0534] The user then corrects their movements based on the feedback and repeats the training. Through this feedback cycle, the user gradually acquires accurate movements. The input is visual and tactile feedback, and the output is the user's corrected movements.

[0535] The above is a specific explanation of each processing step of the program.

[0536] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0537] ---

[0538] The present invention is a system that combines generative AI, AR technology, sensory technology, an emotion engine, and other technologies to enable users to practice Aikido at home. A specific embodiment for implementing this system will be described.

[0539] First, the user starts the application and selects their Aikido level and the type of kata they want to learn. At this time, the terminal accepts the user's input and sends the selection information to the server.

[0540] The server uses generative AI to automatically generate a video of a basic Aikido kata based on the kata selected by the user. The generated video is stored on the server and sent to the user's device.

[0541] The device then uses the received video to display a virtual instructor in the user's field of view using AR technology, allowing the user to practice with the virtual instructor in a real practice space. Additionally, a haptic device is provided using sensory technology to provide feedback on the user's movements, allowing the user to continue practicing while experiencing the accuracy of their movements.

[0542] When a user begins practicing, the device's motion sensor or camera tracks the user's movements in real time. This data is sent from the device to the server. The server then analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The evaluation results are then sent back to the device.

[0543] Based on the evaluation results received from the server, the device provides the user with visual and tactile feedback on the accuracy of their movements and areas for improvement. By correcting their movements and practicing again, the user can master the kata accurately.

[0544] Furthermore, the present invention incorporates an emotion engine that recognizes emotions from the user's facial expressions and voice. The device uses the emotion engine to determine the user's emotions during training in real time and adjust the content and difficulty of the training accordingly. For example, if the user is feeling frustrated, the device sends that information to the server, and the server generates a training plan based on the user's emotional state and sends it to the device.

[0545] For example, when a user practices the advanced kata, Katatori Ikkyo, the server generates a detailed video of the kata and sends it to the device. The user watches the virtual instructor's movements and replicates them. A motion sensor tracks the user's movements and sends the data to the server. The server's AI analysis engine analyzes the user's movements and generates feedback. At the same time, the emotion engine recognizes the user's face and analyzes their voice to determine their emotional state. If the user is feeling frustrated or stressed, the server adjusts the practice and provides advice on how to relax.

[0546] In this way, the present invention provides a system for efficient and accurate Aikido training, not only providing the user with a realistic training environment and accurate feedback, but also providing support according to the user's emotional state.

[0547] The processing flow will be explained below.

[0548] ---

[0549] Step 1:

[0550] The user starts the application and selects their Aikido level (beginner, intermediate, advanced) and the type of kata they want to learn. By entering their input into the interface, the selection information is sent to the terminal.

[0551] Step 2:

[0552] The terminal receives the user's input information and transmits this information to the server.

[0553] Step 3:

[0554] Based on the information received by the server, a generation AI is used to automatically generate a video of the basic Aikido form specified by the user. The generated video is saved on the server.

[0555] Step 4:

[0556] The server transmits the generated video to the user's terminal.

[0557] Step 5:

[0558] The device receives the video, starts the AR engine, and prepares to display the virtual instructor in the user's field of view. At the same time, it initializes the haptics device and configures it to provide tactile feedback according to the user's movements.

[0559] Step 6:

[0560] The device displays a message prompting the user to put on the goggles and sensory device, and waits until they are ready.

[0561] Step 7:

[0562] The user puts on the goggles and sensory device and confirms that they are ready. The user presses the start button.

[0563] Step 8:

[0564] The terminal receives the start signal, and the virtual instructor appears in the user's field of view. The virtual instructor begins the selected form of movement, and the user performs the movement accordingly.

[0565] Step 9:

[0566] The device uses motion sensors or a camera to track the user's movements in real time.

[0567] Step 10:

[0568] The device collects tracking data and sends it to the server. At the same time, the device's emotion engine analyzes the user's facial expressions and voice and sends the results to the server.

[0569] Step 11:

[0570] The tracking data received by the server is analyzed using an AI analysis engine to evaluate the accuracy of the user's movements, and the evaluation results are stored on the server.

[0571] Step 12:

[0572] The server analyzes the received emotional data to determine the user's emotional state. If the user is feeling frustrated or stressed, the content and difficulty of the training will be adjusted based on that information.

[0573] Step 13:

[0574] The server generates feedback and correction instructions appropriate for the user based on the evaluation results and emotional state, and sends them to the terminal.

[0575] Step 14:

[0576] The device receives feedback and correction instructions from the server and provides them to the user visually and tactilely, automatically adjusting the difficulty of the training as needed and suggesting relaxation techniques.

[0577] Step 15:

[0578] Based on the feedback provided by the user, the user practices the form again, correcting the movements.

[0579] Step 16:

[0580] The device again tracks the user's movements and sends the data to the server, and this process is repeated until the user's movements are accurate and emotionally stable.

[0581] ---

[0582] These are the specific processing steps of the program. This process allows the user to receive feedback on their movements and accurately correct them, allowing them to master Aikido forms. Furthermore, the real-time emotional feedback provided by the emotion engine can further improve the quality of training.

[0583] Example 2

[0584] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0585] In recent years, there has been a growing demand for methods to efficiently learn Aikido at home. Conventional methods require training at a dojo or rely on books and video materials, which require the guidance of an instructor. Furthermore, self-practice is difficult because it is difficult to understand and experience the correct movements, making it difficult to practice efficiently. Furthermore, there is also the issue of difficulty in maintaining motivation due to insufficient feedback and emotional awareness during self-practice. There is an urgent need to solve these problems and provide a system that allows users to effectively and accurately learn Aikido at home.

[0586] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0587] In this invention, the server includes means for automatically generating videos of basic Aikido forms based on a user's selection using a generation AI, means for displaying a virtual instructor in the user's field of view based on the generated videos using augmented reality technology, means for providing the user with tactile feedback using sensory technology, means for tracking the user's movements in real time using a motion detector or camera, means for evaluating the accuracy of the movements using an AI analysis engine and generating and transmitting the evaluation results, and means for analyzing the user's facial expressions and voice in real time and adjusting the training content and difficulty based on the user's emotional state. This allows users to receive a realistic training environment and accurate feedback even at home, enabling them to accurately master the movements.

[0588] "Generative AI" refers to a system that uses artificial intelligence technology to automatically generate the necessary content based on user selections.

[0589] "Augmented reality technology" is a technology that overlays computer-generated information onto real-world images and videos.

[0590] "Sensory technology" refers to technology that allows users to obtain information through their senses, such as sight, hearing, and touch.

[0591] A "motion detector" is a device that detects a user's movements in real time and acquires them as data.

[0592] An "artificial intelligence analysis engine" is an artificial intelligence system that analyzes collected data and generates appropriate feedback and actions based on the results.

[0593] "Feedback" refers to information provided to the user by the system regarding corrections and areas for improvement.

[0594] "Real-time tracking" refers to a technology that tracks a user's movements instantly without any time delay and processes the data.

[0595] A "virtual instructor" is a virtual instructor who appears in the user's field of view through AR technology.

[0596] "Adjusting training content" refers to changing the difficulty and content of the training menu based on the user's movements and emotional state.

[0597] "User's emotional state" refers to the emotions (joy, frustration, etc.) that the user is feeling during training.

[0598] The present invention is a system that combines generative AI, augmented reality technology, sensory technology, an artificial intelligence analysis engine, and emotion recognition technology to enable users to effectively practice Aikido at home. Specific embodiments of this system are described below.

[0599] First, the user launches the application on a device such as a smartphone or tablet and selects their Aikido level and the kata they wish to learn. The device then sends this selection information to the server, using the device's input device and communication module.

[0600] Based on the received information, the server uses a generative AI model (e.g., OpenAI's GPT-4) to automatically generate an instructional video for a basic Aikido form. An example of a specific prompt used in the generation process is, "Please generate a detailed video about the basic Aikido form of Shoulder Drill Ikkyo." The generated video is stored on the server and transmitted to the terminal using data streaming technology (e.g., RTMP protocol).

[0601] The device plays the received video and uses augmented reality technology (e.g., ARKit) to display the virtual instructor in the user's field of view, allowing the user to practice with the virtual instructor in the real practice space.

[0602] When a user starts practicing, the device's motion sensors and camera track the user's movements in real time, and this data is sent from the device to the server. The server analyzes the received tracking data using an artificial intelligence analysis engine (e.g., TensorFlow) to accurately evaluate the user's movements. This evaluation result is then sent back to the device.

[0603] Based on the evaluation results, the device provides the user with visual and tactile feedback on the accuracy of their movements and areas for improvement. A haptic device (for example, a wearable device with vibration) is provided that uses sensory technology to provide feedback on the user's movements. Specific feedback might include, "It would be better if you moved your right arm a little more to the left." The user can then correct their movements and continue practicing.

[0604] Furthermore, the device uses an emotion engine (e.g., Affectiva's emotion recognition API) to analyze the user's emotional state in real time. It analyzes the user's facial expressions and voice through the camera and microphone to determine emotions such as joy or frustration. If the user is feeling frustrated, the device sends this information to the server, which then generates a training plan based on the user's emotional state and sends it to the device. For example, it might display a guide message such as "Relax and take a deep breath" and temporarily reduce the difficulty of the exercises.

[0605] This system allows users to practice at home in a realistic environment and receive accurate feedback, enabling them to accurately master movements.

[0606] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0607] Step 1:

[0608] The user launches the application and selects their Aikido level and the kata they wish to practice.

[0609] Input: Aikido level and kata type selected by user operation.

[0610] Output: JSON data containing the selection information.

[0611] Specific operation: The user selects "Advanced Form" or "Shoulder Training" from the menu on the device screen and presses the OK button. The selected information is converted into JSON format and passed to the next process.

[0612] Step 2:

[0613] The terminal transmits the selection information to the server.

[0614] Input: JSON data containing the user's selections.

[0615] Output: HTTPS request sent to the server.

[0616] Specific operation: The device generates JSON data of the selected information and sends it to the server using the HTTPS protocol. Specifically, it sends a POST request to the endpoint URL.

[0617] Step 3:

[0618] The server uses the generative AI model to generate videos of basic Aikido forms and sends them to the device.

[0619] Input: JSON data containing the user's selections.

[0620] Output: Generated video data of Aikido basic forms.

[0621] Specific operation: Based on the received JSON data, the server inputs the prompt "Please generate a detailed video about the basic form of Aikido's Shoulder Pulling Ikkyo" into the generative AI model. The AI ​​model generates the video and stores it on the server. The generated video data is sent to the terminal via the RTMP protocol.

[0622] Step 4:

[0623] The device displays the received video and displays the virtual instructor using AR technology.

[0624] Input: Generated video data of Aikido basic forms.

[0625] Output: Virtual instructor display.

[0626] Specific operation: The device plays the received video data and uses ARKit to display the virtual instructor in the user's field of view. Specifically, a 3D model of the virtual instructor is overlaid on the device's camera image.

[0627] Step 5:

[0628] The user begins practicing and the device tracks the user's movements.

[0629] Input: User movement.

[0630] Output: Tracking data.

[0631] Specific movements: The user watches the movements of the virtual instructor and then reproduces them. The device's motion sensors and camera capture the user's movements in real time, obtaining coordinate data and movement speed data.

[0632] Step 6:

[0633] The device sends the tracking data to the server.

[0634] Input: Tracking data.

[0635] Output: Sends a WebSocket request to the server.

[0636] What it does: The device converts the acquired tracking data into JSON format and sends it to the server using the secure WebSocket protocol.

[0637] Step 7:

[0638] The server analyzes the tracking data, generates feedback, and sends it to the device.

[0639] Input: Tracking data.

[0640] Output: Feedback data.

[0641] Specific operation: The server analyzes the received tracking data using an artificial intelligence analysis engine (e.g., TensorFlow). As a result of the analysis, it evaluates the accuracy of the user's movements and areas for improvement, and generates specific feedback such as "It would be better if you moved your right arm a little more to the left." This feedback data is sent to the device in JSON format.

[0642] Step 8:

[0643] The device provides feedback to the user and instructs them to correct their movements.

[0644] Input: Feedback data.

[0645] Output: Visual and tactile feedback.

[0646] Specific actions: The device reflects the received feedback data on the display screen, provides specific correction instructions to the user, and controls the haptic device to provide feedback to the user's physical sensations.

[0647] Step 9:

[0648] The user corrects the movement and repeats the practice again.

[0649] Input: Feedback information.

[0650] Output: Corrected movement.

[0651] Specific movements: The user corrects their movements based on the feedback and repeats the practice while watching the virtual instructor's movements again.

[0652] Step 10:

[0653] The device uses an emotion engine to analyze the user's emotions.

[0654] Input: User's facial and voice data.

[0655] Output: Emotional state data.

[0656] How it works: The device uses a camera and microphone to capture the user's facial expressions and voice, then analyzes them using an emotion engine to determine a specific emotional state (e.g., joy, frustration) in real time.

[0657] Step 11:

[0658] The server adjusts the training content based on the emotional data and sends it to the device.

[0659] Input: Emotional state data.

[0660] Output: A tailored training plan.

[0661] Specific operation: The server analyzes the emotional state data sent from the device and generates a training plan according to the user's emotions. Specifically, if the user feels frustrated, the server generates a plan that lowers the difficulty of the training or includes a guide message such as "Relax and take a deep breath." This plan is sent to the device and applied to the user.

[0662] (Application example 2)

[0663] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0664] Conventional online training and shopping systems have the problem that it is difficult for users to get the same experience as a real instructor or salesperson at home. In particular, the user experience is poor due to the lack of accurate motion feedback, reproduction of the product touch feeling, and appropriate responses based on the user's emotional state. This leads to issues such as a decrease in user learning and purchasing motivation, and a low overall user satisfaction.

[0665] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for automatically generating a video of a basic Aikido form based on a user's selection using a generation AI, means for displaying a virtual instructor in the user's field of view using AR technology, means for providing the user with haptic feedback using sensory technology, means for tracking the user's movements in real time using a motion sensor or a camera, means for receiving data on the user's movements in the server, evaluating the accuracy of the movements using an AI analysis engine, and generating and transmitting the evaluation results, means for providing the user with the evaluation results received from the server at the terminal and prompting the user to make corrections based on the feedback, means for generating a product explanation video using a generation AI, means for displaying a virtual assistant in the field of view using AR technology, means for tracking the user's line of sight and movements using a motion sensor, analyzing the data, and providing feedback, means for evaluating the user's emotional state using an emotion engine and dynamically adjusting the training content, and means for providing tactile feedback of the product's texture using a haptic device. This allows users to have a realistic practice and shopping experience at home, making it easier to accurately learn movements and experience the texture of products, and by allowing the system to respond appropriately based on the user's emotional state, it is possible to improve overall user satisfaction.

[0666] "Generative AI" is a type of artificial intelligence technology that automatically generates necessary content based on user specifications.

[0667] "AR technology" stands for augmented reality technology, which is a technology that displays digital content overlaid on the real world.

[0668] "Sensory technology" refers to technology that provides feedback to the user's senses, and specifically includes feedback to the sense of touch and vision.

[0669] A "motion sensor" is a device that detects and tracks a user's movements.

[0670] A "camera" is a device that captures video and analyzes the user's movements based on that video.

[0671] A "server" is a computer system connected to a network that processes and stores various types of data.

[0672] An "AI analysis engine" is a system that uses artificial intelligence technology to analyze collected data and make specific evaluations and predictions.

[0673] A "virtual instructor" is a digital instructor that appears in the user's field of vision using AR technology and is responsible for providing training guidance.

[0674] "Feedback" is information for guidance and correction that the system provides based on the user's movements and state.

[0675] A "haptic device" is a device for providing tactile feedback to a user.

[0676] The "emotion engine" is an artificial intelligence technology that detects the user's emotional state from their facial expressions and voice, and adjusts their behavior based on that.

[0677] A "product description video" is a video containing detailed product explanations generated using AI.

[0678] A "virtual assistant" is a digital character that appears in the user's field of vision using AR technology and serves to support and guide the user.

[0679] The present invention is a system that fully utilizes generative AI, AR technology, sensory technology, emotion engine, and haptics technology to enable users to have an online learning or shopping experience at home. Specific embodiments for implementing this system are described below.

[0680] Hardware and Software

[0681] Smart glasses: Uses AR technology to display virtual instructors and virtual assistants.

[0682] Generative AI models (e.g., GPT-4): Generate video or text content based on user selections.

[0683] Motion sensors (e.g. Kinect): Track user movements in real time.

[0684] Haptic devices (e.g., SenseGlove): Provide tactile feedback to the user.

[0685] Server: Responsible for data processing and storage, and equipped with an AI analysis engine and emotion engine.

[0686] Emotion engine (e.g. Microsoft Azure Emotion API): Analyzes the user's emotional state.

[0687] System Operation

[0688] First, the user launches the application and selects information about Aikido training or products. By entering user information, the form they want to learn, and categories of interest, the following steps are carried out.

[0689] Video generation using generative AI

[0690] The server uses a generative AI model (e.g., GPT-4) to automatically generate a basic Aikido form or a product instruction video based on the user's selection. For example, input a prompt such as, "Please explain the features and functions of a new smartphone in a video of no more than 3 minutes."

[0691] AR display

[0692] Through smart glasses, a virtual instructor or virtual assistant is displayed in the user's field of view based on videos generated by AR technology, providing an immersive training or shopping experience.

[0693] Motion sensor tracking

[0694] When a user starts practicing, a motion sensor (e.g., Kinect) tracks the user's movements in real time. This data is sent to a server, where an AI analysis engine evaluates the accuracy of the movements.

[0695] feedback

[0696] The evaluation results are sent from the server to the device and provided to the user. The user can then modify their movements based on this feedback. Tactile feedback is also provided using a haptic device (e.g., SenseGlove).

[0697] Emotion analysis using an emotion engine

[0698] An emotion engine (e.g., Microsoft Azure Emotion API) analyzes the user's emotional state from their facial expressions and voice and dynamically adjusts the training or shopping experience, for example, using prompts such as, "Please rate the user's emotional state (interested, confused, etc.) from their facial expressions and voice while they are watching the smartphone introductory video."

[0699] Specific examples

[0700] 1. If a user wants to practice advanced shoulder-grab techniques, the server generates a detailed video and sends it to the device. The user can then visualize the virtual instructor's movements and replicate them.

[0701] 2. In the case of shopping, when a user is looking for a new smartphone, generative AI will generate a detailed product video, and AR will bring a virtual shopping assistant into view. An emotion engine will read the user's interests and confusions and make appropriate suggestions.

[0702] In this way, the present invention provides a user with a high sense of realism and appropriate feedback, and responds to individual emotional states, thereby providing a more satisfying experience.

[0703] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0704] Step 1:

[0705] The user starts the application and selects information about Aikido training or products. They enter user information, the forms they want to learn, and categories they are interested in. The entered information is sent from the device to the server.

[0706] Step 2:

[0707] The server uses a generative AI model to generate a basic Aikido kata or a product instruction video based on the user's selection. For example, the prompt might be, "Please explain the features and functions of your new smartphone in a video of no more than three minutes." The video generated based on this data processing is then stored on the server.

[0708] Step 3:

[0709] The device receives the generated video from the server and displays a virtual instructor or virtual assistant through the smart glasses using AR technology, allowing users to watch the video and enjoy an immersive training or shopping experience.

[0710] Step 4:

[0711] When a user starts practicing, the motion sensor tracks the user's movements in real time. This movement data is sent from the device to the server. The server analyzes this data using an AI analysis engine and evaluates the accuracy of the movements. This evaluation result is generated.

[0712] Step 5:

[0713] The evaluation results are sent from the server to the device, which then provides visual and tactile feedback to the user. The feedback indicates areas for improvement in the user's movements, and haptic feedback is also provided using a haptic device. This data processing allows for more accurate movement correction.

[0714] Step 6:

[0715] The emotion engine analyzes the user's emotional state from their facial expressions and voice. This analysis data is sent to the server, which then dynamically adjusts the training content based on the user's emotional state. For example, the prompt might be, "Please rate the user's emotional state (interest, confusion, etc.) from their facial expressions and voice while they are watching the smartphone introductory video." Based on this data processing, the training content is adjusted appropriately.

[0716] Step 7:

[0717] The user adjusts their movements based on the feedback and continues practicing or shopping. The device continues to collect data using motion sensors and an emotion engine, and sends it to the server. The server continues to analyze the data in real time and provide feedback, allowing the user to achieve their goals efficiently and accurately.

[0718] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0719] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0720] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0721] [Third embodiment]

[0722] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0723] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0724] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0725] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0726] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0727] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0728] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0729] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0730] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0731] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0732] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0733] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0734] ---

[0735] The present invention is a system that combines generative AI, AR technology, sensory technology, and other technologies to allow users to practice Aikido at home. Specific embodiments for implementing this system will be described below.

[0736] First, the user starts the application and selects their Aikido level and the type of kata they want to learn. At this time, the terminal accepts the user's input and sends the selection information to the server.

[0737] The server uses generative AI to automatically generate a video of a basic Aikido kata based on the kata selected by the user. The generated video is stored on the server and sent to the user's device.

[0738] The device then uses the received video to display a virtual instructor in the user's field of view using AR technology, allowing the user to practice with the virtual instructor in a real practice space. Additionally, a haptic device is provided using sensory technology to provide feedback on the user's movements, allowing the user to continue practicing while experiencing the accuracy of their movements.

[0739] When a user begins practicing, the device's motion sensor or camera tracks the user's movements in real time. This data is sent from the device to the server. The server then analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The evaluation results are then sent back to the device.

[0740] Based on the evaluation results received from the server, the device provides the user with visual and tactile feedback on the accuracy of their movements and areas for improvement. By correcting their movements and practicing again, the user can master the kata accurately.

[0741] For example, when a user practices the advanced kata of Katatori Ikkyo, the server generates a detailed video of the kata and sends it to the device. The user visually watches the virtual instructor's movements through the goggles and performs the movements in sync with them. If there is a misalignment in the movements, the device's haptic device vibrates specific parts of the body, providing feedback on what needs to be corrected. The server's AI analysis engine then analyzes the user's movements in detail and suggests specific corrections.

[0742] This system allows users to have an experience similar to that of a real classroom, even from the comfort of their own home, and aims to accurately master techniques. It also provides custom feedback based on the user's level and the form they select, enabling efficient practice.

[0743] As described above, the present invention provides a system for efficiently and accurately practicing Aikido, and provides the user with a realistic training environment and accurate feedback.

[0744] The processing flow will be explained below.

[0745] ---

[0746] Step 1:

[0747] The user launches the application and fills in the interface to select their Aikido level (beginner, intermediate, advanced) and the type of kata they wish to learn.

[0748] Step 2:

[0749] The terminal receives the user's input information and transmits the selected data to the server.

[0750] Step 3:

[0751] Based on the information received by the server, a generation AI is used to automatically generate a video of the basic Aikido form specified by the user. The generated video is then saved on the server.

[0752] Step 4:

[0753] The server transmits the generated video to the user's terminal.

[0754] Step 5:

[0755] The device receives the video and starts the AR engine to prepare to display the virtual instructor in the user's field of view.

[0756] Step 6:

[0757] The device initializes the haptic device and configures it to provide tactile feedback in response to the user's movements.

[0758] Step 7:

[0759] The user puts on the goggles and sensory device and confirms they are ready.

[0760] Step 8:

[0761] When the user presses the start button, the terminal receives the start signal and the virtual instructor appears in the user's field of vision.

[0762] Step 9:

[0763] The virtual instructor initiates the selected form of movement and the user replicates the same movement.

[0764] Step 10:

[0765] The device uses motion sensors or a camera to track the user's movements in real time.

[0766] Step 11:

[0767] The device collects data on the user's movements and sends it to the server.

[0768] Step 12:

[0769] The tracking data received by the server is analyzed using an AI analysis engine.

[0770] Step 13:

[0771] The server evaluates the accuracy of the user's movements based on the analysis results and identifies areas for improvement. The evaluation results are stored on the server.

[0772] Step 14:

[0773] The server sends the evaluation results and correction instructions to the user's terminal.

[0774] Step 15:

[0775] The terminal receives the evaluation results from the server and provides visual and tactile feedback to the user.

[0776] Step 16:

[0777] The user practices the form again while correcting their movements based on the feedback.

[0778] Step 17:

[0779] The device again tracks the user's movements and sends them to the server, and this process is repeated until the user's movements are accurate.

[0780] ---

[0781] These are the specific processing steps of the program. This process allows the user to receive feedback on their movements and accurately correct them, allowing them to master Aikido forms.

[0782] Example 1

[0783] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0784] Traditional Aikido training required a dedicated dojo and instructor, making it difficult to practice efficiently and accurately at home. Furthermore, there were limited ways to check whether one's movements were accurate, making self-improvement difficult. This created a major hurdle for beginners and intermediates to master accurate techniques.

[0785] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0786] In this invention, the server includes means for automatically generating videos of basic Aikido forms based on user selections using a generative AI model, means for displaying a virtual instructor in the user's field of view based on the generated videos using AR technology, and means for providing the user with tactile feedback using sensory technology, thereby enabling users to efficiently practice Aikido from the comfort of their own homes while receiving accurate feedback in real time.

[0787] A "generative AI model" is an artificial intelligence algorithm that automatically generates information based on data for a specific task.

[0788] A "prompt sentence" is text-based input data used to give instructions to a generative AI model.

[0789] "AR technology" is a technology that displays digital information superimposed on the real world.

[0790] "Sensory technology" is technology that provides users with visual, tactile, or other feedback.

[0791] A "haptic device" is a device that conveys tactile feedback to a user.

[0792] A "server" is a computer system that processes and stores data on a network.

[0793] A "terminal" is a device that a user operates to input and output data.

[0794] "Means for tracking user movements in real time" refers to a function that uses motion sensors and cameras to capture user movements sequentially.

[0795] An "AI analysis engine" is a system that uses AI technology to analyze input data and provide specific insights and evaluations.

[0796] "Means for assessing the accuracy of movements" refers to a method for measuring the accuracy of a user's movements using an AI analysis engine.

[0797] "Means for providing feedback" refers to a function that provides visual and tactile information to the user, informing them of areas for improvement in their actions.

[0798] The present invention relates to a system that combines generative AI models, AR technology, sensory technology, and other technologies to enable users to practice Aikido at home. Specific embodiments for implementing this system are described below.

[0799] The user launches the smartphone application and selects their Aikido level and the type of kata they wish to learn. For example, an intermediate practitioner selects "Intermediate" and the kata "Shouldering Icchi." At this point, the device accepts the selection information and sends it to the server. The information is sent in JSON format via the REST API.

[0800] The server automatically generates a video of the basic Aikido kata based on the kata selected by the user using a generative AI model (e.g., OpenAI's GPT-4). Specifically, the following prompt sentence is input to the generative AI model:

[0801] Example prompt sentence:

[0802] "Create an instructional video with detailed movements for the intermediate Katatori Ikkyo technique. Include a complete explanation of the steps, grips, and throws."

[0803] The server saves the generated video in cloud storage (e.g., AWS S3) and sends the URL of the video to the device.

[0804] The device uses the received video URL to stream the video and uses AR technology (e.g., Unity or ARKit) to display the virtual instructor in the user's field of view, allowing the user to practice with the virtual instructor using their smartphone or AR goggles.

[0805] When a user begins practicing, the device's motion sensors and camera track the user's movements in real time. The device then sends the data obtained from these sensors to a server. The server then analyzes the received tracking data using an AI analysis engine (e.g., TensorFlow or PyTorch) and evaluates the user's movements. The evaluation results are fed back as specific corrections and sent back to the device.

[0806] The device provides the user with visual and tactile feedback based on the evaluation results received from the server. For example, the device may display feedback such as "your right hand is positioned too low" on the screen and vibrate the user's right hand using a haptic device. In this way, the user can correct their movements as they practice.

[0807] As described above, the present invention allows users to experience a similar experience to practicing in a real classroom from the comfort of their own home. It provides an efficient training method aimed at mastering accurate techniques, and provides users with a realistic practice environment and accurate feedback.

[0808] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0809] Step 1:

[0810] The user launches the application on their smartphone and is presented with a home screen to begin Aikido practice. The user taps the "Start Practice" button and selects their Aikido level and the kata they wish to learn. The user's level and kata selection information are obtained as input. The selection information is sent from the device to the server as output.

[0811] Step 2:

[0812] The device sends the user's selected level and type information to the server. During this process, it receives the user's selection information as input and sends it to the server in JSON format via the REST API. The output returns a successful status.

[0813] Step 3:

[0814] Based on the received data, the server uses a generative AI model to automatically generate a video of the basic Aikido forms. As input, the user's selection information and a prompt sentence are provided to the generative AI model. Specifically, the following prompt sentence is used:

[0815] "Create an instructional video with detailed movements for the intermediate Katatori Ikkyo technique. Include a complete explanation of the steps, grips, and throws."

[0816] The output is a video generated by the generative AI model, which the server stores in cloud storage (e.g., AWS S3).

[0817] Step 4:

[0818] The server sends the URL of the stored video to the device. As input, the video URL is retrieved from the cloud storage and sent to the user's device. As output, the video URL is provided to the device.

[0819] Step 5:

[0820] The device streams the video using the received video URL. As input, it receives the video URL and starts the streaming process. It uses AR technology (e.g., Unity or ARKit) to display the virtual instructor in the user's field of view. As output, it displays the virtual instructor in the user's field of view.

[0821] Step 6:

[0822] The user begins practicing while watching the virtual instructor's movements displayed in AR. As input, the user sees the virtual instructor's movements and synchronizes their own movements. As output, the user's movements are tracked in real time.

[0823] Step 7:

[0824] The device's motion sensors and camera track the user's movements in real time. As input, user movement data is obtained from the motion sensors and camera. As output, tracking data is generated.

[0825] Step 8:

[0826] The device sends the captured tracking data to the server. As input, it receives the acquired tracking data and sends it to the server. As output, it returns a status indicating that the transmission was successful.

[0827] Step 9:

[0828] The server analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The tracking data is provided to the AI ​​analysis engine as input. The movement evaluation results are obtained as output. The evaluation results include the accuracy of the user's movements and areas for improvement.

[0829] Step 10:

[0830] The server sends the motion evaluation results to the terminal. As input, the evaluation results are generated and sent to the terminal. As output, the evaluation results are provided to the terminal.

[0831] Step 11:

[0832] The device provides visual and tactile feedback to the user based on the evaluation results received from the server. As input, it receives the evaluation results and presents them to the user. For example, feedback such as "your right hand is positioned low" is displayed on the screen, and the haptic device vibrates the right hand. This allows the user to correct their own movements. As output, feedback information is generated and provided to the user.

[0833] Step 12:

[0834] The user corrects their movements based on the provided feedback and continues practicing again. As input, they receive the feedback and correct their movements. As output, the correct movements are gradually learned.

[0835] (Application example 1)

[0836] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0837] It is difficult for new employees and inexperienced personnel to quickly learn how to operate robots in factories or on production lines. Conventional training methods require training using actual machines, which is costly and time-consuming. Furthermore, it requires a lot of effort from the instructor, making efficient training difficult. The present invention solves these problems by providing a training system that allows employees to efficiently and effectively learn how to operate robots.

[0838] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0839] In this invention, the server includes: means for automatically generating a video of a basic form of a work procedure based on a user's selection using a generation AI; means for displaying a virtual instructor in the user's field of view based on the generated video using AR technology; means for providing the user with tactile feedback using haptic technology; means for tracking the user's movements in real time using a motion sensor or a camera; means for receiving data on the user's movements at the server, evaluating the accuracy of the movements using an AI analysis engine, and generating and transmitting the evaluation results; means for providing the user with the evaluation results received from the server at a terminal and encouraging corrections based on the feedback; and means for supporting training for new employees using virtual instructor display technology for work training, thereby enabling new employees and inexperienced people to quickly learn robot operation.

[0840] "Generative AI" is an artificial intelligence technology that automatically generates videos of actions and work procedures based on user input.

[0841] "AR technology" is an abbreviation for augmented reality technology, which is a technology that displays virtual information overlaid on the real field of view.

[0842] "Haptic technology" refers to technology for providing tactile feedback to users, particularly technology that uses haptic devices.

[0843] A "motion sensor" is a sensor that detects the user's physical movements and acquires them as digital data in real time.

[0844] A "camera" is a device that captures a user's actions as video data and analyzes it.

[0845] The "AI analysis engine" is an artificial intelligence technology that analyzes acquired user movement data and evaluates the accuracy of the movements.

[0846] A "virtual instructor" is a virtual teacher who is displayed in the user's field of view using AR technology.

[0847] "Terminal" refers to the device used by the user for operation and viewing, and is the device that communicates with the server and provides feedback.

[0848] The "training support system" is an integrated system that enables new employees and inexperienced personnel to efficiently learn how to operate robots and other work procedures.

[0849] A "work procedure" is a series of steps or operating methods for performing a specific task.

[0850] MODE FOR CARRYING OUT THE INVENTION

[0851] This invention relates to a robot operation training system for new factory employees. Specifically, it provides an efficient training method using generative AI, AR technology, and haptic technology.

[0852] First, the user starts the training. The user puts on a device (e.g., a smart headset) and launches the application. First, the user is shown a screen where they can select their skill level and the type of work procedure they want to learn. The device accepts the user's input and sends the selection information to the server.

[0853] Next, the server uses a generative AI to automatically generate a basic video based on the work steps selected by the user. The generated video is stored on the server and sent to the user's device. The server uses an AI analysis engine to do this.

[0854] The received video is played on the device, and a virtual instructor is displayed in the user's field of view using AR technology. The user follows the virtual instructor's instructions and begins training to operate the robot. The virtual instructor explains the user's movements in real time and provides visual guidance.

[0855] When a user actually performs an action, a motion sensor or camera tracks the user's movement in real time. This tracking data is sent from the device to a server. The server analyzes the received tracking data using an AI analysis engine and evaluates the user's movement.

[0856] The evaluation results are sent back to the device from the server, and the device then provides feedback to the user. Feedback can be visual or tactile, with tactile feedback provided using a haptic device. Specifically, a haptic glove worn on the user's hand vibrates when the user has performed an accurate movement or when a correction is required.

[0857] This allows users to efficiently train together with the virtual instructor while experiencing the accuracy of their movements in real time. For example, if a user is learning how to "install a part," the server generates a detailed video of the procedure and sends it to the device. The user can visually confirm the virtual instructor's movements through the headset and perform the appropriate movements. If there is any deviation in the movements, the haptic device will vibrate specific parts of the body, providing feedback on what needs to be corrected. The server's AI analysis engine then analyzes the user's movements in detail and suggests specific corrections.

[0858] This system allows new employees and those with no experience to quickly learn how to operate robots, ensuring efficient training.

[0859] Generative AI model prompt example:

[0860] "We are building a training program for new employees to learn the operating procedures for a robotic assembly task. The operating procedures need to be generated as a series of videos."

[0861] The above is an embodiment of the present invention.

[0862] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0863] Step 1:

[0864] The user launches the application on the device. A screen appears in which the user can select their skill level and the type of work procedure they wish to learn. The user's selection (input data) is sent from the device to the server. The input is the user's level and work procedure, and the output is the selection information sent to the server.

[0865] Step 2:

[0866] The server generates a prompt for the generative AI model based on the received selection information. Based on this prompt, the generative AI generates a basic video corresponding to the user's selection. The input is the user's selection information, and the output is a video of the generated work procedure.

[0867] Step 3:

[0868] The server sends the generated video to the terminal. The input is the generated video, and the output is the transmission of video data to the terminal.

[0869] Step 4:

[0870] The device plays the received video and uses AR technology to display the virtual instructor in the user's field of view. The user receives visual guidance through the device's display. The input is the video data, and the output is the display of the virtual instructor.

[0871] Step 5:

[0872] The user follows the instructions of the virtual instructor and begins training. The user's movements are tracked in real time by a motion sensor or camera. The input is the user's movements, and the output is the tracking data.

[0873] Step 6:

[0874] The terminal sends tracking data to the server. The input is tracking data, and the output is data transmission to the server.

[0875] Step 7:

[0876] The server analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The result is an evaluation of the accuracy of the movements and points to be corrected, which is then sent to the device. The input is the tracking data, and the output is the evaluation result.

[0877] Step 8:

[0878] The terminal provides the user with the evaluation results received from the server. The terminal displays visual feedback and simultaneously sends instructions to the haptic device to provide tactile feedback. The input is the evaluation results, and the output is visual and tactile feedback.

[0879] Step 9:

[0880] The user then corrects their movements based on the feedback and repeats the training. Through this feedback cycle, the user gradually acquires accurate movements. The input is visual and tactile feedback, and the output is the user's corrected movements.

[0881] The above is a specific explanation of each processing step of the program.

[0882] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0883] ---

[0884] The present invention is a system that combines generative AI, AR technology, sensory technology, an emotion engine, and other technologies to enable users to practice Aikido at home. A specific embodiment for implementing this system will be described.

[0885] First, the user starts the application and selects their Aikido level and the type of kata they want to learn. At this time, the terminal accepts the user's input and sends the selection information to the server.

[0886] The server uses generative AI to automatically generate a video of a basic Aikido kata based on the kata selected by the user. The generated video is stored on the server and sent to the user's device.

[0887] The device then uses the received video to display a virtual instructor in the user's field of view using AR technology, allowing the user to practice with the virtual instructor in a real practice space. Additionally, a haptic device is provided using sensory technology to provide feedback on the user's movements, allowing the user to continue practicing while experiencing the accuracy of their movements.

[0888] When a user begins practicing, the device's motion sensor or camera tracks the user's movements in real time. This data is sent from the device to the server. The server then analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The evaluation results are then sent back to the device.

[0889] Based on the evaluation results received from the server, the device provides the user with visual and tactile feedback on the accuracy of their movements and areas for improvement. By correcting their movements and practicing again, the user can master the kata accurately.

[0890] Furthermore, the present invention incorporates an emotion engine that recognizes emotions from the user's facial expressions and voice. The device uses the emotion engine to determine the user's emotions during training in real time and adjust the content and difficulty of the training accordingly. For example, if the user is feeling frustrated, the device sends that information to the server, and the server generates a training plan based on the user's emotional state and sends it to the device.

[0891] For example, when a user practices the advanced kata, Katatori Ikkyo, the server generates a detailed video of the kata and sends it to the device. The user watches the virtual instructor's movements and replicates them. A motion sensor tracks the user's movements and sends the data to the server. The server's AI analysis engine analyzes the user's movements and generates feedback. At the same time, the emotion engine recognizes the user's face and analyzes their voice to determine their emotional state. If the user is feeling frustrated or stressed, the server adjusts the practice and provides advice on how to relax.

[0892] In this way, the present invention provides a system for efficient and accurate Aikido training, not only providing the user with a realistic training environment and accurate feedback, but also providing support according to the user's emotional state.

[0893] The processing flow will be explained below.

[0894] ---

[0895] Step 1:

[0896] The user starts the application and selects their Aikido level (beginner, intermediate, advanced) and the type of kata they want to learn. By entering their input into the interface, the selection information is sent to the terminal.

[0897] Step 2:

[0898] The terminal receives the user's input information and transmits this information to the server.

[0899] Step 3:

[0900] Based on the information received by the server, a generation AI is used to automatically generate a video of the basic Aikido form specified by the user. The generated video is saved on the server.

[0901] Step 4:

[0902] The server transmits the generated video to the user's terminal.

[0903] Step 5:

[0904] The device receives the video, starts the AR engine, and prepares to display the virtual instructor in the user's field of view. At the same time, it initializes the haptics device and configures it to provide tactile feedback according to the user's movements.

[0905] Step 6:

[0906] The device displays a message prompting the user to put on the goggles and sensory device, and waits until they are ready.

[0907] Step 7:

[0908] The user puts on the goggles and sensory device and confirms that they are ready. The user presses the start button.

[0909] Step 8:

[0910] The terminal receives the start signal, and the virtual instructor appears in the user's field of view. The virtual instructor begins the selected form of movement, and the user performs the movement accordingly.

[0911] Step 9:

[0912] The device uses motion sensors or a camera to track the user's movements in real time.

[0913] Step 10:

[0914] The device collects tracking data and sends it to the server. At the same time, the device's emotion engine analyzes the user's facial expressions and voice and sends the results to the server.

[0915] Step 11:

[0916] The tracking data received by the server is analyzed using an AI analysis engine to evaluate the accuracy of the user's movements, and the evaluation results are stored on the server.

[0917] Step 12:

[0918] The server analyzes the received emotional data to determine the user's emotional state. If the user is feeling frustrated or stressed, the content and difficulty of the training will be adjusted based on that information.

[0919] Step 13:

[0920] The server generates feedback and correction instructions appropriate for the user based on the evaluation results and emotional state, and sends them to the terminal.

[0921] Step 14:

[0922] The device receives feedback and correction instructions from the server and provides them to the user visually and tactilely, automatically adjusting the difficulty of the training as needed and suggesting relaxation techniques.

[0923] Step 15:

[0924] Based on the feedback provided by the user, the user practices the form again, correcting the movements.

[0925] Step 16:

[0926] The device again tracks the user's movements and sends the data to the server, and this process is repeated until the user's movements are accurate and emotionally stable.

[0927] ---

[0928] These are the specific processing steps of the program. This process allows the user to receive feedback on their movements and accurately correct them, allowing them to master Aikido forms. Furthermore, the real-time emotional feedback provided by the emotion engine can further improve the quality of training.

[0929] Example 2

[0930] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0931] In recent years, there has been a growing demand for methods to efficiently learn Aikido at home. Conventional methods require training at a dojo or rely on books and video materials, which require the guidance of an instructor. Furthermore, self-practice is difficult because it is difficult to understand and experience the correct movements, making it difficult to practice efficiently. Furthermore, there is also the issue of difficulty in maintaining motivation due to insufficient feedback and emotional awareness during self-practice. There is an urgent need to solve these problems and provide a system that allows users to effectively and accurately learn Aikido at home.

[0932] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0933] In this invention, the server includes means for automatically generating videos of basic Aikido forms based on a user's selection using a generation AI, means for displaying a virtual instructor in the user's field of view based on the generated videos using augmented reality technology, means for providing the user with tactile feedback using sensory technology, means for tracking the user's movements in real time using a motion detector or camera, means for evaluating the accuracy of the movements using an AI analysis engine and generating and transmitting the evaluation results, and means for analyzing the user's facial expressions and voice in real time and adjusting the training content and difficulty based on the user's emotional state. This allows users to receive a realistic training environment and accurate feedback even at home, enabling them to accurately master the movements.

[0934] "Generative AI" refers to a system that uses artificial intelligence technology to automatically generate the necessary content based on user selections.

[0935] "Augmented reality technology" is a technology that overlays computer-generated information onto real-world images and videos.

[0936] "Sensory technology" refers to technology that allows users to obtain information through their senses, such as sight, hearing, and touch.

[0937] A "motion detector" is a device that detects a user's movements in real time and acquires them as data.

[0938] An "artificial intelligence analysis engine" is an artificial intelligence system that analyzes collected data and generates appropriate feedback and actions based on the results.

[0939] "Feedback" refers to information provided to the user by the system regarding corrections and areas for improvement.

[0940] "Real-time tracking" refers to a technology that tracks a user's movements instantly without any time delay and processes the data.

[0941] A "virtual instructor" is a virtual instructor who appears in the user's field of view through AR technology.

[0942] "Adjusting training content" refers to changing the difficulty and content of the training menu based on the user's movements and emotional state.

[0943] "User's emotional state" refers to the emotions (joy, frustration, etc.) that the user is feeling during training.

[0944] The present invention is a system that combines generative AI, augmented reality technology, sensory technology, an artificial intelligence analysis engine, and emotion recognition technology to enable users to effectively practice Aikido at home. Specific embodiments of this system are described below.

[0945] First, the user launches the application on a device such as a smartphone or tablet and selects their Aikido level and the kata they wish to learn. The device then sends this selection information to the server, using the device's input device and communication module.

[0946] Based on the received information, the server uses a generative AI model (e.g., OpenAI's GPT-4) to automatically generate an instructional video for a basic Aikido form. An example of a specific prompt used in the generation process is, "Please generate a detailed video about the basic Aikido form of Shoulder Drill Ikkyo." The generated video is stored on the server and transmitted to the terminal using data streaming technology (e.g., RTMP protocol).

[0947] The device plays the received video and uses augmented reality technology (e.g., ARKit) to display the virtual instructor in the user's field of view, allowing the user to practice with the virtual instructor in the real practice space.

[0948] When a user starts practicing, the device's motion sensors and camera track the user's movements in real time, and this data is sent from the device to the server. The server analyzes the received tracking data using an artificial intelligence analysis engine (e.g., TensorFlow) to accurately evaluate the user's movements. This evaluation result is then sent back to the device.

[0949] Based on the evaluation results, the device provides the user with visual and tactile feedback on the accuracy of their movements and areas for improvement. A haptic device (for example, a wearable device with vibration) is provided that uses sensory technology to provide feedback on the user's movements. Specific feedback might include, "It would be better if you moved your right arm a little more to the left." The user can then correct their movements and continue practicing.

[0950] Furthermore, the device uses an emotion engine (e.g., Affectiva's emotion recognition API) to analyze the user's emotional state in real time. It analyzes the user's facial expressions and voice through the camera and microphone to determine emotions such as joy or frustration. If the user is feeling frustrated, the device sends this information to the server, which then generates a training plan based on the user's emotional state and sends it to the device. For example, it might display a guide message such as "Relax and take a deep breath" and temporarily reduce the difficulty of the exercises.

[0951] This system allows users to practice at home in a realistic environment and receive accurate feedback, enabling them to accurately master movements.

[0952] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0953] Step 1:

[0954] The user launches the application and selects their Aikido level and the kata they wish to practice.

[0955] Input: Aikido level and kata type selected by user operation.

[0956] Output: JSON data containing the selection information.

[0957] Specific operation: The user selects "Advanced Form" or "Shoulder Training" from the menu on the device screen and presses the OK button. The selected information is converted into JSON format and passed to the next process.

[0958] Step 2:

[0959] The terminal transmits the selection information to the server.

[0960] Input: JSON data containing the user's selections.

[0961] Output: HTTPS request sent to the server.

[0962] Specific operation: The device generates JSON data of the selected information and sends it to the server using the HTTPS protocol. Specifically, it sends a POST request to the endpoint URL.

[0963] Step 3:

[0964] The server uses the generative AI model to generate videos of basic Aikido forms and sends them to the device.

[0965] Input: JSON data containing the user's selections.

[0966] Output: Generated video data of Aikido basic forms.

[0967] Specific operation: Based on the received JSON data, the server inputs the prompt "Please generate a detailed video about the basic form of Aikido's Shoulder Pulling Ikkyo" into the generative AI model. The AI ​​model generates the video and stores it on the server. The generated video data is sent to the terminal via the RTMP protocol.

[0968] Step 4:

[0969] The device displays the received video and displays the virtual instructor using AR technology.

[0970] Input: Generated video data of Aikido basic forms.

[0971] Output: Virtual instructor display.

[0972] Specific operation: The device plays the received video data and uses ARKit to display the virtual instructor in the user's field of view. Specifically, a 3D model of the virtual instructor is overlaid on the device's camera image.

[0973] Step 5:

[0974] The user begins practicing and the device tracks the user's movements.

[0975] Input: User movement.

[0976] Output: Tracking data.

[0977] Specific movements: The user watches the movements of the virtual instructor and then reproduces them. The device's motion sensors and camera capture the user's movements in real time, obtaining coordinate data and movement speed data.

[0978] Step 6:

[0979] The device sends the tracking data to the server.

[0980] Input: Tracking data.

[0981] Output: Sends a WebSocket request to the server.

[0982] What it does: The device converts the acquired tracking data into JSON format and sends it to the server using the secure WebSocket protocol.

[0983] Step 7:

[0984] The server analyzes the tracking data, generates feedback, and sends it to the device.

[0985] Input: Tracking data.

[0986] Output: Feedback data.

[0987] Specific operation: The server analyzes the received tracking data using an artificial intelligence analysis engine (e.g., TensorFlow). As a result of the analysis, it evaluates the accuracy of the user's movements and areas for improvement, and generates specific feedback such as "It would be better if you moved your right arm a little more to the left." This feedback data is sent to the device in JSON format.

[0988] Step 8:

[0989] The device provides feedback to the user and instructs them to correct their movements.

[0990] Input: Feedback data.

[0991] Output: Visual and tactile feedback.

[0992] Specific actions: The device reflects the received feedback data on the display screen, provides specific correction instructions to the user, and controls the haptic device to provide feedback to the user's physical sensations.

[0993] Step 9:

[0994] The user corrects the movement and repeats the practice again.

[0995] Input: Feedback information.

[0996] Output: Corrected movement.

[0997] Specific movements: The user corrects their movements based on the feedback and repeats the practice while watching the virtual instructor's movements again.

[0998] Step 10:

[0999] The device uses an emotion engine to analyze the user's emotions.

[1000] Input: User's facial and voice data.

[1001] Output: Emotional state data.

[1002] How it works: The device uses a camera and microphone to capture the user's facial expressions and voice, then analyzes them using an emotion engine to determine a specific emotional state (e.g., joy, frustration) in real time.

[1003] Step 11:

[1004] The server adjusts the training content based on the emotional data and sends it to the device.

[1005] Input: Emotional state data.

[1006] Output: A tailored training plan.

[1007] Specific operation: The server analyzes the emotional state data sent from the device and generates a training plan according to the user's emotions. Specifically, if the user feels frustrated, the server generates a plan that lowers the difficulty of the training or includes a guide message such as "Relax and take a deep breath." This plan is sent to the device and applied to the user.

[1008] (Application example 2)

[1009] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1010] Conventional online training and shopping systems have the problem that it is difficult for users to get the same experience as a real instructor or salesperson at home. In particular, the user experience is poor due to the lack of accurate motion feedback, reproduction of the product touch feeling, and appropriate responses based on the user's emotional state. This leads to issues such as a decrease in user learning and purchasing motivation, and a low overall user satisfaction.

[1011] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for automatically generating a video of a basic Aikido form based on a user's selection using a generation AI, means for displaying a virtual instructor in the user's field of view using AR technology, means for providing the user with haptic feedback using sensory technology, means for tracking the user's movements in real time using a motion sensor or a camera, means for receiving data on the user's movements in the server, evaluating the accuracy of the movements using an AI analysis engine, and generating and transmitting the evaluation results, means for providing the user with the evaluation results received from the server at the terminal and prompting the user to make corrections based on the feedback, means for generating a product explanation video using a generation AI, means for displaying a virtual assistant in the field of view using AR technology, means for tracking the user's line of sight and movements using a motion sensor, analyzing the data, and providing feedback, means for evaluating the user's emotional state using an emotion engine and dynamically adjusting the training content, and means for providing tactile feedback of the product's texture using a haptic device. This allows users to have a realistic practice and shopping experience at home, making it easier to accurately learn movements and experience the texture of products, and by allowing the system to respond appropriately based on the user's emotional state, it is possible to improve overall user satisfaction.

[1012] "Generative AI" is a type of artificial intelligence technology that automatically generates necessary content based on user specifications.

[1013] "AR technology" stands for augmented reality technology, which is a technology that displays digital content overlaid on the real world.

[1014] "Sensory technology" refers to technology that provides feedback to the user's senses, and specifically includes feedback to the sense of touch and vision.

[1015] A "motion sensor" is a device that detects and tracks a user's movements.

[1016] A "camera" is a device that captures video and analyzes the user's movements based on that video.

[1017] A "server" is a computer system connected to a network that processes and stores various types of data.

[1018] An "AI analysis engine" is a system that uses artificial intelligence technology to analyze collected data and make specific evaluations and predictions.

[1019] A "virtual instructor" is a digital instructor that appears in the user's field of vision using AR technology and is responsible for providing training guidance.

[1020] "Feedback" is information for guidance and correction that the system provides based on the user's movements and state.

[1021] A "haptic device" is a device for providing tactile feedback to a user.

[1022] The "emotion engine" is an artificial intelligence technology that detects the user's emotional state from their facial expressions and voice, and adjusts their behavior based on that.

[1023] A "product description video" is a video containing detailed product explanations generated using AI.

[1024] A "virtual assistant" is a digital character that appears in the user's field of vision using AR technology and serves to support and guide the user.

[1025] The present invention is a system that fully utilizes generative AI, AR technology, sensory technology, emotion engine, and haptics technology to enable users to have an online learning or shopping experience at home. Specific embodiments for implementing this system are described below.

[1026] Hardware and Software

[1027] Smart glasses: Uses AR technology to display virtual instructors and virtual assistants.

[1028] Generative AI models (e.g., GPT-4): Generate video or text content based on user selections.

[1029] Motion sensors (e.g. Kinect): Track user movements in real time.

[1030] Haptic devices (e.g., SenseGlove): Provide tactile feedback to the user.

[1031] Server: Responsible for data processing and storage, and equipped with an AI analysis engine and emotion engine.

[1032] Emotion engine (e.g. Microsoft Azure Emotion API): Analyzes the user's emotional state.

[1033] System Operation

[1034] First, the user launches the application and selects information about Aikido training or products. By entering user information, the form they want to learn, and categories of interest, the following steps are carried out.

[1035] Video generation using generative AI

[1036] The server uses a generative AI model (e.g., GPT-4) to automatically generate a basic Aikido form or a product instruction video based on the user's selection. For example, input a prompt such as, "Please explain the features and functions of a new smartphone in a video of no more than 3 minutes."

[1037] AR display

[1038] Through smart glasses, a virtual instructor or virtual assistant is displayed in the user's field of view based on videos generated by AR technology, providing an immersive training or shopping experience.

[1039] Motion sensor tracking

[1040] When a user starts practicing, a motion sensor (e.g., Kinect) tracks the user's movements in real time. This data is sent to a server, where an AI analysis engine evaluates the accuracy of the movements.

[1041] feedback

[1042] The evaluation results are sent from the server to the device and provided to the user. The user can then modify their movements based on this feedback. Tactile feedback is also provided using a haptic device (e.g., SenseGlove).

[1043] Emotion analysis using an emotion engine

[1044] An emotion engine (e.g., Microsoft Azure Emotion API) analyzes the user's emotional state from their facial expressions and voice and dynamically adjusts the training or shopping experience, for example, using prompts such as, "Please rate the user's emotional state (interested, confused, etc.) from their facial expressions and voice while they are watching the smartphone introductory video."

[1045] Specific examples

[1046] 1. If a user wants to practice advanced shoulder-grab techniques, the server generates a detailed video and sends it to the device. The user can then visualize the virtual instructor's movements and replicate them.

[1047] 2. In the case of shopping, when a user is looking for a new smartphone, generative AI will generate a detailed product video, and AR will bring a virtual shopping assistant into view. An emotion engine will read the user's interests and confusions and make appropriate suggestions.

[1048] In this way, the present invention provides a user with a high sense of realism and appropriate feedback, and responds to individual emotional states, thereby providing a more satisfying experience.

[1049] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1050] Step 1:

[1051] The user starts the application and selects information about Aikido training or products. They enter user information, the forms they want to learn, and categories they are interested in. The entered information is sent from the device to the server.

[1052] Step 2:

[1053] The server uses a generative AI model to generate a basic Aikido kata or a product instruction video based on the user's selection. For example, the prompt might be, "Please explain the features and functions of your new smartphone in a video of no more than three minutes." The video generated based on this data processing is then stored on the server.

[1054] Step 3:

[1055] The device receives the generated video from the server and displays a virtual instructor or virtual assistant through the smart glasses using AR technology, allowing users to watch the video and enjoy an immersive training or shopping experience.

[1056] Step 4:

[1057] When a user starts practicing, the motion sensor tracks the user's movements in real time. This movement data is sent from the device to the server. The server analyzes this data using an AI analysis engine and evaluates the accuracy of the movements. This evaluation result is generated.

[1058] Step 5:

[1059] The evaluation results are sent from the server to the device, which then provides visual and tactile feedback to the user. The feedback indicates areas for improvement in the user's movements, and haptic feedback is also provided using a haptic device. This data processing allows for more accurate movement correction.

[1060] Step 6:

[1061] The emotion engine analyzes the user's emotional state from their facial expressions and voice. This analysis data is sent to the server, which then dynamically adjusts the training content based on the user's emotional state. For example, the prompt might be, "Please rate the user's emotional state (interest, confusion, etc.) from their facial expressions and voice while they are watching the smartphone introductory video." Based on this data processing, the training content is adjusted appropriately.

[1062] Step 7:

[1063] The user adjusts their movements based on the feedback and continues practicing or shopping. The device continues to collect data using motion sensors and an emotion engine, and sends it to the server. The server continues to analyze the data in real time and provide feedback, allowing the user to achieve their goals efficiently and accurately.

[1064] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1065] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1066] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1067] [Fourth embodiment]

[1068] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1069] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1070] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1071] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1072] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1073] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1074] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1075] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1076] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1077] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1078] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1079] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1080] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1081] ---

[1082] The present invention is a system that combines generative AI, AR technology, sensory technology, and other technologies to allow users to practice Aikido at home. Specific embodiments for implementing this system will be described below.

[1083] First, the user starts the application and selects their Aikido level and the type of kata they want to learn. At this time, the terminal accepts the user's input and sends the selection information to the server.

[1084] The server uses generative AI to automatically generate a video of a basic Aikido kata based on the kata selected by the user. The generated video is stored on the server and sent to the user's device.

[1085] The device then uses the received video to display a virtual instructor in the user's field of view using AR technology, allowing the user to practice with the virtual instructor in a real practice space. Additionally, a haptic device is provided using sensory technology to provide feedback on the user's movements, allowing the user to continue practicing while experiencing the accuracy of their movements.

[1086] When a user begins practicing, the device's motion sensor or camera tracks the user's movements in real time. This data is sent from the device to the server. The server then analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The evaluation results are then sent back to the device.

[1087] Based on the evaluation results received from the server, the device provides the user with visual and tactile feedback on the accuracy of their movements and areas for improvement. By correcting their movements and practicing again, the user can master the kata accurately.

[1088] For example, when a user practices the advanced kata of Katatori Ikkyo, the server generates a detailed video of the kata and sends it to the device. The user visually watches the virtual instructor's movements through the goggles and performs the movements in sync with them. If there is a misalignment in the movements, the device's haptic device vibrates specific parts of the body, providing feedback on what needs to be corrected. The server's AI analysis engine then analyzes the user's movements in detail and suggests specific corrections.

[1089] This system allows users to have an experience similar to that of a real classroom, even from the comfort of their own home, and aims to accurately master techniques. It also provides custom feedback based on the user's level and the form they select, enabling efficient practice.

[1090] As described above, the present invention provides a system for efficiently and accurately practicing Aikido, and provides the user with a realistic training environment and accurate feedback.

[1091] The processing flow will be explained below.

[1092] ---

[1093] Step 1:

[1094] The user launches the application and fills in the interface to select their Aikido level (beginner, intermediate, advanced) and the type of kata they wish to learn.

[1095] Step 2:

[1096] The terminal receives the user's input information and transmits the selected data to the server.

[1097] Step 3:

[1098] Based on the information received by the server, a generation AI is used to automatically generate a video of the basic Aikido form specified by the user. The generated video is then saved on the server.

[1099] Step 4:

[1100] The server transmits the generated video to the user's terminal.

[1101] Step 5:

[1102] The device receives the video and starts the AR engine to prepare to display the virtual instructor in the user's field of view.

[1103] Step 6:

[1104] The device initializes the haptic device and configures it to provide tactile feedback in response to the user's movements.

[1105] Step 7:

[1106] The user puts on the goggles and sensory device and confirms they are ready.

[1107] Step 8:

[1108] When the user presses the start button, the terminal receives the start signal and the virtual instructor appears in the user's field of vision.

[1109] Step 9:

[1110] The virtual instructor initiates the selected form of movement and the user replicates the same movement.

[1111] Step 10:

[1112] The device uses motion sensors or a camera to track the user's movements in real time.

[1113] Step 11:

[1114] The device collects data on the user's movements and sends it to the server.

[1115] Step 12:

[1116] The tracking data received by the server is analyzed using an AI analysis engine.

[1117] Step 13:

[1118] The server evaluates the accuracy of the user's movements based on the analysis results and identifies areas for improvement. The evaluation results are stored on the server.

[1119] Step 14:

[1120] The server sends the evaluation results and correction instructions to the user's terminal.

[1121] Step 15:

[1122] The terminal receives the evaluation results from the server and provides visual and tactile feedback to the user.

[1123] Step 16:

[1124] The user practices the form again while correcting their movements based on the feedback.

[1125] Step 17:

[1126] The device again tracks the user's movements and sends them to the server, and this process is repeated until the user's movements are accurate.

[1127] ---

[1128] These are the specific processing steps of the program. This process allows the user to receive feedback on their movements and accurately correct them, allowing them to master Aikido forms.

[1129] Example 1

[1130] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1131] Traditional Aikido training required a dedicated dojo and instructor, making it difficult to practice efficiently and accurately at home. Furthermore, there were limited ways to check whether one's movements were accurate, making self-improvement difficult. This created a major hurdle for beginners and intermediates to master accurate techniques.

[1132] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1133] In this invention, the server includes means for automatically generating videos of basic Aikido forms based on user selections using a generative AI model, means for displaying a virtual instructor in the user's field of view based on the generated videos using AR technology, and means for providing the user with tactile feedback using sensory technology, thereby enabling users to efficiently practice Aikido from the comfort of their own homes while receiving accurate feedback in real time.

[1134] A "generative AI model" is an artificial intelligence algorithm that automatically generates information based on data for a specific task.

[1135] A "prompt sentence" is text-based input data used to give instructions to a generative AI model.

[1136] "AR technology" is a technology that displays digital information superimposed on the real world.

[1137] "Sensory technology" is technology that provides users with visual, tactile, or other feedback.

[1138] A "haptic device" is a device that conveys tactile feedback to a user.

[1139] A "server" is a computer system that processes and stores data on a network.

[1140] A "terminal" is a device that a user operates to input and output data.

[1141] "Means for tracking user movements in real time" refers to a function that uses motion sensors and cameras to capture user movements sequentially.

[1142] An "AI analysis engine" is a system that uses AI technology to analyze input data and provide specific insights and evaluations.

[1143] "Means for assessing the accuracy of movements" refers to a method for measuring the accuracy of a user's movements using an AI analysis engine.

[1144] "Means for providing feedback" refers to a function that provides visual and tactile information to the user, informing them of areas for improvement in their actions.

[1145] The present invention relates to a system that combines generative AI models, AR technology, sensory technology, and other technologies to enable users to practice Aikido at home. Specific embodiments for implementing this system are described below.

[1146] The user launches the smartphone application and selects their Aikido level and the type of kata they wish to learn. For example, an intermediate practitioner selects "Intermediate" and the kata "Shouldering Icchi." At this point, the device accepts the selection information and sends it to the server. The information is sent in JSON format via the REST API.

[1147] The server automatically generates a video of the basic Aikido kata based on the kata selected by the user using a generative AI model (e.g., OpenAI's GPT-4). Specifically, the following prompt sentence is input to the generative AI model:

[1148] Example prompt sentence:

[1149] "Create an instructional video with detailed movements for the intermediate Katatori Ikkyo technique. Include a complete explanation of the steps, grips, and throws."

[1150] The server saves the generated video in cloud storage (e.g., AWS S3) and sends the URL of the video to the device.

[1151] The device uses the received video URL to stream the video and uses AR technology (e.g., Unity or ARKit) to display the virtual instructor in the user's field of view, allowing the user to practice with the virtual instructor using their smartphone or AR goggles.

[1152] When a user begins practicing, the device's motion sensors and camera track the user's movements in real time. The device then sends the data obtained from these sensors to a server. The server then analyzes the received tracking data using an AI analysis engine (e.g., TensorFlow or PyTorch) and evaluates the user's movements. The evaluation results are fed back as specific corrections and sent back to the device.

[1153] The device provides the user with visual and tactile feedback based on the evaluation results received from the server. For example, the device may display feedback such as "your right hand is positioned too low" on the screen and vibrate the user's right hand using a haptic device. In this way, the user can correct their movements as they practice.

[1154] As described above, the present invention allows users to experience a similar experience to practicing in a real classroom from the comfort of their own home. It provides an efficient training method aimed at mastering accurate techniques, and provides users with a realistic practice environment and accurate feedback.

[1155] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1156] Step 1:

[1157] The user launches the application on their smartphone and is presented with a home screen to begin Aikido practice. The user taps the "Start Practice" button and selects their Aikido level and the kata they wish to learn. The user's level and kata selection information are obtained as input. The selection information is sent from the device to the server as output.

[1158] Step 2:

[1159] The device sends the user's selected level and type information to the server. During this process, it receives the user's selection information as input and sends it to the server in JSON format via the REST API. The output returns a successful status.

[1160] Step 3:

[1161] Based on the received data, the server uses a generative AI model to automatically generate a video of the basic Aikido forms. As input, the user's selection information and a prompt sentence are provided to the generative AI model. Specifically, the following prompt sentence is used:

[1162] "Create an instructional video with detailed movements for the intermediate Katatori Ikkyo technique. Include a complete explanation of the steps, grips, and throws."

[1163] The output is a video generated by the generative AI model, which the server stores in cloud storage (e.g., AWS S3).

[1164] Step 4:

[1165] The server sends the URL of the stored video to the device. As input, the video URL is retrieved from the cloud storage and sent to the user's device. As output, the video URL is provided to the device.

[1166] Step 5:

[1167] The device streams the video using the received video URL. As input, it receives the video URL and starts the streaming process. It uses AR technology (e.g., Unity or ARKit) to display the virtual instructor in the user's field of view. As output, it displays the virtual instructor in the user's field of view.

[1168] Step 6:

[1169] The user begins practicing while watching the virtual instructor's movements displayed in AR. As input, the user sees the virtual instructor's movements and synchronizes their own movements. As output, the user's movements are tracked in real time.

[1170] Step 7:

[1171] The device's motion sensors and camera track the user's movements in real time. As input, user movement data is obtained from the motion sensors and camera. As output, tracking data is generated.

[1172] Step 8:

[1173] The device sends the captured tracking data to the server. As input, it receives the acquired tracking data and sends it to the server. As output, it returns a status indicating that the transmission was successful.

[1174] Step 9:

[1175] The server analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The tracking data is provided to the AI ​​analysis engine as input. The movement evaluation results are obtained as output. The evaluation results include the accuracy of the user's movements and areas for improvement.

[1176] Step 10:

[1177] The server sends the motion evaluation results to the terminal. As input, the evaluation results are generated and sent to the terminal. As output, the evaluation results are provided to the terminal.

[1178] Step 11:

[1179] The device provides visual and tactile feedback to the user based on the evaluation results received from the server. As input, it receives the evaluation results and presents them to the user. For example, feedback such as "your right hand is positioned low" is displayed on the screen, and the haptic device vibrates the right hand. This allows the user to correct their own movements. As output, feedback information is generated and provided to the user.

[1180] Step 12:

[1181] The user corrects their movements based on the provided feedback and continues practicing again. As input, they receive the feedback and correct their movements. As output, the correct movements are gradually learned.

[1182] (Application example 1)

[1183] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1184] It is difficult for new employees and inexperienced personnel to quickly learn how to operate robots in factories or on production lines. Conventional training methods require training using actual machines, which is costly and time-consuming. Furthermore, it requires a lot of effort from the instructor, making efficient training difficult. The present invention solves these problems by providing a training system that allows employees to efficiently and effectively learn how to operate robots.

[1185] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1186] In this invention, the server includes: means for automatically generating a video of a basic form of a work procedure based on a user's selection using a generation AI; means for displaying a virtual instructor in the user's field of view based on the generated video using AR technology; means for providing the user with tactile feedback using haptic technology; means for tracking the user's movements in real time using a motion sensor or a camera; means for receiving data on the user's movements at the server, evaluating the accuracy of the movements using an AI analysis engine, and generating and transmitting the evaluation results; means for providing the user with the evaluation results received from the server at a terminal and encouraging corrections based on the feedback; and means for supporting training for new employees using virtual instructor display technology for work training, thereby enabling new employees and inexperienced people to quickly learn robot operation.

[1187] "Generative AI" is an artificial intelligence technology that automatically generates videos of actions and work procedures based on user input.

[1188] "AR technology" is an abbreviation for augmented reality technology, which is a technology that displays virtual information overlaid on the real field of view.

[1189] "Haptic technology" refers to technology for providing tactile feedback to users, particularly technology that uses haptic devices.

[1190] A "motion sensor" is a sensor that detects the user's physical movements and acquires them as digital data in real time.

[1191] A "camera" is a device that captures a user's actions as video data and analyzes it.

[1192] The "AI analysis engine" is an artificial intelligence technology that analyzes acquired user movement data and evaluates the accuracy of the movements.

[1193] A "virtual instructor" is a virtual teacher who is displayed in the user's field of view using AR technology.

[1194] "Terminal" refers to the device used by the user for operation and viewing, and is the device that communicates with the server and provides feedback.

[1195] The "training support system" is an integrated system that enables new employees and inexperienced personnel to efficiently learn how to operate robots and other work procedures.

[1196] A "work procedure" is a series of steps or operating methods for performing a specific task.

[1197] MODE FOR CARRYING OUT THE INVENTION

[1198] This invention relates to a robot operation training system for new factory employees. Specifically, it provides an efficient training method using generative AI, AR technology, and haptic technology.

[1199] First, the user starts the training. The user puts on a device (e.g., a smart headset) and launches the application. First, the user is shown a screen where they can select their skill level and the type of work procedure they want to learn. The device accepts the user's input and sends the selection information to the server.

[1200] Next, the server uses a generative AI to automatically generate a basic video based on the work steps selected by the user. The generated video is stored on the server and sent to the user's device. The server uses an AI analysis engine to do this.

[1201] The received video is played on the device, and a virtual instructor is displayed in the user's field of view using AR technology. The user follows the virtual instructor's instructions and begins training to operate the robot. The virtual instructor explains the user's movements in real time and provides visual guidance.

[1202] When a user actually performs an action, a motion sensor or camera tracks the user's movement in real time. This tracking data is sent from the device to a server. The server analyzes the received tracking data using an AI analysis engine and evaluates the user's movement.

[1203] The evaluation results are sent back to the device from the server, and the device then provides feedback to the user. Feedback can be visual or tactile, with tactile feedback provided using a haptic device. Specifically, a haptic glove worn on the user's hand vibrates when the user has performed an accurate movement or when a correction is required.

[1204] This allows users to efficiently train together with the virtual instructor while experiencing the accuracy of their movements in real time. For example, if a user is learning how to "install a part," the server generates a detailed video of the procedure and sends it to the device. The user can visually confirm the virtual instructor's movements through the headset and perform the appropriate movements. If there is any deviation in the movements, the haptic device will vibrate specific parts of the body, providing feedback on what needs to be corrected. The server's AI analysis engine then analyzes the user's movements in detail and suggests specific corrections.

[1205] This system allows new employees and those with no experience to quickly learn how to operate robots, ensuring efficient training.

[1206] Generative AI model prompt example:

[1207] "We are building a training program for new employees to learn the operating procedures for a robotic assembly task. The operating procedures need to be generated as a series of videos."

[1208] The above is an embodiment of the present invention.

[1209] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1210] Step 1:

[1211] The user launches the application on the device. A screen appears in which the user can select their skill level and the type of work procedure they wish to learn. The user's selection (input data) is sent from the device to the server. The input is the user's level and work procedure, and the output is the selection information sent to the server.

[1212] Step 2:

[1213] The server generates a prompt for the generative AI model based on the received selection information. Based on this prompt, the generative AI generates a basic video corresponding to the user's selection. The input is the user's selection information, and the output is a video of the generated work procedure.

[1214] Step 3:

[1215] The server sends the generated video to the terminal. The input is the generated video, and the output is the transmission of video data to the terminal.

[1216] Step 4:

[1217] The device plays the received video and uses AR technology to display the virtual instructor in the user's field of view. The user receives visual guidance through the device's display. The input is the video data, and the output is the display of the virtual instructor.

[1218] Step 5:

[1219] The user follows the instructions of the virtual instructor and begins training. The user's movements are tracked in real time by a motion sensor or camera. The input is the user's movements, and the output is the tracking data.

[1220] Step 6:

[1221] The terminal sends tracking data to the server. The input is tracking data, and the output is data transmission to the server.

[1222] Step 7:

[1223] The server analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The result is an evaluation of the accuracy of the movements and points to be corrected, which is then sent to the device. The input is the tracking data, and the output is the evaluation result.

[1224] Step 8:

[1225] The terminal provides the user with the evaluation results received from the server. The terminal displays visual feedback and simultaneously sends instructions to the haptic device to provide tactile feedback. The input is the evaluation results, and the output is visual and tactile feedback.

[1226] Step 9:

[1227] The user then corrects their movements based on the feedback and repeats the training. Through this feedback cycle, the user gradually acquires accurate movements. The input is visual and tactile feedback, and the output is the user's corrected movements.

[1228] The above is a specific explanation of each processing step of the program.

[1229] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1230] ---

[1231] The present invention is a system that combines generative AI, AR technology, sensory technology, an emotion engine, and other technologies to enable users to practice Aikido at home. A specific embodiment for implementing this system will be described.

[1232] First, the user starts the application and selects their Aikido level and the type of kata they want to learn. At this time, the terminal accepts the user's input and sends the selection information to the server.

[1233] The server uses generative AI to automatically generate a video of a basic Aikido kata based on the kata selected by the user. The generated video is stored on the server and sent to the user's device.

[1234] The device then uses the received video to display a virtual instructor in the user's field of view using AR technology, allowing the user to practice with the virtual instructor in a real practice space. Additionally, a haptic device is provided using sensory technology to provide feedback on the user's movements, allowing the user to continue practicing while experiencing the accuracy of their movements.

[1235] When a user begins practicing, the device's motion sensor or camera tracks the user's movements in real time. This data is sent from the device to the server. The server then analyzes the received tracking data using an AI analysis engine and evaluates the user's movements. The evaluation results are then sent back to the device.

[1236] Based on the evaluation results received from the server, the device provides the user with visual and tactile feedback on the accuracy of their movements and areas for improvement. By correcting their movements and practicing again, the user can master the kata accurately.

[1237] Furthermore, the present invention incorporates an emotion engine that recognizes emotions from the user's facial expressions and voice. The device uses the emotion engine to determine the user's emotions during training in real time and adjust the content and difficulty of the training accordingly. For example, if the user is feeling frustrated, the device sends that information to the server, and the server generates a training plan based on the user's emotional state and sends it to the device.

[1238] For example, when a user practices the advanced kata, Katatori Ikkyo, the server generates a detailed video of the kata and sends it to the device. The user watches the virtual instructor's movements and replicates them. A motion sensor tracks the user's movements and sends the data to the server. The server's AI analysis engine analyzes the user's movements and generates feedback. At the same time, the emotion engine recognizes the user's face and analyzes their voice to determine their emotional state. If the user is feeling frustrated or stressed, the server adjusts the practice and provides advice on how to relax.

[1239] In this way, the present invention provides a system for efficient and accurate Aikido training, not only providing the user with a realistic training environment and accurate feedback, but also providing support according to the user's emotional state.

[1240] The processing flow will be explained below.

[1241] ---

[1242] Step 1:

[1243] The user starts the application and selects their Aikido level (beginner, intermediate, advanced) and the type of kata they want to learn. By entering their input into the interface, the selection information is sent to the terminal.

[1244] Step 2:

[1245] The terminal receives the user's input information and transmits this information to the server.

[1246] Step 3:

[1247] Based on the information received by the server, a generation AI is used to automatically generate a video of the basic Aikido form specified by the user. The generated video is saved on the server.

[1248] Step 4:

[1249] The server transmits the generated video to the user's terminal.

[1250] Step 5:

[1251] The device receives the video, starts the AR engine, and prepares to display the virtual instructor in the user's field of view. At the same time, it initializes the haptics device and configures it to provide tactile feedback according to the user's movements.

[1252] Step 6:

[1253] The device displays a message prompting the user to put on the goggles and sensory device, and waits until they are ready.

[1254] Step 7:

[1255] The user puts on the goggles and sensory device and confirms that they are ready. The user presses the start button.

[1256] Step 8:

[1257] The terminal receives the start signal, and the virtual instructor appears in the user's field of view. The virtual instructor begins the selected form of movement, and the user performs the movement accordingly.

[1258] Step 9:

[1259] The device uses motion sensors or a camera to track the user's movements in real time.

[1260] Step 10:

[1261] The device collects tracking data and sends it to the server. At the same time, the device's emotion engine analyzes the user's facial expressions and voice and sends the results to the server.

[1262] Step 11:

[1263] The tracking data received by the server is analyzed using an AI analysis engine to evaluate the accuracy of the user's movements, and the evaluation results are stored on the server.

[1264] Step 12:

[1265] The server analyzes the received emotional data to determine the user's emotional state. If the user is feeling frustrated or stressed, the content and difficulty of the training will be adjusted based on that information.

[1266] Step 13:

[1267] The server generates feedback and correction instructions appropriate for the user based on the evaluation results and emotional state, and sends them to the terminal.

[1268] Step 14:

[1269] The device receives feedback and correction instructions from the server and provides them to the user visually and tactilely, automatically adjusting the difficulty of the training as needed and suggesting relaxation techniques.

[1270] Step 15:

[1271] Based on the feedback provided by the user, the user practices the form again, correcting the movements.

[1272] Step 16:

[1273] The device again tracks the user's movements and sends the data to the server, and this process is repeated until the user's movements are accurate and emotionally stable.

[1274] ---

[1275] These are the specific processing steps of the program. This process allows the user to receive feedback on their movements and accurately correct them, allowing them to master Aikido forms. Furthermore, the real-time emotional feedback provided by the emotion engine can further improve the quality of training.

[1276] Example 2

[1277] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1278] In recent years, there has been a growing demand for methods to efficiently learn Aikido at home. Conventional methods require training at a dojo or rely on books and video materials, which require the guidance of an instructor. Furthermore, self-practice is difficult because it is difficult to understand and experience the correct movements, making it difficult to practice efficiently. Furthermore, there is also the issue of difficulty in maintaining motivation due to insufficient feedback and emotional awareness during self-practice. There is an urgent need to solve these problems and provide a system that allows users to effectively and accurately learn Aikido at home.

[1279] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1280] In this invention, the server includes means for automatically generating videos of basic Aikido forms based on a user's selection using a generation AI, means for displaying a virtual instructor in the user's field of view based on the generated videos using augmented reality technology, means for providing the user with tactile feedback using sensory technology, means for tracking the user's movements in real time using a motion detector or camera, means for evaluating the accuracy of the movements using an AI analysis engine and generating and transmitting the evaluation results, and means for analyzing the user's facial expressions and voice in real time and adjusting the training content and difficulty based on the user's emotional state. This allows users to receive a realistic training environment and accurate feedback even at home, enabling them to accurately master the movements.

[1281] "Generative AI" refers to a system that uses artificial intelligence technology to automatically generate the necessary content based on user selections.

[1282] "Augmented reality technology" is a technology that overlays computer-generated information onto real-world images and videos.

[1283] "Sensory technology" refers to technology that allows users to obtain information through their senses, such as sight, hearing, and touch.

[1284] A "motion detector" is a device that detects a user's movements in real time and acquires them as data.

[1285] An "artificial intelligence analysis engine" is an artificial intelligence system that analyzes collected data and generates appropriate feedback and actions based on the results.

[1286] "Feedback" refers to information provided to the user by the system regarding corrections and areas for improvement.

[1287] "Real-time tracking" refers to a technology that tracks a user's movements instantly without any time delay and processes the data.

[1288] A "virtual instructor" is a virtual instructor who appears in the user's field of view through AR technology.

[1289] "Adjusting training content" refers to changing the difficulty and content of the training menu based on the user's movements and emotional state.

[1290] "User's emotional state" refers to the emotions (joy, frustration, etc.) that the user is feeling during training.

[1291] The present invention is a system that combines generative AI, augmented reality technology, sensory technology, an artificial intelligence analysis engine, and emotion recognition technology to enable users to effectively practice Aikido at home. Specific embodiments of this system are described below.

[1292] First, the user launches the application on a device such as a smartphone or tablet and selects their Aikido level and the kata they wish to learn. The device then sends this selection information to the server, using the device's input device and communication module.

[1293] Based on the received information, the server uses a generative AI model (e.g., OpenAI's GPT-4) to automatically generate an instructional video for a basic Aikido form. An example of a specific prompt used in the generation process is, "Please generate a detailed video about the basic Aikido form of Shoulder Drill Ikkyo." The generated video is stored on the server and transmitted to the terminal using data streaming technology (e.g., RTMP protocol).

[1294] The device plays the received video and uses augmented reality technology (e.g., ARKit) to display the virtual instructor in the user's field of view, allowing the user to practice with the virtual instructor in the real practice space.

[1295] When a user starts practicing, the device's motion sensors and camera track the user's movements in real time, and this data is sent from the device to the server. The server analyzes the received tracking data using an artificial intelligence analysis engine (e.g., TensorFlow) to accurately evaluate the user's movements. This evaluation result is then sent back to the device.

[1296] Based on the evaluation results, the device provides the user with visual and tactile feedback on the accuracy of their movements and areas for improvement. A haptic device (for example, a wearable device with vibration) is provided that uses sensory technology to provide feedback on the user's movements. Specific feedback might include, "It would be better if you moved your right arm a little more to the left." The user can then correct their movements and continue practicing.

[1297] Furthermore, the device uses an emotion engine (e.g., Affectiva's emotion recognition API) to analyze the user's emotional state in real time. It analyzes the user's facial expressions and voice through the camera and microphone to determine emotions such as joy or frustration. If the user is feeling frustrated, the device sends this information to the server, which then generates a training plan based on the user's emotional state and sends it to the device. For example, it might display a guide message such as "Relax and take a deep breath" and temporarily reduce the difficulty of the exercises.

[1298] This system allows users to practice at home in a realistic environment and receive accurate feedback, enabling them to accurately master movements.

[1299] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1300] Step 1:

[1301] The user launches the application and selects their Aikido level and the kata they wish to practice.

[1302] Input: Aikido level and kata type selected by user operation.

[1303] Output: JSON data containing the selection information.

[1304] Specific operation: The user selects "Advanced Form" or "Shoulder Training" from the menu on the device screen and presses the OK button. The selected information is converted into JSON format and passed to the next process.

[1305] Step 2:

[1306] The terminal transmits the selection information to the server.

[1307] Input: JSON data containing the user's selections.

[1308] Output: HTTPS request sent to the server.

[1309] Specific operation: The device generates JSON data of the selected information and sends it to the server using the HTTPS protocol. Specifically, it sends a POST request to the endpoint URL.

[1310] Step 3:

[1311] The server uses the generative AI model to generate videos of basic Aikido forms and sends them to the device.

[1312] Input: JSON data containing the user's selections.

[1313] Output: Generated video data of Aikido basic forms.

[1314] Specific operation: Based on the received JSON data, the server inputs the prompt "Please generate a detailed video about the basic form of Aikido's Shoulder Pulling Ikkyo" into the generative AI model. The AI ​​model generates the video and stores it on the server. The generated video data is sent to the terminal via the RTMP protocol.

[1315] Step 4:

[1316] The device displays the received video and displays the virtual instructor using AR technology.

[1317] Input: Generated video data of Aikido basic forms.

[1318] Output: Virtual instructor display.

[1319] Specific operation: The device plays the received video data and uses ARKit to display the virtual instructor in the user's field of view. Specifically, a 3D model of the virtual instructor is overlaid on the device's camera image.

[1320] Step 5:

[1321] The user begins practicing and the device tracks the user's movements.

[1322] Input: User movement.

[1323] Output: Tracking data.

[1324] Specific movements: The user watches the movements of the virtual instructor and then reproduces them. The device's motion sensors and camera capture the user's movements in real time, obtaining coordinate data and movement speed data.

[1325] Step 6:

[1326] The device sends the tracking data to the server.

[1327] Input: Tracking data.

[1328] Output: Sends a WebSocket request to the server.

[1329] What it does: The device converts the acquired tracking data into JSON format and sends it to the server using the secure WebSocket protocol.

[1330] Step 7:

[1331] The server analyzes the tracking data, generates feedback, and sends it to the device.

[1332] Input: Tracking data.

[1333] Output: Feedback data.

[1334] Specific operation: The server analyzes the received tracking data using an artificial intelligence analysis engine (e.g., TensorFlow). As a result of the analysis, it evaluates the accuracy of the user's movements and areas for improvement, and generates specific feedback such as "It would be better if you moved your right arm a little more to the left." This feedback data is sent to the device in JSON format.

[1335] Step 8:

[1336] The device provides feedback to the user and instructs them to correct their movements.

[1337] Input: Feedback data.

[1338] Output: Visual and tactile feedback.

[1339] Specific actions: The device reflects the received feedback data on the display screen, provides specific correction instructions to the user, and controls the haptic device to provide feedback to the user's physical sensations.

[1340] Step 9:

[1341] The user corrects the movement and repeats the practice again.

[1342] Input: Feedback information.

[1343] Output: Corrected movement.

[1344] Specific movements: The user corrects their movements based on the feedback and repeats the practice while watching the virtual instructor's movements again.

[1345] Step 10:

[1346] The device uses an emotion engine to analyze the user's emotions.

[1347] Input: User's facial and voice data.

[1348] Output: Emotional state data.

[1349] How it works: The device uses a camera and microphone to capture the user's facial expressions and voice, then analyzes them using an emotion engine to determine a specific emotional state (e.g., joy, frustration) in real time.

[1350] Step 11:

[1351] The server adjusts the training content based on the emotional data and sends it to the device.

[1352] Input: Emotional state data.

[1353] Output: A tailored training plan.

[1354] Specific operation: The server analyzes the emotional state data sent from the device and generates a training plan according to the user's emotions. Specifically, if the user feels frustrated, the server generates a plan that lowers the difficulty of the training or includes a guide message such as "Relax and take a deep breath." This plan is sent to the device and applied to the user.

[1355] (Application example 2)

[1356] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1357] Conventional online training and shopping systems have the problem that it is difficult for users to get the same experience as a real instructor or salesperson at home. In particular, the user experience is poor due to the lack of accurate motion feedback, reproduction of the product touch feeling, and appropriate responses based on the user's emotional state. This leads to issues such as a decrease in user learning and purchasing motivation, and a low overall user satisfaction.

[1358] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for automatically generating a video of a basic Aikido form based on a user's selection using a generation AI, means for displaying a virtual instructor in the user's field of view using AR technology, means for providing the user with haptic feedback using sensory technology, means for tracking the user's movements in real time using a motion sensor or a camera, means for receiving data on the user's movements in the server, evaluating the accuracy of the movements using an AI analysis engine, and generating and transmitting the evaluation results, means for providing the user with the evaluation results received from the server at the terminal and prompting the user to make corrections based on the feedback, means for generating a product explanation video using a generation AI, means for displaying a virtual assistant in the field of view using AR technology, means for tracking the user's line of sight and movements using a motion sensor, analyzing the data, and providing feedback, means for evaluating the user's emotional state using an emotion engine and dynamically adjusting the training content, and means for providing tactile feedback of the product's texture using a haptic device. This allows users to have a realistic practice and shopping experience at home, making it easier to accurately learn movements and experience the texture of products, and by allowing the system to respond appropriately based on the user's emotional state, it is possible to improve overall user satisfaction.

[1359] "Generative AI" is a type of artificial intelligence technology that automatically generates necessary content based on user specifications.

[1360] "AR technology" stands for augmented reality technology, which is a technology that displays digital content overlaid on the real world.

[1361] "Sensory technology" refers to technology that provides feedback to the user's senses, and specifically includes feedback to the sense of touch and vision.

[1362] A "motion sensor" is a device that detects and tracks a user's movements.

[1363] A "camera" is a device that captures video and analyzes the user's movements based on that video.

[1364] A "server" is a computer system connected to a network that processes and stores various types of data.

[1365] An "AI analysis engine" is a system that uses artificial intelligence technology to analyze collected data and make specific evaluations and predictions.

[1366] A "virtual instructor" is a digital instructor that appears in the user's field of vision using AR technology and is responsible for providing training guidance.

[1367] "Feedback" is information for guidance and correction that the system provides based on the user's movements and state.

[1368] A "haptic device" is a device for providing tactile feedback to a user.

[1369] The "emotion engine" is an artificial intelligence technology that detects the user's emotional state from their facial expressions and voice, and adjusts their behavior based on that.

[1370] A "product description video" is a video containing detailed product explanations generated using AI.

[1371] A "virtual assistant" is a digital character that appears in the user's field of vision using AR technology and serves to support and guide the user.

[1372] The present invention is a system that fully utilizes generative AI, AR technology, sensory technology, emotion engine, and haptics technology to enable users to have an online learning or shopping experience at home. Specific embodiments for implementing this system are described below.

[1373] Hardware and Software

[1374] Smart glasses: Uses AR technology to display virtual instructors and virtual assistants.

[1375] Generative AI models (e.g., GPT-4): Generate video or text content based on user selections.

[1376] Motion sensors (e.g. Kinect): Track user movements in real time.

[1377] Haptic devices (e.g., SenseGlove): Provide tactile feedback to the user.

[1378] Server: Responsible for data processing and storage, and equipped with an AI analysis engine and emotion engine.

[1379] Emotion engine (e.g. Microsoft Azure Emotion API): Analyzes the user's emotional state.

[1380] System Operation

[1381] First, the user launches the application and selects information about Aikido training or products. By entering user information, the form they want to learn, and categories of interest, the following steps are carried out.

[1382] Video generation using generative AI

[1383] The server uses a generative AI model (e.g., GPT-4) to automatically generate a basic Aikido form or a product instruction video based on the user's selection. For example, input a prompt such as, "Please explain the features and functions of a new smartphone in a video of no more than 3 minutes."

[1384] AR display

[1385] Through smart glasses, a virtual instructor or virtual assistant is displayed in the user's field of view based on videos generated by AR technology, providing an immersive training or shopping experience.

[1386] Motion sensor tracking

[1387] When a user starts practicing, a motion sensor (e.g., Kinect) tracks the user's movements in real time. This data is sent to a server, where an AI analysis engine evaluates the accuracy of the movements.

[1388] feedback

[1389] The evaluation results are sent from the server to the device and provided to the user. The user can then modify their movements based on this feedback. Tactile feedback is also provided using a haptic device (e.g., SenseGlove).

[1390] Emotion analysis using an emotion engine

[1391] An emotion engine (e.g., Microsoft Azure Emotion API) analyzes the user's emotional state from their facial expressions and voice and dynamically adjusts the training or shopping experience, for example, using prompts such as, "Please rate the user's emotional state (interested, confused, etc.) from their facial expressions and voice while they are watching the smartphone introductory video."

[1392] Specific examples

[1393] 1. If a user wants to practice advanced shoulder-grab techniques, the server generates a detailed video and sends it to the device. The user can then visualize the virtual instructor's movements and replicate them.

[1394] 2. In the case of shopping, when a user is looking for a new smartphone, generative AI will generate a detailed product video, and AR will bring a virtual shopping assistant into view. An emotion engine will read the user's interests and confusions and make appropriate suggestions.

[1395] In this way, the present invention provides a user with a high sense of realism and appropriate feedback, and responds to individual emotional states, thereby providing a more satisfying experience.

[1396] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1397] Step 1:

[1398] The user starts the application and selects information about Aikido training or products. They enter user information, the forms they want to learn, and categories they are interested in. The entered information is sent from the device to the server.

[1399] Step 2:

[1400] The server uses a generative AI model to generate a basic Aikido kata or a product instruction video based on the user's selection. For example, the prompt might be, "Please explain the features and functions of your new smartphone in a video of no more than three minutes." The video generated based on this data processing is then stored on the server.

[1401] Step 3:

[1402] The device receives the generated video from the server and displays a virtual instructor or virtual assistant through the smart glasses using AR technology, allowing users to watch the video and enjoy an immersive training or shopping experience.

[1403] Step 4:

[1404] When a user starts practicing, the motion sensor tracks the user's movements in real time. This movement data is sent from the device to the server. The server analyzes this data using an AI analysis engine and evaluates the accuracy of the movements. This evaluation result is generated.

[1405] Step 5:

[1406] The evaluation results are sent from the server to the device, which then provides visual and tactile feedback to the user. The feedback indicates areas for improvement in the user's movements, and haptic feedback is also provided using a haptic device. This data processing allows for more accurate movement correction.

[1407] Step 6:

[1408] The emotion engine analyzes the user's emotional state from their facial expressions and voice. This analysis data is sent to the server, which then dynamically adjusts the training content based on the user's emotional state. For example, the prompt might be, "Please rate the user's emotional state (interest, confusion, etc.) from their facial expressions and voice while they are watching the smartphone introductory video." Based on this data processing, the training content is adjusted appropriately.

[1409] Step 7:

[1410] The user adjusts their movements based on the feedback and continues practicing or shopping. The device continues to collect data using motion sensors and an emotion engine, and sends it to the server. The server continues to analyze the data in real time and provide feedback, allowing the user to achieve their goals efficiently and accurately.

[1411] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1412] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1413] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1414] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1415] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1416] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1417] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1418] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1419] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1420] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1421] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1422] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1423] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1424] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1425] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1426] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1427] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1428] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1429] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1430] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1431] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1432] The following is further disclosed regarding the above embodiment.

[1433] (Claim 1)

[1434] A means for automatically generating videos of basic Aikido forms based on user selections using a generation AI;

[1435] a means for displaying a virtual instructor in the user's field of view based on the generated video using AR technology;

[1436] means for providing tactile feedback to the user using sensory technology;

[1437] a means for tracking user movements in real time using a motion sensor or camera;

[1438] A means for receiving data on the user's movements on a server, evaluating the accuracy of the movements using an AI analysis engine, and generating and transmitting evaluation results;

[1439] a means for providing the user with the evaluation results received from the server at the terminal and prompting the user to make corrections based on the feedback;

[1440] A system including:

[1441] (Claim 2)

[1442] 2. The system of claim 1, further comprising a user input means for selecting the user's Aikido level and the type of kata they wish to learn.

[1443] (Claim 3)

[1444] 10. The system of claim 1, comprising a haptic device that provides tactile feedback to a user based on sensory technology.

[1445] "Example 1"

[1446] (Claim 1)

[1447] A means for a user to launch an application for starting Aikido training and select an Aikido level and the type of kata they wish to learn;

[1448] means via the terminal for transmitting the selection information to a server;

[1449] A means for automatically generating videos of basic Aikido forms based on user selections using a generative AI model;

[1450] a means for displaying a virtual instructor in the user's field of view based on the generated video using AR technology;

[1451] means for providing tactile feedback to the user using sensory technology;

[1452] a means for tracking user movements in real time using a motion sensor or camera;

[1453] A means for receiving data on the user's movements on a server, evaluating the accuracy of the movements using an AI analysis engine, and generating and transmitting evaluation results;

[1454] a means for providing the user with the evaluation results received from the server at the terminal and prompting the user to make corrections based on the feedback;

[1455] A system including:

[1456] (Claim 2)

[1457] 2. The system of claim 1, further comprising a user input means for selecting the user's Aikido level and the type of kata they wish to learn.

[1458] (Claim 3)

[1459] 10. The system of claim 1, comprising a haptic device that provides tactile feedback to a user based on sensory technology.

[1460] "Application Example 1"

[1461] (Claim 1)

[1462] A means for automatically generating a basic video of the work procedure based on the user's selection using a generation AI;

[1463] a means for displaying a virtual instructor in the user's field of view based on the generated video using AR technology;

[1464] means for providing tactile feedback to the user using haptic technology;

[1465] a means for tracking user movements in real time using motion sensors or cameras;

[1466] A means for receiving data on the user's movements on a server, evaluating the accuracy of the movements using an AI analysis engine, and generating and transmitting evaluation results;

[1467] a means for providing the user with the evaluation results received from the server at the terminal and prompting the user to make corrections based on the feedback;

[1468] A means to assist in training new employees with virtual instructor viewing technology for job training;

[1469] A system including:

[1470] (Claim 2)

[1471] 10. The system of claim 1, further comprising a user input means for selecting the user's skill level and the type of work procedure they wish to learn.

[1472] (Claim 3)

[1473] 10. The system of claim 1, comprising a haptic device that provides tactile feedback to a user based on haptic technology.

[1474] "Example 2: Combining Emotion Engines"

[1475] (Claim 1)

[1476] A means for automatically generating videos of basic Aikido forms based on user selections using a generation AI;

[1477] means for displaying a virtual instructor in the user's field of view based on the generated video using augmented reality technology;

[1478] means for providing tactile feedback to the user using sensory technology;

[1479] means for tracking user movements in real time using a motion detector or camera;

[1480] A means for receiving data on the user's movements in a server, evaluating the accuracy of the movements using an artificial intelligence analysis engine, and generating and transmitting evaluation results;

[1481] a means for providing the user with the evaluation results received from the server at the terminal and prompting the user to make corrections based on the feedback;

[1482] A means for analyzing the user's facial expressions and voice in real time and adjusting the training content and difficulty level based on the user's emotional state;

[1483] A system including:

[1484] (Claim 2)

[1485] 2. The system of claim 1, further comprising a user input means for selecting the user's Aikido level and the type of kata they wish to learn.

[1486] (Claim 3)

[1487] 10. The system of claim 1, comprising a haptic device that provides tactile feedback to a user based on sensory technology.

[1488] "Application example 2 when combining emotion engines"

[1489] (Claim 1)

[1490] A means for automatically generating videos of basic Aikido forms based on user selections using a generation AI;

[1491] a means for displaying a virtual instructor in the user's field of view based on the generated video using AR technology;

[1492] means for providing tactile feedback to the user using sensory technology;

[1493] a means for tracking user movements in real time using a motion sensor or camera;

[1494] A means for receiving data on the user's movements on a server, evaluating the accuracy of the movements using an AI analysis engine, and generating and transmitting evaluation results;

[1495] a means for providing the user with the evaluation results received from the server at the terminal and prompting the user to make corrections based on the feedback;

[1496] A method to generate product explanation videos using generative AI and display a virtual assistant in the field of view using AR technology.

[1497] A means of tracking the user's gaze and movements using motion sensors, analyzing the data, and providing feedback.

[1498] a means for evaluating the user's emotional state using an emotion engine and dynamically adjusting the training content;

[1499] A means for providing tactile feedback of the texture of a product using a haptic device;

[1500] A system including:

[1501] (Claim 2)

[1502] 2. The system of claim 1, further comprising a user input means for selecting the user's Aikido level and the type of kata they wish to learn.

[1503] (Claim 3)

[1504] 10. The system of claim 1, comprising a haptic device that provides tactile feedback to a user based on sensory technology. [Explanation of symbols]

[1505] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for automatically generating videos of basic Aikido forms based on user selections using a generation AI; a means for displaying a virtual instructor in the user's field of view based on the generated video using AR technology; means for providing tactile feedback to the user using sensory technology; a means for tracking user movements in real time using a motion sensor or camera; A means for receiving data on the user's movements on a server, evaluating the accuracy of the movements using an AI analysis engine, and generating and transmitting evaluation results; a means for providing the user with the evaluation results received from the server at the terminal and prompting the user to make corrections based on the feedback; A system including:

2. 2. The system of claim 1, further comprising user input means for selecting the user's Aikido level and the type of kata they wish to learn.

3. The system of claim 1 , comprising a haptic device that provides tactile feedback to a user based on sensory technology.

Images