System

Abstract

A system is provided.SOLUTION: A system, comprising: means for capturing a training scene of a user by a camera installed in a training area; means for receiving and storing a video captured by the camera in real time; means for preprocessing the received video and inputting the preprocessed video to a generative AI model to analyze a motion of the user; and means for generating a feedback in the form of a text, an image, or a video based on the analyzed video.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Recently, the number of sports gyms without instructors has been increasing, but in such environments, it is difficult for users to receive appropriate training feedback. In particular, if the training method is incorrect, continuing the same training method will be inefficient and, in the worst case, may even lead to injury. Furthermore, many users train on their own while watching training videos, but the lack of accurate real-time advice reduces the effectiveness of the training. This invention solves these problems and enables efficient and safe training by providing accurate feedback even during self-training. [Means for solving the problem]

[0005] In the system of the present invention, a camera installed in the training area first captures the user's training. The video data captured by the camera is sent in real time to a server, which receives and stores it. The received video data is preprocessed and input into a generative AI model. The generative AI model analyzes the video data and evaluates the user's movements. The evaluation results are generated as feedback in the form of text, images, or videos. The generated feedback data is sent to a tablet and displayed to the user on the tablet. This allows the user to receive feedback in real time and improve their form and movements. The system also detects errors in form during training and allows them to compare it with a pre-defined ideal form, providing even more accurate feedback.

[0006] The "camera" is a device that captures the user's training movements as video and transmits the video data to a server.

[0007] "Video data" refers to digital data that includes footage of a user training, captured by a camera.

[0008] "Real-time" refers to processing and sending and receiving data instantly, with almost no delay.

[0009] The "server" is a central processing unit that receives and stores the video data sent from the camera and analyzes it using the generative AI model.

[0010] "Preprocessing" refers to preparatory processing for analysis, such as noise removal and resolution adjustment of received video data.

[0011] A "generative AI model" is an artificial intelligence program that analyzes a user's training movements and generates appropriate feedback.

[0012] "Analysis" is the process by which the generative AI model evaluates the user's actions based on video data and identifies problems.

[0013] "Text, image, or video feedback" means advice or instructions to a user generated by a generative AI model in the form of text, images, or video.

[0014] A "tablet" is a mobile device that allows users to view feedback. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The system of this invention allows users to receive appropriate training feedback in an instructor-less sports gym. Specifically, it is realized using a camera, a server, a generative AI model, and a tablet.

[0037] System Configuration

[0038] 1. Camera

[0039] The camera is installed in the training area and captures the user's training in real time.

[0040] The camera continuously captures video data and transmits it to a server over a network.

[0041] 2. Server

[0042] The server receives and stores the video data sent from the camera in real time.

[0043] The server preprocesses the received video data and prepares it for input into the generative AI model.

[0044] 3. Generative AI Models

[0045] The generative AI model analyzes the video data input by the server and evaluates the user's actions.

[0046] The model compares the user's movements with ideal training form and identifies any corrections or improvements that need to be made.

[0047] 4. Generate feedback

[0048] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[0049] The feedback is specific and includes information that users can immediately understand and act on.

[0050] 5. Tablets

[0051] The tablet receives the feedback data sent from the server and displays it to the user.

[0052] Based on the feedback provided, users can modify their forms and actions.

[0053] Specific examples

[0054] Squat training

[0055] 1. Users

[0056] The user performs a squat.

[0057] The camera captures the squat movement in real time.

[0058] 2. Device (camera)

[0059] The camera transmits the video data to the server.

[0060] 3. Server

[0061] The server preprocesses the received squat video data and inputs it into the generative AI model.

[0062] A generative AI model analyzes the video data and evaluates the user's actions.

[0063] 4. Generative AI model (server)

[0064] The generative AI model compares the user's squat motion with ideal form.

[0065] Generates feedback such as "Your knee angle is too deep, adjust it to 90 degrees."

[0066] 5. Server

[0067] Send the generated feedback to the tablet.

[0068] 6. Device (tablet)

[0069] The tablet receives the feedback and displays it in an easy-to-understand way for the user.

[0070] For example, images or videos can be displayed to visually confirm that the knee angle is not appropriate.

[0071] 7. Users

[0072] The user checks the feedback displayed on the tablet and corrects their squat form.

[0073] In this way, users can train efficiently and safely with real-time feedback, maximizing the effectiveness of their training and reducing the risk of injury.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] Terminal (camera): When the user starts training, the camera starts operating. The camera continuously captures live images of the training area and transmits the images to the server in real time.

[0077] Step 2:

[0078] Server: The server receives video data sent from the camera in real time, stores the received video data in a buffer, and prepares it for pre-processing.

[0079] Step 3:

[0080] Server: Preprocesses the video data. Preprocessing includes frame extraction, noise reduction, and resolution adjustment. This converts the data into a format that is easy for the generative AI model to analyze.

[0081] Step 4:

[0082] Server: Inputs the preprocessed video data into the generative AI model, which analyzes the input data and evaluates the user's training movements.

[0083] Step 5:

[0084] Generative AI model (server): Detects the user's bone structure and joint angles from video data and compares them with pre-registered ideal form. For example, in the case of squats, the angle of the knees and the degree of curvature of the back are evaluated.

[0085] Step 6:

[0086] Generative AI model (server): Generates feedback based on the results of behavior analysis. The feedback is created in the form of text, images, and videos, and provides specific advice to the user.

[0087] Step 7:

[0088] Server: Sends the generated feedback to the tablet. The feedback data is sent to the tablet in real time, allowing the user to view it immediately.

[0089] Step 8:

[0090] Terminal (tablet): The tablet receives the feedback data sent from the server and displays it to the user. The feedback is easy for the user to understand and indicates corrections and improvements.

[0091] Step 9:

[0092] User: The user checks the feedback displayed on the tablet and follows the instructions to correct their training form, for example, by performing specific actions such as "change the angle of your knees" or "keep your back straight."

[0093] Through these steps, users can receive appropriate feedback in real time, improving the efficiency and safety of their training.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] In modern sports gyms, users often find it difficult to maintain proper form without the guidance of an instructor. This increases the risk of injury due to incorrect training form or improper movements. This creates a need for a system that provides users with real-time feedback to ensure safe and effective training.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes means for capturing images of the user's training movements with a camera installed in the training area, means for receiving and saving the video data captured by the camera in real time, means for preprocessing the received video data and inputting it into an analysis model to analyze the user's movements, means for generating feedback in the form of text, images, or video based on the analyzed data, and means for transmitting the generated feedback data to a display terminal and displaying it to the user. This allows the user to receive appropriate feedback in real time, improving the quality of their training and further reducing the risk of injury.

[0099] The "photography device" is a device for capturing images of the user's training movements in real time and acquiring the video data.

[0100] "Video data" refers to visual information of training movements captured by a camera.

[0101] "Preprocessing" refers to a data processing procedure for converting video data into a format that can be analyzed by the generative AI model.

[0102] An "analysis model" is a computational model that analyzes received video data and evaluates the user's training movements, and this mainly applies to generative AI models.

[0103] "Feedback" is information, including corrections and improvements, provided to users based on analyzed data, and is provided in the form of text, images, or videos.

[0104] A "display terminal" is a device on which a user can view the feedback they have received, and is typically a display device such as a tablet.

[0105] "Real-time" refers to the fact that feedback is generated and displayed within a very short time (almost instantaneously) after the user performs a training action.

[0106] This invention provides a system that allows users to receive real-time feedback on their training movements without the need for an instructor. Specifically, this system is realized using a camera, a server, an analysis model, and a display terminal.

[0107] System Configuration

[0108] 1. Imaging equipment

[0109] The imaging device is installed in the training area and captures images of the user's training movements in real time.

[0110] The imaging device is composed of a high-resolution camera, which continuously captures video data of the training movements and transmits it to a server via a network.

[0111] 2. Server

[0112] The server is a high-performance computing device that receives and stores the video data transmitted from the imaging device in real time.

[0113] The server preprocesses the received video data to prepare it for input to the generative AI model, including noise reduction, frame resizing, and data format conversion.

[0114] 3. Analysis Model

[0115] Using the generative AI model, the server analyzes the pre-processed video data.

[0116] The generative AI model compares the user's movements with ideal training form and identifies areas for correction and improvement.

[0117] 4. Generating and displaying feedback

[0118] The server generates specific feedback based on the analysis results of the generative AI model, which can be in the form of text, images, or videos.

[0119] The feedback data is sent to a display terminal (e.g., a tablet) and displayed to the user.

[0120] Specific examples

[0121] Squat training

[0122] 1. User Actions

[0123] A user performs squats in the training area, making sure to stay within the camera's field of view.

[0124] 2. Acquiring video data

[0125] The camera captures the user's squatting motion in real time, and the captured video data is sent to a server.

[0126] 3. Data Analysis

[0127] The server receives the video data, preprocesses it, and then inputs it into the generative AI model, which analyzes the video data and compares the user's movements with the ideal form.

[0128] 4. Generate feedback

[0129] The generative AI model generates feedback such as "The knee angle is too deep, please adjust it to 90 degrees." The server sends this feedback to the tablet.

[0130] 5. Viewing Feedback

[0131] The tablet receives the feedback and displays it to the user in an easy-to-understand way, such as showing an image or video that indicates the knee angle is not correct.

[0132] 6. User Modifications

[0133] The user checks the feedback displayed on the tablet and corrects their squat form.

[0134] Prompt Sentence Examples

[0135] "Please evaluate whether this training form is correct and advise me on how to correct it."

[0136] In this way, the present invention aims to allow users to receive appropriate feedback in real time, improving the quality of their training and reducing the risk of injury.

[0137] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0138] Step 1:

[0139] Start user training behavior

[0140] Input: A user enters a training area and begins a particular training movement (e.g., a squat).

[0141] Specific Action: User performs squats with proper form in the designated training area, taking care to perform the movements within the camera's field of view.

[0142] Step 2:

[0143] Acquisition of video data using imaging equipment

[0144] Input: User training behavior

[0145] Output: Real-time video data of training movements

[0146] Specific operation: The camera records the user's training movements in real time, continuously capturing high-resolution video data and storing it in a temporary buffer.

[0147] Step 3:

[0148] Video data transmission

[0149] Input: Real-time video data of training movements

[0150] Output: Video data sent to the server

[0151] Specific operation: The video data captured by the camera is sent to the server via the network. Data transfer is performed with low latency.

[0152] Step 4:

[0153] Receiving and storing video data on the server

[0154] Input: Image data sent from the imaging device

[0155] Output: Saved video data

[0156] Specific operation: The server receives the received video data in real time and stores it in a database, ensuring the integrity of the data required for subsequent processing.

[0157] Step 5:

[0158] Video data preprocessing

[0159] Input: Saved video data

[0160] Output: Pre-processed video data

[0161] What it does: The server preprocesses the video data, specifically removing noise, resizing frames, and converting data formats, so that it is properly prepared for input to the generative AI model.

[0162] Step 6:

[0163] Motion analysis using generative AI models

[0164] Input: Preprocessed video data

[0165] Output: Analyzed data (user behavior evaluation)

[0166] Specific operation: The server inputs preprocessed video data into the generative AI model, which then identifies the user's skeletal structure and the positions of major joints from the video data and analyzes their movements.

[0167] Step 7:

[0168] Generate feedback from analysis results

[0169] Input: Data analyzed by the generative AI model

[0170] Output: Feedback in text, image and video formats

[0171] Specific operation: The server generates feedback based on the analysis results of the generative AI model. The feedback includes specific corrections and improvements, and is provided in a format that is easy for the user to understand.

[0172] Step 8:

[0173] Sending feedback data to a tablet

[0174] Input: Generated feedback

[0175] Output: Feedback data sent to the tablet

[0176] What it does: The server sends the generated feedback to the tablet. The communication is real-time, ensuring that the feedback reaches the user immediately.

[0177] Step 9:

[0178] Viewing feedback on a tablet

[0179] Input: Feedback data sent from the server

[0180] Output: Feedback that is displayed to the user

[0181] Specific behavior: The tablet receives feedback and displays it to the user. The feedback displayed can include text, images, and videos, and includes specific corrections.

[0182] Step 10:

[0183] Modifying user behavior

[0184] Input: Feedback displayed on tablet

[0185] Output: Corrected training behavior

[0186] Specific actions: The user checks the feedback displayed on the tablet and corrects their training form, for example, by adjusting the knee angle properly when squatting.

[0187] (Application example 1)

[0188] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0189] Currently, in many factories, worker movements are dependent on visual evaluation and manual checks, resulting in issues with efficiency and safety. In particular, when new or inexperienced workers work alone, their movements may lack accuracy and appropriateness, posing risks to the quality and safety of the work. Furthermore, the lack of real-time feedback increases the likelihood of incorrect movements and postures being continued, resulting in reduced production efficiency. The purpose of this invention is to solve these issues.

[0190] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0191] In this invention, the server includes: means for capturing images of the user's training with a camera installed in the training area; means for receiving and saving the image data captured by the camera in real time; means for preprocessing the received image data and inputting it into a generative AI model to analyze the user's movements; means for generating feedback in the form of text, images, or videos based on the analyzed data; means for sending the generated feedback data to a tablet and displaying it to the user; and means installed in the work area of ​​the factory for analyzing the worker's movements in real time and sending feedback. This allows workers to receive feedback in real time, improving the accuracy and safety of their movements and ultimately increasing the efficiency of the entire factory.

[0192] "Training Area" means an area designated for a user to exercise or work.

[0193] A "camera" is a device that captures video data and transmits it to a server in real time.

[0194] "Video data" refers to digital data that includes the user's movements and training footage captured by a camera.

[0195] "Real-time" means that data is processed immediately with little delay.

[0196] "Storage" refers to the act of recording data on a storage device such as a server.

[0197] "Preprocessing" refers to the preparation work carried out before data analysis, and includes data normalization and resizing.

[0198] A "generative AI model" refers to an artificial intelligence model trained for a specified task, performing behavior analysis and feedback generation.

[0199] "Analysis" is the process of extracting information contained in data and finding meaning and patterns.

[0200] "Feedback" refers to advice or suggestions provided to users based on the analysis results.

[0201] "Text" means written information.

[0202] An "image" is a form of representing information as a visual (still image).

[0203] "Video" is a data format that contains moving visual information.

[0204] A "tablet" is a portable display device used by a user to view feedback.

[0205] "Factory work area" refers to the area within a factory used by workers to perform their designated tasks.

[0206] This invention is a system for analyzing the actions of users and workers in training areas and factory work areas in real time and providing feedback. Specifically, it uses cameras, servers, generative AI models, and tablets to provide feedback to users so that they can perform appropriate training and actions. This can improve work efficiency and safety.

[0207] System Configuration

[0208] 1. Camera

[0209] The cameras are installed in training areas or factory work areas and capture the actions of users and workers in real time.

[0210] The captured video data is transmitted to a server via a network.

[0211] 2. Server

[0212] The server receives and stores the video data sent from the camera.

[0213] The server preprocesses the received video data and prepares it for input into the generative AI model.

[0214] Specific preprocessing includes resizing and normalizing the video data.

[0215] 3. Generative AI Models

[0216] The generative AI model analyzes the video data input by the server and evaluates the actions of the user or worker.

[0217] The model compares the user's movements with ideal training form and work procedures to identify needed corrections and improvements.

[0218] 4. Generate feedback

[0219] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[0220] The feedback is specific, easy to understand, and includes content that users can immediately act on.

[0221] 5. Tablets

[0222] The tablet receives the feedback data sent from the server and displays it to the user or worker.

[0223] Based on the displayed feedback, users or workers can correct their form and actions.

[0224] Hardware and software used

[0225] Hardware: high-resolution cameras, servers, tablets

[0226] software:

[0227] OpenCV: Used for real-time video capture and display

[0228] TensorFlow: used for loading and inferencing generative AI models

[0229] requests: Used to communicate with the server

[0230] Specific examples

[0231] For example, consider a factory worker using a robotic arm to install parts. A camera captures the image to determine whether the movement is appropriate, and a generative AI model analyzes it. If the movement is incorrect, feedback such as "The robotic arm is out of position. Please move it 5 cm to the right" is displayed on a tablet.

[0232] Prompt Sentence Examples

[0233] "Generative AI models, robot behavior evaluation, real-time feedback, worker safety, and improved factory efficiency"

[0234] This allows the invention to provide real-time feedback to assist the worker or user in taking the appropriate action.

[0235] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0236] Step 1:

[0237] The camera captures the user or worker's movements in real time in the training area or factory work area.

[0238] Input: User or worker actions

[0239] Output: Real-time video data

[0240] Specific operation: The camera captures the movements of the user or worker and continuously captures frames.

[0241] Step 2:

[0242] The server receives and stores the video data captured by the camera in real time.

[0243] Input: Real-time video data

[0244] Output: Saved video data

[0245] Specific operation: The server receives the video data sent from the camera via the network and saves it in a file format.

[0246] Step 3:

[0247] The server preprocesses the received video data and prepares it for input into the generative AI model.

[0248] Input: Saved video data

[0249] Output: Preprocessed data (resized and normalized)

[0250] What it does: The server resizes the video data, normalizes pixel values, and converts the data format to one that is more easily accepted by the generative AI model.

[0251] Step 4:

[0252] The server inputs the preprocessed data into a generative AI model to analyze the actions of users and workers.

[0253] Input: Preprocessed data

[0254] Output: Analysis results (operation evaluation)

[0255] Specific actions: The generative AI model analyzes the video data and evaluates the actions of the user or worker. The analysis results identify whether the actions are correct and which areas need improvement.

[0256] Step 5:

[0257] The server generates feedback in the form of text, images, and videos based on the analyzed data.

[0258] Input: Analysis results

[0259] Output: Feedback data (text, images, videos)

[0260] Specific actions: The server generates feedback based on the analysis results, such as text instructions like "Move the robot arm 5 cm to the right," or images or videos showing the incorrect action.

[0261] Step 6:

[0262] The server sends the generated feedback data to a tablet and displays it to the user or worker.

[0263] Input: Feedback data

[0264] Output: Feedback displayed on the tablet

[0265] Specific operation: Feedback data is sent to a tablet and displayed in a form that can be visually understood by the user or worker. The user or worker then corrects their operation based on the feedback.

[0266] Step 7:

[0267] Users and workers can check the feedback displayed on the tablet and correct their actions.

[0268] Input: Feedback displayed on tablet

[0269] Output: Corrected behavior

[0270] Specific actions: The user or worker checks the feedback and corrects their actions or form, for example, adjusting the position of the robot arm as instructed.

[0271] The above are the specific processing steps for carrying out the present invention.

[0272] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0273] The system of this invention allows users to receive appropriate training feedback when an instructor is not present, and also has the ability to recognize the user's emotional state and dynamically adjust the feedback and training menu. Specifically, it is realized using a camera, a server, a generative AI model, a tablet, and an emotion engine.

[0274] System Configuration

[0275] 1. Camera

[0276] The camera is installed in the training area and captures the user's training in real time.

[0277] The camera continuously captures video data and transmits it to a server over a network.

[0278] 2. Server

[0279] The server receives and stores the video data sent from the camera in real time.

[0280] The server preprocesses the received video data and prepares it for input into the generative AI model.

[0281] 3. Generative AI Models

[0282] The generative AI model analyzes the video data input by the server and evaluates the user's actions.

[0283] The model compares the user's movements with ideal training form and identifies any corrections or improvements that need to be made.

[0284] 4. Generate feedback

[0285] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[0286] The feedback is specific and includes information that users can immediately understand and act on.

[0287] 5. Emotion Engine

[0288] The emotion engine uses video and audio data captured by the camera to analyze changes in the user's facial expressions and voice and estimate their emotional state.

[0289] Based on the analysis results of the emotion engine, the content and tone of the feedback is adjusted to improve the user's training experience.

[0290] 6. Tablets

[0291] The tablet receives the feedback data sent from the server and the analysis results of the emotion engine and displays them to the user.

[0292] Based on the feedback provided, users can improve their form and behavior.

[0293] Specific examples

[0294] Squat training

[0295] 1. Users

[0296] The user performs a squat.

[0297] The camera captures the squat movements and facial expressions in real time.

[0298] 2. Device (camera)

[0299] The camera transmits the video data to the server.

[0300] 3. Server

[0301] The server preprocesses the received squat video data and inputs it into the generative AI model.

[0302] A generative AI model analyzes the video data and evaluates the user's actions.

[0303] 4. Generative AI model (server)

[0304] The generative AI model compares the user's squat motion with ideal form.

[0305] Generates feedback such as "Your knee angle is too deep, adjust it to 90 degrees."

[0306] 5. Server

[0307] Send the generated feedback to the tablet.

[0308] 6. Device (tablet)

[0309] The tablet receives the feedback and displays it in an easy-to-understand way for the user.

[0310] For example, images or videos can be displayed to visually confirm that the knee angle is not appropriate.

[0311] 7. Emotion engine (server)

[0312] The emotion engine analyzes the user's facial expressions and voice to estimate their emotional state.

[0313] If the user is feeling tired or frustrated, for example, the system generates feedback that is more encouraging or offers simple points for improvement depending on the user's emotional state.

[0314] 8. Server

[0315] Based on the results of the emotion engine, the content and tone of the feedback are appropriately adjusted and sent to the tablet.

[0316] 9. Device (tablet)

[0317] The tablet displays feedback based on your emotional state.

[0318] For example, it provides personalized advice such as, "We recommend you take a short break or drink some water."

[0319] 10. Users

[0320] The user checks the feedback displayed on the tablet and follows the instructions to modify their training form and menu.

[0321] Emotional feedback is also taken into consideration to take appropriate actions such as taking breaks at appropriate times.

[0322] In this way, users can receive real-time feedback on not only their training movements but also their emotional state, which can improve the efficiency and safety of training and help users maintain their motivation.

[0323] The processing flow will be explained below.

[0324] Step 1:

[0325] User: The user starts a workout, for example, performing a squat.

[0326] Step 2:

[0327] Device (camera): A camera is installed in the training area and captures the user's movements and facial expressions in real time. Video data is continuously captured.

[0328] Step 3:

[0329] Terminal (camera): The camera sends video data to the server in real time. The video data is transferred via the network.

[0330] Step 4:

[0331] Server: The server receives and stores video data sent from the camera in real time. The video data is stored in a buffer.

[0332] Step 5:

[0333] Server: Pre-processes the received video data, including frame extraction, noise reduction, and resolution adjustment.

[0334] Step 6:

[0335] Server: Inputs preprocessed video data into the generative AI model.

[0336] Step 7:

[0337] Generative AI model (server): Analyzes video data and evaluates the user's movements. It compares the user's squat movements with the ideal form.

[0338] Step 8:

[0339] Generative AI model (server): Generates specific feedback such as "Your knee angle is too deep, please adjust it to 90 degrees." Feedback is generated in the form of text, images, and videos.

[0340] Step 9:

[0341] Server: Sends the generated feedback to the tablet. The feedback data is sent to the tablet in real time.

[0342] Step 10:

[0343] Terminal (tablet): The tablet receives the feedback data and displays it to the user, for example, an image showing the knee angle and text indicating correction points.

[0344] Step 11:

[0345] User: The user checks the feedback displayed on the tablet and corrects their squat form.

[0346] Step 12:

[0347] Server: Passes part of the video data to the emotion engine.

[0348] Step 13:

[0349] Emotion engine (server): Analyzes video and audio data, detects changes in the user's facial expressions and voice, and infers their emotional state. For example, it infers whether the user is tired or frustrated.

[0350] Step 14:

[0351] Emotion engine (server): Adjusts the content and tone of the feedback based on the results of the emotion engine. For example, if the user is estimated to be tired, it generates encouraging feedback such as "Take a short break."

[0352] Step 15:

[0353] Server: Sends feedback reflecting the results of the emotion engine back to the tablet.

[0354] Step 16:

[0355] Device (tablet): The tablet displays feedback based on the user's emotional state. If the user's facial expression indicates fatigue, it displays words of encouragement or a message recommending rest.

[0356] Step 17:

[0357] User: The user checks the newly displayed emotional feedback and takes appropriate action, such as taking a short break or drinking more water.

[0358] Through these steps, users can receive real-time feedback on their training movements and emotional state, which can improve the efficiency and safety of their training and help maintain their motivation.

[0359] Example 2

[0360] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0361] Conventional training systems have the problem that it is difficult for users to receive appropriate feedback when an instructor is not present, preventing them from correcting their form or maximizing the effectiveness of their training. Additionally, feedback does not take into account the user's emotional state, making it difficult to maintain motivation and continue training.

[0362] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0363] In this invention, the server includes means for capturing images of the user's training using a camera installed in the training area, means for receiving and saving the video data captured by the camera in real time, means for preprocessing the received video data and inputting it into an artificial intelligence model to analyze the user's movements, means for generating feedback in the form of text, images, or video based on the analyzed data, means for transmitting the generated feedback data to a display terminal and displaying it to the user, means for analyzing changes in the user's facial expressions and voice to estimate their emotional state, and means for adjusting the content and tone of the feedback based on their emotional state. This allows the user to receive feedback on their training movements and emotional state in real time, improving the efficiency and safety of their training and helping them maintain their motivation.

[0364] The "photography device" is a device that is installed in the training area and is used to capture the user's training scene in real time.

[0365] A "server" is a device that receives video data sent from an imaging device and performs a series of processes such as storage, pre-processing, analysis, and feedback generation.

[0366] The "artificial intelligence model" is a program that analyzes video data input into the server and evaluates the user's actions.

[0367] "Preprocessing" refers to the process of removing noise from the received video data, extracting frames, adjusting the resolution, and so on.

[0368] "Feedback" refers to formal advice or instructions to users that are generated based on the analysis results of an artificial intelligence model and are provided in the form of text, images, videos, etc.

[0369] A "display terminal" is a device that receives feedback data sent from the server and visually presents it to the user.

[0370] The "emotion engine" is a program that analyzes changes in the user's facial expressions and voice to estimate their emotional state.

[0371] "Real-time" refers to the immediacy of data processing and feedback during user training.

[0372] "Tone" refers to the wording and delivery of feedback or advice, which is adjusted according to the user's emotional state.

[0373] The system of this invention is designed to allow users to receive appropriate training feedback even when an instructor is not present. It also has the ability to recognize the user's emotional state and dynamically adjust the feedback and training menu. The main components of the system are as follows:

[0374] 1. Imaging equipment

[0375] It is installed in the training area and records the user's training in real time, and the captured video data is sent sequentially to a server via the network.

[0376] 2. Server

[0377] The server receives and stores video data sent from the imaging device in real time. The first step is pre-processing of the received video data, which includes noise removal, frame extraction, and resolution adjustment. The pre-processed data is then formatted so that it can be effectively analyzed by the generative AI model.

[0378] The pre-processed video data is then fed into a generative AI model, which analyzes the data and evaluates the user's movements, for example comparing them with ideal training form to identify necessary corrections or improvements.

[0379] Based on the analysis results, the server generates specific feedback in the form of text, images, or videos, which are then sent to a display device and presented to the user.

[0380] The server is also equipped with an emotion engine that analyzes the user's facial expressions and voice to estimate their emotional state, and adjusts the content and tone of the feedback based on the results of this analysis.

[0381] 3. Display terminal

[0382] The system receives feedback data sent from the server and displays it to the user. The feedback is designed to allow users to understand and act immediately, and specific images and videos are also presented.

[0383] Specific examples

[0384] Squat training

[0385] 1. Users

[0386] The user stands in the training area and begins training by pressing the "Start Training" button on the tablet screen.

[0387] 2. Terminal (photography device)

[0388] The camera captures the user's squatting movements and facial expressions in real time and sends the video data to a server.

[0389] 3. Server

[0390] The server preprocesses the received video data of the squat and inputs it into a generative AI model, which then analyzes the video data and evaluates the user's movements.

[0391] The generative AI model compares the user's squat movement with ideal training form and generates specific feedback such as, "Your knee angle is too deep; adjust it to 90 degrees."

[0392] This feedback can be generated in the form of text, images, or videos and sent to a display device.

[0393] The emotion engine analyzes the user's facial expressions and voice to estimate their emotional state. For example, if the user is feeling tired or frustrated, it generates feedback content and tone appropriate to that emotional state and sends it to the display device.

[0394] 4. Terminal (display terminal)

[0395] The display device receives the feedback and displays it in an easy-to-understand way to the user. For example, it can show images or videos that show that the knee angle is not appropriate, and provide personalized advice such as "We recommend that you take a short break or drink some water."

[0396] Prompt Sentence Examples

[0397] "Your knees are at too deep an angle when squatting; adjust them to 90 degrees."

[0398] "I suggest you take a short break or drink some water."

[0399] In this way, users can receive real-time feedback on their training movements and emotional state, which can improve training efficiency and safety and help users maintain their motivation.

[0400] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0401] Step 1:

[0402] User starts training

[0403] Specific behavior:

[0404] The user stands in the training area and presses the "Start Training" button on the tablet screen, which starts the system.

[0405] Input: User presses the "Start Training" button on the tablet.

[0406] Output: The system goes into training start mode.

[0407] Step 2:

[0408] The device (camera) captures the training video.

[0409] Specific behavior:

[0410] The camera is installed in a position that captures the entire training area. When the user starts to move, the camera captures their movements in real time and transmits the video data sequentially to a server via a network.

[0411] Input: User action

[0412] Output: Real-time video data

[0413] Step 3:

[0414] The server receives the video data

[0415] Specific behavior:

[0416] The server receives the image data sent from the imaging device and stores it in real time. At this point, the data is raw data that has not yet been pre-processed.

[0417] Input: Raw image data transmitted from the imaging device

[0418] Output: Saved video data

[0419] Step 4:

[0420] The server preprocesses the video data

[0421] Specific behavior:

[0422] The server preprocesses the received video data, which includes removing noise, cropping frames, and adjusting resolution, so that the preprocessed data is in a format that can be effectively analyzed by the generative AI model.

[0423] Input: Stored raw video data

[0424] Output: Pre-processed video data

[0425] Step 5:

[0426] The server inputs data into the generative AI model

[0427] Specific behavior:

[0428] The pre-processed video data is then input into the generative AI model, a process that prepares the data to fit the model's input format.

[0429] Input: Preprocessed video data

[0430] Output: Data input to the generative AI model

[0431] Step 6:

[0432] Generative AI models analyze video data

[0433] Specific behavior:

[0434] A generative AI model analyzes the input data and evaluates the user's movements. The model compares the user's movements with ideal training form and identifies necessary corrections and improvements.

[0435] Input: Data input to the generative AI model

[0436] Output: Parsed feedback data

[0437] Step 7:

[0438] Generative AI models generate feedback

[0439] Specific behavior:

[0440] The generative AI model generates specific feedback based on the analysis results, such as "Your knee angle is too deep, please adjust it to 90 degrees."

[0441] Input: Parsed feedback data

[0442] Output: Generated feedback

[0443] Step 8:

[0444] The server sends feedback to the display device

[0445] Specific behavior:

[0446] The generative AI model generates feedback that is sent to the display device in the form of text, images, or videos.

[0447] Input: Generated feedback

[0448] Output: Feedback sent to the display device

[0449] Step 9:

[0450] Emotion engine analyzes emotional state

[0451] Specific behavior:

[0452] The emotion engine analyzes the user's facial expressions and voice to estimate their emotional state. For example, if frustration is detected, the engine returns that data to the server.

[0453] Input: User's facial expressions and voice

[0454] Output: Emotional state data

[0455] Step 10:

[0456] The server sends feedback based on the emotional state to the display device.

[0457] Specific behavior:

[0458] The feedback content is adjusted based on the analysis results of the emotion engine, changing the tone and message content, and then the adjusted feedback is sent to the display device.

[0459] Input: Emotional state data

[0460] Output: Feedback based on emotional state

[0461] Step 11:

[0462] The display device displays the feedback to the user.

[0463] Specific behavior:

[0464] The display device then displays the received feedback to the user, for example, showing an image or video that indicates the knee angle is not appropriate and providing personalized advice such as "We recommend you take a short break or drink some water."

[0465] Input: Feedback based on emotional state

[0466] Output: Feedback displayed to the user

[0467] Step 12:

[0468] Users modify their training according to the feedback

[0469] Specific behavior:

[0470] The user checks the feedback displayed on the display terminal and follows the instructions to correct their training form and movements. They also refer to the emotional feedback and take appropriate actions such as taking breaks at appropriate times.

[0471] Input: Feedback displayed to the user

[0472] Output: Corrected training form and movements

[0473] (Application example 2)

[0474] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0475] Conventional fitness training systems struggle to provide users with proper form and movement feedback when an instructor is not present. Furthermore, providing uniform feedback without considering the user's emotional state can reduce the user's motivation and training efficiency. This raises concerns about the risk of injury and reduced training effectiveness.

[0476] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing images of the user's training using a camera installed in the training area, means for receiving and saving the video data captured by the camera in real time, means for preprocessing the received video data and inputting it into a generative AI model to analyze the user's movements, means for generating feedback in the form of text, images, or videos based on the analyzed data, means for sending the generated feedback data to a tablet and displaying it to the user, and means for analyzing the user's emotional state and dynamically adjusting the feedback and training menu. This allows users to receive appropriate feedback in real time even when an instructor is not present, and also enables them to obtain feedback optimized according to their individual emotional state.

[0477] The "camera" is an image capturing device for capturing images of the user's training.

[0478] "Video data" is digital data containing visual information captured by a camera.

[0479] "Means for receiving and storing in real time" refers to a function for receiving captured video data to a server without delay and storing the data as needed.

[0480] "Preprocessing" refers to the process of shaping and processing video data to make it suitable for analysis by a generative AI model.

[0481] A "generative AI model" is an artificial intelligence model that analyzes and evaluates a user's training movements.

[0482] "Input" refers to feeding pre-processed video data into a generative AI model.

[0483] The "means of analysis" refers to a method of analyzing and evaluating a user's training movements using a generative AI model.

[0484] "Feedback" refers to instructions or advice provided to the user based on the analysis results of the generative AI model.

[0485] "Text, image, video format" refers to a format in which the feedback content is displayed using different means of expression.

[0486] A "tablet" is a portable display device for displaying generated feedback to a user.

[0487] "Emotional state" refers to the psychological state estimated based on information such as the user's facial expression and voice.

[0488] "Dynamic adjustment means" refers to a function that flexibly changes the feedback and training menu content according to the user's emotional state.

[0489] This invention is a system that allows users to receive appropriate training feedback when an instructor is not present, and also has the ability to recognize the user's emotional state and dynamically adjust the feedback and training menu. This system is realized using a camera, a server, a generative AI model, a tablet, and an emotion engine.

[0490] System Configuration

[0491] 1. Camera

[0492] The camera is installed in the training area and captures the user's training in real time.

[0493] As a specific example, a high-resolution camera (e.g., Logitech StreamCam) is used.

[0494] 2. Server

[0495] The server receives and stores the video data sent from the camera in real time.

[0496] After preprocessing, the received video data is input into the generative AI model.

[0497] As a specific example, a high-performance cloud server (e.g., Amazon Web Services EC2) is used.

[0498] 3. Generative AI Models

[0499] The generative AI model analyzes the video data input by the server and evaluates the user's actions.

[0500] The model compares the user's movements with ideal training form and identifies any corrections or improvements that need to be made.

[0501] A specific example is using a custom AI model (e.g., TensorFlow or PyTorch) for behavior analysis.

[0502] 4. Generate feedback

[0503] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[0504] The feedback is specific and includes information that users can immediately understand and act on.

[0505] Example: The knee angle is too deep, please adjust it to 90 degrees.

[0506] 5. Emotion Engine

[0507] The emotion engine uses video and audio data captured by the camera to analyze changes in the user's facial expressions and voice and estimate their emotional state.

[0508] Based on the analysis results, the content and tone of the feedback can be adjusted to improve the user's training experience.

[0509] As a specific example, we use an emotion analysis engine (e.g., Microsoft Azure Emotion API).

[0510] 6. Tablets

[0511] The tablet receives the feedback data sent from the server and the analysis results of the emotion engine and displays them to the user.

[0512] Based on the feedback provided, users can improve their form and behavior.

[0513] Processing flow

[0514] 1. User Operation

[0515] A user taps the start button on a tablet or smartphone in the fitness training area.

[0516] 2. Camera Operation

[0517] The camera captures the user's training movements and sends the video data to a server.

[0518] 3. Server Processing

[0519] The server receives the video data, pre-processes it, and then feeds the data into the generative AI model.

[0520] 4. Analysis of generative AI models

[0521] A generative AI model analyzes the data and compares the user's movements with ideal form.

[0522] For example: "Your knee angle is too deep, please adjust it to 90 degrees."

[0523] 5. Generate and send feedback

[0524] The server generates feedback in the form of text, images and videos and sends it to the tablet.

[0525] 6. Emotion Engine Analysis

[0526] The emotion engine analyzes the user's emotional state and dynamically adjusts the feedback content.

[0527] 7. Displaying feedback to users

[0528] The tablet receives the feedback and displays it in an easy-to-understand way.

[0529] For example: "Bend your knees more."

[0530] Prompt Sentence Examples

[0531] Analyze footage of a user squatting, assess knee angle, hip position, and back curvature, and generate feedback to correct for ideal posture.

[0532] This allows users to receive both motion and emotional feedback in real time, improving training efficiency and safety.

[0533] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0534] Step 1:

[0535] A user taps the start button on a tablet or smartphone in the fitness training area. This causes the camera to start capturing video data. The input is the user's operation information, and the output is a signal to the camera to start recording.

[0536] Step 2:

[0537] The camera captures the user's training movements in real time and captures the video data, which is then sent to a server via a network. The input is the user's movement data, and the output is the video data sent to the server.

[0538] Step 3:

[0539] The server receives and stores video data from the camera in real time. It then preprocesses the received video data and prepares it for input into the generative AI model. The input is video data from the camera, and the output is preprocessed data. Specifically, noise removal and frame correction are performed.

[0540] Step 4:

[0541] The server inputs the preprocessed video data into a generative AI model. The generative AI model analyzes and evaluates the user's training movements. The input is the preprocessed data, and the output is the evaluation result of the user's movements. Specifically, the movement data is compared with the ideal form to detect errors.

[0542] Step 5:

[0543] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model. The input is the evaluation results of the user's movements, and the output is feedback data. Specifically, it describes in detail the corrections needed for knee angle and hip position.

[0544] Step 6:

[0545] The emotion engine uses video and audio data captured by the camera to analyze changes in the user's facial expressions and voice and estimate their emotional state. The input is video and audio data, and the output is the analysis result of the emotional state. The emotion engine uses, for example, the Microsoft Azure Emotion API.

[0546] Step 7:

[0547] The server adjusts the content and tone of the feedback based on the analysis results of the emotion engine. The input is the analysis result of the emotional state, and the output is dynamically adjusted feedback. If the user is fatigued, the content will be changed to be more encouraging.

[0548] Step 8:

[0549] The server sends the final feedback data to the tablet. The input is the dynamically adjusted feedback data, and the output is the data transmission to the tablet.

[0550] Step 9:

[0551] The tablet receives the feedback data sent from the server and the analysis results of the emotion engine, and displays them in an easy-to-understand manner for the user. The input is the feedback data, and the output is the display on the tablet.

[0552] Step 10:

[0553] The user checks the feedback displayed on the tablet and corrects their training movements. They also refer to emotional feedback to take appropriate actions such as taking breaks at appropriate times. The input is feedback data, and the output is corrected movements and actions such as resting.

[0554] This series of steps allows users to receive both behavioral and emotional feedback in real time, improving training efficiency and safety.

[0555] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0556] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0557] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0558] [Second embodiment]

[0559] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0560] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0561] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0562] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0563] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0564] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0565] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0566] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0567] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0568] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0569] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0570] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0571] The system of this invention allows users to receive appropriate training feedback in an instructor-less sports gym. Specifically, it is realized using a camera, a server, a generative AI model, and a tablet.

[0572] System Configuration

[0573] 1. Camera

[0574] The camera is installed in the training area and captures the user's training in real time.

[0575] The camera continuously captures video data and transmits it to a server over a network.

[0576] 2. Server

[0577] The server receives and stores the video data sent from the camera in real time.

[0578] The server preprocesses the received video data and prepares it for input into the generative AI model.

[0579] 3. Generative AI Models

[0580] The generative AI model analyzes the video data input by the server and evaluates the user's actions.

[0581] The model compares the user's movements with ideal training form and identifies any corrections or improvements that need to be made.

[0582] 4. Generate feedback

[0583] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[0584] The feedback is specific and includes information that users can immediately understand and act on.

[0585] 5. Tablets

[0586] The tablet receives the feedback data sent from the server and displays it to the user.

[0587] Based on the feedback provided, users can modify their forms and actions.

[0588] Specific examples

[0589] Squat training

[0590] 1. Users

[0591] The user performs a squat.

[0592] The camera captures the squat movement in real time.

[0593] 2. Device (camera)

[0594] The camera transmits the video data to the server.

[0595] 3. Server

[0596] The server preprocesses the received squat video data and inputs it into the generative AI model.

[0597] A generative AI model analyzes the video data and evaluates the user's actions.

[0598] 4. Generative AI model (server)

[0599] The generative AI model compares the user's squat motion with ideal form.

[0600] Generates feedback such as "Your knee angle is too deep, adjust it to 90 degrees."

[0601] 5. Server

[0602] Send the generated feedback to the tablet.

[0603] 6. Device (tablet)

[0604] The tablet receives the feedback and displays it in an easy-to-understand way for the user.

[0605] For example, images or videos can be displayed to visually confirm that the knee angle is not appropriate.

[0606] 7. Users

[0607] The user checks the feedback displayed on the tablet and corrects their squat form.

[0608] In this way, users can train efficiently and safely with real-time feedback, maximizing the effectiveness of their training and reducing the risk of injury.

[0609] The processing flow will be explained below.

[0610] Step 1:

[0611] Terminal (camera): When the user starts training, the camera starts operating. The camera continuously captures live images of the training area and transmits the images to the server in real time.

[0612] Step 2:

[0613] Server: The server receives video data sent from the camera in real time, stores the received video data in a buffer, and prepares it for pre-processing.

[0614] Step 3:

[0615] Server: Preprocesses the video data. Preprocessing includes frame extraction, noise reduction, and resolution adjustment. This converts the data into a format that is easy for the generative AI model to analyze.

[0616] Step 4:

[0617] Server: Inputs the preprocessed video data into the generative AI model, which analyzes the input data and evaluates the user's training movements.

[0618] Step 5:

[0619] Generative AI model (server): Detects the user's bone structure and joint angles from video data and compares them with pre-registered ideal form. For example, in the case of squats, the angle of the knees and the degree of curvature of the back are evaluated.

[0620] Step 6:

[0621] Generative AI model (server): Generates feedback based on the results of behavior analysis. The feedback is created in the form of text, images, and videos, and provides specific advice to the user.

[0622] Step 7:

[0623] Server: Sends the generated feedback to the tablet. The feedback data is sent to the tablet in real time, allowing the user to view it immediately.

[0624] Step 8:

[0625] Terminal (tablet): The tablet receives the feedback data sent from the server and displays it to the user. The feedback is easy for the user to understand and indicates corrections and improvements.

[0626] Step 9:

[0627] User: The user checks the feedback displayed on the tablet and follows the instructions to correct their training form, for example, by performing specific actions such as "change the angle of your knees" or "keep your back straight."

[0628] Through these steps, users can receive appropriate feedback in real time, improving the efficiency and safety of their training.

[0629] Example 1

[0630] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0631] In modern sports gyms, users often find it difficult to maintain proper form without the guidance of an instructor. This increases the risk of injury due to incorrect training form or improper movements. This creates a need for a system that provides users with real-time feedback to ensure safe and effective training.

[0632] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0633] In this invention, the server includes means for capturing images of the user's training movements with a camera installed in the training area, means for receiving and saving the video data captured by the camera in real time, means for preprocessing the received video data and inputting it into an analysis model to analyze the user's movements, means for generating feedback in the form of text, images, or video based on the analyzed data, and means for transmitting the generated feedback data to a display terminal and displaying it to the user. This allows the user to receive appropriate feedback in real time, improving the quality of their training and further reducing the risk of injury.

[0634] The "photography device" is a device for capturing images of the user's training movements in real time and acquiring the video data.

[0635] "Video data" refers to visual information of training movements captured by a camera.

[0636] "Preprocessing" refers to a data processing procedure for converting video data into a format that can be analyzed by the generative AI model.

[0637] An "analysis model" is a computational model that analyzes received video data and evaluates the user's training movements, and this mainly applies to generative AI models.

[0638] "Feedback" is information, including corrections and improvements, provided to users based on analyzed data, and is provided in the form of text, images, or videos.

[0639] A "display terminal" is a device on which a user can view the feedback they have received, and is typically a display device such as a tablet.

[0640] "Real-time" refers to the fact that feedback is generated and displayed within a very short time (almost instantaneously) after the user performs a training action.

[0641] This invention provides a system that allows users to receive real-time feedback on their training movements without the need for an instructor. Specifically, this system is realized using a camera, a server, an analysis model, and a display terminal.

[0642] System Configuration

[0643] 1. Imaging equipment

[0644] The imaging device is installed in the training area and captures images of the user's training movements in real time.

[0645] The imaging device is composed of a high-resolution camera, which continuously captures video data of the training movements and transmits it to a server via a network.

[0646] 2. Server

[0647] The server is a high-performance computing device that receives and stores the video data transmitted from the imaging device in real time.

[0648] The server preprocesses the received video data to prepare it for input to the generative AI model, including noise reduction, frame resizing, and data format conversion.

[0649] 3. Analysis Model

[0650] Using the generative AI model, the server analyzes the pre-processed video data.

[0651] The generative AI model compares the user's movements with ideal training form and identifies areas for correction and improvement.

[0652] 4. Generating and displaying feedback

[0653] The server generates specific feedback based on the analysis results of the generative AI model, which can be in the form of text, images, or videos.

[0654] The feedback data is sent to a display terminal (e.g., a tablet) and displayed to the user.

[0655] Specific examples

[0656] Squat training

[0657] 1. User Actions

[0658] A user performs squats in the training area, making sure to perform the movements within the camera's field of view.

[0659] 2. Acquiring video data

[0660] The camera captures the user's squatting motion in real time, and the captured video data is sent to a server.

[0661] 3. Data Analysis

[0662] The server receives the video data, preprocesses it, and then inputs it into the generative AI model, which analyzes the video data and compares the user's movements with the ideal form.

[0663] 4. Generate feedback

[0664] The generative AI model generates feedback such as, "The knee angle is too deep, please adjust it to 90 degrees." The server sends this feedback to the tablet.

[0665] 5. Viewing Feedback

[0666] The tablet receives the feedback and displays it to the user in an easy-to-understand way, such as showing an image or video that indicates the knee angle is not correct.

[0667] 6. User Modifications

[0668] The user checks the feedback displayed on the tablet and corrects their squat form.

[0669] Prompt Sentence Examples

[0670] "Please evaluate whether this training form is correct and advise me on how to correct it."

[0671] In this way, the present invention aims to allow users to receive appropriate feedback in real time, improving the quality of their training and reducing the risk of injury.

[0672] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0673] Step 1:

[0674] Start user training behavior

[0675] Input: A user enters a training area and begins a particular training movement (e.g., a squat).

[0676] Specific Action: User performs squats with proper form in the designated training area, taking care to perform the movements within the camera's field of view.

[0677] Step 2:

[0678] Acquisition of video data using imaging equipment

[0679] Input: User training behavior

[0680] Output: Real-time video data of training movements

[0681] Specific operation: The camera records the user's training movements in real time, continuously capturing high-resolution video data and storing it in a temporary buffer.

[0682] Step 3:

[0683] Video data transmission

[0684] Input: Real-time video data of training movements

[0685] Output: Video data sent to the server

[0686] Specific operation: The video data captured by the camera is sent to the server via the network. Data transfer is performed with low latency.

[0687] Step 4:

[0688] Receiving and storing video data on the server

[0689] Input: Image data sent from the imaging device

[0690] Output: Saved video data

[0691] Specific operation: The server receives the received video data in real time and stores it in a database, ensuring the integrity of the data required for subsequent processing.

[0692] Step 5:

[0693] Video data preprocessing

[0694] Input: Saved video data

[0695] Output: Pre-processed video data

[0696] What it does: The server preprocesses the video data, specifically removing noise, resizing frames, and converting data formats, so that it is properly prepared for input to the generative AI model.

[0697] Step 6:

[0698] Motion analysis using generative AI models

[0699] Input: Preprocessed video data

[0700] Output: Analyzed data (user behavior evaluation)

[0701] Specific operation: The server inputs preprocessed video data into the generative AI model, which then identifies the user's skeletal structure and the positions of major joints from the video data and analyzes their movements.

[0702] Step 7:

[0703] Generate feedback from analysis results

[0704] Input: Data analyzed by the generative AI model

[0705] Output: Feedback in text, image and video formats

[0706] Specific operation: The server generates feedback based on the analysis results of the generative AI model. The feedback includes specific corrections and improvements, and is provided in a format that is easy for the user to understand.

[0707] Step 8:

[0708] Sending feedback data to a tablet

[0709] Input: Generated feedback

[0710] Output: Feedback data sent to the tablet

[0711] What it does: The server sends the generated feedback to the tablet. The communication is real-time, ensuring that the feedback reaches the user immediately.

[0712] Step 9:

[0713] Viewing feedback on a tablet

[0714] Input: Feedback data sent from the server

[0715] Output: Feedback that is displayed to the user

[0716] Specific behavior: The tablet receives feedback and displays it to the user. The feedback displayed can include text, images, and videos, and includes specific corrections.

[0717] Step 10:

[0718] Modifying user behavior

[0719] Input: Feedback displayed on tablet

[0720] Output: Corrected training behavior

[0721] Specific actions: The user checks the feedback displayed on the tablet and corrects their training form, for example, by adjusting the knee angle properly when squatting.

[0722] (Application example 1)

[0723] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0724] Currently, in many factories, worker movements are dependent on visual evaluation and manual checks, resulting in issues with efficiency and safety. In particular, when new or inexperienced workers work alone, their movements may lack accuracy and appropriateness, posing risks to the quality and safety of the work. Furthermore, the lack of real-time feedback increases the likelihood of incorrect movements and postures being continued, resulting in reduced production efficiency. The purpose of this invention is to solve these issues.

[0725] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0726] In this invention, the server includes: means for capturing images of the user's training with a camera installed in the training area; means for receiving and saving the image data captured by the camera in real time; means for preprocessing the received image data and inputting it into a generative AI model to analyze the user's movements; means for generating feedback in the form of text, images, or videos based on the analyzed data; means for sending the generated feedback data to a tablet and displaying it to the user; and means installed in the work area of ​​the factory for analyzing the worker's movements in real time and sending feedback. This allows workers to receive feedback in real time, improving the accuracy and safety of their movements and ultimately increasing the efficiency of the entire factory.

[0727] "Training Area" means an area designated for a user to exercise or work.

[0728] A "camera" is a device that captures video data and transmits it to a server in real time.

[0729] "Video data" refers to digital data that includes the user's movements and training footage captured by a camera.

[0730] "Real-time" means that data is processed immediately with little delay.

[0731] "Storage" refers to the act of recording data on a storage device such as a server.

[0732] "Preprocessing" refers to the preparation work carried out before data analysis, and includes data normalization and resizing.

[0733] A "generative AI model" refers to an artificial intelligence model trained for a specified task, performing behavior analysis and feedback generation.

[0734] "Analysis" is the process of extracting information contained in data and finding meaning and patterns.

[0735] "Feedback" refers to advice or suggestions provided to users based on the analysis results.

[0736] "Text" means written information.

[0737] An "image" is a form of representing information as a visual (still image).

[0738] "Video" is a data format that contains moving visual information.

[0739] A "tablet" is a portable display device used by a user to view feedback.

[0740] "Factory work area" refers to the area within a factory used by workers to perform their designated tasks.

[0741] This invention is a system for analyzing the actions of users and workers in training areas and factory work areas in real time and providing feedback. Specifically, it uses cameras, servers, generative AI models, and tablets to provide feedback to users so that they can perform appropriate training and actions. This can improve work efficiency and safety.

[0742] System Configuration

[0743] 1. Camera

[0744] The cameras are installed in training areas or factory work areas and capture the actions of users and workers in real time.

[0745] The captured video data is transmitted to a server via a network.

[0746] 2. Server

[0747] The server receives and stores the video data sent from the camera.

[0748] The server preprocesses the received video data and prepares it for input into the generative AI model.

[0749] Specific preprocessing includes resizing and normalizing the video data.

[0750] 3. Generative AI Models

[0751] The generative AI model analyzes the video data input by the server and evaluates the actions of the user or worker.

[0752] The model compares the user's movements with ideal training form and work procedures to identify needed corrections and improvements.

[0753] 4. Generate feedback

[0754] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[0755] The feedback is specific, easy to understand, and includes content that users can immediately act on.

[0756] 5. Tablets

[0757] The tablet receives the feedback data sent from the server and displays it to the user or worker.

[0758] Based on the displayed feedback, users or workers can correct their form and actions.

[0759] Hardware and software used

[0760] Hardware: high-resolution cameras, servers, tablets

[0761] software:

[0762] OpenCV: Used for real-time video capture and display

[0763] TensorFlow: used for loading and inferencing generative AI models

[0764] requests: Used to communicate with the server

[0765] Specific examples

[0766] For example, consider a factory worker using a robotic arm to install parts. A camera captures the image to determine whether the movement is appropriate, and a generative AI model analyzes it. If the movement is incorrect, feedback such as "The robotic arm is out of position. Please move it 5 cm to the right" is displayed on a tablet.

[0767] Prompt Sentence Examples

[0768] "Generative AI models, robot behavior evaluation, real-time feedback, worker safety, and improved factory efficiency"

[0769] This allows the invention to provide real-time feedback to assist the worker or user in taking the appropriate action.

[0770] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0771] Step 1:

[0772] The camera captures the user or worker's movements in real time in the training area or factory work area.

[0773] Input: User or worker actions

[0774] Output: Real-time video data

[0775] Specific operation: The camera captures the movements of the user or worker and continuously captures frames.

[0776] Step 2:

[0777] The server receives and stores the video data captured by the camera in real time.

[0778] Input: Real-time video data

[0779] Output: Saved video data

[0780] Specific operation: The server receives the video data sent from the camera via the network and saves it in a file format.

[0781] Step 3:

[0782] The server preprocesses the received video data and prepares it for input into the generative AI model.

[0783] Input: Saved video data

[0784] Output: Preprocessed data (resized and normalized)

[0785] What it does: The server resizes the video data, normalizes pixel values, and converts the data format to one that is more easily accepted by the generative AI model.

[0786] Step 4:

[0787] The server inputs the preprocessed data into a generative AI model to analyze the actions of users and workers.

[0788] Input: Preprocessed data

[0789] Output: Analysis results (operation evaluation)

[0790] Specific actions: The generative AI model analyzes the video data and evaluates the actions of the user or worker. The analysis results identify whether the actions are correct and which areas need improvement.

[0791] Step 5:

[0792] The server generates feedback in the form of text, images, and videos based on the analyzed data.

[0793] Input: Analysis results

[0794] Output: Feedback data (text, images, videos)

[0795] Specific actions: The server generates feedback based on the analysis results, such as text instructions like "Move the robot arm 5 cm to the right," or images or videos showing the incorrect action.

[0796] Step 6:

[0797] The server sends the generated feedback data to a tablet and displays it to the user or worker.

[0798] Input: Feedback data

[0799] Output: Feedback displayed on the tablet

[0800] Specific operation: Feedback data is sent to a tablet and displayed in a form that can be visually understood by the user or worker. The user or worker then corrects their operation based on the feedback.

[0801] Step 7:

[0802] Users and workers can check the feedback displayed on the tablet and correct their actions.

[0803] Input: Feedback displayed on tablet

[0804] Output: Corrected behavior

[0805] Specific actions: The user or worker checks the feedback and corrects their actions or form, for example, adjusting the position of the robot arm as instructed.

[0806] The above are the specific processing steps for carrying out the present invention.

[0807] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0808] The system of this invention allows users to receive appropriate training feedback when an instructor is not present, and also has the ability to recognize the user's emotional state and dynamically adjust the feedback and training menu. Specifically, it is realized using a camera, a server, a generative AI model, a tablet, and an emotion engine.

[0809] System Configuration

[0810] 1. Camera

[0811] The camera is installed in the training area and captures the user's training in real time.

[0812] The camera continuously captures video data and transmits it to a server over a network.

[0813] 2. Server

[0814] The server receives and stores the video data sent from the camera in real time.

[0815] The server preprocesses the received video data and prepares it for input into the generative AI model.

[0816] 3. Generative AI Models

[0817] The generative AI model analyzes the video data input by the server and evaluates the user's actions.

[0818] The model compares the user's movements with ideal training form and identifies any corrections or improvements that need to be made.

[0819] 4. Generate feedback

[0820] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[0821] The feedback is specific and includes information that users can immediately understand and act on.

[0822] 5. Emotion Engine

[0823] The emotion engine uses video and audio data captured by the camera to analyze changes in the user's facial expressions and voice and estimate their emotional state.

[0824] Based on the analysis results of the emotion engine, the content and tone of the feedback is adjusted to improve the user's training experience.

[0825] 6. Tablets

[0826] The tablet receives the feedback data sent from the server and the analysis results of the emotion engine and displays them to the user.

[0827] Based on the feedback provided, users can improve their form and behavior.

[0828] Specific examples

[0829] Squat training

[0830] 1. Users

[0831] The user performs a squat.

[0832] The camera captures the squat movements and facial expressions in real time.

[0833] 2. Device (camera)

[0834] The camera transmits the video data to the server.

[0835] 3. Server

[0836] The server preprocesses the received squat video data and inputs it into the generative AI model.

[0837] A generative AI model analyzes the video data and evaluates the user's actions.

[0838] 4. Generative AI model (server)

[0839] The generative AI model compares the user's squat motion with ideal form.

[0840] Generates feedback such as "Your knee angle is too deep, adjust it to 90 degrees."

[0841] 5. Server

[0842] Send the generated feedback to the tablet.

[0843] 6. Device (tablet)

[0844] The tablet receives the feedback and displays it in an easy-to-understand way for the user.

[0845] For example, images or videos can be displayed to visually confirm that the knee angle is not appropriate.

[0846] 7. Emotion engine (server)

[0847] The emotion engine analyzes the user's facial expressions and voice to estimate their emotional state.

[0848] If the user is feeling tired or frustrated, for example, the system generates feedback that is more encouraging or offers simple points for improvement depending on the user's emotional state.

[0849] 8. Server

[0850] Based on the results of the emotion engine, the content and tone of the feedback are appropriately adjusted and sent to the tablet.

[0851] 9. Device (tablet)

[0852] The tablet displays feedback based on your emotional state.

[0853] For example, it provides personalized advice such as, "We recommend you take a short break or drink some water."

[0854] 10. Users

[0855] The user checks the feedback displayed on the tablet and follows the instructions to modify their training form and menu.

[0856] Emotional feedback is also taken into consideration to take appropriate actions such as taking breaks at appropriate times.

[0857] In this way, users can receive real-time feedback on not only their training movements but also their emotional state, which can improve the efficiency and safety of training and help users maintain their motivation.

[0858] The processing flow will be explained below.

[0859] Step 1:

[0860] User: The user starts a workout, for example, performing a squat.

[0861] Step 2:

[0862] Device (camera): A camera is installed in the training area and captures the user's movements and facial expressions in real time. Video data is continuously captured.

[0863] Step 3:

[0864] Terminal (camera): The camera sends video data to the server in real time. The video data is transferred via the network.

[0865] Step 4:

[0866] Server: The server receives and stores video data sent from the camera in real time. The video data is stored in a buffer.

[0867] Step 5:

[0868] Server: Pre-processes the received video data, including frame extraction, noise reduction, and resolution adjustment.

[0869] Step 6:

[0870] Server: Inputs preprocessed video data into the generative AI model.

[0871] Step 7:

[0872] Generative AI model (server): Analyzes video data and evaluates the user's movements. It compares the user's squat movements with the ideal form.

[0873] Step 8:

[0874] Generative AI model (server): Generates specific feedback such as "Your knee angle is too deep, please adjust it to 90 degrees." Feedback is generated in the form of text, images, and videos.

[0875] Step 9:

[0876] Server: Sends the generated feedback to the tablet. The feedback data is sent to the tablet in real time.

[0877] Step 10:

[0878] Terminal (tablet): The tablet receives the feedback data and displays it to the user, for example, an image showing the knee angle and text indicating correction points.

[0879] Step 11:

[0880] User: The user checks the feedback displayed on the tablet and corrects their squat form.

[0881] Step 12:

[0882] Server: Passes part of the video data to the emotion engine.

[0883] Step 13:

[0884] Emotion engine (server): Analyzes video and audio data, detects changes in the user's facial expressions and voice, and infers their emotional state. For example, it infers whether the user is tired or frustrated.

[0885] Step 14:

[0886] Emotion engine (server): Adjusts the content and tone of the feedback based on the results of the emotion engine. For example, if the user is estimated to be tired, it generates encouraging feedback such as "Take a short break."

[0887] Step 15:

[0888] Server: Sends feedback reflecting the results of the emotion engine back to the tablet.

[0889] Step 16:

[0890] Device (tablet): The tablet displays feedback based on the user's emotional state. If the user's facial expression indicates fatigue, it displays words of encouragement or a message recommending rest.

[0891] Step 17:

[0892] User: The user checks the newly displayed emotional feedback and takes appropriate action, such as taking a short break or drinking more water.

[0893] Through these steps, users can receive real-time feedback on their training movements and emotional state, which can improve the efficiency and safety of their training and help maintain their motivation.

[0894] Example 2

[0895] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0896] Conventional training systems have the problem that it is difficult for users to receive appropriate feedback when an instructor is not present, preventing them from correcting their form or maximizing the effectiveness of their training. Additionally, feedback does not take into account the user's emotional state, making it difficult to maintain motivation and continue training.

[0897] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0898] In this invention, the server includes means for capturing images of the user's training using a camera installed in the training area, means for receiving and saving the video data captured by the camera in real time, means for preprocessing the received video data and inputting it into an artificial intelligence model to analyze the user's movements, means for generating feedback in the form of text, images, or video based on the analyzed data, means for transmitting the generated feedback data to a display terminal and displaying it to the user, means for analyzing changes in the user's facial expressions and voice to estimate their emotional state, and means for adjusting the content and tone of the feedback based on their emotional state. This allows the user to receive feedback on their training movements and emotional state in real time, improving the efficiency and safety of their training and helping them maintain their motivation.

[0899] The "photography device" is a device that is installed in the training area and is used to capture the user's training scene in real time.

[0900] A "server" is a device that receives video data sent from an imaging device and performs a series of processes such as storage, pre-processing, analysis, and feedback generation.

[0901] The "artificial intelligence model" is a program that analyzes video data input into the server and evaluates the user's actions.

[0902] "Preprocessing" refers to the process of removing noise from the received video data, extracting frames, adjusting the resolution, and so on.

[0903] "Feedback" refers to formal advice or instructions to users that are generated based on the analysis results of an artificial intelligence model and are provided in the form of text, images, videos, etc.

[0904] A "display terminal" is a device that receives feedback data sent from the server and visually presents it to the user.

[0905] The "emotion engine" is a program that analyzes changes in the user's facial expressions and voice to estimate their emotional state.

[0906] "Real-time" refers to the immediacy of data processing and feedback during a user's training.

[0907] "Tone" refers to the wording and delivery of feedback or advice, which is adjusted according to the user's emotional state.

[0908] The system of this invention is designed to allow users to receive appropriate training feedback even when an instructor is not present. It also has the ability to recognize the user's emotional state and dynamically adjust the feedback and training menu. The main components of the system are as follows:

[0909] 1. Imaging equipment

[0910] It is installed in the training area and records the user's training in real time, and the captured video data is sent sequentially to a server via the network.

[0911] 2. Server

[0912] The server receives and stores video data sent from the imaging device in real time. The first step is pre-processing of the received video data, which includes noise removal, frame extraction, and resolution adjustment. The pre-processed data is then formatted so that it can be effectively analyzed by the generative AI model.

[0913] The pre-processed video data is then fed into a generative AI model, which analyzes the data and evaluates the user's movements, for example comparing them with ideal training form to identify necessary corrections or improvements.

[0914] Based on the analysis results, the server generates specific feedback in the form of text, images, or videos, which are then sent to a display device and presented to the user.

[0915] The server is also equipped with an emotion engine that analyzes the user's facial expressions and voice to estimate their emotional state, and adjusts the content and tone of the feedback based on the results of this analysis.

[0916] 3. Display terminal

[0917] The system receives feedback data sent from the server and displays it to the user. The feedback is designed to allow users to understand and act immediately, and specific images and videos are also presented.

[0918] Specific examples

[0919] Squat training

[0920] 1. Users

[0921] The user stands in the training area and begins training by pressing the "Start Training" button on the tablet screen.

[0922] 2. Terminal (photography device)

[0923] The camera captures the user's squatting movements and facial expressions in real time and sends the video data to a server.

[0924] 3. Server

[0925] The server preprocesses the received video data of the squat and inputs it into a generative AI model, which then analyzes the video data and evaluates the user's movements.

[0926] The generative AI model compares the user's squat movement with ideal training form and generates specific feedback such as, "Your knee angle is too deep; adjust it to 90 degrees."

[0927] This feedback is generated in the form of text, images, or videos and sent to a display device.

[0928] The emotion engine analyzes the user's facial expressions and voice to estimate their emotional state. For example, if the user is feeling tired or frustrated, it generates feedback content and tone appropriate to that emotional state and sends it to the display device.

[0929] 4. Terminal (display terminal)

[0930] The display device receives the feedback and displays it in an easy-to-understand way to the user. For example, it can show images or videos that show that the knee angle is not appropriate, and provide personalized advice such as "We recommend that you take a short break or drink some water."

[0931] Prompt Sentence Examples

[0932] "Your knees are at too deep an angle when squatting; adjust them to 90 degrees."

[0933] "I suggest you take a short break or drink some water."

[0934] In this way, users can receive real-time feedback on their training movements and emotional state, which can improve training efficiency and safety and help users maintain their motivation.

[0935] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0936] Step 1:

[0937] User starts training

[0938] Specific behavior:

[0939] The user stands in the training area and presses the "Start Training" button on the tablet screen, which starts the system.

[0940] Input: User presses the "Start Training" button on the tablet.

[0941] Output: The system goes into training start mode.

[0942] Step 2:

[0943] The device (camera) captures the training video.

[0944] Specific behavior:

[0945] The camera is installed in a position that captures the entire training area. When the user starts to move, the camera captures their movements in real time and transmits the video data sequentially to a server via a network.

[0946] Input: User action

[0947] Output: Real-time video data

[0948] Step 3:

[0949] The server receives the video data

[0950] Specific behavior:

[0951] The server receives the image data sent from the imaging device and stores it in real time. At this point, the data is raw data that has not yet been pre-processed.

[0952] Input: Raw image data transmitted from the imaging device

[0953] Output: Saved video data

[0954] Step 4:

[0955] The server preprocesses the video data

[0956] Specific behavior:

[0957] The server preprocesses the received video data, which includes removing noise, cropping frames, and adjusting resolution, so that the preprocessed data is in a format that can be effectively analyzed by the generative AI model.

[0958] Input: Stored raw video data

[0959] Output: Pre-processed video data

[0960] Step 5:

[0961] The server inputs data into the generative AI model

[0962] Specific behavior:

[0963] The pre-processed video data is then input into the generative AI model, a process that prepares the data to fit the model's input format.

[0964] Input: Preprocessed video data

[0965] Output: Data input to the generative AI model

[0966] Step 6:

[0967] Generative AI models analyze video data

[0968] Specific behavior:

[0969] A generative AI model analyzes the input data and evaluates the user's movements. The model compares the user's movements with ideal training form and identifies necessary corrections and improvements.

[0970] Input: Data input to the generative AI model

[0971] Output: Parsed feedback data

[0972] Step 7:

[0973] Generative AI models generate feedback

[0974] Specific behavior:

[0975] The generative AI model generates specific feedback based on the analysis results, such as "Your knee angle is too deep, please adjust it to 90 degrees."

[0976] Input: Parsed feedback data

[0977] Output: Generated feedback

[0978] Step 8:

[0979] The server sends feedback to the display device

[0980] Specific behavior:

[0981] The generative AI model generates feedback that is sent to the display device in the form of text, images, or videos.

[0982] Input: Generated feedback

[0983] Output: Feedback sent to the display device

[0984] Step 9:

[0985] Emotion engine analyzes emotional state

[0986] Specific behavior:

[0987] The emotion engine analyzes the user's facial expressions and voice to estimate their emotional state. For example, if frustration is detected, the engine returns that data to the server.

[0988] Input: User's facial expressions and voice

[0989] Output: Emotional state data

[0990] Step 10:

[0991] The server sends feedback based on the emotional state to the display device.

[0992] Specific behavior:

[0993] The feedback content is adjusted based on the analysis results of the emotion engine, changing the tone and message content, and then the adjusted feedback is sent to the display device.

[0994] Input: Emotional state data

[0995] Output: Feedback based on emotional state

[0996] Step 11:

[0997] The display device displays the feedback to the user.

[0998] Specific behavior:

[0999] The display device then displays the received feedback to the user, for example, showing an image or video that indicates the knee angle is not appropriate and providing personalized advice such as "We recommend you take a short break or drink some water."

[1000] Input: Feedback based on emotional state

[1001] Output: Feedback displayed to the user

[1002] Step 12:

[1003] Users modify their training according to the feedback

[1004] Specific behavior:

[1005] The user checks the feedback displayed on the display terminal and follows the instructions to correct their training form and movements. They also refer to the emotional feedback and take appropriate actions such as taking breaks at appropriate times.

[1006] Input: Feedback displayed to the user

[1007] Output: Corrected training form and movements

[1008] (Application example 2)

[1009] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1010] Conventional fitness training systems struggle to provide users with proper form and movement feedback when an instructor is not present. Furthermore, providing uniform feedback without considering the user's emotional state can reduce the user's motivation and training efficiency. This raises concerns about the risk of injury and reduced training effectiveness.

[1011] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing images of the user's training using a camera installed in the training area, means for receiving and saving the video data captured by the camera in real time, means for preprocessing the received video data and inputting it into a generative AI model to analyze the user's movements, means for generating feedback in the form of text, images, or videos based on the analyzed data, means for sending the generated feedback data to a tablet and displaying it to the user, and means for analyzing the user's emotional state and dynamically adjusting the feedback and training menu. This allows users to receive appropriate feedback in real time even when an instructor is not present, and also enables them to obtain feedback optimized according to their individual emotional state.

[1012] The "camera" is an image capturing device for capturing images of the user's training.

[1013] "Video data" is digital data containing visual information captured by a camera.

[1014] "Means for receiving and storing in real time" refers to a function for receiving captured video data to a server without delay and storing the data as needed.

[1015] "Preprocessing" refers to the process of shaping and processing video data to make it suitable for analysis by a generative AI model.

[1016] A "generative AI model" is an artificial intelligence model that analyzes and evaluates a user's training movements.

[1017] "Input" refers to feeding pre-processed video data into a generative AI model.

[1018] The "means of analysis" refers to a method of analyzing and evaluating a user's training movements using a generative AI model.

[1019] "Feedback" refers to instructions or advice provided to the user based on the analysis results of the generative AI model.

[1020] "Text, image, video format" refers to a format in which the feedback content is displayed using different means of expression.

[1021] A "tablet" is a portable display device for displaying generated feedback to a user.

[1022] "Emotional state" refers to the psychological state estimated based on information such as the user's facial expression and voice.

[1023] "Dynamic adjustment means" refers to a function that flexibly changes the feedback and training menu content according to the user's emotional state.

[1024] This invention is a system that allows users to receive appropriate training feedback when an instructor is not present, and also has the ability to recognize the user's emotional state and dynamically adjust the feedback and training menu. This system is realized using a camera, a server, a generative AI model, a tablet, and an emotion engine.

[1025] System Configuration

[1026] 1. Camera

[1027] The camera is installed in the training area and captures the user's training in real time.

[1028] As a specific example, a high-resolution camera (e.g., Logitech StreamCam) is used.

[1029] 2. Server

[1030] The server receives and stores the video data sent from the camera in real time.

[1031] After preprocessing, the received video data is input into the generative AI model.

[1032] As a specific example, a high-performance cloud server (e.g., Amazon Web Services EC2) is used.

[1033] 3. Generative AI Models

[1034] The generative AI model analyzes the video data input by the server and evaluates the user's actions.

[1035] The model compares the user's movements with ideal training form and identifies any corrections or improvements that need to be made.

[1036] A specific example is using a custom AI model (e.g., TensorFlow or PyTorch) for behavior analysis.

[1037] 4. Generate feedback

[1038] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[1039] The feedback is specific and includes information that users can immediately understand and act on.

[1040] Example: The knee angle is too deep, please adjust it to 90 degrees.

[1041] 5. Emotion Engine

[1042] The emotion engine uses video and audio data captured by the camera to analyze changes in the user's facial expressions and voice and estimate their emotional state.

[1043] Based on the analysis results, the content and tone of the feedback can be adjusted to improve the user's training experience.

[1044] As a specific example, we use an emotion analysis engine (e.g., Microsoft Azure Emotion API).

[1045] 6. Tablets

[1046] The tablet receives the feedback data sent from the server and the analysis results of the emotion engine and displays them to the user.

[1047] Based on the feedback provided, users can improve their form and behavior.

[1048] Processing flow

[1049] 1. User Operation

[1050] A user taps the start button on a tablet or smartphone in the fitness training area.

[1051] 2. Camera Operation

[1052] The camera captures the user's training movements and sends the video data to a server.

[1053] 3. Server Processing

[1054] The server receives the video data, pre-processes it, and then feeds the data into the generative AI model.

[1055] 4. Analysis of generative AI models

[1056] A generative AI model analyzes the data and compares the user's movements with ideal form.

[1057] For example: "Your knee angle is too deep, please adjust it to 90 degrees."

[1058] 5. Generate and send feedback

[1059] The server generates feedback in the form of text, images and videos and sends it to the tablet.

[1060] 6. Emotion Engine Analysis

[1061] The emotion engine analyzes the user's emotional state and dynamically adjusts the feedback content.

[1062] 7. Displaying feedback to users

[1063] The tablet receives the feedback and displays it in an easy-to-understand way.

[1064] For example: "Bend your knees more."

[1065] Prompt Sentence Examples

[1066] Analyze footage of a user squatting, assess knee angle, hip position, and back curvature, and generate feedback to correct for ideal posture.

[1067] This allows users to receive both motion- and emotion-based feedback in real time, improving training efficiency and safety.

[1068] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1069] Step 1:

[1070] A user taps the start button on a tablet or smartphone in the fitness training area. This causes the camera to start capturing video data. The input is the user's operation information, and the output is a signal to the camera to start recording.

[1071] Step 2:

[1072] The camera captures the user's training movements in real time and captures the video data, which is then sent to a server via a network. The input is the user's movement data, and the output is the video data sent to the server.

[1073] Step 3:

[1074] The server receives and stores video data from the camera in real time. It then preprocesses the received video data and prepares it for input into the generative AI model. The input is video data from the camera, and the output is preprocessed data. Specifically, noise removal and frame correction are performed.

[1075] Step 4:

[1076] The server inputs the preprocessed video data into a generative AI model. The generative AI model analyzes and evaluates the user's training movements. The input is the preprocessed data, and the output is the evaluation result of the user's movements. Specifically, the movement data is compared with the ideal form to detect errors.

[1077] Step 5:

[1078] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model. The input is the evaluation results of the user's movements, and the output is feedback data. Specifically, it describes in detail the corrections needed for knee angle and hip position.

[1079] Step 6:

[1080] The emotion engine uses video and audio data captured by the camera to analyze changes in the user's facial expressions and voice and estimate their emotional state. The input is video and audio data, and the output is the analysis result of the emotional state. The emotion engine uses, for example, the Microsoft Azure Emotion API.

[1081] Step 7:

[1082] The server adjusts the content and tone of the feedback based on the analysis results of the emotion engine. The input is the analysis result of the emotional state, and the output is dynamically adjusted feedback. If the user is fatigued, the content will be changed to be more encouraging.

[1083] Step 8:

[1084] The server sends the final feedback data to the tablet. The input is the dynamically adjusted feedback data, and the output is the data transmission to the tablet.

[1085] Step 9:

[1086] The tablet receives the feedback data sent from the server and the analysis results of the emotion engine, and displays them in an easy-to-understand manner for the user. The input is the feedback data, and the output is the display on the tablet.

[1087] Step 10:

[1088] The user checks the feedback displayed on the tablet and corrects their training movements. They also refer to emotional feedback to take appropriate actions such as taking breaks at appropriate times. The input is feedback data, and the output is corrected movements and actions such as resting.

[1089] This series of steps allows users to receive both behavioral and emotional feedback in real time, improving training efficiency and safety.

[1090] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1091] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1092] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1093] [Third embodiment]

[1094] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1095] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1096] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1097] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1098] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1099] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1100] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1101] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1102] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1103] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1104] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1105] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1106] The system of this invention allows users to receive appropriate training feedback in an instructor-less sports gym. Specifically, it is realized using a camera, a server, a generative AI model, and a tablet.

[1107] System Configuration

[1108] 1. Camera

[1109] The camera is installed in the training area and captures the user's training in real time.

[1110] The camera continuously captures video data and transmits it to a server over a network.

[1111] 2. Server

[1112] The server receives and stores the video data sent from the camera in real time.

[1113] The server preprocesses the received video data and prepares it for input into the generative AI model.

[1114] 3. Generative AI Models

[1115] The generative AI model analyzes the video data input by the server and evaluates the user's actions.

[1116] The model compares the user's movements with ideal training form and identifies any corrections or improvements that need to be made.

[1117] 4. Generate feedback

[1118] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[1119] The feedback is specific and includes information that users can immediately understand and act on.

[1120] 5. Tablets

[1121] The tablet receives the feedback data sent from the server and displays it to the user.

[1122] Based on the feedback provided, users can modify their forms and actions.

[1123] Specific examples

[1124] Squat training

[1125] 1. Users

[1126] The user performs a squat.

[1127] The camera captures the squat movement in real time.

[1128] 2. Device (camera)

[1129] The camera transmits the video data to the server.

[1130] 3. Server

[1131] The server preprocesses the received squat video data and inputs it into the generative AI model.

[1132] A generative AI model analyzes the video data and evaluates the user's actions.

[1133] 4. Generative AI model (server)

[1134] The generative AI model compares the user's squat motion with ideal form.

[1135] Generates feedback such as "Your knee angle is too deep, adjust it to 90 degrees."

[1136] 5. Server

[1137] Send the generated feedback to the tablet.

[1138] 6. Device (tablet)

[1139] The tablet receives the feedback and displays it in an easy-to-understand way for the user.

[1140] For example, images or videos can be displayed to visually confirm that the knee angle is not appropriate.

[1141] 7. Users

[1142] The user checks the feedback displayed on the tablet and corrects their squat form.

[1143] In this way, users can train efficiently and safely with real-time feedback, maximizing the effectiveness of their training and reducing the risk of injury.

[1144] The processing flow will be explained below.

[1145] Step 1:

[1146] Terminal (camera): When the user starts training, the camera starts operating. The camera continuously captures live images of the training area and transmits the images to the server in real time.

[1147] Step 2:

[1148] Server: The server receives video data sent from the camera in real time, stores the received video data in a buffer, and prepares it for pre-processing.

[1149] Step 3:

[1150] Server: Preprocesses the video data. Preprocessing includes frame extraction, noise reduction, and resolution adjustment. This converts the data into a format that is easy for the generative AI model to analyze.

[1151] Step 4:

[1152] Server: Inputs the preprocessed video data into the generative AI model, which analyzes the input data and evaluates the user's training movements.

[1153] Step 5:

[1154] Generative AI model (server): Detects the user's bone structure and joint angles from video data and compares them with pre-registered ideal form. For example, in the case of squats, the angle of the knees and the degree of curvature of the back are evaluated.

[1155] Step 6:

[1156] Generative AI model (server): Generates feedback based on the results of behavior analysis. The feedback is created in the form of text, images, and videos, and provides specific advice to the user.

[1157] Step 7:

[1158] Server: Sends the generated feedback to the tablet. The feedback data is sent to the tablet in real time, allowing the user to view it immediately.

[1159] Step 8:

[1160] Terminal (tablet): The tablet receives the feedback data sent from the server and displays it to the user. The feedback is easy for the user to understand and indicates corrections and improvements.

[1161] Step 9:

[1162] User: The user checks the feedback displayed on the tablet and follows the instructions to correct their training form, for example, by performing specific actions such as "change the angle of your knees" or "keep your back straight."

[1163] Through these steps, users can receive appropriate feedback in real time, improving the efficiency and safety of their training.

[1164] Example 1

[1165] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1166] In modern sports gyms, users often find it difficult to maintain proper form without the guidance of an instructor. This increases the risk of injury due to incorrect training form or improper movements. This creates a need for a system that provides users with real-time feedback to ensure safe and effective training.

[1167] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1168] In this invention, the server includes means for capturing images of the user's training movements with a camera installed in the training area, means for receiving and saving the video data captured by the camera in real time, means for preprocessing the received video data and inputting it into an analysis model to analyze the user's movements, means for generating feedback in the form of text, images, or video based on the analyzed data, and means for transmitting the generated feedback data to a display terminal and displaying it to the user. This allows the user to receive appropriate feedback in real time, improving the quality of their training and further reducing the risk of injury.

[1169] The "photography device" is a device for capturing images of the user's training movements in real time and acquiring the video data.

[1170] "Video data" refers to visual information of training movements captured by a camera.

[1171] "Preprocessing" refers to a data processing procedure for converting video data into a format that can be analyzed by the generative AI model.

[1172] An "analysis model" is a computational model that analyzes received video data and evaluates the user's training movements, and this mainly applies to generative AI models.

[1173] "Feedback" is information, including corrections and improvements, provided to users based on analyzed data, and is provided in the form of text, images, or videos.

[1174] A "display terminal" is a device on which a user can view the feedback they have received, and is typically a display device such as a tablet.

[1175] "Real-time" refers to the fact that feedback is generated and displayed within a very short time (almost instantaneously) after the user performs a training action.

[1176] This invention provides a system that allows users to receive real-time feedback on their training movements without the need for an instructor. Specifically, this system is realized using a camera, a server, an analysis model, and a display terminal.

[1177] System Configuration

[1178] 1. Imaging equipment

[1179] The imaging device is installed in the training area and captures images of the user's training movements in real time.

[1180] The imaging device is composed of a high-resolution camera, which continuously captures video data of the training movements and transmits it to a server via a network.

[1181] 2. Server

[1182] The server is a high-performance computing device that receives and stores the video data transmitted from the imaging device in real time.

[1183] The server preprocesses the received video data to prepare it for input to the generative AI model, including noise reduction, frame resizing, and data format conversion.

[1184] 3. Analysis Model

[1185] Using the generative AI model, the server analyzes the pre-processed video data.

[1186] The generative AI model compares the user's movements with ideal training form and identifies areas for correction and improvement.

[1187] 4. Generating and displaying feedback

[1188] The server generates specific feedback based on the analysis results of the generative AI model, which can be in the form of text, images, or videos.

[1189] The feedback data is sent to a display terminal (e.g., a tablet) and displayed to the user.

[1190] Specific examples

[1191] Squat training

[1192] 1. User Actions

[1193] A user performs squats in the training area, making sure to perform the movements within the camera's field of view.

[1194] 2. Acquiring video data

[1195] The camera captures the user's squatting motion in real time, and the captured video data is sent to a server.

[1196] 3. Data Analysis

[1197] The server receives the video data, preprocesses it, and then inputs it into the generative AI model, which analyzes the video data and compares the user's movements with the ideal form.

[1198] 4. Generate feedback

[1199] The generative AI model generates feedback such as, "The knee angle is too deep, please adjust it to 90 degrees." The server sends this feedback to the tablet.

[1200] 5. Viewing Feedback

[1201] The tablet receives the feedback and displays it to the user in an easy-to-understand way, such as showing an image or video that indicates the knee angle is not correct.

[1202] 6. User Modifications

[1203] The user checks the feedback displayed on the tablet and corrects their squat form.

[1204] Prompt Sentence Examples

[1205] "Please evaluate whether this training form is correct and advise me on how to correct it."

[1206] In this way, the present invention aims to allow users to receive appropriate feedback in real time, improving the quality of their training and reducing the risk of injury.

[1207] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1208] Step 1:

[1209] Start user training behavior

[1210] Input: A user enters a training area and begins a particular training movement (e.g., a squat).

[1211] Specific Action: User performs squats with proper form in the designated training area, taking care to perform the movements within the camera's field of view.

[1212] Step 2:

[1213] Acquisition of video data using imaging equipment

[1214] Input: User training behavior

[1215] Output: Real-time video data of training movements

[1216] Specific operation: The camera records the user's training movements in real time, continuously capturing high-resolution video data and storing it in a temporary buffer.

[1217] Step 3:

[1218] Video data transmission

[1219] Input: Real-time video data of training movements

[1220] Output: Video data sent to the server

[1221] Specific operation: The video data captured by the camera is sent to the server via the network. Data transfer is performed with low latency.

[1222] Step 4:

[1223] Receiving and storing video data on the server

[1224] Input: Image data sent from the imaging device

[1225] Output: Saved video data

[1226] Specific operation: The server receives the received video data in real time and stores it in a database, ensuring the integrity of the data required for subsequent processing.

[1227] Step 5:

[1228] Video data preprocessing

[1229] Input: Saved video data

[1230] Output: Pre-processed video data

[1231] What it does: The server preprocesses the video data, specifically removing noise, resizing frames, and converting data formats, so that it is properly prepared for input to the generative AI model.

[1232] Step 6:

[1233] Motion analysis using generative AI models

[1234] Input: Preprocessed video data

[1235] Output: Analyzed data (user behavior evaluation)

[1236] Specific operation: The server inputs preprocessed video data into the generative AI model, which then identifies the user's skeletal structure and the positions of major joints from the video data and analyzes their movements.

[1237] Step 7:

[1238] Generate feedback from analysis results

[1239] Input: Data analyzed by the generative AI model

[1240] Output: Feedback in text, image and video formats

[1241] Specific operation: The server generates feedback based on the analysis results of the generative AI model. The feedback includes specific corrections and improvements, and is provided in a format that is easy for the user to understand.

[1242] Step 8:

[1243] Sending feedback data to a tablet

[1244] Input: Generated feedback

[1245] Output: Feedback data sent to the tablet

[1246] What it does: The server sends the generated feedback to the tablet. The communication is real-time, ensuring that the feedback reaches the user immediately.

[1247] Step 9:

[1248] Viewing feedback on a tablet

[1249] Input: Feedback data sent from the server

[1250] Output: Feedback that is displayed to the user

[1251] Specific behavior: The tablet receives feedback and displays it to the user. The feedback displayed can include text, images, and videos, and includes specific corrections.

[1252] Step 10:

[1253] Modifying user behavior

[1254] Input: Feedback displayed on tablet

[1255] Output: Corrected training behavior

[1256] Specific actions: The user checks the feedback displayed on the tablet and corrects their training form, for example, by adjusting the knee angle properly when squatting.

[1257] (Application example 1)

[1258] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1259] Currently, in many factories, worker movements are dependent on visual evaluation and manual checks, resulting in issues with efficiency and safety. In particular, when new or inexperienced workers work alone, their movements may lack accuracy and appropriateness, posing risks to the quality and safety of the work. Furthermore, the lack of real-time feedback increases the likelihood of incorrect movements and postures being continued, resulting in reduced production efficiency. The purpose of this invention is to solve these issues.

[1260] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1261] In this invention, the server includes: means for capturing images of the user's training with a camera installed in the training area; means for receiving and saving the image data captured by the camera in real time; means for preprocessing the received image data and inputting it into a generative AI model to analyze the user's movements; means for generating feedback in the form of text, images, or videos based on the analyzed data; means for sending the generated feedback data to a tablet and displaying it to the user; and means installed in the work area of ​​the factory for analyzing the worker's movements in real time and sending feedback. This allows workers to receive feedback in real time, improving the accuracy and safety of their movements and ultimately increasing the efficiency of the entire factory.

[1262] "Training Area" means an area designated for a user to exercise or work.

[1263] A "camera" is a device that captures video data and transmits it to a server in real time.

[1264] "Video data" refers to digital data that includes the user's movements and training footage captured by a camera.

[1265] "Real-time" means that data is processed immediately with little delay.

[1266] "Storage" refers to the act of recording data on a storage device such as a server.

[1267] "Preprocessing" refers to the preparation work carried out before data analysis, and includes data normalization and resizing.

[1268] A "generative AI model" refers to an artificial intelligence model trained for a specified task, performing behavior analysis and feedback generation.

[1269] "Analysis" is the process of extracting information contained in data and finding meaning and patterns.

[1270] "Feedback" refers to advice or suggestions provided to users based on the analysis results.

[1271] "Text" means written information.

[1272] An "image" is a form of representing information as a visual (still image).

[1273] "Video" is a data format that contains moving visual information.

[1274] A "tablet" is a portable display device used by a user to view feedback.

[1275] "Factory work area" refers to the area within a factory used by workers to perform their designated tasks.

[1276] This invention is a system for analyzing the actions of users and workers in training areas and factory work areas in real time and providing feedback. Specifically, it uses cameras, servers, generative AI models, and tablets to provide feedback to users so that they can perform appropriate training and actions. This can improve work efficiency and safety.

[1277] System Configuration

[1278] 1. Camera

[1279] The cameras are installed in training areas or factory work areas and capture the actions of users and workers in real time.

[1280] The captured video data is transmitted to a server via a network.

[1281] 2. Server

[1282] The server receives and stores the video data sent from the camera.

[1283] The server preprocesses the received video data and prepares it for input into the generative AI model.

[1284] Specific preprocessing includes resizing and normalizing the video data.

[1285] 3. Generative AI Models

[1286] The generative AI model analyzes the video data input by the server and evaluates the actions of the user or worker.

[1287] The model compares the user's movements with ideal training form and work procedures to identify needed corrections and improvements.

[1288] 4. Generate feedback

[1289] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[1290] The feedback is specific, easy to understand, and includes content that users can immediately act on.

[1291] 5. Tablets

[1292] The tablet receives the feedback data sent from the server and displays it to the user or worker.

[1293] Based on the displayed feedback, users or workers can correct their form and actions.

[1294] Hardware and software used

[1295] Hardware: high-resolution cameras, servers, tablets

[1296] software:

[1297] OpenCV: Used for real-time video capture and display

[1298] TensorFlow: used for loading and inferencing generative AI models

[1299] requests: Used to communicate with the server

[1300] Specific examples

[1301] For example, consider a factory worker using a robotic arm to install parts. A camera captures the image to determine whether the movement is appropriate, and a generative AI model analyzes it. If the movement is incorrect, feedback such as "The robotic arm is out of position. Please move it 5 cm to the right" is displayed on a tablet.

[1302] Prompt Sentence Examples

[1303] "Generative AI models, robot behavior evaluation, real-time feedback, worker safety, and improved factory efficiency"

[1304] This allows the invention to provide real-time feedback to assist the worker or user in taking the appropriate action.

[1305] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1306] Step 1:

[1307] The camera captures the user or worker's movements in real time in the training area or factory work area.

[1308] Input: User or worker actions

[1309] Output: Real-time video data

[1310] Specific operation: The camera captures the movements of the user or worker and continuously captures frames.

[1311] Step 2:

[1312] The server receives and stores the video data captured by the camera in real time.

[1313] Input: Real-time video data

[1314] Output: Saved video data

[1315] Specific operation: The server receives the video data sent from the camera via the network and saves it in a file format.

[1316] Step 3:

[1317] The server preprocesses the received video data and prepares it for input into the generative AI model.

[1318] Input: Saved video data

[1319] Output: Preprocessed data (resized and normalized)

[1320] What it does: The server resizes the video data, normalizes pixel values, and converts the data format to one that is more easily accepted by the generative AI model.

[1321] Step 4:

[1322] The server inputs the preprocessed data into a generative AI model to analyze the actions of users and workers.

[1323] Input: Preprocessed data

[1324] Output: Analysis results (operation evaluation)

[1325] Specific actions: The generative AI model analyzes the video data and evaluates the actions of the user or worker. The analysis results identify whether the actions are correct and which areas need improvement.

[1326] Step 5:

[1327] The server generates feedback in the form of text, images, and videos based on the analyzed data.

[1328] Input: Analysis results

[1329] Output: Feedback data (text, images, videos)

[1330] Specific actions: The server generates feedback based on the analysis results, such as text instructions like "Move the robot arm 5 cm to the right," or images or videos showing the incorrect action.

[1331] Step 6:

[1332] The server sends the generated feedback data to a tablet and displays it to the user or worker.

[1333] Input: Feedback data

[1334] Output: Feedback displayed on the tablet

[1335] Specific operation: Feedback data is sent to a tablet and displayed in a form that can be visually understood by the user or worker. The user or worker then corrects their operation based on the feedback.

[1336] Step 7:

[1337] Users and workers can check the feedback displayed on the tablet and correct their actions.

[1338] Input: Feedback displayed on tablet

[1339] Output: Corrected behavior

[1340] Specific actions: The user or worker checks the feedback and corrects their actions or form, for example, adjusting the position of the robot arm as instructed.

[1341] The above are the specific processing steps for carrying out the present invention.

[1342] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1343] The system of this invention allows users to receive appropriate training feedback when an instructor is not present, and also has the ability to recognize the user's emotional state and dynamically adjust the feedback and training menu. Specifically, it is realized using a camera, a server, a generative AI model, a tablet, and an emotion engine.

[1344] System Configuration

[1345] 1. Camera

[1346] The camera is installed in the training area and captures the user's training in real time.

[1347] The camera continuously captures video data and transmits it to a server over a network.

[1348] 2. Server

[1349] The server receives and stores the video data sent from the camera in real time.

[1350] The server preprocesses the received video data and prepares it for input into the generative AI model.

[1351] 3. Generative AI Models

[1352] The generative AI model analyzes the video data input by the server and evaluates the user's actions.

[1353] The model compares the user's movements with ideal training form and identifies any corrections or improvements that need to be made.

[1354] 4. Generate feedback

[1355] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[1356] The feedback is specific and includes information that users can immediately understand and act on.

[1357] 5. Emotion Engine

[1358] The emotion engine uses video and audio data captured by the camera to analyze changes in the user's facial expressions and voice and estimate their emotional state.

[1359] Based on the analysis results of the emotion engine, the content and tone of the feedback is adjusted to improve the user's training experience.

[1360] 6. Tablets

[1361] The tablet receives the feedback data sent from the server and the analysis results of the emotion engine and displays them to the user.

[1362] Based on the feedback provided, users can improve their form and behavior.

[1363] Specific examples

[1364] Squat training

[1365] 1. Users

[1366] The user performs a squat.

[1367] The camera captures the squat movements and facial expressions in real time.

[1368] 2. Device (camera)

[1369] The camera transmits the video data to the server.

[1370] 3. Server

[1371] The server preprocesses the received squat video data and inputs it into the generative AI model.

[1372] A generative AI model analyzes the video data and evaluates the user's actions.

[1373] 4. Generative AI model (server)

[1374] The generative AI model compares the user's squat motion with ideal form.

[1375] Generates feedback such as "Your knee angle is too deep, adjust it to 90 degrees."

[1376] 5. Server

[1377] Send the generated feedback to the tablet.

[1378] 6. Device (tablet)

[1379] The tablet receives the feedback and displays it in an easy-to-understand way for the user.

[1380] For example, images or videos can be displayed to visually confirm that the knee angle is not appropriate.

[1381] 7. Emotion engine (server)

[1382] The emotion engine analyzes the user's facial expressions and voice to estimate their emotional state.

[1383] If the user is feeling tired or frustrated, for example, the system generates feedback that is more encouraging or offers simple points for improvement depending on the user's emotional state.

[1384] 8. Server

[1385] Based on the results of the emotion engine, the content and tone of the feedback are appropriately adjusted and sent to the tablet.

[1386] 9. Device (tablet)

[1387] The tablet displays feedback based on your emotional state.

[1388] For example, it provides personalized advice such as, "We recommend you take a short break or drink some water."

[1389] 10. Users

[1390] The user checks the feedback displayed on the tablet and follows the instructions to modify their training form and menu.

[1391] Emotional feedback is also taken into consideration to take appropriate actions such as taking breaks at appropriate times.

[1392] In this way, users can receive real-time feedback on not only their training movements but also their emotional state, which can improve the efficiency and safety of training and help users maintain their motivation.

[1393] The processing flow will be explained below.

[1394] Step 1:

[1395] User: The user starts a workout, for example, performing a squat.

[1396] Step 2:

[1397] Device (camera): A camera is installed in the training area and captures the user's movements and facial expressions in real time. Video data is continuously captured.

[1398] Step 3:

[1399] Terminal (camera): The camera sends video data to the server in real time. The video data is transferred via the network.

[1400] Step 4:

[1401] Server: The server receives and stores video data sent from the camera in real time. The video data is stored in a buffer.

[1402] Step 5:

[1403] Server: Pre-processes the received video data, including frame extraction, noise reduction, and resolution adjustment.

[1404] Step 6:

[1405] Server: Inputs preprocessed video data into the generative AI model.

[1406] Step 7:

[1407] Generative AI model (server): Analyzes video data and evaluates the user's movements. It compares the user's squat movements with the ideal form.

[1408] Step 8:

[1409] Generative AI model (server): Generates specific feedback such as "Your knee angle is too deep, please adjust it to 90 degrees." Feedback is generated in the form of text, images, and videos.

[1410] Step 9:

[1411] Server: Sends the generated feedback to the tablet. The feedback data is sent to the tablet in real time.

[1412] Step 10:

[1413] Terminal (tablet): The tablet receives the feedback data and displays it to the user, for example, an image showing the knee angle and text indicating correction points.

[1414] Step 11:

[1415] User: The user checks the feedback displayed on the tablet and corrects their squat form.

[1416] Step 12:

[1417] Server: Passes part of the video data to the emotion engine.

[1418] Step 13:

[1419] Emotion engine (server): Analyzes video and audio data, detects changes in the user's facial expressions and voice, and infers their emotional state. For example, it infers whether the user is tired or frustrated.

[1420] Step 14:

[1421] Emotion engine (server): Adjusts the content and tone of the feedback based on the results of the emotion engine. For example, if the user is estimated to be tired, it generates encouraging feedback such as "Take a short break."

[1422] Step 15:

[1423] Server: Sends feedback reflecting the results of the emotion engine back to the tablet.

[1424] Step 16:

[1425] Device (tablet): The tablet displays feedback based on the user's emotional state. If the user's facial expression indicates fatigue, it displays words of encouragement or a message recommending rest.

[1426] Step 17:

[1427] User: The user checks the newly displayed emotional feedback and takes appropriate action, such as taking a short break or drinking more water.

[1428] Through these steps, users can receive real-time feedback on their training movements and emotional state, which can improve the efficiency and safety of their training and help maintain their motivation.

[1429] Example 2

[1430] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1431] Conventional training systems have the problem that it is difficult for users to receive appropriate feedback when an instructor is not present, preventing them from correcting their form or maximizing the effectiveness of their training. Additionally, feedback does not take into account the user's emotional state, making it difficult to maintain motivation and continue training.

[1432] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1433] In this invention, the server includes means for capturing images of the user's training using a camera installed in the training area, means for receiving and saving the video data captured by the camera in real time, means for preprocessing the received video data and inputting it into an artificial intelligence model to analyze the user's movements, means for generating feedback in the form of text, images, or video based on the analyzed data, means for transmitting the generated feedback data to a display terminal and displaying it to the user, means for analyzing changes in the user's facial expressions and voice to estimate their emotional state, and means for adjusting the content and tone of the feedback based on their emotional state. This allows the user to receive feedback on their training movements and emotional state in real time, improving the efficiency and safety of their training and helping them maintain their motivation.

[1434] The "photography device" is a device that is installed in the training area and is used to capture the user's training scene in real time.

[1435] A "server" is a device that receives video data sent from an imaging device and performs a series of processes such as storage, pre-processing, analysis, and feedback generation.

[1436] The "artificial intelligence model" is a program that analyzes video data input into the server and evaluates the user's actions.

[1437] "Preprocessing" refers to the process of removing noise from the received video data, extracting frames, adjusting the resolution, and so on.

[1438] "Feedback" refers to formal advice or instructions to users that are generated based on the analysis results of an artificial intelligence model and are provided in the form of text, images, videos, etc.

[1439] A "display terminal" is a device that receives feedback data sent from the server and visually presents it to the user.

[1440] The "emotion engine" is a program that analyzes changes in the user's facial expressions and voice to estimate their emotional state.

[1441] "Real-time" refers to the immediacy of data processing and feedback during a user's training.

[1442] "Tone" refers to the wording and delivery of feedback or advice, which is adjusted according to the user's emotional state.

[1443] The system of this invention is designed to allow users to receive appropriate training feedback even when an instructor is not present. It also has the ability to recognize the user's emotional state and dynamically adjust the feedback and training menu. The main components of the system are as follows:

[1444] 1. Imaging equipment

[1445] It is installed in the training area and records the user's training in real time, and the captured video data is sent sequentially to a server via the network.

[1446] 2. Server

[1447] The server receives and stores video data sent from the imaging device in real time. The first step is pre-processing of the received video data, which includes noise removal, frame extraction, and resolution adjustment. The pre-processed data is then formatted so that it can be effectively analyzed by the generative AI model.

[1448] The pre-processed video data is then fed into a generative AI model, which analyzes the data and evaluates the user's movements, for example comparing them with ideal training form to identify necessary corrections or improvements.

[1449] Based on the analysis results, the server generates specific feedback in the form of text, images, or videos, which are then sent to a display device and presented to the user.

[1450] The server is also equipped with an emotion engine that analyzes the user's facial expressions and voice to estimate their emotional state, and adjusts the content and tone of the feedback based on the results of this analysis.

[1451] 3. Display terminal

[1452] The system receives feedback data sent from the server and displays it to the user. The feedback is designed to allow users to understand and act immediately, and specific images and videos are also presented.

[1453] Specific examples

[1454] Squat training

[1455] 1. Users

[1456] The user stands in the training area and begins training by pressing the "Start Training" button on the tablet screen.

[1457] 2. Terminal (photography device)

[1458] The camera captures the user's squatting movements and facial expressions in real time and sends the video data to a server.

[1459] 3. Server

[1460] The server preprocesses the received video data of the squat and inputs it into a generative AI model, which then analyzes the video data and evaluates the user's movements.

[1461] The generative AI model compares the user's squat movement with ideal training form and generates specific feedback such as, "Your knee angle is too deep; adjust it to 90 degrees."

[1462] This feedback is generated in the form of text, images, or videos and sent to a display device.

[1463] The emotion engine analyzes the user's facial expressions and voice to estimate their emotional state. For example, if the user is feeling tired or frustrated, it generates feedback content and tone appropriate to that emotional state and sends it to the display device.

[1464] 4. Terminal (display terminal)

[1465] The display device receives the feedback and displays it in an easy-to-understand way to the user. For example, it can show images or videos that show that the knee angle is not appropriate, and provide personalized advice such as "We recommend that you take a short break or drink some water."

[1466] Prompt Sentence Examples

[1467] "Your knees are at too deep an angle when squatting; adjust them to 90 degrees."

[1468] "I suggest you take a short break or drink some water."

[1469] In this way, users can receive real-time feedback on their training movements and emotional state, which can improve training efficiency and safety and help users maintain their motivation.

[1470] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1471] Step 1:

[1472] User starts training

[1473] Specific behavior:

[1474] The user stands in the training area and presses the "Start Training" button on the tablet screen, which starts the system.

[1475] Input: User presses the "Start Training" button on the tablet.

[1476] Output: The system goes into training start mode.

[1477] Step 2:

[1478] The device (camera) captures the training video.

[1479] Specific behavior:

[1480] The camera is installed in a position that captures the entire training area. When the user starts to move, the camera captures their movements in real time and transmits the video data sequentially to a server via a network.

[1481] Input: User action

[1482] Output: Real-time video data

[1483] Step 3:

[1484] The server receives the video data

[1485] Specific behavior:

[1486] The server receives the image data sent from the imaging device and stores it in real time. At this point, the data is raw data that has not yet been pre-processed.

[1487] Input: Raw image data transmitted from the imaging device

[1488] Output: Saved video data

[1489] Step 4:

[1490] The server preprocesses the video data

[1491] Specific behavior:

[1492] The server preprocesses the received video data, which includes removing noise, cropping frames, and adjusting resolution, so that the preprocessed data is in a format that can be effectively analyzed by the generative AI model.

[1493] Input: Stored raw video data

[1494] Output: Pre-processed video data

[1495] Step 5:

[1496] The server inputs data into the generative AI model

[1497] Specific behavior:

[1498] The pre-processed video data is then input into the generative AI model, a process that prepares the data to fit the model's input format.

[1499] Input: Preprocessed video data

[1500] Output: Data input to the generative AI model

[1501] Step 6:

[1502] Generative AI models analyze video data

[1503] Specific behavior:

[1504] A generative AI model analyzes the input data and evaluates the user's movements. The model compares the user's movements with ideal training form and identifies necessary corrections and improvements.

[1505] Input: Data input to the generative AI model

[1506] Output: Parsed feedback data

[1507] Step 7:

[1508] Generative AI models generate feedback

[1509] Specific behavior:

[1510] The generative AI model generates specific feedback based on the analysis results, such as "Your knee angle is too deep, please adjust it to 90 degrees."

[1511] Input: Parsed feedback data

[1512] Output: Generated feedback

[1513] Step 8:

[1514] The server sends feedback to the display device

[1515] Specific behavior:

[1516] The generative AI model generates feedback that is sent to the display device in the form of text, images, or videos.

[1517] Input: Generated feedback

[1518] Output: Feedback sent to the display device

[1519] Step 9:

[1520] Emotion engine analyzes emotional state

[1521] Specific behavior:

[1522] The emotion engine analyzes the user's facial expressions and voice to estimate their emotional state. For example, if frustration is detected, the engine returns that data to the server.

[1523] Input: User's facial expressions and voice

[1524] Output: Emotional state data

[1525] Step 10:

[1526] The server sends feedback based on the emotional state to the display device.

[1527] Specific behavior:

[1528] The feedback content is adjusted based on the analysis results of the emotion engine, changing the tone and message content, and then the adjusted feedback is sent to the display device.

[1529] Input: Emotional state data

[1530] Output: Feedback based on emotional state

[1531] Step 11:

[1532] The display device displays the feedback to the user.

[1533] Specific behavior:

[1534] The display device then displays the received feedback to the user, for example, showing an image or video that indicates the knee angle is not appropriate and providing personalized advice such as "We recommend you take a short break or drink some water."

[1535] Input: Feedback based on emotional state

[1536] Output: Feedback displayed to the user

[1537] Step 12:

[1538] Users modify their training according to the feedback

[1539] Specific behavior:

[1540] The user checks the feedback displayed on the display terminal and follows the instructions to correct their training form and movements. They also refer to the emotional feedback and take appropriate actions such as taking breaks at appropriate times.

[1541] Input: Feedback displayed to the user

[1542] Output: Corrected training form and movements

[1543] (Application example 2)

[1544] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1545] Conventional fitness training systems struggle to provide users with proper form and movement feedback when an instructor is not present. Furthermore, providing uniform feedback without considering the user's emotional state can reduce the user's motivation and training efficiency. This raises concerns about the risk of injury and reduced training effectiveness.

[1546] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing images of the user's training using a camera installed in the training area, means for receiving and saving the video data captured by the camera in real time, means for preprocessing the received video data and inputting it into a generative AI model to analyze the user's movements, means for generating feedback in the form of text, images, or videos based on the analyzed data, means for sending the generated feedback data to a tablet and displaying it to the user, and means for analyzing the user's emotional state and dynamically adjusting the feedback and training menu. This allows users to receive appropriate feedback in real time even when an instructor is not present, and also enables them to obtain feedback optimized according to their individual emotional state.

[1547] The "camera" is an image capturing device for capturing images of the user's training.

[1548] "Video data" is digital data containing visual information captured by a camera.

[1549] "Means for receiving and storing in real time" refers to a function for receiving captured video data to a server without delay and storing the data as needed.

[1550] "Preprocessing" refers to the process of shaping and processing video data to make it suitable for analysis by a generative AI model.

[1551] A "generative AI model" is an artificial intelligence model that analyzes and evaluates a user's training movements.

[1552] "Input" refers to feeding pre-processed video data into a generative AI model.

[1553] The "means of analysis" refers to a method of analyzing and evaluating a user's training movements using a generative AI model.

[1554] "Feedback" refers to instructions or advice provided to the user based on the analysis results of the generative AI model.

[1555] "Text, image, video format" refers to a format in which the feedback content is displayed using different means of expression.

[1556] A "tablet" is a portable display device for displaying generated feedback to a user.

[1557] "Emotional state" refers to the psychological state estimated based on information such as the user's facial expression and voice.

[1558] "Dynamic adjustment means" refers to a function that flexibly changes the feedback and training menu content according to the user's emotional state.

[1559] This invention is a system that allows users to receive appropriate training feedback when an instructor is not present, and also has the ability to recognize the user's emotional state and dynamically adjust the feedback and training menu. This system is realized using a camera, a server, a generative AI model, a tablet, and an emotion engine.

[1560] System Configuration

[1561] 1. Camera

[1562] The camera is installed in the training area and captures the user's training in real time.

[1563] As a specific example, a high-resolution camera (e.g., Logitech StreamCam) is used.

[1564] 2. Server

[1565] The server receives and stores the video data sent from the camera in real time.

[1566] After preprocessing, the received video data is input into the generative AI model.

[1567] As a specific example, a high-performance cloud server (e.g., Amazon Web Services EC2) is used.

[1568] 3. Generative AI Models

[1569] The generative AI model analyzes the video data input by the server and evaluates the user's actions.

[1570] The model compares the user's movements with ideal training form and identifies any corrections or improvements that need to be made.

[1571] A specific example is using a custom AI model (e.g., TensorFlow or PyTorch) for behavior analysis.

[1572] 4. Generate feedback

[1573] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[1574] The feedback is specific and includes information that users can immediately understand and act on.

[1575] Example: The knee angle is too deep, please adjust it to 90 degrees.

[1576] 5. Emotion Engine

[1577] The emotion engine uses video and audio data captured by the camera to analyze changes in the user's facial expressions and voice and estimate their emotional state.

[1578] Based on the analysis results, the content and tone of the feedback can be adjusted to improve the user's training experience.

[1579] As a specific example, we use an emotion analysis engine (e.g., Microsoft Azure Emotion API).

[1580] 6. Tablets

[1581] The tablet receives the feedback data sent from the server and the analysis results of the emotion engine and displays them to the user.

[1582] Based on the feedback provided, users can improve their form and behavior.

[1583] Processing flow

[1584] 1. User Operation

[1585] A user taps the start button on a tablet or smartphone in the fitness training area.

[1586] 2. Camera Operation

[1587] The camera captures the user's training movements and sends the video data to a server.

[1588] 3. Server Processing

[1589] The server receives the video data, pre-processes it, and then feeds the data into the generative AI model.

[1590] 4. Analysis of generative AI models

[1591] A generative AI model analyzes the data and compares the user's movements with ideal form.

[1592] For example: "Your knee angle is too deep, please adjust it to 90 degrees."

[1593] 5. Generate and send feedback

[1594] The server generates feedback in the form of text, images and videos and sends it to the tablet.

[1595] 6. Emotion Engine Analysis

[1596] The emotion engine analyzes the user's emotional state and dynamically adjusts the feedback content.

[1597] 7. Displaying feedback to users

[1598] The tablet receives the feedback and displays it in an easy-to-understand way.

[1599] For example: "Bend your knees more."

[1600] Prompt Sentence Examples

[1601] Analyze footage of a user squatting, assess knee angle, hip position, and back curvature, and generate feedback to correct for ideal posture.

[1602] This allows users to receive both motion- and emotion-based feedback in real time, improving training efficiency and safety.

[1603] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1604] Step 1:

[1605] A user taps the start button on a tablet or smartphone in the fitness training area. This causes the camera to start capturing video data. The input is the user's operation information, and the output is a signal to the camera to start recording.

[1606] Step 2:

[1607] The camera captures the user's training movements in real time and captures the video data, which is then sent to a server via a network. The input is the user's movement data, and the output is the video data sent to the server.

[1608] Step 3:

[1609] The server receives and stores video data from the camera in real time. It then preprocesses the received video data and prepares it for input into the generative AI model. The input is video data from the camera, and the output is preprocessed data. Specifically, noise removal and frame correction are performed.

[1610] Step 4:

[1611] The server inputs the preprocessed video data into a generative AI model. The generative AI model analyzes and evaluates the user's training movements. The input is the preprocessed data, and the output is the evaluation result of the user's movements. Specifically, the movement data is compared with the ideal form to detect errors.

[1612] Step 5:

[1613] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model. The input is the evaluation results of the user's movements, and the output is feedback data. Specifically, it describes in detail the corrections needed for knee angle and hip position.

[1614] Step 6:

[1615] The emotion engine uses video and audio data captured by the camera to analyze changes in the user's facial expressions and voice and estimate their emotional state. The input is video and audio data, and the output is the analysis result of the emotional state. The emotion engine uses, for example, the Microsoft Azure Emotion API.

[1616] Step 7:

[1617] The server adjusts the content and tone of the feedback based on the analysis results of the emotion engine. The input is the analysis result of the emotional state, and the output is dynamically adjusted feedback. If the user is fatigued, the content will be changed to be more encouraging.

[1618] Step 8:

[1619] The server sends the final feedback data to the tablet. The input is the dynamically adjusted feedback data, and the output is the data transmission to the tablet.

[1620] Step 9:

[1621] The tablet receives the feedback data sent from the server and the analysis results of the emotion engine, and displays them in an easy-to-understand manner for the user. The input is the feedback data, and the output is the display on the tablet.

[1622] Step 10:

[1623] The user checks the feedback displayed on the tablet and corrects their training movements. They also refer to emotional feedback to take appropriate actions such as taking breaks at appropriate times. The input is feedback data, and the output is corrected movements and actions such as resting.

[1624] This series of steps allows users to receive both behavioral and emotional feedback in real time, improving training efficiency and safety.

[1625] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1626] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1627] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1628] [Fourth embodiment]

[1629] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1630] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1631] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1632] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1633] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1634] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1635] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1636] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1637] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1638] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1639] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1640] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1641] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1642] The system of this invention allows users to receive appropriate training feedback in an instructor-less sports gym. Specifically, it is realized using a camera, a server, a generative AI model, and a tablet.

[1643] System Configuration

[1644] 1. Camera

[1645] The camera is installed in the training area and captures the user's training in real time.

[1646] The camera continuously captures video data and transmits it to a server over a network.

[1647] 2. Server

[1648] The server receives and stores the video data sent from the camera in real time.

[1649] The server preprocesses the received video data and prepares it for input into the generative AI model.

[1650] 3. Generative AI Models

[1651] The generative AI model analyzes the video data input by the server and evaluates the user's actions.

[1652] The model compares the user's movements with ideal training form and identifies any corrections or improvements that need to be made.

[1653] 4. Generate feedback

[1654] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[1655] The feedback is specific and includes information that users can immediately understand and act on.

[1656] 5. Tablets

[1657] The tablet receives the feedback data sent from the server and displays it to the user.

[1658] Based on the feedback provided, users can modify their forms and actions.

[1659] Specific examples

[1660] Squat training

[1661] 1. Users

[1662] The user performs a squat.

[1663] The camera captures the squat movement in real time.

[1664] 2. Device (camera)

[1665] The camera transmits the video data to the server.

[1666] 3. Server

[1667] The server preprocesses the received squat video data and inputs it into the generative AI model.

[1668] A generative AI model analyzes the video data and evaluates the user's actions.

[1669] 4. Generative AI model (server)

[1670] The generative AI model compares the user's squat motion with ideal form.

[1671] Generates feedback such as "Your knee angle is too deep, adjust it to 90 degrees."

[1672] 5. Server

[1673] Send the generated feedback to the tablet.

[1674] 6. Device (tablet)

[1675] The tablet receives the feedback and displays it in an easy-to-understand way for the user.

[1676] For example, images or videos can be displayed to visually confirm that the knee angle is not appropriate.

[1677] 7. Users

[1678] The user checks the feedback displayed on the tablet and corrects their squat form.

[1679] In this way, users can train efficiently and safely with real-time feedback, maximizing the effectiveness of their training and reducing the risk of injury.

[1680] The processing flow will be explained below.

[1681] Step 1:

[1682] Terminal (camera): When the user starts training, the camera starts operating. The camera continuously captures live images of the training area and transmits the images to the server in real time.

[1683] Step 2:

[1684] Server: The server receives video data sent from the camera in real time, stores the received video data in a buffer, and prepares it for pre-processing.

[1685] Step 3:

[1686] Server: Preprocesses the video data. Preprocessing includes frame extraction, noise reduction, and resolution adjustment. This converts the data into a format that is easy for the generative AI model to analyze.

[1687] Step 4:

[1688] Server: Inputs the preprocessed video data into the generative AI model, which analyzes the input data and evaluates the user's training movements.

[1689] Step 5:

[1690] Generative AI model (server): Detects the user's bone structure and joint angles from video data and compares them with pre-registered ideal form. For example, in the case of squats, the angle of the knees and the degree of curvature of the back are evaluated.

[1691] Step 6:

[1692] Generative AI model (server): Generates feedback based on the results of behavior analysis. The feedback is created in the form of text, images, and videos, and provides specific advice to the user.

[1693] Step 7:

[1694] Server: Sends the generated feedback to the tablet. The feedback data is sent to the tablet in real time, allowing the user to view it immediately.

[1695] Step 8:

[1696] Terminal (tablet): The tablet receives the feedback data sent from the server and displays it to the user. The feedback is easy for the user to understand and indicates corrections and improvements.

[1697] Step 9:

[1698] User: The user checks the feedback displayed on the tablet and follows the instructions to correct their training form, for example, by performing specific actions such as "change the angle of your knees" or "keep your back straight."

[1699] Through these steps, users can receive appropriate feedback in real time, improving the efficiency and safety of their training.

[1700] Example 1

[1701] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1702] In modern sports gyms, users often find it difficult to maintain proper form without the guidance of an instructor. This increases the risk of injury due to incorrect training form or improper movements. This creates a need for a system that provides users with real-time feedback to ensure safe and effective training.

[1703] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1704] In this invention, the server includes means for capturing images of the user's training movements with a camera installed in the training area, means for receiving and saving the video data captured by the camera in real time, means for preprocessing the received video data and inputting it into an analysis model to analyze the user's movements, means for generating feedback in the form of text, images, or video based on the analyzed data, and means for transmitting the generated feedback data to a display terminal and displaying it to the user. This allows the user to receive appropriate feedback in real time, improving the quality of their training and further reducing the risk of injury.

[1705] The "photography device" is a device for capturing images of the user's training movements in real time and acquiring the video data.

[1706] "Video data" refers to visual information of training movements captured by a camera.

[1707] "Preprocessing" refers to a data processing procedure for converting video data into a format that can be analyzed by the generative AI model.

[1708] An "analysis model" is a computational model that analyzes received video data and evaluates the user's training movements, and this mainly applies to generative AI models.

[1709] "Feedback" is information, including corrections and improvements, provided to users based on analyzed data, and is provided in the form of text, images, or videos.

[1710] A "display terminal" is a device on which a user can view the feedback they have received, and is typically a display device such as a tablet.

[1711] "Real-time" refers to the fact that feedback is generated and displayed within a very short time (almost instantaneously) after the user performs a training action.

[1712] This invention provides a system that allows users to receive real-time feedback on their training movements without the need for an instructor. Specifically, this system is realized using a camera, a server, an analysis model, and a display terminal.

[1713] System Configuration

[1714] 1. Imaging equipment

[1715] The imaging device is installed in the training area and captures images of the user's training movements in real time.

[1716] The imaging device is composed of a high-resolution camera, which continuously captures video data of the training movements and transmits it to a server via a network.

[1717] 2. Server

[1718] The server is a high-performance computing device that receives and stores the video data transmitted from the imaging device in real time.

[1719] The server preprocesses the received video data to prepare it for input to the generative AI model, including noise reduction, frame resizing, and data format conversion.

[1720] 3. Analysis Model

[1721] Using the generative AI model, the server analyzes the pre-processed video data.

[1722] The generative AI model compares the user's movements with ideal training form and identifies areas for correction and improvement.

[1723] 4. Generating and displaying feedback

[1724] The server generates specific feedback based on the analysis results of the generative AI model, which can be in the form of text, images, or videos.

[1725] The feedback data is sent to a display terminal (e.g., a tablet) and displayed to the user.

[1726] Specific examples

[1727] Squat training

[1728] 1. User Actions

[1729] A user performs squats in the training area, making sure to perform the movements within the camera's field of view.

[1730] 2. Acquiring video data

[1731] The camera captures the user's squatting motion in real time, and the captured video data is sent to a server.

[1732] 3. Data Analysis

[1733] The server receives the video data, preprocesses it, and then inputs it into the generative AI model, which analyzes the video data and compares the user's movements with the ideal form.

[1734] 4. Generate feedback

[1735] The generative AI model generates feedback such as, "The knee angle is too deep, please adjust it to 90 degrees." The server sends this feedback to the tablet.

[1736] 5. Viewing Feedback

[1737] The tablet receives the feedback and displays it to the user in an easy-to-understand way, such as showing an image or video that indicates the knee angle is not correct.

[1738] 6. User Modifications

[1739] The user checks the feedback displayed on the tablet and corrects their squat form.

[1740] Prompt Sentence Examples

[1741] "Please evaluate whether this training form is correct and advise me on how to correct it."

[1742] In this way, the present invention aims to allow users to receive appropriate feedback in real time, improving the quality of their training and reducing the risk of injury.

[1743] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1744] Step 1:

[1745] Start user training behavior

[1746] Input: A user enters a training area and begins a particular training movement (e.g., a squat).

[1747] Specific Action: User performs squats with proper form in the designated training area, taking care to perform the movements within the camera's field of view.

[1748] Step 2:

[1749] Acquisition of video data using imaging equipment

[1750] Input: User training behavior

[1751] Output: Real-time video data of training movements

[1752] Specific operation: The camera records the user's training movements in real time, continuously capturing high-resolution video data and storing it in a temporary buffer.

[1753] Step 3:

[1754] Video data transmission

[1755] Input: Real-time video data of training movements

[1756] Output: Video data sent to the server

[1757] Specific operation: The video data captured by the camera is sent to the server via the network. Data transfer is performed with low latency.

[1758] Step 4:

[1759] Receiving and storing video data on the server

[1760] Input: Image data sent from the imaging device

[1761] Output: Saved video data

[1762] Specific operation: The server receives the received video data in real time and stores it in a database, ensuring the integrity of the data required for subsequent processing.

[1763] Step 5:

[1764] Video data preprocessing

[1765] Input: Saved video data

[1766] Output: Pre-processed video data

[1767] What it does: The server preprocesses the video data, specifically removing noise, resizing frames, and converting data formats, so that it is properly prepared for input to the generative AI model.

[1768] Step 6:

[1769] Motion analysis using generative AI models

[1770] Input: Preprocessed video data

[1771] Output: Analyzed data (user behavior evaluation)

[1772] Specific operation: The server inputs preprocessed video data into the generative AI model, which then identifies the user's skeletal structure and the positions of major joints from the video data and analyzes their movements.

[1773] Step 7:

[1774] Generate feedback from analysis results

[1775] Input: Data analyzed by the generative AI model

[1776] Output: Feedback in text, image and video formats

[1777] Specific operation: The server generates feedback based on the analysis results of the generative AI model. The feedback includes specific corrections and improvements, and is provided in a format that is easy for the user to understand.

[1778] Step 8:

[1779] Sending feedback data to a tablet

[1780] Input: Generated feedback

[1781] Output: Feedback data sent to the tablet

[1782] What it does: The server sends the generated feedback to the tablet. The communication is real-time, ensuring that the feedback reaches the user immediately.

[1783] Step 9:

[1784] Viewing feedback on a tablet

[1785] Input: Feedback data sent from the server

[1786] Output: Feedback that is displayed to the user

[1787] Specific behavior: The tablet receives feedback and displays it to the user. The feedback displayed can include text, images, and videos, and includes specific corrections.

[1788] Step 10:

[1789] Modifying user behavior

[1790] Input: Feedback displayed on tablet

[1791] Output: Corrected training behavior

[1792] Specific actions: The user checks the feedback displayed on the tablet and corrects their training form, for example, by adjusting the knee angle properly when squatting.

[1793] (Application example 1)

[1794] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1795] Currently, in many factories, worker movements are dependent on visual evaluation and manual checks, resulting in issues with efficiency and safety. In particular, when new or inexperienced workers work alone, their movements may lack accuracy and appropriateness, posing risks to the quality and safety of the work. Furthermore, the lack of real-time feedback increases the likelihood of incorrect movements and postures being continued, resulting in reduced production efficiency. The purpose of this invention is to solve these issues.

[1796] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1797] In this invention, the server includes: means for capturing images of the user's training with a camera installed in the training area; means for receiving and saving the image data captured by the camera in real time; means for preprocessing the received image data and inputting it into a generative AI model to analyze the user's movements; means for generating feedback in the form of text, images, or videos based on the analyzed data; means for sending the generated feedback data to a tablet and displaying it to the user; and means installed in the work area of ​​the factory for analyzing the worker's movements in real time and sending feedback. This allows workers to receive feedback in real time, improving the accuracy and safety of their movements and ultimately increasing the efficiency of the entire factory.

[1798] "Training Area" means an area designated for a user to exercise or work.

[1799] A "camera" is a device that captures video data and transmits it to a server in real time.

[1800] "Video data" refers to digital data that includes the user's movements and training footage captured by a camera.

[1801] "Real-time" means that data is processed immediately with little delay.

[1802] "Storage" refers to the act of recording data on a storage device such as a server.

[1803] "Preprocessing" refers to the preparation work carried out before data analysis, and includes data normalization and resizing.

[1804] A "generative AI model" refers to an artificial intelligence model trained for a specified task, performing behavior analysis and feedback generation.

[1805] "Analysis" is the process of extracting information contained in data and finding meaning and patterns.

[1806] "Feedback" refers to advice or suggestions provided to users based on the analysis results.

[1807] "Text" means written information.

[1808] An "image" is a form of representing information as a visual (still image).

[1809] "Video" is a data format that contains moving visual information.

[1810] A "tablet" is a portable display device used by a user to view feedback.

[1811] "Factory work area" refers to the area within a factory used by workers to perform their designated tasks.

[1812] This invention is a system for analyzing the actions of users and workers in training areas and factory work areas in real time and providing feedback. Specifically, it uses cameras, servers, generative AI models, and tablets to provide feedback to users so that they can perform appropriate training and actions. This can improve work efficiency and safety.

[1813] System Configuration

[1814] 1. Camera

[1815] The cameras are installed in training areas or factory work areas and capture the actions of users and workers in real time.

[1816] The captured video data is transmitted to a server via a network.

[1817] 2. Server

[1818] The server receives and stores the video data sent from the camera.

[1819] The server preprocesses the received video data and prepares it for input into the generative AI model.

[1820] Specific preprocessing includes resizing and normalizing the video data.

[1821] 3. Generative AI Models

[1822] The generative AI model analyzes the video data input by the server and evaluates the actions of the user or worker.

[1823] The model compares the user's movements with ideal training form and work procedures to identify needed corrections and improvements.

[1824] 4. Generate feedback

[1825] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[1826] The feedback is specific, easy to understand, and includes content that users can immediately act on.

[1827] 5. Tablets

[1828] The tablet receives the feedback data sent from the server and displays it to the user or worker.

[1829] Based on the displayed feedback, users or workers can correct their form and actions.

[1830] Hardware and software used

[1831] Hardware: high-resolution cameras, servers, tablets

[1832] software:

[1833] OpenCV: Used for real-time video capture and display

[1834] TensorFlow: used for loading and inferencing generative AI models

[1835] requests: Used to communicate with the server

[1836] Specific examples

[1837] For example, consider a factory worker using a robotic arm to install parts. A camera captures the image to determine whether the movement is appropriate, and a generative AI model analyzes it. If the movement is incorrect, feedback such as "The robotic arm is out of position. Please move it 5 cm to the right" is displayed on a tablet.

[1838] Prompt Sentence Examples

[1839] "Generative AI models, robot behavior evaluation, real-time feedback, worker safety, and improved factory efficiency"

[1840] This allows the invention to provide real-time feedback to assist the worker or user in taking the appropriate action.

[1841] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1842] Step 1:

[1843] The camera captures the user or worker's movements in real time in the training area or factory work area.

[1844] Input: User or worker actions

[1845] Output: Real-time video data

[1846] Specific operation: The camera captures the movements of the user or worker and continuously captures frames.

[1847] Step 2:

[1848] The server receives and stores the video data captured by the camera in real time.

[1849] Input: Real-time video data

[1850] Output: Saved video data

[1851] Specific operation: The server receives the video data sent from the camera via the network and saves it in a file format.

[1852] Step 3:

[1853] The server preprocesses the received video data and prepares it for input into the generative AI model.

[1854] Input: Saved video data

[1855] Output: Preprocessed data (resized and normalized)

[1856] What it does: The server resizes the video data, normalizes pixel values, and converts the data format to one that is more easily accepted by the generative AI model.

[1857] Step 4:

[1858] The server inputs the preprocessed data into a generative AI model to analyze the actions of users and workers.

[1859] Input: Preprocessed data

[1860] Output: Analysis results (operation evaluation)

[1861] Specific actions: The generative AI model analyzes the video data and evaluates the actions of the user or worker. The analysis results identify whether the actions are correct and which areas need improvement.

[1862] Step 5:

[1863] The server generates feedback in the form of text, images, and videos based on the analyzed data.

[1864] Input: Analysis results

[1865] Output: Feedback data (text, images, videos)

[1866] Specific actions: The server generates feedback based on the analysis results, such as text instructions like "Move the robot arm 5 cm to the right," or images or videos showing the incorrect action.

[1867] Step 6:

[1868] The server sends the generated feedback data to a tablet and displays it to the user or worker.

[1869] Input: Feedback data

[1870] Output: Feedback displayed on the tablet

[1871] Specific operation: Feedback data is sent to a tablet and displayed in a form that can be visually understood by the user or worker. The user or worker then corrects their operation based on the feedback.

[1872] Step 7:

[1873] Users and workers can check the feedback displayed on the tablet and correct their actions.

[1874] Input: Feedback displayed on tablet

[1875] Output: Corrected behavior

[1876] Specific actions: The user or worker checks the feedback and corrects their actions or form, for example, adjusting the position of the robot arm as instructed.

[1877] The above are the specific processing steps for carrying out the present invention.

[1878] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1879] The system of this invention allows users to receive appropriate training feedback when an instructor is not present, and also has the ability to recognize the user's emotional state and dynamically adjust the feedback and training menu. Specifically, it is realized using a camera, a server, a generative AI model, a tablet, and an emotion engine.

[1880] System Configuration

[1881] 1. Camera

[1882] The camera is installed in the training area and captures the user's training in real time.

[1883] The camera continuously captures video data and transmits it to a server over a network.

[1884] 2. Server

[1885] The server receives and stores the video data sent from the camera in real time.

[1886] The server preprocesses the received video data and prepares it for input into the generative AI model.

[1887] 3. Generative AI Models

[1888] The generative AI model analyzes the video data input by the server and evaluates the user's actions.

[1889] The model compares the user's movements with ideal training form and identifies any corrections or improvements that need to be made.

[1890] 4. Generate feedback

[1891] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[1892] The feedback is specific and includes information that users can immediately understand and act on.

[1893] 5. Emotion Engine

[1894] The emotion engine uses video and audio data captured by the camera to analyze changes in the user's facial expressions and voice and estimate their emotional state.

[1895] Based on the analysis results of the emotion engine, the content and tone of the feedback is adjusted to improve the user's training experience.

[1896] 6. Tablets

[1897] The tablet receives the feedback data sent from the server and the analysis results of the emotion engine and displays them to the user.

[1898] Based on the feedback provided, users can improve their form and behavior.

[1899] Specific examples

[1900] Squat training

[1901] 1. Users

[1902] The user performs a squat.

[1903] The camera captures the squat movements and facial expressions in real time.

[1904] 2. Device (camera)

[1905] The camera transmits the video data to the server.

[1906] 3. Server

[1907] The server preprocesses the received squat video data and inputs it into the generative AI model.

[1908] A generative AI model analyzes the video data and evaluates the user's actions.

[1909] 4. Generative AI model (server)

[1910] The generative AI model compares the user's squat motion with ideal form.

[1911] Generates feedback such as "Your knee angle is too deep, adjust it to 90 degrees."

[1912] 5. Server

[1913] Send the generated feedback to the tablet.

[1914] 6. Device (tablet)

[1915] The tablet receives the feedback and displays it in an easy-to-understand way for the user.

[1916] For example, images or videos can be displayed to visually confirm that the knee angle is not appropriate.

[1917] 7. Emotion engine (server)

[1918] The emotion engine analyzes the user's facial expressions and voice to estimate their emotional state.

[1919] If the user is feeling tired or frustrated, for example, the system generates feedback that is more encouraging or offers simple points for improvement depending on the user's emotional state.

[1920] 8. Server

[1921] Based on the results of the emotion engine, the content and tone of the feedback are appropriately adjusted and sent to the tablet.

[1922] 9. Device (tablet)

[1923] The tablet displays feedback based on your emotional state.

[1924] For example, it provides personalized advice such as, "We recommend you take a short break or drink some water."

[1925] 10. Users

[1926] The user checks the feedback displayed on the tablet and follows the instructions to modify their training form and menu.

[1927] Emotional feedback is also taken into consideration to take appropriate actions such as taking breaks at appropriate times.

[1928] In this way, users can receive real-time feedback on not only their training movements but also their emotional state, which can improve the efficiency and safety of training and help users maintain their motivation.

[1929] The processing flow will be explained below.

[1930] Step 1:

[1931] User: The user starts a workout, for example, performing a squat.

[1932] Step 2:

[1933] Device (camera): A camera is installed in the training area and captures the user's movements and facial expressions in real time. Video data is continuously captured.

[1934] Step 3:

[1935] Terminal (camera): The camera sends video data to the server in real time. The video data is transferred via the network.

[1936] Step 4:

[1937] Server: The server receives and stores video data sent from the camera in real time. The video data is stored in a buffer.

[1938] Step 5:

[1939] Server: Pre-processes the received video data, including frame extraction, noise reduction, and resolution adjustment.

[1940] Step 6:

[1941] Server: Inputs preprocessed video data into the generative AI model.

[1942] Step 7:

[1943] Generative AI model (server): Analyzes video data and evaluates the user's movements. It compares the user's squat movements with the ideal form.

[1944] Step 8:

[1945] Generative AI model (server): Generates specific feedback such as "Your knee angle is too deep, please adjust it to 90 degrees." Feedback is generated in the form of text, images, and videos.

[1946] Step 9:

[1947] Server: Sends the generated feedback to the tablet. The feedback data is sent to the tablet in real time.

[1948] Step 10:

[1949] Terminal (tablet): The tablet receives the feedback data and displays it to the user, for example, an image showing the knee angle and text indicating correction points.

[1950] Step 11:

[1951] User: The user checks the feedback displayed on the tablet and corrects their squat form.

[1952] Step 12:

[1953] Server: Passes part of the video data to the emotion engine.

[1954] Step 13:

[1955] Emotion engine (server): Analyzes video and audio data, detects changes in the user's facial expressions and voice, and infers their emotional state. For example, it infers whether the user is tired or frustrated.

[1956] Step 14:

[1957] Emotion engine (server): Adjusts the content and tone of the feedback based on the results of the emotion engine. For example, if the user is estimated to be tired, it generates encouraging feedback such as "Take a short break."

[1958] Step 15:

[1959] Server: Sends feedback reflecting the results of the emotion engine back to the tablet.

[1960] Step 16:

[1961] Device (tablet): The tablet displays feedback based on the user's emotional state. If the user's facial expression indicates fatigue, it displays words of encouragement or a message recommending rest.

[1962] Step 17:

[1963] User: The user checks the newly displayed emotional feedback and takes appropriate action, such as taking a short break or drinking more water.

[1964] Through these steps, users can receive real-time feedback on their training movements and emotional state, which can improve the efficiency and safety of their training and help maintain their motivation.

[1965] Example 2

[1966] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1967] Conventional training systems have the problem that it is difficult for users to receive appropriate feedback when an instructor is not present, preventing them from correcting their form or maximizing the effectiveness of their training. Additionally, feedback does not take into account the user's emotional state, making it difficult to maintain motivation and continue training.

[1968] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1969] In this invention, the server includes means for capturing images of the user's training using a camera installed in the training area, means for receiving and saving the video data captured by the camera in real time, means for preprocessing the received video data and inputting it into an artificial intelligence model to analyze the user's movements, means for generating feedback in the form of text, images, or video based on the analyzed data, means for transmitting the generated feedback data to a display terminal and displaying it to the user, means for analyzing changes in the user's facial expressions and voice to estimate their emotional state, and means for adjusting the content and tone of the feedback based on their emotional state. This allows the user to receive feedback on their training movements and emotional state in real time, improving the efficiency and safety of their training and helping them maintain their motivation.

[1970] The "photography device" is a device that is installed in the training area and is used to capture the user's training scene in real time.

[1971] A "server" is a device that receives video data sent from an imaging device and performs a series of processes such as storage, pre-processing, analysis, and feedback generation.

[1972] The "artificial intelligence model" is a program that analyzes video data input into the server and evaluates the user's actions.

[1973] "Preprocessing" refers to the process of removing noise from the received video data, extracting frames, adjusting the resolution, and so on.

[1974] "Feedback" refers to formal advice or instructions to users that are generated based on the analysis results of an artificial intelligence model and are provided in the form of text, images, videos, etc.

[1975] A "display terminal" is a device that receives feedback data sent from the server and visually presents it to the user.

[1976] The "emotion engine" is a program that analyzes changes in the user's facial expressions and voice to estimate their emotional state.

[1977] "Real-time" refers to the immediacy of data processing and feedback during a user's training.

[1978] "Tone" refers to the wording and delivery of feedback or advice, which is adjusted according to the user's emotional state.

[1979] The system of this invention is designed to allow users to receive appropriate training feedback even when an instructor is not present. It also has the ability to recognize the user's emotional state and dynamically adjust the feedback and training menu. The main components of the system are as follows:

[1980] 1. Imaging equipment

[1981] It is installed in the training area and records the user's training in real time, and the captured video data is sent sequentially to a server via the network.

[1982] 2. Server

[1983] The server receives and stores video data sent from the imaging device in real time. The first step is pre-processing of the received video data, which includes noise removal, frame extraction, and resolution adjustment. The pre-processed data is then formatted so that it can be effectively analyzed by the generative AI model.

[1984] The pre-processed video data is then fed into a generative AI model, which analyzes the data and evaluates the user's movements, for example comparing them with ideal training form to identify necessary corrections or improvements.

[1985] Based on the analysis results, the server generates specific feedback in the form of text, images, or videos, which are then sent to a display device and presented to the user.

[1986] The server is also equipped with an emotion engine that analyzes the user's facial expressions and voice to estimate their emotional state, and adjusts the content and tone of the feedback based on the results of this analysis.

[1987] 3. Display terminal

[1988] The system receives feedback data sent from the server and displays it to the user. The feedback is designed to allow users to understand and act immediately, and specific images and videos are also presented.

[1989] Specific examples

[1990] Squat training

[1991] 1. Users

[1992] The user stands in the training area and begins training by pressing the "Start Training" button on the tablet screen.

[1993] 2. Terminal (photography device)

[1994] The camera captures the user's squatting movements and facial expressions in real time and sends the video data to a server.

[1995] 3. Server

[1996] The server preprocesses the received video data of the squat and inputs it into a generative AI model, which then analyzes the video data and evaluates the user's movements.

[1997] The generative AI model compares the user's squat movement with ideal training form and generates specific feedback such as, "Your knee angle is too deep; adjust it to 90 degrees."

[1998] This feedback is generated in the form of text, images, or videos and sent to a display device.

[1999] The emotion engine analyzes the user's facial expressions and voice to estimate their emotional state. For example, if the user is feeling tired or frustrated, it generates feedback content and tone appropriate to that emotional state and sends it to the display device.

[2000] 4. Terminal (display terminal)

[2001] The display device receives the feedback and displays it in an easy-to-understand way to the user. For example, it can show images or videos that show that the knee angle is not appropriate, and provide personalized advice such as "We recommend that you take a short break or drink some water."

[2002] Prompt Sentence Examples

[2003] "Your knees are at too deep an angle when squatting; adjust them to 90 degrees."

[2004] "I suggest you take a short break or drink some water."

[2005] In this way, users can receive real-time feedback on their training movements and emotional state, which can improve training efficiency and safety and help users maintain their motivation.

[2006] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2007] Step 1:

[2008] User starts training

[2009] Specific behavior:

[2010] The user stands in the training area and presses the "Start Training" button on the tablet screen, which starts the system.

[2011] Input: User presses the "Start Training" button on the tablet.

[2012] Output: The system goes into training start mode.

[2013] Step 2:

[2014] The device (camera) captures the training video.

[2015] Specific behavior:

[2016] The camera is installed in a position that captures the entire training area. When the user starts to move, the camera captures their movements in real time and transmits the video data sequentially to a server via a network.

[2017] Input: User action

[2018] Output: Real-time video data

[2019] Step 3:

[2020] The server receives the video data

[2021] Specific behavior:

[2022] The server receives the image data sent from the imaging device and stores it in real time. At this point, the data is raw data that has not yet been pre-processed.

[2023] Input: Raw image data transmitted from the imaging device

[2024] Output: Saved video data

[2025] Step 4:

[2026] The server preprocesses the video data

[2027] Specific behavior:

[2028] The server preprocesses the received video data, which includes removing noise, cropping frames, and adjusting resolution, so that the preprocessed data is in a format that can be effectively analyzed by the generative AI model.

[2029] Input: Stored raw video data

[2030] Output: Pre-processed video data

[2031] Step 5:

[2032] The server inputs data into the generative AI model

[2033] Specific behavior:

[2034] The pre-processed video data is then input into the generative AI model, a process that prepares the data to fit the model's input format.

[2035] Input: Preprocessed video data

[2036] Output: Data input to the generative AI model

[2037] Step 6:

[2038] Generative AI models analyze video data

[2039] Specific behavior:

[2040] A generative AI model analyzes the input data and evaluates the user's movements. The model compares the user's movements with ideal training form and identifies necessary corrections and improvements.

[2041] Input: Data input to the generative AI model

[2042] Output: Parsed feedback data

[2043] Step 7:

[2044] Generative AI models generate feedback

[2045] Specific behavior:

[2046] The generative AI model generates specific feedback based on the analysis results, such as "Your knee angle is too deep, please adjust it to 90 degrees."

[2047] Input: Parsed feedback data

[2048] Output: Generated feedback

[2049] Step 8:

[2050] The server sends feedback to the display device

[2051] Specific behavior:

[2052] The generative AI model generates feedback that is sent to the display device in the form of text, images, or videos.

[2053] Input: Generated feedback

[2054] Output: Feedback sent to the display device

[2055] Step 9:

[2056] Emotion engine analyzes emotional state

[2057] Specific behavior:

[2058] The emotion engine analyzes the user's facial expressions and voice to estimate their emotional state. For example, if frustration is detected, the engine returns that data to the server.

[2059] Input: User's facial expressions and voice

[2060] Output: Emotional state data

[2061] Step 10:

[2062] The server sends feedback based on the emotional state to the display device.

[2063] Specific behavior:

[2064] The feedback content is adjusted based on the analysis results of the emotion engine, changing the tone and message content, and then the adjusted feedback is sent to the display device.

[2065] Input: Emotional state data

[2066] Output: Feedback based on emotional state

[2067] Step 11:

[2068] The display device displays the feedback to the user.

[2069] Specific behavior:

[2070] The display device then displays the received feedback to the user, for example, showing an image or video that indicates the knee angle is not appropriate and providing personalized advice such as "We recommend you take a short break or drink some water."

[2071] Input: Feedback based on emotional state

[2072] Output: Feedback displayed to the user

[2073] Step 12:

[2074] Users modify their training according to the feedback

[2075] Specific behavior:

[2076] The user checks the feedback displayed on the display terminal and follows the instructions to correct their training form and movements. They also refer to the emotional feedback and take appropriate actions such as taking breaks at appropriate times.

[2077] Input: Feedback displayed to the user

[2078] Output: Corrected training form and movements

[2079] (Application example 2)

[2080] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2081] Conventional fitness training systems struggle to provide users with proper form and movement feedback when an instructor is not present. Furthermore, providing uniform feedback without considering the user's emotional state can reduce the user's motivation and training efficiency. This raises concerns about the risk of injury and reduced training effectiveness.

[2082] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing images of the user's training using a camera installed in the training area, means for receiving and saving the video data captured by the camera in real time, means for preprocessing the received video data and inputting it into a generative AI model to analyze the user's movements, means for generating feedback in the form of text, images, or videos based on the analyzed data, means for sending the generated feedback data to a tablet and displaying it to the user, and means for analyzing the user's emotional state and dynamically adjusting the feedback and training menu. This allows users to receive appropriate feedback in real time even when an instructor is not present, and also enables them to obtain feedback optimized according to their individual emotional state.

[2083] The "camera" is an image capturing device for capturing images of the user's training.

[2084] "Video data" is digital data containing visual information captured by a camera.

[2085] "Means for receiving and storing in real time" refers to a function for receiving captured video data to a server without delay and storing the data as needed.

[2086] "Preprocessing" refers to the process of shaping and processing video data to make it suitable for analysis by a generative AI model.

[2087] A "generative AI model" is an artificial intelligence model that analyzes and evaluates a user's training movements.

[2088] "Input" refers to feeding pre-processed video data into a generative AI model.

[2089] The "means of analysis" refers to a method of analyzing and evaluating a user's training movements using a generative AI model.

[2090] "Feedback" refers to instructions or advice provided to the user based on the analysis results of the generative AI model.

[2091] "Text, image, video format" refers to a format in which the feedback content is displayed using different means of expression.

[2092] A "tablet" is a portable display device for displaying generated feedback to a user.

[2093] "Emotional state" refers to the psychological state estimated based on information such as the user's facial expression and voice.

[2094] "Dynamic adjustment means" refers to a function that flexibly changes the feedback and training menu content according to the user's emotional state.

[2095] This invention is a system that allows users to receive appropriate training feedback when an instructor is not present, and also has the ability to recognize the user's emotional state and dynamically adjust the feedback and training menu. This system is realized using a camera, a server, a generative AI model, a tablet, and an emotion engine.

[2096] System Configuration

[2097] 1. Camera

[2098] The camera is installed in the training area and captures the user's training in real time.

[2099] As a specific example, a high-resolution camera (e.g., Logitech StreamCam) is used.

[2100] 2. Server

[2101] The server receives and stores the video data sent from the camera in real time.

[2102] After preprocessing, the received video data is input into the generative AI model.

[2103] As a specific example, a high-performance cloud server (e.g., Amazon Web Services EC2) is used.

[2104] 3. Generative AI Models

[2105] The generative AI model analyzes the video data input by the server and evaluates the user's actions.

[2106] The model compares the user's movements with ideal training form and identifies any corrections or improvements that need to be made.

[2107] A specific example is using a custom AI model (e.g., TensorFlow or PyTorch) for behavior analysis.

[2108] 4. Generate feedback

[2109] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model.

[2110] The feedback is specific and includes information that users can immediately understand and act on.

[2111] Example: The knee angle is too deep, please adjust it to 90 degrees.

[2112] 5. Emotion Engine

[2113] The emotion engine uses video and audio data captured by the camera to analyze changes in the user's facial expressions and voice and estimate their emotional state.

[2114] Based on the analysis results, the content and tone of the feedback can be adjusted to improve the user's training experience.

[2115] As a specific example, we use an emotion analysis engine (e.g., Microsoft Azure Emotion API).

[2116] 6. Tablets

[2117] The tablet receives the feedback data sent from the server and the analysis results of the emotion engine and displays them to the user.

[2118] Based on the feedback provided, users can improve their form and behavior.

[2119] Processing flow

[2120] 1. User Operation

[2121] A user taps the start button on a tablet or smartphone in the fitness training area.

[2122] 2. Camera Operation

[2123] The camera captures the user's training movements and sends the video data to a server.

[2124] 3. Server Processing

[2125] The server receives the video data, pre-processes it, and then feeds the data into the generative AI model.

[2126] 4. Analysis of generative AI models

[2127] A generative AI model analyzes the data and compares the user's movements with ideal form.

[2128] For example: "Your knee angle is too deep, please adjust it to 90 degrees."

[2129] 5. Generate and send feedback

[2130] The server generates feedback in the form of text, images and videos and sends it to the tablet.

[2131] 6. Emotion Engine Analysis

[2132] The emotion engine analyzes the user's emotional state and dynamically adjusts the feedback content.

[2133] 7. Displaying feedback to users

[2134] The tablet receives the feedback and displays it in an easy-to-understand way.

[2135] For example: "Bend your knees more."

[2136] Prompt Sentence Examples

[2137] Analyze footage of a user squatting, assess knee angle, hip position, and back curvature, and generate feedback to correct for ideal posture.

[2138] This allows users to receive both motion- and emotion-based feedback in real time, improving training efficiency and safety.

[2139] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2140] Step 1:

[2141] A user taps the start button on a tablet or smartphone in the fitness training area. This causes the camera to start capturing video data. The input is the user's operation information, and the output is a signal to the camera to start recording.

[2142] Step 2:

[2143] The camera captures the user's training movements in real time and captures the video data, which is then sent to a server via a network. The input is the user's movement data, and the output is the video data sent to the server.

[2144] Step 3:

[2145] The server receives and stores video data from the camera in real time. It then preprocesses the received video data and prepares it for input into the generative AI model. The input is video data from the camera, and the output is preprocessed data. Specifically, noise removal and frame correction are performed.

[2146] Step 4:

[2147] The server inputs the preprocessed video data into a generative AI model. The generative AI model analyzes and evaluates the user's training movements. The input is the preprocessed data, and the output is the evaluation result of the user's movements. Specifically, the movement data is compared with the ideal form to detect errors.

[2148] Step 5:

[2149] The server generates feedback in the form of text, images, and videos based on the analysis results of the generative AI model. The input is the evaluation results of the user's movements, and the output is feedback data. Specifically, it describes in detail the corrections needed for knee angle and hip position.

[2150] Step 6:

[2151] The emotion engine uses video and audio data captured by the camera to analyze changes in the user's facial expressions and voice and estimate their emotional state. The input is video and audio data, and the output is the analysis result of the emotional state. The emotion engine uses, for example, the Microsoft Azure Emotion API.

[2152] Step 7:

[2153] The server adjusts the content and tone of the feedback based on the analysis results of the emotion engine. The input is the analysis result of the emotional state, and the output is dynamically adjusted feedback. If the user is fatigued, the content will be changed to be more encouraging.

[2154] Step 8:

[2155] The server sends the final feedback data to the tablet. The input is the dynamically adjusted feedback data, and the output is the data transmission to the tablet.

[2156] Step 9:

[2157] The tablet receives the feedback data sent from the server and the analysis results of the emotion engine, and displays them in an easy-to-understand manner for the user. The input is the feedback data, and the output is the display on the tablet.

[2158] Step 10:

[2159] The user checks the feedback displayed on the tablet and corrects their training movements. They also refer to emotional feedback to take appropriate actions such as taking breaks at appropriate times. The input is feedback data, and the output is corrected movements and actions such as resting.

[2160] This series of steps allows users to receive both behavioral and emotional feedback in real time, improving training efficiency and safety.

[2161] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2162] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2163] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2164] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2165] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2166] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2167] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2168] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2169] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2170] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2171] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2172] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2173] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2174] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2175] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2176] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2177] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2178] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2179] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2180] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2181] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2182] The following is further disclosed regarding the above embodiment.

[2183] (Claim 1)

[2184] A means for taking pictures of the user's training with a camera installed in the training area;

[2185] A means for receiving and storing video data captured by the camera in real time;

[2186] A means to preprocess the received video data and input it into a generative AI model to analyze user behavior;

[2187] A means for generating feedback in the form of text, images, or videos based on the analyzed data;

[2188] The generated feedback data is sent to the tablet and displayed to the user.

[2189] A system including:

[2190] (Claim 2)

[2191] 2. The system according to claim 1, wherein a series of processes from filming the training scene to generating and displaying feedback is performed in real time.

[2192] (Claim 3)

[2193] 10. The system of claim 1, further comprising means for analyzing the accuracy of a user's form during exercises in the training area and detecting errors by comparing the form with a predetermined ideal form.

[2194] "Example 1"

[2195] (Claim 1)

[2196] a means for capturing images of a user's training movements using a photographing device installed in the training area;

[2197] means for receiving and storing video data captured by the imaging device in real time;

[2198] A means for preprocessing the received video data and inputting it into an analytical model to analyze user behavior;

[2199] A means for generating feedback in the form of text, images, or videos based on the analyzed data;

[2200] A means for transmitting the generated feedback data to a display terminal and displaying it to the user;

[2201] A system including:

[2202] (Claim 2)

[2203] 2. The system according to claim 1, wherein the entire process from capturing the image to generating and displaying the feedback is performed in real time.

[2204] (Claim 3)

[2205] 10. The system of claim 1, further comprising means for analyzing the accuracy of a user's form during exercises in the training area and detecting errors by comparing the form with a predetermined ideal form.

[2206] "Application Example 1"

[2207] (Claim 1)

[2208] A means for taking pictures of the user's training with a camera installed in the training area;

[2209] A means for receiving and storing video data captured by the camera in real time;

[2210] A means to preprocess the received video data and input it into a generative AI model to analyze user behavior;

[2211] A means for generating feedback in the form of text, images, or videos based on the analyzed data;

[2212] means for transmitting the generated feedback data to a tablet for display to the user;

[2213] A means for analyzing the worker's movements in real time and sending feedback to the worker, which is installed in the work area of ​​the factory;

[2214] A system including:

[2215] (Claim 2)

[2216] 2. The system according to claim 1, wherein a series of processes from filming the training scene to generating and displaying feedback is performed in real time.

[2217] (Claim 3)

[2218] The system of claim 1 further comprises means for analyzing the accuracy of a user's form during practice in the training area, comparing it with a predetermined ideal form to detect errors, and performing motion analysis and feedback generation in the work area of ​​the factory.

[2219] "Example 2: Combining Emotion Engines"

[2220] (Claim 1)

[2221] a means for taking pictures of the user's training scene with a photographing device installed in the training area;

[2222] means for receiving and storing video data captured by the imaging device in real time;

[2223] A means for preprocessing the received video data and inputting it into an artificial intelligence model to analyze the user's behavior;

[2224] A means for generating feedback in the form of text, images, or videos based on the analyzed data;

[2225] means for transmitting the generated feedback data to a display terminal and displaying it to a user;

[2226] A means of estimating the user's emotional state by analyzing changes in facial expressions and voice;

[2227] A means to tailor feedback content and tone based on emotional state; and

[2228] A system including:

[2229] (Claim 2)

[2230] 2. The system according to claim 1, wherein a series of processes from filming the training scene to generating and displaying feedback is performed in real time.

[2231] (Claim 3)

[2232] 10. The system of claim 1, further comprising means for analyzing the accuracy of a user's form during exercises in the training area and detecting errors by comparing the form with a predetermined ideal form.

[2233] "Application example 2 when combining emotion engines"

[2234] (Claim 1)

[2235] A means for taking pictures of the user's training with a camera installed in the training area;

[2236] A means for receiving and storing video data captured by the camera in real time;

[2237] A means to preprocess the received video data and input it into a generative AI model to analyze user behavior;

[2238] A means for generating feedback in the form of text, images, or videos based on the analyzed data;

[2239] means for transmitting the generated feedback data to a tablet for display to the user;

[2240] A means of analyzing the user's emotional state and dynamically adjusting feedback and training menus;

[2241] A system including:

[2242] (Claim 2)

[2243] 2. The system according to claim 1, wherein a series of processes from filming the training scene to generating and displaying feedback is performed in real time.

[2244] (Claim 3)

[2245] The system of claim 1 includes a means for analyzing the accuracy of a user's form during exercises in the training area, comparing it with a pre-defined ideal form to detect any discrepancies, and adjusting the content and tone of the feedback based on the user's emotional state. [Explanation of symbols]

[2246] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for taking pictures of the user's training with a camera installed in the training area; A means for receiving and storing video data captured by the camera in real time; A means to preprocess the received video data and input it into a generative AI model to analyze user behavior; A means for generating feedback in the form of text, images, or videos based on the analyzed data; The generated feedback data is sent to the tablet and displayed to the user. A system including:

2. 2. The system according to claim 1, wherein a series of processes from photographing the training scene to generating and displaying the feedback is performed in real time.

3. 2. The system of claim 1, further comprising means for analyzing the accuracy of a user's form during exercise in the training area and detecting errors by comparing it with a predetermined ideal form.

Images