System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing a training motion of a user in real-time with an image capture device; means for analyzing the captured training motion with an image recognition device to estimate the coordinates of each joint of the user; means for comparing the estimated coordinate data with standard training motion data to detect errors; and means for providing feedback to the user based on the errors.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention relates to a system that detects posture during training in real time and provides appropriate feedback. Its purpose is to replace the expensive training provided by trainers at traditional gyms and enable individuals to train efficiently and accurately at home. There is a demand for a system that can detect posture errors during individual training and provide real-time feedback to correct those errors, thereby improving training effectiveness. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including: means for capturing images of a user's training movements in real time using an image capture device; means for analyzing the captured training movements using an image recognition device and estimating the coordinates of each of the user's joints; means for comparing the estimated coordinate data with standard training movement data to detect errors; and means for providing feedback to the user based on the errors. This allows individuals to train accurately and effectively from the comfort of their own homes, reducing the need for conventional, expensive trainers.

[0006] Below are definitions of important terms included in the claims.

[0007] "Image capture device" refers to a device that captures the user's movements in real time during training.

[0008] "User" refers to an individual who uses this system to train.

[0009] "Training movements" refers to the exercise or workout movements performed by the user.

[0010] "Real-time" refers to the acquisition and processing of data at the exact moment the user is training.

[0011] An "image recognition device" refers to a device that analyzes image data of captured training movements and recognizes the user's posture and movements.

[0012] "Coordinates" refers to data indicating the position of each part of the user's body.

[0013] "Standard training movement data" refers to data on correct training movements by professional trainers, etc.

[0014] The "error" refers to a part where the user's training motion differs from the standard training motion data.

[0015] "Feedback" refers to instructions and advice to correct errors so that the user can perform the training movements correctly.

[0016] "System" refers to an integrated device or program that includes the above means. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback to users while they are training at home, using an image capture device, an image recognition device, a standard training movement database, and a feedback device.

[0039] System configuration

[0040] 1. Image Capture Device

[0041] The image capture device captures the user's training movements in real time, typically using a smartphone camera.

[0042] 2. Image Recognition Device

[0043] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each joint of the user using a posture estimation model.

[0044] 3. Standard training motion database

[0045] The standard training movement database stores coordinate data of correct training movements shown by professional trainers.

[0046] 4. Feedback Devices

[0047] The feedback device is a device for comparing the data obtained from the image recognition device and the standard training motion database and providing feedback to the user when an error is detected. This feedback can be provided in the form of voice or text.

[0048] Program processing

[0049] User Actions

[0050] The user launches a fitness app on their smartphone, selects a training menu and goal, and then begins training in front of the camera.

[0051] Device behavior

[0052] The device captures the user's movements in real time through a camera and temporarily stores the video data. The stored data is analyzed using a posture estimation model to generate three-dimensional coordinate data for each joint. This coordinate data is then sent from the device to a server.

[0053] Server Operation

[0054] The server receives the transmitted coordinate data and compares it with data in a standard training motion database. If an error is detected as a result of the comparison, the content of the error (e.g., knee angle is too shallow, back is bent, etc.) is identified. The server generates appropriate feedback and sends it to the terminal.

[0055] User Feedback

[0056] The device receives feedback from the server and provides it to the user in the form of voice or text, such as instructions like "Bend your knees a little more" or "Keep your back straight." The user can then correct their posture and continue training.

[0057] Specific examples

[0058] For squat training

[0059] User action: The user performs squats in front of the smartphone camera.

[0060] Device behavior: The device camera captures the user's squatting motion, and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[0061] Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[0062] Device behavior: The feedback sent from the server, "Bend your knees to 90 degrees," is notified to the user via voice or text.

[0063] User Action: The user follows the instructions to correct the knee angle and continue squatting.

[0064] In the case of dietary management

[0065] User action: The user uploads a photo of their lunch to the app.

[0066] Device action: The photo is sent to the server.

[0067] Server operation: Analyzes the contents of lunch using image recognition and calculates nutritional value.

[0068] Device behavior: Displays a nutrition report and next meal recommendation to the user, for example, "You're lacking in vitamin C today. Add some fruit to your next meal."

[0069] In this way, by using this system, users can receive high-quality training and dietary management at home. Crawling allows for efficient and accurate training, enabling sustainable health maintenance.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] Users launch a fitness app on their smartphone and select a training menu and goals.

[0073] Step 2:

[0074] The device will perform initial setup based on the user's selections, start the camera, and notify the user that they are ready to use the camera.

[0075] Step 3:

[0076] The user stands in front of the camera and starts a training movement, for example, a squat movement.

[0077] Step 4:

[0078] The device's camera captures the user's actions in real time and temporarily stores the video data.

[0079] Step 5:

[0080] The device preprocesses the stored video data, for example changing the resolution or removing noise.

[0081] Step 6:

[0082] The device inputs the preprocessed video data into a posture estimation model, which generates 3D coordinate data for each of the user's joints.

[0083] Step 7:

[0084] The terminal transmits the generated three-dimensional coordinate data to the server.

[0085] Step 8:

[0086] The server receives the transmitted three-dimensional coordinate data and retrieves corresponding data from a standard training motion database.

[0087] Step 9:

[0088] The server compares the 3D coordinate data with standard training motion data to detect errors, such as the angle of the knees or the position of the hips.

[0089] Step 10:

[0090] The server generates appropriate feedback based on the detected error, such as instructions like "bend your knees more."

[0091] Step 11:

[0092] The server transmits the generated feedback to the terminal.

[0093] Step 12:

[0094] The device will then notify the user of the received feedback in the form of voice or text, for example by displaying a message on the screen saying, "Bend your knees to 90 degrees."

[0095] Step 13:

[0096] The user corrects their posture according to the feedback and continues training.

[0097] Step 14:

[0098] After training, the device stores all data locally (e.g., number of repetitions, time, posture accuracy, etc. for each set).

[0099] Step 15:

[0100] The device sends the saved data to the server as training history.

[0101] Step 16:

[0102] Based on the received data, the server updates the user's training history and analyzes long-term effects.

[0103] Step 17:

[0104] When a user uploads a photo of their meal to the app, the device sends the photo to the server.

[0105] Step 18:

[0106] The server uses image recognition to analyze the contents of the meal and calculate its nutritional value.

[0107] Step 19:

[0108] The server generates analysis results and nutritional advice and sends them to the terminal.

[0109] Step 20:

[0110] The device will then display the received nutrition report and advice to the user, for example, a message saying "You are lacking in vitamin C today. Add some fruit to your next meal."

[0111] This allows users to receive accurate feedback when training at home, enabling them to train efficiently and effectively, and also provides appropriate advice on dietary management, improving overall health management.

[0112] Example 1

[0113] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0114] With conventional training systems, when users train at home, it is difficult to maintain proper form and posture, making it difficult to train effectively. Incorrect form also increases the risk of injury. Furthermore, the timing and quality of feedback is insufficient, making it difficult for users to correct their form in real time.

[0115] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0116] In this invention, the server includes means for capturing images of training movements in real time using an image capture device, means for analyzing the captured training movements using an image recognition device and estimating the coordinates of each joint of the user, means for comparing the estimated coordinate data with standard training movement data and detecting errors, means for providing feedback to the user based on the errors, means for transmitting the analysis data from a terminal to the server, and means for providing feedback to the user in audio or text format. This allows users to train effectively and safely at home and correct their form and posture in real time.

[0117] An "image capture device" is a device for capturing images of a user's training movements in real time.

[0118] The "image recognition device" is a device that analyzes photographed training movements and estimates the coordinates of each of the user's joints.

[0119] "Standard training movement data" is a database containing coordinate data of correct training movements.

[0120] The "means for detecting errors" is a mechanism for comparing the estimated coordinate data with standard training motion data to identify errors or inaccurate motions.

[0121] A "means for providing feedback" is a mechanism for providing users with correction instructions in audio or text format based on errors.

[0122] A "terminal" is an electronic device that allows a user to launch a fitness app, capture images of their movements using a camera, and send the analysis data to a server.

[0123] The "server" is a device that receives the analysis data sent from the terminal, compares it with standard training motion data to detect errors, and generates appropriate feedback.

[0124] A "network" is a communications infrastructure for sending and receiving data between terminals and servers.

[0125] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home. The system uses an image capture device, an image recognition device, a standard training movement database, and a feedback device.

[0126] System configuration

[0127] 1. Image Capture Device

[0128] The image capture device captures the user's training movements in real time. Typically, a camera on a mobile information terminal (smartphone) is used. For example, the user performs training movements in front of the smartphone camera.

[0129] 2. Image Recognition Device

[0130] The image recognition device receives the captured video data and estimates the coordinates of each of the user's joints. This analysis uses a posture estimation model such as "MediaPipe." The analysis generates three-dimensional coordinate data for each joint.

[0131] 3. Standard training motion database

[0132] The standard training motion database stores coordinate data of correct training motions shown by professional trainers. This database is managed by a server.

[0133] 4. Feedback Devices

[0134] The feedback device compares the coordinate data obtained from the image recognition device with standard training movement data and provides feedback to the user when an error is detected. The feedback can be in the form of audio or text. For example, instructions such as "Please keep your back more straight" are displayed.

[0135] 5. Terminal

[0136] The device is an electronic device that allows the user to launch the fitness app, capture their movements with a camera, and send the analysis data to the server. The user selects the desired training menu and performs the exercise in front of the camera, and the device captures their movements.

[0137] 6. Server

[0138] The server receives the analysis data transmitted from the terminal, compares it with standard training motion data to detect errors, and generates feedback based on the errors, which is then transmitted to the terminal via a network.

[0139] Specific examples

[0140] For squat training

[0141] 1. User action: The user performs squats in front of the smartphone camera.

[0142] 2. Device movement: The device camera captures the squat movement and analyzes the knee and hip angles through a pose estimation model such as "MediaPipe."

[0143] 3. Device sends the analysis results to the server.

[0144] 4. Server operation: The server compares the data with the standard data and detects, for example, if the knee angle is not appropriate.

[0145] 5. Providing feedback: The server sends feedback such as "Bend your knees to 90 degrees" and notifies the user via voice or text.

[0146] 6. User Correction: The user follows the instructions to correct the knee angle and continue squatting.

[0147] Prompt Sentence Examples

[0148] "Design an AI model that can accurately measure the knee angle during training and provide appropriate feedback. This system analyzes squat training movements in real time and sends instructions to the user if the knee angle is not appropriate. The hardware used will be a smartphone, and the software used will be MediaPipe, which acts as a posture estimation model."

[0149] In this way, by using this system, users can train effectively and safely at home and correct their form and posture in real time.

[0150] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0151] Step 1:

[0152] The user launches a fitness app.

[0153] The user taps to launch a fitness app on their smartphone and selects a training menu and goal. For example, the user selects the "Squat" menu and sets the goal to "20 repetitions." In this step, the user provides input, and the selected training menu and goal are output.

[0154] Step 2:

[0155] The device captures the user's actions.

[0156] The device (smartphone) camera captures the user's training movements in real time and generates video data. When the user starts squatting in front of the camera, the camera captures the movements at 30 frames per second. The input of this step is the user's movements, and the output is real-time video data.

[0157] Step 3:

[0158] The device analyzes the video data.

[0159] The device inputs the captured video data into the posture estimation model "MediaPipe" to generate 3D coordinate data for each joint. The joint coordinate data is extracted through analysis. The input for this step is video data, and the output is 3D coordinate data for each joint.

[0160] Step 4:

[0161] The device sends the analysis data to the server.

[0162] The device converts the generated 3D coordinate data into JSON format and sends it to the server via the network. Specific operations include packaging the coordinate data and setting up communication. The input of this step is the 3D coordinate data, and the output is the data to be sent to the server.

[0163] Step 5:

[0164] The server compares the data and detects errors.

[0165] The server compares the received 3D coordinate data with standard data in a standard training movement database. For example, it compares it with standard movement data for squats to detect if the knee angle is inappropriate. The server receives the data, compares it, and performs an error analysis. The input of this step is the 3D coordinate data, and the output is the error detection result.

[0166] Step 6:

[0167] The server generates the feedback.

[0168] The server generates appropriate feedback based on the error. For example, it generates feedback such as "Bend your knees to 90 degrees." The feedback content is determined and the generated feedback message is output. The input of this step is the error detection result, and the output is the feedback message.

[0169] Step 7:

[0170] The terminal provides feedback to the user.

[0171] The device notifies the user of the feedback received from the server in the form of voice or text. Specific actions include playing voice or displaying on the screen. For example, the instruction "Please bend your knees to 90 degrees" is played to the user by voice. The input of this step is the feedback message, and the output is the feedback notification to the user.

[0172] (Application example 1)

[0173] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0174] Conventional training methods have the problem that it is difficult to maintain correct posture and movement, and that training in incorrect posture can lead to injury or reduced effectiveness. Another problem is that real-time feedback is not provided, making it difficult to immediately correct movements. The purpose of this invention is to solve these problems and enable users to train efficiently and safely.

[0175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0176] In this invention, the server includes means for capturing images of the user's movements in real time, means for analyzing the captured images and estimating the coordinates of each joint, means for comparing the estimated coordinate data with standard behavior data to detect errors, means for providing feedback to the user based on the errors and displaying the feedback on a video frame, means for transmitting data to the server and receiving feedback, and means for including a posture estimation model used to analyze the joint coordinates. This allows the user to receive accurate feedback in real time and perform safe and effective training.

[0177] An "image capture device" is a device for capturing images of a user's actions in real time.

[0178] An "image recognition device" is a device that analyzes movements captured by an image capture device and estimates the coordinates of each of the user's joints.

[0179] "Standard behavior data" is a database that stores coordinate data of correct training movements.

[0180] An "error" is an inappropriate point in the training movement detected by comparing the estimated joint coordinate data of the user with the standard action data.

[0181] "Feedback" is the instruction or advice provided to the user based on a detected error.

[0182] "Means for displaying on the video frame" refers to a method for visually notifying the user of feedback.

[0183] The "means for transmitting data to the server" is a method for transmitting the user's joint coordinate data to the server.

[0184] The "means for receiving feedback" is a method for receiving feedback sent from the server.

[0185] A "pose estimation model" is a neural network model used to analyze the user's joint coordinate data.

[0186] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home or in a physical training facility. The system uses an image capture device, an image recognition device, a standard behavior database, and a feedback device.

[0187] 1. System Configuration

[0188] 1. Image Capture Device

[0189] The image capture device is a device that captures the user's movements in real time, and a camera on a smartphone or smart glasses is used.

[0190] 2. Image Recognition Device

[0191] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each user's joints using a posture estimation model (TENSORFLOW (registered trademark), Keras).

[0192] 3. Standard Behavior Database

[0193] The coordinate data of correct training movements is stored in a standard behavior database, and errors are detected by comparing with this database.

[0194] 4. Feedback Devices

[0195] The feedback device is a device that provides appropriate feedback to the user when an error is detected. This feedback is provided to the user in the form of voice or text and is displayed on the video frame.

[0196] 2. Program operation overview

[0197] Device behavior

[0198] 1. The user launches a fitness app on their smartphone or smart glasses.

[0199] 2. Start training movements in front of the camera, which will capture them in real time.

[0200] 3. The acquired video data is temporarily stored and preprocessed (resizing, normalization).

[0201] Server Operation

[0202] 1. Analyzed joint coordinate data is sent from the device to the server.

[0203] 2. The server receives this data and compares it with data in a standard behavior database.

[0204] 3. If the comparison reveals any errors, they are identified and feedback is generated.

[0205] User Feedback

[0206] 1. The device receives the feedback sent from the server and provides it to the user in voice or text format.

[0207] 2. Feedback is also displayed on the video frame, allowing the user to refer to it and correct their training movements.

[0208] 3. Specific Examples

[0209] For squat training

[0210] 1. User action: The user performs squats in front of the smartphone camera.

[0211] 2. Device movement: The camera captures the user's movements and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[0212] 3. Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[0213] 4. Device operation: The feedback sent from the server, "Please bend your knees to 90 degrees," is notified to the user via voice or text.

[0214] 5. User Action: The user follows the instructions to correct the knee angle and continue squatting.

[0215] 4. Example prompts for generative AI models

[0216] TXT

[0217] Imagine a user performing a squat in a physical store. Design an application that analyzes the joint angles in real time based on the video captured by a camera and provides appropriate feedback. The hardware used is a smartphone camera, and the software uses a pose estimation model built with TensorFlow.

[0218] This embodiment allows the user to receive accurate feedback in real time while training safely and effectively.

[0219] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0220] Step 1:

[0221] The user launches a fitness app on their smartphone or smart glasses and selects a training menu.

[0222] Input: Operation instructions from the user

[0223] Output: Training menu selection status

[0224] Specific operation: The user taps the screen of their smartphone or smart glasses to select the desired training menu (e.g., squats).

[0225] Step 2:

[0226] The device activates the camera and captures the user's training movements in real time.

[0227] Input: User action

[0228] Output: Real-time video data

[0229] Specific operation: The device's camera captures the user's movements and obtains the video as data in real time.

[0230] Step 3:

[0231] The device preprocesses (resizes and normalizes) the acquired video data and inputs it into the pose estimation model.

[0232] Input: Real-time video data

[0233] Output: Preprocessed video data, joint coordinate data

[0234] Specific operation: The device resizes and normalizes the video data, then inputs the preprocessed data into a posture estimation model (TensorFlow, Keras) to generate coordinate data for each of the user's joints.

[0235] Step 4:

[0236] The terminal transmits the generated joint coordinate data to the server.

[0237] Input: joint coordinate data

[0238] Output: Data transfer to the server

[0239] Specific operation: The device sends joint coordinate data to the server using an appropriate communication protocol (e.g., HTTP request).

[0240] Step 5:

[0241] The server compares the received joint coordinate data with a standard behavior database to detect errors.

[0242] Input: joint coordinate data, standard action data

[0243] Output: Error detection result

[0244] Specific operation: The server compares the data in the standard behavior database with the joint coordinate data and detects errors (e.g., a shallow knee angle).

[0245] Step 6:

[0246] The server generates feedback based on the detected errors and sends it to the terminal.

[0247] Input: Error detection result

[0248] Output: Feedback data

[0249] Specific operation: Based on the error, the server generates appropriate feedback (e.g., "Bend your knees to 90 degrees") and sends it to the device.

[0250] Step 7:

[0251] The terminal displays the feedback received from the server on the video frame and notifies the user in voice or text format.

[0252] Input: Feedback data

[0253] Output: Audio or text feedback, feedback displayed on the video frame

[0254] What it does: The device overlays feedback text and images on the video frame and simultaneously plays an audio notification to the user.

[0255] Step 8:

[0256] The user uses the feedback to modify their training movements.

[0257] Input: Feedback information

[0258] Output: Corrected training behavior

[0259] Specific actions: The user corrects their training actions according to the feedback provided by the device (e.g., "Bend your knees a little more").

[0260] Through these steps, users can receive appropriate feedback in real time and undergo safe and effective training.

[0261] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0262] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home, using an image capture device, an image recognition device, a standard training movement database, a feedback device, and an emotion engine.

[0263] System configuration

[0264] 1. Image Capture Device

[0265] The image capture device is a device that captures images of the user's training movements in real time, and is generally a smartphone camera.

[0266] 2. Image Recognition Device

[0267] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each joint of the user using a posture estimation model.

[0268] 3. Standard training motion database

[0269] The standard training movement database stores coordinate data of correct training movements shown by professional trainers.

[0270] 4. Feedback Devices

[0271] The feedback device is a device for comparing the data obtained from the image recognition device and the standard training motion database and providing feedback to the user when an error is detected. This feedback can be provided in the form of voice or text.

[0272] 5. Emotion Engine

[0273] The emotion engine is a device that recognizes emotions by analyzing the user's facial expressions and voice, and generates more appropriate and effective feedback based on the user's emotional state.

[0274] Program processing

[0275] User Actions

[0276] The user launches a fitness app on their smartphone, selects a training menu and goal, and then begins training in front of the camera.

[0277] Device behavior

[0278] The device captures the user's movements in real time through a camera and temporarily stores the video data. The stored data is analyzed using a posture estimation model to generate three-dimensional coordinate data for each joint. This coordinate data is then sent from the device to a server.

[0279] Server Operation

[0280] The server receives the coordinate data and compares it with data in a standard training motion database. If an error is detected as a result of the comparison, the nature of the error (e.g., knee angle is too shallow, back is bent, etc.) is identified. The server generates appropriate feedback and sends it to the device.

[0281] Emotion Engine Operation

[0282] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. For example, it can detect when the user is tired, irritated, or unable to concentrate. The emotion engine takes these emotional states into account and adjusts the content of the feedback generated by the server.

[0283] User Feedback

[0284] The device receives feedback from the server and provides it to the user in the form of voice or text. The feedback, adjusted by the emotion engine, includes encouragement and advice based on the user's emotional state. For example, specific messages such as "Great, keep it up" or "Take a short break" are provided.

[0285] Specific examples

[0286] For squat training

[0287] User action: The user performs squats in front of the smartphone camera.

[0288] Device behavior: The device camera captures the user's squatting motion, and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[0289] Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[0290] How the emotion engine works: The emotion engine analyzes the user's facial expressions and recognizes that they are tired.

[0291] Device behavior: In addition to the feedback sent from the server, "Bend your knees to 90 degrees," the device displays the message "Feeling good, take a short break" via voice or text.

[0292] User Actions: The user follows the instructions, corrects the knee angle, and takes breaks at appropriate times.

[0293] In the case of dietary management

[0294] User action: The user uploads a photo of their lunch to the app.

[0295] Device action: The photo is sent to the server.

[0296] Server operation: Analyzes the contents of lunch using image recognition and calculates nutritional value.

[0297] How the emotion engine works: The emotion engine analyzes the user's voice and recognizes that they are depressed.

[0298] What happens on the device: In addition to the nutrition report, the device will display encouraging messages based on the analysis of the emotion engine (e.g., "You're doing great! Make sure you get more vitamin C at your next meal").

[0299] In this way, by using this system, users can receive high-quality training and dietary management at home. Furthermore, the emotion engine provides feedback appropriate to the user's emotional state, increasing motivation for training and dietary management.

[0300] The processing flow will be explained below.

[0301] Step 1:

[0302] Users launch a fitness app on their smartphone and select a training menu and goals.

[0303] Step 2:

[0304] The device will perform initial setup based on the user's selections, start the camera, and notify the user that they are ready to use the camera.

[0305] Step 3:

[0306] The user stands in front of the camera and starts a training movement, for example, a squat movement.

[0307] Step 4:

[0308] The device's camera captures the user's actions in real time and temporarily stores the video data.

[0309] Step 5:

[0310] The device preprocesses the stored video data, for example changing the resolution or removing noise.

[0311] Step 6:

[0312] The device inputs the preprocessed video data into a posture estimation model, which generates 3D coordinate data for each of the user's joints.

[0313] Step 7:

[0314] The terminal transmits the generated three-dimensional coordinate data to the server.

[0315] Step 8:

[0316] The server receives the transmitted three-dimensional coordinate data and retrieves corresponding data from a standard training motion database.

[0317] Step 9:

[0318] The server compares the 3D coordinate data with standard training motion data to detect errors, such as the angle of the knees or the position of the hips.

[0319] Step 10:

[0320] The server generates appropriate feedback based on the detected error, such as instructions like "bend your knees more."

[0321] Step 11:

[0322] The server transmits the generated feedback to the terminal.

[0323] Step 12:

[0324] The device captures the user's facial expressions and voice and sends them to the emotion engine.

[0325] Step 13:

[0326] The emotion engine analyzes the user's facial expressions and voice to recognize emotions, such as fatigue, concentration, and frustration.

[0327] Step 14:

[0328] The emotion engine sends the recognized emotion data to the server.

[0329] Step 15:

[0330] The server receives data from the emotion engine and adjusts the feedback, for example adding a message to a tired user saying "Please take a break."

[0331] Step 16:

[0332] The server sends the adjusted feedback to the terminal.

[0333] Step 17:

[0334] The device will then provide the user with the feedback in the form of voice or text, for example displaying a message such as "Bend your knees to 90 degrees. Feels good, take a short break."

[0335] Step 18:

[0336] The user corrects their posture according to the feedback and continues training.

[0337] Step 19:

[0338] After training, the device stores all data locally (e.g., number of repetitions, time, posture accuracy, etc. for each set).

[0339] Step 20:

[0340] The device sends the saved data to the server as training history.

[0341] Step 21:

[0342] Based on the received data, the server updates the user's training history and analyzes long-term effects.

[0343] Step 22:

[0344] When a user uploads a photo of their meal to the app, the device sends the photo to the server.

[0345] Step 23:

[0346] The server uses image recognition to analyze the contents of the meal and calculate its nutritional value.

[0347] Step 24:

[0348] The emotion engine analyzes the user's voice and recognizes food-related emotions, such as satisfaction, stress, and appetite.

[0349] Step 25:

[0350] The server generates a nutrition report based on the analysis results and emotion engine data and sends it to the device.

[0351] Step 26:

[0352] The device displays the received nutrition report and feedback based on the emotion engine's results. For example, it provides a message like, "You're lacking in vitamin C today. Add some fruit to your next meal. You're doing a great job!"

[0353] This process allows users to receive appropriate feedback in real time during their workouts, enabling them to effectively train at home. Furthermore, the emotion engine provides encouragement and advice appropriate to the user's emotional state, helping to maintain motivation.

[0354] Example 2

[0355] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0356] Conventional training support systems only detect errors in the user's movements and provide feedback, but they have the problem of not being able to provide appropriate feedback that takes into account the user's emotional state. This can make it difficult for users to maintain their motivation and can result in ineffective training.

[0357] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0358] In this invention, the server includes means for capturing images of the user's training movements in real time using an image capture device, means for analyzing the captured training movements using an image recognition device and estimating the coordinates of each of the user's joints, means for comparing the estimated coordinate data with standard training movement data and detecting errors, means for providing feedback to the user based on the errors, and means for analyzing the user's emotional state and adjusting the content of the feedback. This makes it possible to provide appropriate feedback according to the user's emotional state, thereby achieving effective training while increasing motivation.

[0359] An "image capture device" is a device for capturing images of a user's training movements in real time, such as a camera on a mobile device.

[0360] An "image recognition device" is a device that analyzes video data captured by an image capture device and estimates the coordinates of each of the user's joints.

[0361] "Standard training movement data" refers to a database that stores coordinate data of correct training movements shown by professional trainers.

[0362] "Feedback" refers to advice or guidance provided based on errors in the user's training movements, and is provided in audio or text format.

[0363] An "emotion engine" refers to a device or system that has the function of analyzing the user's emotional state and adjusting the content of the feedback.

[0364] "Emotional state" refers to psychological states such as fatigue, concentration, and stress, which are recognized by analyzing the user's facial expressions and voice.

[0365] "Coordinate data" refers to numerical data that indicates the position of each joint of the user in three-dimensional space.

[0366] "Error" refers to the difference in the estimated coordinate data compared to the standard training motion data.

[0367] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home, using an image capture device, an image recognition device, a standard training movement database, a feedback device, and an emotion engine.

[0368] System configuration

[0369] 1. Image Capture Device

[0370] The image capture device is a device that captures images of the user's training movements in real time, and is typically a camera on a mobile device.

[0371] 2. Image Recognition Device

[0372] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each of the user's joints using a pose estimation model such as TensorFlow or OpenPose.

[0373] 3. Standard training motion database

[0374] The standard training movement database stores coordinate data of correct training movements shown by a professional trainer.

[0375] 4. Feedback Devices

[0376] The feedback device compares the data obtained from the image recognition device and the standard training motion database, and provides feedback to the user when an error is detected. The feedback can be provided in the form of voice or text using voice recognition technology such as Google® Text-to-Speech.

[0377] 5. Emotion Engine

[0378] The emotion engine uses Microsoft® Azure® emotion recognition APIs to analyze the user's facial expressions and voice, recognizing their emotional state. Based on this emotional state, the server adjusts the content of the feedback it generates.

[0379] Description

[0380] The user launches a fitness app on their mobile device and selects a training menu. They then begin training in front of the camera. The device captures the user's movements in real time through the camera and temporarily stores the video data. This data is analyzed using a pose estimation model such as TensorFlow or OpenPose to generate three-dimensional coordinate data for each joint. This coordinate data is then sent from the device to a server via the Internet.

[0381] The server runs on a cloud platform and receives coordinate data sent from the device. The server compares this data with standard training movement data of professional trainers stored in a database of standard training movements, and identifies errors such as a shallow knee angle or a bent back. If an error is identified, the server generates feedback according to the error, converts it into voice or text using a speech synthesis service (e.g., Google Text-to-Speech), and sends it to the device.

[0382] The emotion engine analyzes the user's facial expressions and voice through the microphone and camera connected to the device, recognizing their emotional state, such as fatigue, concentration, and stress. This emotional data is also sent to the server and reflected in the generated feedback. For example, if the user is tired, an encouraging message such as "Take a short break" will be added.

[0383] The device receives feedback from the server and provides it to the user in the form of voice or text, allowing the user to correct their next actions. This system allows users to receive high-quality training at home and increases their motivation through appropriate feedback provided by the emotion engine.

[0384] Specific examples

[0385] For squat training

[0386] User action: The user performs squats in front of the smartphone camera.

[0387] Device behavior: The device camera captures the user's squatting motion, and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[0388] Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[0389] How the emotion engine works: The emotion engine analyzes the user's facial expressions and recognizes that they are tired.

[0390] Device behavior: In addition to the feedback sent from the server, "Bend your knees to 90 degrees," the device displays the message "Feeling good, take a short break" via voice or text.

[0391] User Actions: The user follows the instructions, corrects the knee angle, and takes breaks at appropriate times.

[0392] Example prompts to input to the generative AI model

[0393] "Please explain the detailed processing steps of a system that analyzes home training in real time and provides appropriate feedback. Furthermore, please provide a concrete example of how the emotion engine analyzes the user's emotional state and adjusts the feedback content."

[0394] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0395] Step 1:

[0396] User training menu selection

[0397] A user starts a fitness app on their smartphone and selects the exercise they want to do from the provided training menu. The input is the user's menu selection, and the output is the selected menu (e.g., squats, push-ups, etc.).

[0398] Step 2:

[0399] User action starts and camera captures

[0400] The user begins training in front of the camera according to the selected training menu. The device starts capturing images in real time through the camera and temporarily stores this video data. The input is the camera image, and the output is the temporarily stored video data.

[0401] Step 3:

[0402] Pose estimation and joint coordinate generation

[0403] The device analyzes the temporarily stored video data using a posture estimation model (e.g., TensorFlow, OpenPose) to generate three-dimensional coordinate data for each of the user's joints. Based on this analysis, position data for each joint is obtained. The input is the temporarily stored video data, and the output is three-dimensional coordinate data.

[0404] Step 4:

[0405] Coordinate data sent to server

[0406] The terminal sends the generated 3D coordinate data to a server via the Internet. The input is 3D coordinate data, and the output is data transmission to the server. Specifically, the terminal sends the data using the network protocol.

[0407] Step 5:

[0408] Server error detection

[0409] The server compares the received 3D coordinate data with data in a standard training motion database to detect errors, such as a knee angle that is shallower than normal. The input is the 3D coordinate data sent to the server, and the output is the content of the error.

[0410] Step 6:

[0411] Feedback Generation

[0412] The server generates appropriate feedback based on the error. This feedback is provided via a speech synthesis service (e.g., Google Text-to-Speech) or as a text message. The input is the error, and the output is the feedback (e.g., "Bend your knees a little deeper").

[0413] Step 7:

[0414] Emotional state analysis

[0415] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state (e.g., tired, unfocused). The input is the user's facial expression data and voice data, and the output is the analysis result of their emotional state.

[0416] Step 8:

[0417] Adjusting feedback

[0418] Based on the analysis results of the emotion engine, the server adjusts the feedback content. For example, if the user is tired, an encouraging message is added. The input is the analysis results of the emotion engine, and the output is the adjusted feedback content.

[0419] Step 9:

[0420] Providing feedback to users

[0421] The terminal provides the user with the feedback sent from the server in the form of voice or text. The user receives this feedback and corrects their training movements. The input is the feedback content from the server, and the output is the feedback display and audio playback for the user. Specifically, the terminal displays a message on the screen and plays audio from the speaker.

[0422] This allows users to receive high-quality training at home and increases their motivation through appropriate feedback provided by the emotion engine.

[0423] (Application example 2)

[0424] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0425] Previous fitness applications provided single, one-way feedback on training movements and nutritional management, and were unable to provide customized feedback based on the user's physical condition or emotional state. Furthermore, they lacked a system that could simultaneously provide advice that took into account the nutritional value of food and the user's emotional state. This resulted in users being unable to receive a more effective fitness experience or health management, leading to a loss of motivation and reduced effectiveness.

[0426] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0427] In this invention, the server includes a means for capturing food images and calculating nutritional values, a means for analyzing a user's emotional state and providing emotion-based feedback, and a means for analyzing training movements and providing feedback, thereby enabling the provision of comprehensive and personalized feedback that takes into account not only the user's training movements but also the nutritional information of meals and their emotional state.

[0428] An "image capture device" is a device that takes real-time images of a user's movements and meals, and generally refers to a smartphone camera.

[0429] An "image recognition device" is a device that analyzes video data captured by an image capture device and estimates the coordinates of the user's joints and food items.

[0430] "Standard training movement data" is a database that stores coordinate data of correct training movements shown by professional trainers.

[0431] The "feedback device" is a device that detects errors based on the analysis results of the image recognition device and provides feedback to the user. The feedback is provided in the form of voice or text.

[0432] An "emotion engine" is a device that analyzes a user's facial expressions and voice to recognize their emotional state and generate appropriate feedback based on that emotion.

[0433] The "nutritional value calculation means" is a means for recognizing food items from food images and calculating their nutritional information.

[0434] The "emotion analysis means" is a means for analyzing the user's emotional state, and recognizes emotions by analyzing the user's facial expressions and voice.

[0435] The "training motion analysis means" is a means for analyzing the training motion of the user and estimating the coordinates of each joint.

[0436] A "feedback providing means" is a means for providing customized feedback to a user based on errors, nutritional information, and emotional state.

[0437] "User" refers to a person who uses the system to train and manage their diet.

[0438] The present invention relates to a system for managing training movements and dietary intake. A specific embodiment for realizing this system will be described below.

[0439] First, the user uses the smartphone camera to record their training movements in real time. An image capture device captures images of the user's movements and meals. The hardware used in this process is the smartphone's built-in camera. The captured images are analyzed by an image recognition device. The image recognition device uses a posture estimation model to estimate the coordinates of each of the user's joints.

[0440] The server compares these coordinate data with a database of standard training movements to detect errors. If an error is detected, the feedback device provides specific advice to the user. This feedback can be provided in the form of voice or text.

[0441] The system also includes a function to calculate the nutritional value of a user's meal. When a user captures an image of the meal using a camera, the image is sent to a server, which uses a food identification model to analyze the meal and calculate the nutritional value.

[0442] Furthermore, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. The results of the emotion analysis are used to customize the feedback. For example, if the user is recognized as tired, an encouraging message such as "Take a short break" will be added to the feedback.

[0443] As a concrete example, when a user is doing squat training, the movement is filmed with a smartphone camera and the angles of the knees and hips are analyzed. If the analysis results differ from the standard data, feedback such as "Bend your knees more" is provided. At the same time, if the user takes a photo of "salad" and "grilled chicken" for lunch, the nutritional information is calculated and advice is provided such as "Salad contains vitamins A and C. You seem tired, so try to consume more vitamins at your next meal."

[0444] Feedback can be further personalized using generative AI models, with specific prompt examples like:

[0445] "Generate a nutritional report and feedback based on the user's emotional state based on the following meal: Meal: Salad, grilled chicken. User's emotional state: Tired."

[0446] This allows the user to receive more effective and personalized support in their training movements and dietary management.

[0447] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0448] Step 1:

[0449] The user uses the smartphone camera to capture images of their workouts and meals. The input is image data captured by the smartphone camera, and the output is a saved image file. Specifically, the user launches a fitness app and takes photos of their workouts and meals with the smartphone camera.

[0450] Step 2:

[0451] The device saves image data captured through the camera. The input is the captured image data, and the output is an image file in the local storage. Specifically, the captured image is saved in the smartphone's internal storage in JPEG format or similar.

[0452] Step 3:

[0453] The device sends the stored image data to an image recognition device for analysis. The input is the stored image file, and the output is the analyzed coordinate data of each joint and food item identification data. Specifically, the image data is sent to an image recognition model on a cloud server and analyzed using a pose estimation model and a food identification model.

[0454] Step 4:

[0455] The server receives and processes the coordinate data and food item identification data sent from the image recognition device. The input is the recognized coordinate data and food item identification data, and the output is the comparison result with standard training data and nutritional information. Specifically, the server refers to a standard training motion database and compares the recognized coordinate data.

[0456] Step 5:

[0457] The server detects errors and generates feedback. The input is the coordinate data comparison result and the nutritional information of the food item, and the output is a feedback message. Specifically, the server identifies errors in the coordinate data and generates feedback such as "Bend your knees more."

[0458] Step 6:

[0459] The server uses an emotion engine to analyze the user's emotional state. The input is the user's facial expression and voice data, and the output is the recognized emotional state. Specifically, the emotion engine recognizes the user's face and analyzes their voice to identify emotional states such as fatigue or stress.

[0460] Step 7:

[0461] The device provides the user with the feedback message received from the server. The input is the feedback message sent from the server, and the output is feedback in the form of voice or text. Specifically, the feedback is presented to the user via the smartphone's speaker or screen.

[0462] Step 8:

[0463] Based on the feedback provided by the user, the system modifies training movements or adjusts diet. The input is a feedback message, and the output is modified movements or dietary choices. Specifically, the user optimizes their movements by "bending their knees more" or takes actions such as "taking more vitamins at their next meal."

[0464] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0465] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0466] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0467] [Second embodiment]

[0468] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0469] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0470] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0471] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0472] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0473] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0474] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0475] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0476] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0477] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0478] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0479] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0480] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback to users while they are training at home, using an image capture device, an image recognition device, a standard training movement database, and a feedback device.

[0481] System configuration

[0482] 1. Image Capture Device

[0483] The image capture device captures the user's training movements in real time, typically using a smartphone camera.

[0484] 2. Image Recognition Device

[0485] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each joint of the user using a posture estimation model.

[0486] 3. Standard training motion database

[0487] The standard training movement database stores coordinate data of correct training movements shown by professional trainers.

[0488] 4. Feedback Devices

[0489] The feedback device is a device for comparing the data obtained from the image recognition device and the standard training motion database and providing feedback to the user when an error is detected. This feedback can be provided in the form of voice or text.

[0490] Program processing

[0491] User Actions

[0492] The user launches a fitness app on their smartphone, selects a training menu and goal, and then begins training in front of the camera.

[0493] Device behavior

[0494] The device captures the user's movements in real time through a camera and temporarily stores the video data. The stored data is analyzed using a posture estimation model to generate three-dimensional coordinate data for each joint. This coordinate data is then sent from the device to a server.

[0495] Server Operation

[0496] The server receives the transmitted coordinate data and compares it with data in a standard training motion database. If an error is detected as a result of the comparison, the content of the error (e.g., knee angle is too shallow, back is bent, etc.) is identified. The server generates appropriate feedback and sends it to the terminal.

[0497] User Feedback

[0498] The device receives feedback from the server and provides it to the user in the form of voice or text, such as instructions like "Bend your knees a little more" or "Keep your back straight." The user can then correct their posture and continue training.

[0499] Specific examples

[0500] For squat training

[0501] User action: The user performs squats in front of the smartphone camera.

[0502] Device behavior: The device camera captures the user's squatting motion, and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[0503] Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[0504] Device behavior: The feedback sent from the server, "Bend your knees to 90 degrees," is notified to the user via voice or text.

[0505] User Action: The user follows the instructions to correct the knee angle and continue squatting.

[0506] In the case of dietary management

[0507] User action: The user uploads a photo of their lunch to the app.

[0508] Device action: The photo is sent to the server.

[0509] Server operation: Analyzes the contents of lunch using image recognition and calculates nutritional value.

[0510] Device behavior: Displays a nutrition report and next meal recommendation to the user, for example, "You're lacking in vitamin C today. Add some fruit to your next meal."

[0511] In this way, by using this system, users can receive high-quality training and dietary management at home. Crawling allows for efficient and accurate training, enabling sustainable health maintenance.

[0512] The processing flow will be explained below.

[0513] Step 1:

[0514] Users launch a fitness app on their smartphone and select a training menu and goals.

[0515] Step 2:

[0516] The device will perform initial setup based on the user's selections, start the camera, and notify the user that they are ready to use the camera.

[0517] Step 3:

[0518] The user stands in front of the camera and starts a training movement, for example, a squat movement.

[0519] Step 4:

[0520] The device's camera captures the user's actions in real time and temporarily stores the video data.

[0521] Step 5:

[0522] The device preprocesses the stored video data, for example changing the resolution or removing noise.

[0523] Step 6:

[0524] The device inputs the preprocessed video data into a posture estimation model, which generates 3D coordinate data for each of the user's joints.

[0525] Step 7:

[0526] The terminal transmits the generated three-dimensional coordinate data to the server.

[0527] Step 8:

[0528] The server receives the transmitted three-dimensional coordinate data and retrieves corresponding data from a standard training motion database.

[0529] Step 9:

[0530] The server compares the 3D coordinate data with standard training motion data to detect errors, such as the angle of the knees or the position of the hips.

[0531] Step 10:

[0532] The server generates appropriate feedback based on the detected error, such as instructions like "bend your knees more."

[0533] Step 11:

[0534] The server transmits the generated feedback to the terminal.

[0535] Step 12:

[0536] The device will then notify the user of the received feedback in the form of voice or text, for example by displaying a message on the screen saying, "Bend your knees to 90 degrees."

[0537] Step 13:

[0538] The user corrects their posture according to the feedback and continues training.

[0539] Step 14:

[0540] After training, the device stores all data locally (e.g., number of repetitions, time, posture accuracy, etc. for each set).

[0541] Step 15:

[0542] The device sends the saved data to the server as training history.

[0543] Step 16:

[0544] Based on the received data, the server updates the user's training history and analyzes long-term effects.

[0545] Step 17:

[0546] When a user uploads a photo of their meal to the app, the device sends the photo to the server.

[0547] Step 18:

[0548] The server uses image recognition to analyze the contents of the meal and calculate its nutritional value.

[0549] Step 19:

[0550] The server generates analysis results and nutritional advice and sends them to the terminal.

[0551] Step 20:

[0552] The device will then display the received nutrition report and advice to the user, for example, a message saying "You are lacking in vitamin C today. Add some fruit to your next meal."

[0553] This allows users to receive accurate feedback when training at home, enabling them to train efficiently and effectively, and also provides appropriate advice on dietary management, improving overall health management.

[0554] Example 1

[0555] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0556] With conventional training systems, when users train at home, it is difficult to maintain proper form and posture, making it difficult to train effectively. Incorrect form also increases the risk of injury. Furthermore, the timing and quality of feedback is insufficient, making it difficult for users to correct their form in real time.

[0557] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0558] In this invention, the server includes means for capturing images of training movements in real time using an image capture device, means for analyzing the captured training movements using an image recognition device and estimating the coordinates of each joint of the user, means for comparing the estimated coordinate data with standard training movement data and detecting errors, means for providing feedback to the user based on the errors, means for transmitting the analysis data from a terminal to the server, and means for providing feedback to the user in audio or text format. This allows users to train effectively and safely at home and correct their form and posture in real time.

[0559] An "image capture device" is a device for capturing images of a user's training movements in real time.

[0560] The "image recognition device" is a device that analyzes photographed training movements and estimates the coordinates of each of the user's joints.

[0561] "Standard training movement data" is a database containing coordinate data of correct training movements.

[0562] The "means for detecting errors" is a mechanism for comparing the estimated coordinate data with standard training motion data to identify errors or inaccurate motions.

[0563] A "means for providing feedback" is a mechanism for providing users with correction instructions in audio or text format based on errors.

[0564] A "terminal" is an electronic device that allows a user to launch a fitness app, capture images of their movements using a camera, and send the analysis data to a server.

[0565] The "server" is a device that receives the analysis data sent from the terminal, compares it with standard training motion data to detect errors, and generates appropriate feedback.

[0566] A "network" is a communications infrastructure for sending and receiving data between terminals and servers.

[0567] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home. The system uses an image capture device, an image recognition device, a standard training movement database, and a feedback device.

[0568] System configuration

[0569] 1. Image Capture Device

[0570] The image capture device captures the user's training movements in real time. Typically, a camera on a mobile information terminal (smartphone) is used. For example, the user performs training movements in front of the smartphone camera.

[0571] 2. Image Recognition Device

[0572] The image recognition device receives the captured video data and estimates the coordinates of each of the user's joints. This analysis uses a posture estimation model such as "MediaPipe." The analysis generates three-dimensional coordinate data for each joint.

[0573] 3. Standard training motion database

[0574] The standard training motion database stores coordinate data of correct training motions shown by professional trainers. This database is managed by a server.

[0575] 4. Feedback Devices

[0576] The feedback device compares the coordinate data obtained from the image recognition device with standard training movement data and provides feedback to the user when an error is detected. The feedback can be in the form of audio or text. For example, instructions such as "Please keep your back more straight" are displayed.

[0577] 5. Terminal

[0578] The device is an electronic device that allows the user to launch the fitness app, capture their movements with a camera, and send the analysis data to the server. The user selects the desired training menu and performs the exercise in front of the camera, and the device captures their movements.

[0579] 6. Server

[0580] The server receives the analysis data transmitted from the terminal, compares it with standard training motion data to detect errors, and generates feedback based on the errors, which is then transmitted to the terminal via a network.

[0581] Specific examples

[0582] For squat training

[0583] 1. User action: The user performs squats in front of the smartphone camera.

[0584] 2. Device movement: The device camera captures the squat movement and analyzes the knee and hip angles through a pose estimation model such as "MediaPipe."

[0585] 3. Device sends the analysis results to the server.

[0586] 4. Server operation: The server compares the data with the standard data and detects, for example, if the knee angle is not appropriate.

[0587] 5. Providing feedback: The server sends feedback such as "Bend your knees to 90 degrees" and notifies the user via voice or text.

[0588] 6. User Correction: The user follows the instructions to correct the knee angle and continue squatting.

[0589] Prompt Sentence Examples

[0590] "Design an AI model that can accurately measure the knee angle during training and provide appropriate feedback. This system analyzes squat training movements in real time and sends instructions to the user if the knee angle is not appropriate. The hardware used will be a smartphone, and the software used will be MediaPipe, which acts as a posture estimation model."

[0591] In this way, by using this system, users can train effectively and safely at home and correct their form and posture in real time.

[0592] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0593] Step 1:

[0594] The user launches a fitness app.

[0595] The user taps to launch a fitness app on their smartphone and selects a training menu and goal. For example, the user selects the "Squat" menu and sets the goal to "20 repetitions." In this step, the user provides input, and the selected training menu and goal are output.

[0596] Step 2:

[0597] The device captures the user's actions.

[0598] The device (smartphone) camera captures the user's training movements in real time and generates video data. When the user starts squatting in front of the camera, the camera captures the movements at 30 frames per second. The input of this step is the user's movements, and the output is real-time video data.

[0599] Step 3:

[0600] The device analyzes the video data.

[0601] The device inputs the captured video data into the posture estimation model "MediaPipe" to generate 3D coordinate data for each joint. The joint coordinate data is extracted through analysis. The input for this step is video data, and the output is 3D coordinate data for each joint.

[0602] Step 4:

[0603] The device sends the analysis data to the server.

[0604] The device converts the generated 3D coordinate data into JSON format and sends it to the server via the network. Specific operations include packaging the coordinate data and setting up communication. The input of this step is the 3D coordinate data, and the output is the data to be sent to the server.

[0605] Step 5:

[0606] The server compares the data and detects errors.

[0607] The server compares the received 3D coordinate data with standard data in a standard training movement database. For example, it compares it with standard movement data for squats to detect if the knee angle is inappropriate. The server receives the data, compares it, and performs an error analysis. The input of this step is the 3D coordinate data, and the output is the error detection result.

[0608] Step 6:

[0609] The server generates the feedback.

[0610] The server generates appropriate feedback based on the error. For example, it generates feedback such as "Bend your knees to 90 degrees." The feedback content is determined and the generated feedback message is output. The input of this step is the error detection result, and the output is the feedback message.

[0611] Step 7:

[0612] The terminal provides feedback to the user.

[0613] The device notifies the user of the feedback received from the server in the form of voice or text. Specific actions include playing voice or displaying on the screen. For example, the instruction "Please bend your knees to 90 degrees" is played to the user by voice. The input of this step is the feedback message, and the output is the feedback notification to the user.

[0614] (Application example 1)

[0615] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0616] Conventional training methods have the problem that it is difficult to maintain correct posture and movement, and that training in incorrect posture can lead to injury or reduced effectiveness. Another problem is that real-time feedback is not provided, making it difficult to immediately correct movements. The purpose of this invention is to solve these problems and enable users to train efficiently and safely.

[0617] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0618] In this invention, the server includes means for capturing images of the user's movements in real time, means for analyzing the captured images and estimating the coordinates of each joint, means for comparing the estimated coordinate data with standard behavior data to detect errors, means for providing feedback to the user based on the errors and displaying the feedback on a video frame, means for transmitting data to the server and receiving feedback, and means for including a posture estimation model used to analyze the joint coordinates. This allows the user to receive accurate feedback in real time and perform safe and effective training.

[0619] An "image capture device" is a device for capturing images of a user's actions in real time.

[0620] An "image recognition device" is a device that analyzes movements captured by an image capture device and estimates the coordinates of each of the user's joints.

[0621] "Standard behavior data" is a database that stores coordinate data of correct training movements.

[0622] An "error" is an inappropriate point in the training movement detected by comparing the estimated joint coordinate data of the user with the standard action data.

[0623] "Feedback" is the instruction or advice provided to the user based on a detected error.

[0624] "Means for displaying on the video frame" refers to a method for visually notifying the user of feedback.

[0625] The "means for transmitting data to the server" is a method for transmitting the user's joint coordinate data to the server.

[0626] The "means for receiving feedback" is a method for receiving feedback sent from the server.

[0627] A "pose estimation model" is a neural network model used to analyze the user's joint coordinate data.

[0628] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home or in a physical training facility. The system uses an image capture device, an image recognition device, a standard behavior database, and a feedback device.

[0629] 1. System Configuration

[0630] 1. Image Capture Device

[0631] The image capture device is a device that captures the user's movements in real time, and a camera on a smartphone or smart glasses is used.

[0632] 2. Image Recognition Device

[0633] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each user's joints using a pose estimation model (TensorFlow, Keras).

[0634] 3. Standard Behavior Database

[0635] The coordinate data of correct training movements is stored in a standard behavior database, and errors are detected by comparing with this database.

[0636] 4. Feedback Devices

[0637] The feedback device is a device that provides appropriate feedback to the user when an error is detected. This feedback is provided to the user in the form of voice or text and is displayed on the video frame.

[0638] 2. Program operation overview

[0639] Device behavior

[0640] 1. The user launches a fitness app on their smartphone or smart glasses.

[0641] 2. Start training movements in front of the camera, which will capture them in real time.

[0642] 3. The acquired video data is temporarily stored and preprocessed (resizing, normalization).

[0643] Server Operation

[0644] 1. Analyzed joint coordinate data is sent from the device to the server.

[0645] 2. The server receives this data and compares it with data in a standard behavior database.

[0646] 3. If the comparison reveals any errors, they are identified and feedback is generated.

[0647] User Feedback

[0648] 1. The device receives the feedback sent from the server and provides it to the user in voice or text format.

[0649] 2. Feedback is also displayed on the video frame, allowing the user to refer to it and correct their training movements.

[0650] 3. Specific Examples

[0651] For squat training

[0652] 1. User action: The user performs squats in front of the smartphone camera.

[0653] 2. Device movement: The camera captures the user's movements and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[0654] 3. Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[0655] 4. Device operation: The feedback sent from the server, "Please bend your knees to 90 degrees," is notified to the user via voice or text.

[0656] 5. User Action: The user follows the instructions to correct the knee angle and continue squatting.

[0657] 4. Example prompts for generative AI models

[0658] TXT

[0659] Imagine a user performing a squat in a physical store. Design an application that analyzes the joint angles in real time based on the video captured by a camera and provides appropriate feedback. The hardware used is a smartphone camera, and the software uses a pose estimation model built with TensorFlow.

[0660] This embodiment allows the user to receive accurate feedback in real time while training safely and effectively.

[0661] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0662] Step 1:

[0663] The user launches a fitness app on their smartphone or smart glasses and selects a training menu.

[0664] Input: Operation instructions from the user

[0665] Output: Training menu selection status

[0666] Specific operation: The user taps the screen of their smartphone or smart glasses to select the desired training menu (e.g., squats).

[0667] Step 2:

[0668] The device activates the camera and captures the user's training movements in real time.

[0669] Input: User action

[0670] Output: Real-time video data

[0671] Specific operation: The device's camera captures the user's movements and obtains the video as data in real time.

[0672] Step 3:

[0673] The device preprocesses (resizes and normalizes) the acquired video data and inputs it into the pose estimation model.

[0674] Input: Real-time video data

[0675] Output: Preprocessed video data, joint coordinate data

[0676] Specific operation: The device resizes and normalizes the video data, then inputs the preprocessed data into a posture estimation model (TensorFlow, Keras) to generate coordinate data for each of the user's joints.

[0677] Step 4:

[0678] The terminal transmits the generated joint coordinate data to the server.

[0679] Input: joint coordinate data

[0680] Output: Data transfer to the server

[0681] Specific operation: The device sends joint coordinate data to the server using an appropriate communication protocol (e.g., HTTP request).

[0682] Step 5:

[0683] The server compares the received joint coordinate data with a standard behavior database to detect errors.

[0684] Input: joint coordinate data, standard action data

[0685] Output: Error detection result

[0686] Specific operation: The server compares the data in the standard behavior database with the joint coordinate data and detects errors (e.g., a shallow knee angle).

[0687] Step 6:

[0688] The server generates feedback based on the detected errors and sends it to the terminal.

[0689] Input: Error detection result

[0690] Output: Feedback data

[0691] Specific operation: Based on the error, the server generates appropriate feedback (e.g., "Bend your knees to 90 degrees") and sends it to the device.

[0692] Step 7:

[0693] The terminal displays the feedback received from the server on the video frame and notifies the user in voice or text format.

[0694] Input: Feedback data

[0695] Output: Audio or text feedback, feedback displayed on the video frame

[0696] What it does: The device overlays feedback text and images on the video frame and simultaneously plays an audio notification to the user.

[0697] Step 8:

[0698] The user uses the feedback to modify their training movements.

[0699] Input: Feedback information

[0700] Output: Corrected training behavior

[0701] Specific actions: The user corrects their training actions according to the feedback provided by the device (e.g., "Bend your knees a little more").

[0702] Through these steps, users can receive appropriate feedback in real time and undergo safe and effective training.

[0703] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0704] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home, using an image capture device, an image recognition device, a standard training movement database, a feedback device, and an emotion engine.

[0705] System configuration

[0706] 1. Image Capture Device

[0707] The image capture device is a device that captures images of the user's training movements in real time, and is generally a smartphone camera.

[0708] 2. Image Recognition Device

[0709] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each joint of the user using a posture estimation model.

[0710] 3. Standard training motion database

[0711] The standard training movement database stores coordinate data of correct training movements shown by professional trainers.

[0712] 4. Feedback Devices

[0713] The feedback device is a device for comparing the data obtained from the image recognition device and the standard training motion database and providing feedback to the user when an error is detected. This feedback can be provided in the form of voice or text.

[0714] 5. Emotion Engine

[0715] The emotion engine is a device that recognizes emotions by analyzing the user's facial expressions and voice, and generates more appropriate and effective feedback based on the user's emotional state.

[0716] Program processing

[0717] User Actions

[0718] The user launches a fitness app on their smartphone, selects a training menu and goal, and then begins training in front of the camera.

[0719] Device behavior

[0720] The device captures the user's movements in real time through a camera and temporarily stores the video data. The stored data is analyzed using a posture estimation model to generate three-dimensional coordinate data for each joint. This coordinate data is then sent from the device to a server.

[0721] Server Operation

[0722] The server receives the coordinate data and compares it with data in a standard training motion database. If an error is detected as a result of the comparison, the nature of the error (e.g., knee angle is too shallow, back is bent, etc.) is identified. The server generates appropriate feedback and sends it to the device.

[0723] Emotion Engine Operation

[0724] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. For example, it can detect when the user is tired, irritated, or unable to concentrate. The emotion engine takes these emotional states into account and adjusts the content of the feedback generated by the server.

[0725] User Feedback

[0726] The device receives feedback from the server and provides it to the user in the form of voice or text. The feedback, adjusted by the emotion engine, includes encouragement and advice based on the user's emotional state. For example, specific messages such as "Great, keep it up" or "Take a short break" are provided.

[0727] Specific examples

[0728] For squat training

[0729] User action: The user performs squats in front of the smartphone camera.

[0730] Device behavior: The device camera captures the user's squatting motion, and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[0731] Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[0732] How the emotion engine works: The emotion engine analyzes the user's facial expressions and recognizes that they are tired.

[0733] Device behavior: In addition to the feedback sent from the server, "Bend your knees to 90 degrees," the device displays the message "Feeling good, take a short break" via voice or text.

[0734] User Actions: The user follows the instructions, corrects the knee angle, and takes breaks at appropriate times.

[0735] In the case of dietary management

[0736] User action: The user uploads a photo of their lunch to the app.

[0737] Device action: The photo is sent to the server.

[0738] Server operation: Analyzes the contents of lunch using image recognition and calculates nutritional value.

[0739] How the emotion engine works: The emotion engine analyzes the user's voice and recognizes that they are depressed.

[0740] What happens on the device: In addition to the nutrition report, the device will display encouraging messages based on the analysis of the emotion engine (e.g., "You're doing great! Make sure you get more vitamin C at your next meal").

[0741] In this way, by using this system, users can receive high-quality training and dietary management at home. Furthermore, the emotion engine provides feedback appropriate to the user's emotional state, increasing motivation for training and dietary management.

[0742] The processing flow will be explained below.

[0743] Step 1:

[0744] Users launch a fitness app on their smartphone and select a training menu and goals.

[0745] Step 2:

[0746] The device will perform initial setup based on the user's selections, start the camera, and notify the user that they are ready to use the camera.

[0747] Step 3:

[0748] The user stands in front of the camera and starts a training movement, for example, a squat movement.

[0749] Step 4:

[0750] The device's camera captures the user's actions in real time and temporarily stores the video data.

[0751] Step 5:

[0752] The device preprocesses the stored video data, for example changing the resolution or removing noise.

[0753] Step 6:

[0754] The device inputs the preprocessed video data into a posture estimation model, which generates 3D coordinate data for each of the user's joints.

[0755] Step 7:

[0756] The terminal transmits the generated three-dimensional coordinate data to the server.

[0757] Step 8:

[0758] The server receives the transmitted three-dimensional coordinate data and retrieves corresponding data from a standard training motion database.

[0759] Step 9:

[0760] The server compares the 3D coordinate data with standard training motion data to detect errors, such as the angle of the knees or the position of the hips.

[0761] Step 10:

[0762] The server generates appropriate feedback based on the detected error, such as instructions like "bend your knees more."

[0763] Step 11:

[0764] The server transmits the generated feedback to the terminal.

[0765] Step 12:

[0766] The device captures the user's facial expressions and voice and sends them to the emotion engine.

[0767] Step 13:

[0768] The emotion engine analyzes the user's facial expressions and voice to recognize emotions, such as fatigue, concentration, and frustration.

[0769] Step 14:

[0770] The emotion engine sends the recognized emotion data to the server.

[0771] Step 15:

[0772] The server receives data from the emotion engine and adjusts the feedback, for example adding a message to a tired user saying "Please take a break."

[0773] Step 16:

[0774] The server sends the adjusted feedback to the terminal.

[0775] Step 17:

[0776] The device will then provide the user with the feedback in the form of voice or text, for example displaying a message such as "Bend your knees to 90 degrees. Feels good, take a short break."

[0777] Step 18:

[0778] The user corrects their posture according to the feedback and continues training.

[0779] Step 19:

[0780] After training, the device stores all data locally (e.g., number of repetitions, time, posture accuracy, etc. for each set).

[0781] Step 20:

[0782] The device sends the saved data to the server as training history.

[0783] Step 21:

[0784] Based on the received data, the server updates the user's training history and analyzes long-term effects.

[0785] Step 22:

[0786] When a user uploads a photo of their meal to the app, the device sends the photo to the server.

[0787] Step 23:

[0788] The server uses image recognition to analyze the contents of the meal and calculate its nutritional value.

[0789] Step 24:

[0790] The emotion engine analyzes the user's voice and recognizes food-related emotions, such as satisfaction, stress, and appetite.

[0791] Step 25:

[0792] The server generates a nutrition report based on the analysis results and emotion engine data and sends it to the device.

[0793] Step 26:

[0794] The device displays the received nutrition report and feedback based on the emotion engine's results. For example, it provides a message like, "You're lacking in vitamin C today. Add some fruit to your next meal. You're doing a great job!"

[0795] This process allows users to receive appropriate feedback in real time during their workouts, enabling them to effectively train at home. Furthermore, the emotion engine provides encouragement and advice appropriate to the user's emotional state, helping to maintain motivation.

[0796] Example 2

[0797] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0798] Conventional training support systems only detect errors in the user's movements and provide feedback, but they have the problem of not being able to provide appropriate feedback that takes into account the user's emotional state. This can make it difficult for users to maintain their motivation and can result in ineffective training.

[0799] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0800] In this invention, the server includes means for capturing images of the user's training movements in real time using an image capture device, means for analyzing the captured training movements using an image recognition device and estimating the coordinates of each of the user's joints, means for comparing the estimated coordinate data with standard training movement data and detecting errors, means for providing feedback to the user based on the errors, and means for analyzing the user's emotional state and adjusting the content of the feedback. This makes it possible to provide appropriate feedback according to the user's emotional state, thereby achieving effective training while increasing motivation.

[0801] An "image capture device" is a device for capturing images of a user's training movements in real time, such as a camera on a mobile device.

[0802] An "image recognition device" is a device that analyzes video data captured by an image capture device and estimates the coordinates of each of the user's joints.

[0803] "Standard training movement data" refers to a database that stores coordinate data of correct training movements shown by professional trainers.

[0804] "Feedback" refers to advice or guidance provided based on errors in the user's training movements, and is provided in audio or text format.

[0805] An "emotion engine" refers to a device or system that has the function of analyzing the user's emotional state and adjusting the content of the feedback.

[0806] "Emotional state" refers to psychological states such as fatigue, concentration, and stress, which are recognized by analyzing the user's facial expressions and voice.

[0807] "Coordinate data" refers to numerical data that indicates the position of each joint of the user in three-dimensional space.

[0808] "Error" refers to the difference in the estimated coordinate data compared to the standard training motion data.

[0809] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home, using an image capture device, an image recognition device, a standard training movement database, a feedback device, and an emotion engine.

[0810] System configuration

[0811] 1. Image Capture Device

[0812] The image capture device is a device that captures images of the user's training movements in real time, and is typically a camera on a mobile device.

[0813] 2. Image Recognition Device

[0814] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each of the user's joints using a pose estimation model such as TensorFlow or OpenPose.

[0815] 3. Standard training motion database

[0816] The standard training movement database stores coordinate data of correct training movements shown by a professional trainer.

[0817] 4. Feedback Devices

[0818] The feedback device compares the data obtained from the image recognition device and the standard training motion database, and provides feedback to the user when an error is detected. The feedback is provided in the form of voice or text using voice recognition technology such as Google Text-to-Speech.

[0819] 5. Emotion Engine

[0820] The emotion engine uses Microsoft Azure's emotion recognition API to analyze the user's facial expressions and voice to recognize their emotional state, and adjusts the content of the feedback generated by the server based on their emotional state.

[0821] Description

[0822] The user launches a fitness app on their mobile device and selects a training menu. They then begin training in front of the camera. The device captures the user's movements in real time through the camera and temporarily stores the video data. This data is analyzed using a pose estimation model such as TensorFlow or OpenPose to generate three-dimensional coordinate data for each joint. This coordinate data is then sent from the device to a server via the Internet.

[0823] The server runs on a cloud platform and receives coordinate data sent from the device. The server compares this data with standard training movement data of professional trainers stored in a database of standard training movements, and identifies errors such as a shallow knee angle or a bent back. If an error is identified, the server generates feedback according to the error, converts it into voice or text using a speech synthesis service (e.g., Google Text-to-Speech), and sends it to the device.

[0824] The emotion engine analyzes the user's facial expressions and voice through the microphone and camera connected to the device, recognizing their emotional state, such as fatigue, concentration, and stress. This emotional data is also sent to the server and reflected in the generated feedback. For example, if the user is tired, an encouraging message such as "Take a short break" will be added.

[0825] The device receives feedback from the server and provides it to the user in the form of voice or text, allowing the user to correct their next actions. This system allows users to receive high-quality training at home and increases their motivation through appropriate feedback provided by the emotion engine.

[0826] Specific examples

[0827] For squat training

[0828] User action: The user performs squats in front of the smartphone camera.

[0829] Device behavior: The device camera captures the user's squatting motion, and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[0830] Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[0831] How the emotion engine works: The emotion engine analyzes the user's facial expressions and recognizes that they are tired.

[0832] Device behavior: In addition to the feedback sent from the server, "Bend your knees to 90 degrees," the device displays the message "Feeling good, take a short break" via voice or text.

[0833] User Actions: The user follows the instructions, corrects the knee angle, and takes breaks at appropriate times.

[0834] Example prompts to input to the generative AI model

[0835] "Please explain the detailed processing steps of a system that analyzes home training in real time and provides appropriate feedback. Furthermore, please provide a concrete example of how the emotion engine analyzes the user's emotional state and adjusts the feedback content."

[0836] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0837] Step 1:

[0838] User training menu selection

[0839] A user starts a fitness app on their smartphone and selects the exercise they want to do from the provided training menu. The input is the user's menu selection, and the output is the selected menu (e.g., squats, push-ups, etc.).

[0840] Step 2:

[0841] User action starts and camera captures

[0842] The user begins training in front of the camera according to the selected training menu. The device starts capturing images in real time through the camera and temporarily stores this video data. The input is the camera image, and the output is the temporarily stored video data.

[0843] Step 3:

[0844] Pose estimation and joint coordinate generation

[0845] The device analyzes the temporarily stored video data using a posture estimation model (e.g., TensorFlow, OpenPose) to generate three-dimensional coordinate data for each of the user's joints. Based on this analysis, position data for each joint is obtained. The input is the temporarily stored video data, and the output is three-dimensional coordinate data.

[0846] Step 4:

[0847] Coordinate data sent to server

[0848] The terminal sends the generated 3D coordinate data to a server via the Internet. The input is 3D coordinate data, and the output is data transmission to the server. Specifically, the terminal sends the data using the network protocol.

[0849] Step 5:

[0850] Server error detection

[0851] The server compares the received 3D coordinate data with data in a standard training motion database to detect errors, such as a knee angle that is shallower than normal. The input is the 3D coordinate data sent to the server, and the output is the content of the error.

[0852] Step 6:

[0853] Feedback Generation

[0854] The server generates appropriate feedback based on the error. This feedback is provided via a speech synthesis service (e.g., Google Text-to-Speech) or as a text message. The input is the error, and the output is the feedback (e.g., "Bend your knees a little deeper").

[0855] Step 7:

[0856] Emotional state analysis

[0857] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state (e.g., tired, unfocused). The input is the user's facial expression data and voice data, and the output is the analysis result of their emotional state.

[0858] Step 8:

[0859] Adjusting feedback

[0860] Based on the analysis results of the emotion engine, the server adjusts the feedback content. For example, if the user is tired, an encouraging message is added. The input is the analysis results of the emotion engine, and the output is the adjusted feedback content.

[0861] Step 9:

[0862] Providing feedback to users

[0863] The terminal provides the user with the feedback sent from the server in the form of voice or text. The user receives this feedback and corrects their training movements. The input is the feedback content from the server, and the output is the feedback display and audio playback for the user. Specifically, the terminal displays a message on the screen and plays audio from the speaker.

[0864] This allows users to receive high-quality training at home and increases their motivation through appropriate feedback provided by the emotion engine.

[0865] (Application example 2)

[0866] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0867] Previous fitness applications provided single, one-way feedback on training movements and nutritional management, and were unable to provide customized feedback based on the user's physical condition or emotional state. Furthermore, they lacked a system that could simultaneously provide advice that took into account the nutritional value of food and the user's emotional state. This resulted in users being unable to receive a more effective fitness experience or health management, leading to a loss of motivation and reduced effectiveness.

[0868] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0869] In this invention, the server includes a means for capturing food images and calculating nutritional values, a means for analyzing a user's emotional state and providing emotion-based feedback, and a means for analyzing training movements and providing feedback, thereby enabling the provision of comprehensive and personalized feedback that takes into account not only the user's training movements but also the nutritional information of meals and their emotional state.

[0870] An "image capture device" is a device that takes real-time images of a user's movements and meals, and generally refers to a smartphone camera.

[0871] An "image recognition device" is a device that analyzes video data captured by an image capture device and estimates the coordinates of the user's joints and food items.

[0872] "Standard training movement data" is a database that stores coordinate data of correct training movements shown by professional trainers.

[0873] The "feedback device" is a device that detects errors based on the analysis results of the image recognition device and provides feedback to the user. The feedback is provided in the form of voice or text.

[0874] An "emotion engine" is a device that analyzes a user's facial expressions and voice to recognize their emotional state and generate appropriate feedback based on that emotion.

[0875] The "nutritional value calculation means" is a means for recognizing food items from food images and calculating their nutritional information.

[0876] The "emotion analysis means" is a means for analyzing the user's emotional state, and recognizes emotions by analyzing the user's facial expressions and voice.

[0877] The "training motion analysis means" is a means for analyzing the training motion of the user and estimating the coordinates of each joint.

[0878] A "feedback providing means" is a means for providing customized feedback to a user based on errors, nutritional information, and emotional state.

[0879] "User" refers to a person who uses the system to train and manage their diet.

[0880] The present invention relates to a system for managing training movements and dietary intake. A specific embodiment for realizing this system will be described below.

[0881] First, the user uses the smartphone camera to record their training movements in real time. An image capture device captures images of the user's movements and meals. The hardware used in this process is the smartphone's built-in camera. The captured images are analyzed by an image recognition device. The image recognition device uses a posture estimation model to estimate the coordinates of each of the user's joints.

[0882] The server compares these coordinate data with a database of standard training movements to detect errors. If an error is detected, the feedback device provides specific advice to the user. This feedback can be provided in the form of voice or text.

[0883] The system also includes a function to calculate the nutritional value of a user's meal. When a user captures an image of the meal using a camera, the image is sent to a server, which uses a food identification model to analyze the meal and calculate the nutritional value.

[0884] Furthermore, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. The results of the emotion analysis are used to customize the feedback. For example, if the user is recognized as tired, an encouraging message such as "Take a short break" will be added to the feedback.

[0885] As a concrete example, when a user is doing squat training, the movement is filmed with a smartphone camera and the angles of the knees and hips are analyzed. If the analysis results differ from the standard data, feedback such as "Bend your knees more" is provided. At the same time, if the user takes a photo of "salad" and "grilled chicken" for lunch, the nutritional information is calculated and advice is provided such as "Salad contains vitamins A and C. You seem tired, so try to consume more vitamins at your next meal."

[0886] Feedback can be further personalized using generative AI models, with specific prompt examples like:

[0887] "Generate a nutritional report and feedback based on the user's emotional state based on the following meal: Meal: Salad, grilled chicken. User's emotional state: Tired."

[0888] This allows the user to receive more effective and personalized support in their training movements and dietary management.

[0889] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0890] Step 1:

[0891] The user uses the smartphone camera to capture images of their workouts and meals. The input is image data captured by the smartphone camera, and the output is a saved image file. Specifically, the user launches a fitness app and takes photos of their workouts and meals with the smartphone camera.

[0892] Step 2:

[0893] The device saves image data captured through the camera. The input is the captured image data, and the output is an image file in the local storage. Specifically, the captured image is saved in the smartphone's internal storage in JPEG format or similar.

[0894] Step 3:

[0895] The device sends the stored image data to an image recognition device for analysis. The input is the stored image file, and the output is the analyzed coordinate data of each joint and food item identification data. Specifically, the image data is sent to an image recognition model on a cloud server and analyzed using a pose estimation model and a food identification model.

[0896] Step 4:

[0897] The server receives and processes the coordinate data and food item identification data sent from the image recognition device. The input is the recognized coordinate data and food item identification data, and the output is the comparison result with standard training data and nutritional information. Specifically, the server refers to a standard training motion database and compares the recognized coordinate data.

[0898] Step 5:

[0899] The server detects errors and generates feedback. The input is the coordinate data comparison result and the nutritional information of the food item, and the output is a feedback message. Specifically, the server identifies errors in the coordinate data and generates feedback such as "Bend your knees more."

[0900] Step 6:

[0901] The server uses an emotion engine to analyze the user's emotional state. The input is the user's facial expression and voice data, and the output is the recognized emotional state. Specifically, the emotion engine recognizes the user's face and analyzes their voice to identify emotional states such as fatigue or stress.

[0902] Step 7:

[0903] The device provides the user with the feedback message received from the server. The input is the feedback message sent from the server, and the output is feedback in the form of voice or text. Specifically, the feedback is presented to the user via the smartphone's speaker or screen.

[0904] Step 8:

[0905] Based on the feedback provided by the user, the system modifies training movements or adjusts diet. The input is a feedback message, and the output is modified movements or dietary choices. Specifically, the user optimizes their movements by "bending their knees more" or takes actions such as "taking more vitamins at their next meal."

[0906] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0907] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0908] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0909] [Third embodiment]

[0910] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0911] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0912] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0913] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0914] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0915] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0916] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0917] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0918] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0919] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0920] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0921] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0922] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback to users while they are training at home, using an image capture device, an image recognition device, a standard training movement database, and a feedback device.

[0923] System configuration

[0924] 1. Image Capture Device

[0925] The image capture device captures the user's training movements in real time, typically using a smartphone camera.

[0926] 2. Image Recognition Device

[0927] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each joint of the user using a posture estimation model.

[0928] 3. Standard training motion database

[0929] The standard training movement database stores coordinate data of correct training movements shown by professional trainers.

[0930] 4. Feedback Devices

[0931] The feedback device is a device for comparing the data obtained from the image recognition device and the standard training motion database and providing feedback to the user when an error is detected. This feedback can be provided in the form of voice or text.

[0932] Program processing

[0933] User Actions

[0934] The user launches a fitness app on their smartphone, selects a training menu and goal, and then begins training in front of the camera.

[0935] Device behavior

[0936] The device captures the user's movements in real time through a camera and temporarily stores the video data. The stored data is analyzed using a posture estimation model to generate three-dimensional coordinate data for each joint. This coordinate data is then sent from the device to a server.

[0937] Server Operation

[0938] The server receives the transmitted coordinate data and compares it with data in a standard training motion database. If an error is detected as a result of the comparison, the content of the error (e.g., knee angle is too shallow, back is bent, etc.) is identified. The server generates appropriate feedback and sends it to the terminal.

[0939] User Feedback

[0940] The device receives feedback from the server and provides it to the user in the form of voice or text, such as instructions like "Bend your knees a little more" or "Keep your back straight." The user can then correct their posture and continue training.

[0941] Specific examples

[0942] For squat training

[0943] User action: The user performs squats in front of the smartphone camera.

[0944] Device behavior: The device camera captures the user's squatting motion, and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[0945] Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[0946] Device behavior: The feedback sent from the server, "Bend your knees to 90 degrees," is notified to the user via voice or text.

[0947] User Action: The user follows the instructions to correct the knee angle and continue squatting.

[0948] In the case of dietary management

[0949] User action: The user uploads a photo of their lunch to the app.

[0950] Device action: The photo is sent to the server.

[0951] Server operation: Analyzes the contents of lunch using image recognition and calculates nutritional value.

[0952] Device behavior: Displays a nutrition report and next meal recommendation to the user, for example, "You're lacking in vitamin C today. Add some fruit to your next meal."

[0953] In this way, by using this system, users can receive high-quality training and dietary management at home. Crawling allows for efficient and accurate training, enabling sustainable health maintenance.

[0954] The processing flow will be explained below.

[0955] Step 1:

[0956] Users launch a fitness app on their smartphone and select a training menu and goals.

[0957] Step 2:

[0958] The device will perform initial setup based on the user's selections, start the camera, and notify the user that they are ready to use the camera.

[0959] Step 3:

[0960] The user stands in front of the camera and starts a training movement, for example, a squat movement.

[0961] Step 4:

[0962] The device's camera captures the user's actions in real time and temporarily stores the video data.

[0963] Step 5:

[0964] The device preprocesses the stored video data, for example changing the resolution or removing noise.

[0965] Step 6:

[0966] The device inputs the preprocessed video data into a posture estimation model, which generates 3D coordinate data for each of the user's joints.

[0967] Step 7:

[0968] The terminal transmits the generated three-dimensional coordinate data to the server.

[0969] Step 8:

[0970] The server receives the transmitted three-dimensional coordinate data and retrieves corresponding data from a standard training motion database.

[0971] Step 9:

[0972] The server compares the 3D coordinate data with standard training motion data to detect errors, such as the angle of the knees or the position of the hips.

[0973] Step 10:

[0974] The server generates appropriate feedback based on the detected error, such as instructions like "bend your knees more."

[0975] Step 11:

[0976] The server transmits the generated feedback to the terminal.

[0977] Step 12:

[0978] The device will then notify the user of the received feedback in the form of voice or text, for example by displaying a message on the screen saying, "Bend your knees to 90 degrees."

[0979] Step 13:

[0980] The user corrects their posture according to the feedback and continues training.

[0981] Step 14:

[0982] After training, the device stores all data locally (e.g., number of repetitions, time, posture accuracy, etc. for each set).

[0983] Step 15:

[0984] The device sends the saved data to the server as training history.

[0985] Step 16:

[0986] Based on the received data, the server updates the user's training history and analyzes long-term effects.

[0987] Step 17:

[0988] When a user uploads a photo of their meal to the app, the device sends the photo to the server.

[0989] Step 18:

[0990] The server uses image recognition to analyze the contents of the meal and calculate its nutritional value.

[0991] Step 19:

[0992] The server generates analysis results and nutritional advice and sends them to the terminal.

[0993] Step 20:

[0994] The device will then display the received nutrition report and advice to the user, for example, a message saying "You are lacking in vitamin C today. Add some fruit to your next meal."

[0995] This allows users to receive accurate feedback when training at home, enabling them to train efficiently and effectively, and also provides appropriate advice on dietary management, improving overall health management.

[0996] Example 1

[0997] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0998] With conventional training systems, when users train at home, it is difficult to maintain proper form and posture, making it difficult to train effectively. Incorrect form also increases the risk of injury. Furthermore, the timing and quality of feedback is insufficient, making it difficult for users to correct their form in real time.

[0999] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1000] In this invention, the server includes means for capturing images of training movements in real time using an image capture device, means for analyzing the captured training movements using an image recognition device and estimating the coordinates of each joint of the user, means for comparing the estimated coordinate data with standard training movement data and detecting errors, means for providing feedback to the user based on the errors, means for transmitting the analysis data from a terminal to the server, and means for providing feedback to the user in audio or text format. This allows users to train effectively and safely at home and correct their form and posture in real time.

[1001] An "image capture device" is a device for capturing images of a user's training movements in real time.

[1002] The "image recognition device" is a device that analyzes photographed training movements and estimates the coordinates of each of the user's joints.

[1003] "Standard training movement data" is a database containing coordinate data of correct training movements.

[1004] The "means for detecting errors" is a mechanism for comparing the estimated coordinate data with standard training motion data to identify errors or inaccurate motions.

[1005] A "means for providing feedback" is a mechanism for providing users with correction instructions in audio or text format based on errors.

[1006] A "terminal" is an electronic device that allows a user to launch a fitness app, capture images of their movements using a camera, and send the analysis data to a server.

[1007] The "server" is a device that receives the analysis data sent from the terminal, compares it with standard training motion data to detect errors, and generates appropriate feedback.

[1008] A "network" is a communications infrastructure for sending and receiving data between terminals and servers.

[1009] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home. The system uses an image capture device, an image recognition device, a standard training movement database, and a feedback device.

[1010] System configuration

[1011] 1. Image Capture Device

[1012] The image capture device captures the user's training movements in real time. Typically, a camera on a mobile information terminal (smartphone) is used. For example, the user performs training movements in front of the smartphone camera.

[1013] 2. Image Recognition Device

[1014] The image recognition device receives the captured video data and estimates the coordinates of each of the user's joints. This analysis uses a posture estimation model such as "MediaPipe." The analysis generates three-dimensional coordinate data for each joint.

[1015] 3. Standard training motion database

[1016] The standard training motion database stores coordinate data of correct training motions shown by professional trainers. This database is managed by a server.

[1017] 4. Feedback Devices

[1018] The feedback device compares the coordinate data obtained from the image recognition device with standard training movement data and provides feedback to the user when an error is detected. The feedback can be in the form of audio or text. For example, instructions such as "Please keep your back more straight" are displayed.

[1019] 5. Terminal

[1020] The device is an electronic device that allows the user to launch the fitness app, capture their movements with a camera, and send the analysis data to the server. The user selects the desired training menu and performs the exercise in front of the camera, and the device captures their movements.

[1021] 6. Server

[1022] The server receives the analysis data transmitted from the terminal, compares it with standard training motion data to detect errors, and generates feedback based on the errors, which is then transmitted to the terminal via a network.

[1023] Specific examples

[1024] For squat training

[1025] 1. User action: The user performs squats in front of the smartphone camera.

[1026] 2. Device movement: The device camera captures the squat movement and analyzes the knee and hip angles through a pose estimation model such as "MediaPipe."

[1027] 3. Device sends the analysis results to the server.

[1028] 4. Server operation: The server compares the data with the standard data and detects, for example, if the knee angle is not appropriate.

[1029] 5. Providing feedback: The server sends feedback such as "Bend your knees to 90 degrees" and notifies the user via voice or text.

[1030] 6. User Correction: The user follows the instructions to correct the knee angle and continue squatting.

[1031] Prompt Sentence Examples

[1032] "Design an AI model that can accurately measure the knee angle during training and provide appropriate feedback. This system analyzes squat training movements in real time and sends instructions to the user if the knee angle is not appropriate. The hardware used will be a smartphone, and the software used will be MediaPipe, which acts as a posture estimation model."

[1033] In this way, by using this system, users can train effectively and safely at home and correct their form and posture in real time.

[1034] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1035] Step 1:

[1036] The user launches a fitness app.

[1037] The user taps to launch a fitness app on their smartphone and selects a training menu and goal. For example, the user selects the "Squat" menu and sets the goal to "20 repetitions." In this step, the user provides input, and the selected training menu and goal are output.

[1038] Step 2:

[1039] The device captures the user's actions.

[1040] The device (smartphone) camera captures the user's training movements in real time and generates video data. When the user starts squatting in front of the camera, the camera captures the movements at 30 frames per second. The input of this step is the user's movements, and the output is real-time video data.

[1041] Step 3:

[1042] The device analyzes the video data.

[1043] The device inputs the captured video data into the posture estimation model "MediaPipe" to generate 3D coordinate data for each joint. The joint coordinate data is extracted through analysis. The input for this step is video data, and the output is 3D coordinate data for each joint.

[1044] Step 4:

[1045] The device sends the analysis data to the server.

[1046] The device converts the generated 3D coordinate data into JSON format and sends it to the server via the network. Specific operations include packaging the coordinate data and setting up communication. The input of this step is the 3D coordinate data, and the output is the data to be sent to the server.

[1047] Step 5:

[1048] The server compares the data and detects errors.

[1049] The server compares the received 3D coordinate data with standard data in a standard training movement database. For example, it compares it with standard movement data for squats to detect if the knee angle is inappropriate. The server receives the data, compares it, and performs an error analysis. The input of this step is the 3D coordinate data, and the output is the error detection result.

[1050] Step 6:

[1051] The server generates the feedback.

[1052] The server generates appropriate feedback based on the error. For example, it generates feedback such as "Bend your knees to 90 degrees." The feedback content is determined and the generated feedback message is output. The input of this step is the error detection result, and the output is the feedback message.

[1053] Step 7:

[1054] The terminal provides feedback to the user.

[1055] The device notifies the user of the feedback received from the server in the form of voice or text. Specific actions include playing voice or displaying on the screen. For example, the instruction "Please bend your knees to 90 degrees" is played to the user by voice. The input of this step is the feedback message, and the output is the feedback notification to the user.

[1056] (Application example 1)

[1057] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1058] Conventional training methods have the problem that it is difficult to maintain correct posture and movement, and that training in incorrect posture can lead to injury or reduced effectiveness. Another problem is that real-time feedback is not provided, making it difficult to immediately correct movements. The purpose of this invention is to solve these problems and enable users to train efficiently and safely.

[1059] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1060] In this invention, the server includes means for capturing images of the user's movements in real time, means for analyzing the captured images and estimating the coordinates of each joint, means for comparing the estimated coordinate data with standard behavior data to detect errors, means for providing feedback to the user based on the errors and displaying the feedback on a video frame, means for transmitting data to the server and receiving feedback, and means for including a posture estimation model used to analyze the joint coordinates. This allows the user to receive accurate feedback in real time and perform safe and effective training.

[1061] An "image capture device" is a device for capturing images of a user's actions in real time.

[1062] An "image recognition device" is a device that analyzes movements captured by an image capture device and estimates the coordinates of each of the user's joints.

[1063] "Standard behavior data" is a database that stores coordinate data of correct training movements.

[1064] An "error" is an inappropriate point in the training movement detected by comparing the estimated joint coordinate data of the user with the standard action data.

[1065] "Feedback" is the instruction or advice provided to the user based on a detected error.

[1066] "Means for displaying on the video frame" refers to a method for visually notifying the user of feedback.

[1067] The "means for transmitting data to the server" is a method for transmitting the user's joint coordinate data to the server.

[1068] The "means for receiving feedback" is a method for receiving feedback sent from the server.

[1069] A "pose estimation model" is a neural network model used to analyze the user's joint coordinate data.

[1070] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home or in a physical training facility. The system uses an image capture device, an image recognition device, a standard behavior database, and a feedback device.

[1071] 1. System Configuration

[1072] 1. Image Capture Device

[1073] The image capture device is a device that captures the user's movements in real time, and a camera on a smartphone or smart glasses is used.

[1074] 2. Image Recognition Device

[1075] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each user's joints using a pose estimation model (TensorFlow, Keras).

[1076] 3. Standard Behavior Database

[1077] The coordinate data of correct training movements is stored in a standard behavior database, and errors are detected by comparing with this database.

[1078] 4. Feedback Devices

[1079] The feedback device is a device that provides appropriate feedback to the user when an error is detected. This feedback is provided to the user in the form of voice or text and is displayed on the video frame.

[1080] 2. Program operation overview

[1081] Device behavior

[1082] 1. The user launches a fitness app on their smartphone or smart glasses.

[1083] 2. Start training movements in front of the camera, which will capture them in real time.

[1084] 3. The acquired video data is temporarily stored and preprocessed (resizing, normalization).

[1085] Server Operation

[1086] 1. Analyzed joint coordinate data is sent from the device to the server.

[1087] 2. The server receives this data and compares it with data in a standard behavior database.

[1088] 3. If the comparison reveals any errors, they are identified and feedback is generated.

[1089] User Feedback

[1090] 1. The device receives the feedback sent from the server and provides it to the user in voice or text format.

[1091] 2. Feedback is also displayed on the video frame, allowing the user to refer to it and correct their training movements.

[1092] 3. Specific Examples

[1093] For squat training

[1094] 1. User action: The user performs squats in front of the smartphone camera.

[1095] 2. Device movement: The camera captures the user's movements and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[1096] 3. Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[1097] 4. Device operation: The feedback sent from the server, "Please bend your knees to 90 degrees," is notified to the user via voice or text.

[1098] 5. User Action: The user follows the instructions to correct the knee angle and continue squatting.

[1099] 4. Example prompts for generative AI models

[1100] TXT

[1101] Imagine a user performing a squat in a physical store. Design an application that analyzes the joint angles in real time based on the video captured by a camera and provides appropriate feedback. The hardware used is a smartphone camera, and the software uses a pose estimation model built with TensorFlow.

[1102] This embodiment allows the user to receive accurate feedback in real time while training safely and effectively.

[1103] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1104] Step 1:

[1105] The user launches a fitness app on their smartphone or smart glasses and selects a training menu.

[1106] Input: Operation instructions from the user

[1107] Output: Training menu selection status

[1108] Specific operation: The user taps the screen of their smartphone or smart glasses to select the desired training menu (e.g., squats).

[1109] Step 2:

[1110] The device activates the camera and captures the user's training movements in real time.

[1111] Input: User action

[1112] Output: Real-time video data

[1113] Specific operation: The device's camera captures the user's movements and obtains the video as data in real time.

[1114] Step 3:

[1115] The device preprocesses (resizes and normalizes) the acquired video data and inputs it into the pose estimation model.

[1116] Input: Real-time video data

[1117] Output: Preprocessed video data, joint coordinate data

[1118] Specific operation: The device resizes and normalizes the video data, then inputs the preprocessed data into a posture estimation model (TensorFlow, Keras) to generate coordinate data for each of the user's joints.

[1119] Step 4:

[1120] The terminal transmits the generated joint coordinate data to the server.

[1121] Input: joint coordinate data

[1122] Output: Data transfer to the server

[1123] Specific operation: The device sends joint coordinate data to the server using an appropriate communication protocol (e.g., HTTP request).

[1124] Step 5:

[1125] The server compares the received joint coordinate data with a standard behavior database to detect errors.

[1126] Input: joint coordinate data, standard action data

[1127] Output: Error detection result

[1128] Specific operation: The server compares the data in the standard behavior database with the joint coordinate data and detects errors (e.g., a shallow knee angle).

[1129] Step 6:

[1130] The server generates feedback based on the detected errors and sends it to the terminal.

[1131] Input: Error detection result

[1132] Output: Feedback data

[1133] Specific operation: Based on the error, the server generates appropriate feedback (e.g., "Bend your knees to 90 degrees") and sends it to the device.

[1134] Step 7:

[1135] The terminal displays the feedback received from the server on the video frame and notifies the user in voice or text format.

[1136] Input: Feedback data

[1137] Output: Audio or text feedback, feedback displayed on the video frame

[1138] What it does: The device overlays feedback text and images on the video frame and simultaneously plays an audio notification to the user.

[1139] Step 8:

[1140] The user uses the feedback to modify their training movements.

[1141] Input: Feedback information

[1142] Output: Corrected training behavior

[1143] Specific actions: The user corrects their training actions according to the feedback provided by the device (e.g., "Bend your knees a little more").

[1144] Through these steps, users can receive appropriate feedback in real time and undergo safe and effective training.

[1145] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1146] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home, using an image capture device, an image recognition device, a standard training movement database, a feedback device, and an emotion engine.

[1147] System configuration

[1148] 1. Image Capture Device

[1149] The image capture device is a device that captures images of the user's training movements in real time, and is generally a smartphone camera.

[1150] 2. Image Recognition Device

[1151] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each joint of the user using a posture estimation model.

[1152] 3. Standard training motion database

[1153] The standard training movement database stores coordinate data of correct training movements shown by professional trainers.

[1154] 4. Feedback Devices

[1155] The feedback device is a device for comparing the data obtained from the image recognition device and the standard training motion database and providing feedback to the user when an error is detected. This feedback can be provided in the form of voice or text.

[1156] 5. Emotion Engine

[1157] The emotion engine is a device that recognizes emotions by analyzing the user's facial expressions and voice, and generates more appropriate and effective feedback based on the user's emotional state.

[1158] Program processing

[1159] User Actions

[1160] The user launches a fitness app on their smartphone, selects a training menu and goal, and then begins training in front of the camera.

[1161] Device behavior

[1162] The device captures the user's movements in real time through a camera and temporarily stores the video data. The stored data is analyzed using a posture estimation model to generate three-dimensional coordinate data for each joint. This coordinate data is then sent from the device to a server.

[1163] Server Operation

[1164] The server receives the coordinate data and compares it with data in a standard training motion database. If an error is detected as a result of the comparison, the nature of the error (e.g., knee angle is too shallow, back is bent, etc.) is identified. The server generates appropriate feedback and sends it to the device.

[1165] Emotion Engine Operation

[1166] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. For example, it can detect when the user is tired, irritated, or unable to concentrate. The emotion engine takes these emotional states into account and adjusts the content of the feedback generated by the server.

[1167] User Feedback

[1168] The device receives feedback from the server and provides it to the user in the form of voice or text. The feedback, adjusted by the emotion engine, includes encouragement and advice based on the user's emotional state. For example, specific messages such as "Great, keep it up" or "Take a short break" are provided.

[1169] Specific examples

[1170] For squat training

[1171] User action: The user performs squats in front of the smartphone camera.

[1172] Device behavior: The device camera captures the user's squatting motion, and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[1173] Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[1174] How the emotion engine works: The emotion engine analyzes the user's facial expressions and recognizes that they are tired.

[1175] Device behavior: In addition to the feedback sent from the server, "Bend your knees to 90 degrees," the device displays the message "Feeling good, take a short break" via voice or text.

[1176] User Actions: The user follows the instructions, corrects the knee angle, and takes breaks at appropriate times.

[1177] In the case of dietary management

[1178] User action: The user uploads a photo of their lunch to the app.

[1179] Device action: The photo is sent to the server.

[1180] Server operation: Analyzes the contents of lunch using image recognition and calculates nutritional value.

[1181] How the emotion engine works: The emotion engine analyzes the user's voice and recognizes that they are depressed.

[1182] What happens on the device: In addition to the nutrition report, the device will display encouraging messages based on the analysis of the emotion engine (e.g., "You're doing great! Make sure you get more vitamin C at your next meal").

[1183] In this way, by using this system, users can receive high-quality training and dietary management at home. Furthermore, the emotion engine provides feedback appropriate to the user's emotional state, increasing motivation for training and dietary management.

[1184] The processing flow will be explained below.

[1185] Step 1:

[1186] Users launch a fitness app on their smartphone and select a training menu and goals.

[1187] Step 2:

[1188] The device will perform initial setup based on the user's selections, start the camera, and notify the user that they are ready to use the camera.

[1189] Step 3:

[1190] The user stands in front of the camera and starts a training movement, for example, a squat movement.

[1191] Step 4:

[1192] The device's camera captures the user's actions in real time and temporarily stores the video data.

[1193] Step 5:

[1194] The device preprocesses the stored video data, for example changing the resolution or removing noise.

[1195] Step 6:

[1196] The device inputs the preprocessed video data into a posture estimation model, which generates 3D coordinate data for each of the user's joints.

[1197] Step 7:

[1198] The terminal transmits the generated three-dimensional coordinate data to the server.

[1199] Step 8:

[1200] The server receives the transmitted three-dimensional coordinate data and retrieves corresponding data from a standard training motion database.

[1201] Step 9:

[1202] The server compares the 3D coordinate data with standard training motion data to detect errors, such as the angle of the knees or the position of the hips.

[1203] Step 10:

[1204] The server generates appropriate feedback based on the detected error, such as instructions like "bend your knees more."

[1205] Step 11:

[1206] The server transmits the generated feedback to the terminal.

[1207] Step 12:

[1208] The device captures the user's facial expressions and voice and sends them to the emotion engine.

[1209] Step 13:

[1210] The emotion engine analyzes the user's facial expressions and voice to recognize emotions, such as fatigue, concentration, and frustration.

[1211] Step 14:

[1212] The emotion engine sends the recognized emotion data to the server.

[1213] Step 15:

[1214] The server receives data from the emotion engine and adjusts the feedback, for example adding a message to a tired user saying "Please take a break."

[1215] Step 16:

[1216] The server sends the adjusted feedback to the terminal.

[1217] Step 17:

[1218] The device will then provide the user with the feedback in the form of voice or text, for example displaying a message such as "Bend your knees to 90 degrees. Feels good, take a short break."

[1219] Step 18:

[1220] The user corrects their posture according to the feedback and continues training.

[1221] Step 19:

[1222] After training, the device stores all data locally (e.g., number of repetitions, time, posture accuracy, etc. for each set).

[1223] Step 20:

[1224] The device sends the saved data to the server as training history.

[1225] Step 21:

[1226] Based on the received data, the server updates the user's training history and analyzes long-term effects.

[1227] Step 22:

[1228] When a user uploads a photo of their meal to the app, the device sends the photo to the server.

[1229] Step 23:

[1230] The server uses image recognition to analyze the contents of the meal and calculate its nutritional value.

[1231] Step 24:

[1232] The emotion engine analyzes the user's voice and recognizes food-related emotions, such as satisfaction, stress, and appetite.

[1233] Step 25:

[1234] The server generates a nutrition report based on the analysis results and emotion engine data and sends it to the device.

[1235] Step 26:

[1236] The device displays the received nutrition report and feedback based on the emotion engine's results. For example, it provides a message like, "You're lacking in vitamin C today. Add some fruit to your next meal. You're doing a great job!"

[1237] This process allows users to receive appropriate feedback in real time during their workouts, enabling them to effectively train at home. Furthermore, the emotion engine provides encouragement and advice appropriate to the user's emotional state, helping to maintain motivation.

[1238] Example 2

[1239] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1240] Conventional training support systems only detect errors in the user's movements and provide feedback, but they have the problem of not being able to provide appropriate feedback that takes into account the user's emotional state. This can make it difficult for users to maintain their motivation and can result in ineffective training.

[1241] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1242] In this invention, the server includes means for capturing images of the user's training movements in real time using an image capture device, means for analyzing the captured training movements using an image recognition device and estimating the coordinates of each of the user's joints, means for comparing the estimated coordinate data with standard training movement data and detecting errors, means for providing feedback to the user based on the errors, and means for analyzing the user's emotional state and adjusting the content of the feedback. This makes it possible to provide appropriate feedback according to the user's emotional state, thereby achieving effective training while increasing motivation.

[1243] An "image capture device" is a device for capturing images of a user's training movements in real time, such as a camera on a mobile device.

[1244] An "image recognition device" is a device that analyzes video data captured by an image capture device and estimates the coordinates of each of the user's joints.

[1245] "Standard training movement data" refers to a database that stores coordinate data of correct training movements shown by professional trainers.

[1246] "Feedback" refers to advice or guidance provided based on errors in the user's training movements, and is provided in audio or text format.

[1247] An "emotion engine" refers to a device or system that has the function of analyzing the user's emotional state and adjusting the content of the feedback.

[1248] "Emotional state" refers to psychological states such as fatigue, concentration, and stress, which are recognized by analyzing the user's facial expressions and voice.

[1249] "Coordinate data" refers to numerical data that indicates the position of each joint of the user in three-dimensional space.

[1250] "Error" refers to the difference in the estimated coordinate data compared to the standard training motion data.

[1251] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home, using an image capture device, an image recognition device, a standard training movement database, a feedback device, and an emotion engine.

[1252] System configuration

[1253] 1. Image Capture Device

[1254] The image capture device is a device that captures images of the user's training movements in real time, and is typically a camera on a mobile device.

[1255] 2. Image Recognition Device

[1256] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each of the user's joints using a pose estimation model such as TensorFlow or OpenPose.

[1257] 3. Standard training motion database

[1258] The standard training movement database stores coordinate data of correct training movements shown by a professional trainer.

[1259] 4. Feedback Devices

[1260] The feedback device compares the data obtained from the image recognition device and the standard training motion database, and provides feedback to the user when an error is detected. The feedback is provided in the form of voice or text using voice recognition technology such as Google Text-to-Speech.

[1261] 5. Emotion Engine

[1262] The emotion engine uses Microsoft Azure's emotion recognition API to analyze the user's facial expressions and voice to recognize their emotional state, and adjusts the content of the feedback generated by the server based on their emotional state.

[1263] Description

[1264] The user launches a fitness app on their mobile device and selects a training menu. They then begin training in front of the camera. The device captures the user's movements in real time through the camera and temporarily stores the video data. This data is analyzed using a pose estimation model such as TensorFlow or OpenPose to generate three-dimensional coordinate data for each joint. This coordinate data is then sent from the device to a server via the Internet.

[1265] The server runs on a cloud platform and receives coordinate data sent from the device. The server compares this data with standard training movement data of professional trainers stored in a database of standard training movements, and identifies errors such as a shallow knee angle or a bent back. If an error is identified, the server generates feedback according to the error, converts it into voice or text using a speech synthesis service (e.g., Google Text-to-Speech), and sends it to the device.

[1266] The emotion engine analyzes the user's facial expressions and voice through the microphone and camera connected to the device, recognizing their emotional state, such as fatigue, concentration, and stress. This emotional data is also sent to the server and reflected in the generated feedback. For example, if the user is tired, an encouraging message such as "Take a short break" will be added.

[1267] The device receives feedback from the server and provides it to the user in the form of voice or text, allowing the user to correct their next actions. This system allows users to receive high-quality training at home and increases their motivation through appropriate feedback provided by the emotion engine.

[1268] Specific examples

[1269] For squat training

[1270] User action: The user performs squats in front of the smartphone camera.

[1271] Device behavior: The device camera captures the user's squatting motion, and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[1272] Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[1273] How the emotion engine works: The emotion engine analyzes the user's facial expressions and recognizes that they are tired.

[1274] Device behavior: In addition to the feedback sent from the server, "Bend your knees to 90 degrees," the device displays the message "Feeling good, take a short break" via voice or text.

[1275] User Actions: The user follows the instructions, corrects the knee angle, and takes breaks at appropriate times.

[1276] Example prompts to input to the generative AI model

[1277] "Please explain the detailed processing steps of a system that analyzes home training in real time and provides appropriate feedback. Furthermore, please provide a concrete example of how the emotion engine analyzes the user's emotional state and adjusts the feedback content."

[1278] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1279] Step 1:

[1280] User training menu selection

[1281] A user starts a fitness app on their smartphone and selects the exercise they want to do from the provided training menu. The input is the user's menu selection, and the output is the selected menu (e.g., squats, push-ups, etc.).

[1282] Step 2:

[1283] User action starts and camera captures

[1284] The user begins training in front of the camera according to the selected training menu. The device starts capturing images in real time through the camera and temporarily stores this video data. The input is the camera image, and the output is the temporarily stored video data.

[1285] Step 3:

[1286] Pose estimation and joint coordinate generation

[1287] The device analyzes the temporarily stored video data using a posture estimation model (e.g., TensorFlow, OpenPose) to generate three-dimensional coordinate data for each of the user's joints. Based on this analysis, position data for each joint is obtained. The input is the temporarily stored video data, and the output is three-dimensional coordinate data.

[1288] Step 4:

[1289] Coordinate data sent to server

[1290] The terminal sends the generated 3D coordinate data to a server via the Internet. The input is 3D coordinate data, and the output is data transmission to the server. Specifically, the terminal sends the data using the network protocol.

[1291] Step 5:

[1292] Server error detection

[1293] The server compares the received 3D coordinate data with data in a standard training motion database to detect errors, such as a knee angle that is shallower than normal. The input is the 3D coordinate data sent to the server, and the output is the content of the error.

[1294] Step 6:

[1295] Feedback Generation

[1296] The server generates appropriate feedback based on the error. This feedback is provided via a speech synthesis service (e.g., Google Text-to-Speech) or as a text message. The input is the error, and the output is the feedback (e.g., "Bend your knees a little deeper").

[1297] Step 7:

[1298] Emotional state analysis

[1299] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state (e.g., tired, unfocused). The input is the user's facial expression data and voice data, and the output is the analysis result of their emotional state.

[1300] Step 8:

[1301] Adjusting feedback

[1302] Based on the analysis results of the emotion engine, the server adjusts the feedback content. For example, if the user is tired, an encouraging message is added. The input is the analysis results of the emotion engine, and the output is the adjusted feedback content.

[1303] Step 9:

[1304] Providing feedback to users

[1305] The terminal provides the user with the feedback sent from the server in the form of voice or text. The user receives this feedback and corrects their training movements. The input is the feedback content from the server, and the output is the feedback display and audio playback for the user. Specifically, the terminal displays a message on the screen and plays audio from the speaker.

[1306] This allows users to receive high-quality training at home and increases their motivation through appropriate feedback provided by the emotion engine.

[1307] (Application example 2)

[1308] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1309] Previous fitness applications provided single, one-way feedback on training movements and nutritional management, and were unable to provide customized feedback based on the user's physical condition or emotional state. Furthermore, they lacked a system that could simultaneously provide advice that took into account the nutritional value of food and the user's emotional state. This resulted in users being unable to receive a more effective fitness experience or health management, leading to a loss of motivation and reduced effectiveness.

[1310] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1311] In this invention, the server includes a means for capturing food images and calculating nutritional values, a means for analyzing a user's emotional state and providing emotion-based feedback, and a means for analyzing training movements and providing feedback, thereby enabling the provision of comprehensive and personalized feedback that takes into account not only the user's training movements but also the nutritional information of meals and their emotional state.

[1312] An "image capture device" is a device that takes real-time images of a user's movements and meals, and generally refers to a smartphone camera.

[1313] An "image recognition device" is a device that analyzes video data captured by an image capture device and estimates the coordinates of the user's joints and food items.

[1314] "Standard training movement data" is a database that stores coordinate data of correct training movements shown by professional trainers.

[1315] The "feedback device" is a device that detects errors based on the analysis results of the image recognition device and provides feedback to the user. The feedback is provided in the form of voice or text.

[1316] An "emotion engine" is a device that analyzes a user's facial expressions and voice to recognize their emotional state and generate appropriate feedback based on that emotion.

[1317] The "nutritional value calculation means" is a means for recognizing food items from food images and calculating their nutritional information.

[1318] The "emotion analysis means" is a means for analyzing the user's emotional state, and recognizes emotions by analyzing the user's facial expressions and voice.

[1319] The "training motion analysis means" is a means for analyzing the training motion of the user and estimating the coordinates of each joint.

[1320] A "feedback providing means" is a means for providing customized feedback to a user based on errors, nutritional information, and emotional state.

[1321] "User" refers to a person who uses the system to train and manage their diet.

[1322] The present invention relates to a system for managing training movements and dietary intake. A specific embodiment for realizing this system will be described below.

[1323] First, the user uses the smartphone camera to record their training movements in real time. An image capture device captures images of the user's movements and meals. The hardware used in this process is the smartphone's built-in camera. The captured images are analyzed by an image recognition device. The image recognition device uses a posture estimation model to estimate the coordinates of each of the user's joints.

[1324] The server compares these coordinate data with a database of standard training movements to detect errors. If an error is detected, the feedback device provides specific advice to the user. This feedback can be provided in the form of voice or text.

[1325] The system also includes a function to calculate the nutritional value of a user's meal. When a user captures an image of the meal using a camera, the image is sent to a server, which uses a food identification model to analyze the meal and calculate the nutritional value.

[1326] Furthermore, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. The results of the emotion analysis are used to customize the feedback. For example, if the user is recognized as tired, an encouraging message such as "Take a short break" will be added to the feedback.

[1327] As a concrete example, when a user is doing squat training, the movement is filmed with a smartphone camera and the angles of the knees and hips are analyzed. If the analysis results differ from the standard data, feedback such as "Bend your knees more" is provided. At the same time, if the user takes a photo of "salad" and "grilled chicken" for lunch, the nutritional information is calculated and advice is provided such as "Salad contains vitamins A and C. You seem tired, so try to consume more vitamins at your next meal."

[1328] Feedback can be further personalized using generative AI models, with specific prompt examples like:

[1329] "Generate a nutritional report and feedback based on the user's emotional state based on the following meal: Meal: Salad, grilled chicken. User's emotional state: Tired."

[1330] This allows the user to receive more effective and personalized support in their training movements and dietary management.

[1331] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1332] Step 1:

[1333] The user uses the smartphone camera to capture images of their workouts and meals. The input is image data captured by the smartphone camera, and the output is a saved image file. Specifically, the user launches a fitness app and takes photos of their workouts and meals with the smartphone camera.

[1334] Step 2:

[1335] The device saves image data captured through the camera. The input is the captured image data, and the output is an image file in the local storage. Specifically, the captured image is saved in the smartphone's internal storage in JPEG format or similar.

[1336] Step 3:

[1337] The device sends the stored image data to an image recognition device for analysis. The input is the stored image file, and the output is the analyzed coordinate data of each joint and food item identification data. Specifically, the image data is sent to an image recognition model on a cloud server and analyzed using a pose estimation model and a food identification model.

[1338] Step 4:

[1339] The server receives and processes the coordinate data and food item identification data sent from the image recognition device. The input is the recognized coordinate data and food item identification data, and the output is the comparison result with standard training data and nutritional information. Specifically, the server refers to a standard training motion database and compares the recognized coordinate data.

[1340] Step 5:

[1341] The server detects errors and generates feedback. The input is the coordinate data comparison result and the nutritional information of the food item, and the output is a feedback message. Specifically, the server identifies errors in the coordinate data and generates feedback such as "Bend your knees more."

[1342] Step 6:

[1343] The server uses an emotion engine to analyze the user's emotional state. The input is the user's facial expression and voice data, and the output is the recognized emotional state. Specifically, the emotion engine recognizes the user's face and analyzes their voice to identify emotional states such as fatigue or stress.

[1344] Step 7:

[1345] The device provides the user with the feedback message received from the server. The input is the feedback message sent from the server, and the output is feedback in the form of voice or text. Specifically, the feedback is presented to the user via the smartphone's speaker or screen.

[1346] Step 8:

[1347] Based on the feedback provided by the user, the system modifies training movements or adjusts diet. The input is a feedback message, and the output is modified movements or dietary choices. Specifically, the user optimizes their movements by "bending their knees more" or takes actions such as "taking more vitamins at their next meal."

[1348] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1349] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1350] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1351] [Fourth embodiment]

[1352] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1353] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1354] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1355] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1356] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1357] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1358] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1359] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1360] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1361] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1362] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1363] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1364] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1365] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback to users while they are training at home, using an image capture device, an image recognition device, a standard training movement database, and a feedback device.

[1366] System configuration

[1367] 1. Image Capture Device

[1368] The image capture device captures the user's training movements in real time, typically using a smartphone camera.

[1369] 2. Image Recognition Device

[1370] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each joint of the user using a posture estimation model.

[1371] 3. Standard training motion database

[1372] The standard training movement database stores coordinate data of correct training movements shown by professional trainers.

[1373] 4. Feedback Devices

[1374] The feedback device is a device for comparing the data obtained from the image recognition device and the standard training motion database and providing feedback to the user when an error is detected. This feedback can be provided in the form of voice or text.

[1375] Program processing

[1376] User Actions

[1377] The user launches a fitness app on their smartphone, selects a training menu and goal, and then begins training in front of the camera.

[1378] Device behavior

[1379] The device captures the user's movements in real time through a camera and temporarily stores the video data. The stored data is analyzed using a posture estimation model to generate three-dimensional coordinate data for each joint. This coordinate data is then sent from the device to a server.

[1380] Server Operation

[1381] The server receives the transmitted coordinate data and compares it with data in a standard training motion database. If an error is detected as a result of the comparison, the content of the error (e.g., knee angle is too shallow, back is bent, etc.) is identified. The server generates appropriate feedback and sends it to the terminal.

[1382] User Feedback

[1383] The device receives feedback from the server and provides it to the user in the form of voice or text, such as instructions like "Bend your knees a little more" or "Keep your back straight." The user can then correct their posture and continue training.

[1384] Specific examples

[1385] For squat training

[1386] User action: The user performs squats in front of the smartphone camera.

[1387] Device behavior: The device camera captures the user's squatting motion, and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[1388] Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[1389] Device behavior: The feedback sent from the server, "Bend your knees to 90 degrees," is notified to the user via voice or text.

[1390] User Action: The user follows the instructions to correct the knee angle and continue squatting.

[1391] In the case of dietary management

[1392] User action: The user uploads a photo of their lunch to the app.

[1393] Device action: The photo is sent to the server.

[1394] Server operation: Analyzes the contents of lunch using image recognition and calculates nutritional value.

[1395] Device behavior: Displays a nutrition report and next meal recommendation to the user, for example, "You're lacking in vitamin C today. Add some fruit to your next meal."

[1396] In this way, by using this system, users can receive high-quality training and dietary management at home. Crawling allows for efficient and accurate training, enabling sustainable health maintenance.

[1397] The processing flow will be explained below.

[1398] Step 1:

[1399] Users launch a fitness app on their smartphone and select a training menu and goals.

[1400] Step 2:

[1401] The device will perform initial setup based on the user's selections, start the camera, and notify the user that they are ready to use the camera.

[1402] Step 3:

[1403] The user stands in front of the camera and starts a training movement, for example, a squat movement.

[1404] Step 4:

[1405] The device's camera captures the user's actions in real time and temporarily stores the video data.

[1406] Step 5:

[1407] The device preprocesses the stored video data, for example changing the resolution or removing noise.

[1408] Step 6:

[1409] The device inputs the preprocessed video data into a posture estimation model, which generates 3D coordinate data for each of the user's joints.

[1410] Step 7:

[1411] The terminal transmits the generated three-dimensional coordinate data to the server.

[1412] Step 8:

[1413] The server receives the transmitted three-dimensional coordinate data and retrieves corresponding data from a standard training motion database.

[1414] Step 9:

[1415] The server compares the 3D coordinate data with standard training motion data to detect errors, such as the angle of the knees or the position of the hips.

[1416] Step 10:

[1417] The server generates appropriate feedback based on the detected error, such as instructions like "bend your knees more."

[1418] Step 11:

[1419] The server transmits the generated feedback to the terminal.

[1420] Step 12:

[1421] The device will then notify the user of the received feedback in the form of voice or text, for example by displaying a message on the screen saying, "Bend your knees to 90 degrees."

[1422] Step 13:

[1423] The user corrects their posture according to the feedback and continues training.

[1424] Step 14:

[1425] After training, the device stores all data locally (e.g., number of repetitions, time, posture accuracy, etc. for each set).

[1426] Step 15:

[1427] The device sends the saved data to the server as training history.

[1428] Step 16:

[1429] Based on the received data, the server updates the user's training history and analyzes long-term effects.

[1430] Step 17:

[1431] When a user uploads a photo of their meal to the app, the device sends the photo to the server.

[1432] Step 18:

[1433] The server uses image recognition to analyze the contents of the meal and calculate its nutritional value.

[1434] Step 19:

[1435] The server generates analysis results and nutritional advice and sends them to the terminal.

[1436] Step 20:

[1437] The device will then display the received nutrition report and advice to the user, for example, a message saying "You are lacking in vitamin C today. Add some fruit to your next meal."

[1438] This allows users to receive accurate feedback when training at home, enabling them to train efficiently and effectively, and also provides appropriate advice on dietary management, improving overall health management.

[1439] Example 1

[1440] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1441] With conventional training systems, when users train at home, it is difficult to maintain proper form and posture, making it difficult to train effectively. Incorrect form also increases the risk of injury. Furthermore, the timing and quality of feedback is insufficient, making it difficult for users to correct their form in real time.

[1442] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1443] In this invention, the server includes means for capturing images of training movements in real time using an image capture device, means for analyzing the captured training movements using an image recognition device and estimating the coordinates of each joint of the user, means for comparing the estimated coordinate data with standard training movement data and detecting errors, means for providing feedback to the user based on the errors, means for transmitting the analysis data from a terminal to the server, and means for providing feedback to the user in audio or text format. This allows users to train effectively and safely at home and correct their form and posture in real time.

[1444] An "image capture device" is a device for capturing images of a user's training movements in real time.

[1445] The "image recognition device" is a device that analyzes photographed training movements and estimates the coordinates of each of the user's joints.

[1446] "Standard training movement data" is a database containing coordinate data of correct training movements.

[1447] The "means for detecting errors" is a mechanism for comparing the estimated coordinate data with standard training motion data to identify errors or inaccurate motions.

[1448] A "means for providing feedback" is a mechanism for providing users with correction instructions in audio or text format based on errors.

[1449] A "terminal" is an electronic device that allows a user to launch a fitness app, capture images of their movements using a camera, and send the analysis data to a server.

[1450] The "server" is a device that receives the analysis data sent from the terminal, compares it with standard training motion data to detect errors, and generates appropriate feedback.

[1451] A "network" is a communications infrastructure for sending and receiving data between terminals and servers.

[1452] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home. The system uses an image capture device, an image recognition device, a standard training movement database, and a feedback device.

[1453] System configuration

[1454] 1. Image Capture Device

[1455] The image capture device captures the user's training movements in real time. Typically, a camera on a mobile information terminal (smartphone) is used. For example, the user performs training movements in front of the smartphone camera.

[1456] 2. Image Recognition Device

[1457] The image recognition device receives the captured video data and estimates the coordinates of each of the user's joints. This analysis uses a posture estimation model such as "MediaPipe." The analysis generates three-dimensional coordinate data for each joint.

[1458] 3. Standard training motion database

[1459] The standard training motion database stores coordinate data of correct training motions shown by professional trainers. This database is managed by a server.

[1460] 4. Feedback Devices

[1461] The feedback device compares the coordinate data obtained from the image recognition device with standard training movement data and provides feedback to the user when an error is detected. The feedback can be in the form of audio or text. For example, instructions such as "Please keep your back more straight" are displayed.

[1462] 5. Terminal

[1463] The device is an electronic device that allows the user to launch the fitness app, capture their movements with a camera, and send the analysis data to the server. The user selects the desired training menu and performs the exercise in front of the camera, and the device captures their movements.

[1464] 6. Server

[1465] The server receives the analysis data transmitted from the terminal, compares it with standard training motion data to detect errors, and generates feedback based on the errors, which is then transmitted to the terminal via a network.

[1466] Specific examples

[1467] For squat training

[1468] 1. User action: The user performs squats in front of the smartphone camera.

[1469] 2. Device movement: The device camera captures the squat movement and analyzes the knee and hip angles through a pose estimation model such as "MediaPipe."

[1470] 3. Device sends the analysis results to the server.

[1471] 4. Server operation: The server compares the data with the standard data and detects, for example, if the knee angle is not appropriate.

[1472] 5. Providing feedback: The server sends feedback such as "Bend your knees to 90 degrees" and notifies the user via voice or text.

[1473] 6. User Correction: The user follows the instructions to correct the knee angle and continue squatting.

[1474] Prompt Sentence Examples

[1475] "Design an AI model that can accurately measure the knee angle during training and provide appropriate feedback. This system analyzes squat training movements in real time and sends instructions to the user if the knee angle is not appropriate. The hardware used will be a smartphone, and the software used will be MediaPipe, which acts as a posture estimation model."

[1476] In this way, by using this system, users can train effectively and safely at home and correct their form and posture in real time.

[1477] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1478] Step 1:

[1479] The user launches a fitness app.

[1480] The user taps to launch a fitness app on their smartphone and selects a training menu and goal. For example, the user selects the "Squat" menu and sets the goal to "20 repetitions." In this step, the user provides input, and the selected training menu and goal are output.

[1481] Step 2:

[1482] The device captures the user's actions.

[1483] The device (smartphone) camera captures the user's training movements in real time and generates video data. When the user starts squatting in front of the camera, the camera captures the movements at 30 frames per second. The input of this step is the user's movements, and the output is real-time video data.

[1484] Step 3:

[1485] The device analyzes the video data.

[1486] The device inputs the captured video data into the posture estimation model "MediaPipe" to generate 3D coordinate data for each joint. The joint coordinate data is extracted through analysis. The input for this step is video data, and the output is 3D coordinate data for each joint.

[1487] Step 4:

[1488] The device sends the analysis data to the server.

[1489] The device converts the generated 3D coordinate data into JSON format and sends it to the server via the network. Specific operations include packaging the coordinate data and setting up communication. The input of this step is the 3D coordinate data, and the output is the data to be sent to the server.

[1490] Step 5:

[1491] The server compares the data and detects errors.

[1492] The server compares the received 3D coordinate data with standard data in a standard training movement database. For example, it compares it with standard movement data for squats to detect if the knee angle is inappropriate. The server receives the data, compares it, and performs an error analysis. The input of this step is the 3D coordinate data, and the output is the error detection result.

[1493] Step 6:

[1494] The server generates the feedback.

[1495] The server generates appropriate feedback based on the error. For example, it generates feedback such as "Bend your knees to 90 degrees." The feedback content is determined and the generated feedback message is output. The input of this step is the error detection result, and the output is the feedback message.

[1496] Step 7:

[1497] The terminal provides feedback to the user.

[1498] The device notifies the user of the feedback received from the server in the form of voice or text. Specific actions include playing voice or displaying on the screen. For example, the instruction "Please bend your knees to 90 degrees" is played to the user by voice. The input of this step is the feedback message, and the output is the feedback notification to the user.

[1499] (Application example 1)

[1500] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1501] Conventional training methods have the problem that it is difficult to maintain correct posture and movement, and that training in incorrect posture can lead to injury or reduced effectiveness. Another problem is that real-time feedback is not provided, making it difficult to immediately correct movements. The purpose of this invention is to solve these problems and enable users to train efficiently and safely.

[1502] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1503] In this invention, the server includes means for capturing images of the user's movements in real time, means for analyzing the captured images and estimating the coordinates of each joint, means for comparing the estimated coordinate data with standard behavior data to detect errors, means for providing feedback to the user based on the errors and displaying the feedback on a video frame, means for transmitting data to the server and receiving feedback, and means for including a posture estimation model used to analyze the joint coordinates. This allows the user to receive accurate feedback in real time and perform safe and effective training.

[1504] An "image capture device" is a device for capturing images of a user's actions in real time.

[1505] An "image recognition device" is a device that analyzes movements captured by an image capture device and estimates the coordinates of each of the user's joints.

[1506] "Standard behavior data" is a database that stores coordinate data of correct training movements.

[1507] An "error" is an inappropriate point in the training movement detected by comparing the estimated joint coordinate data of the user with the standard action data.

[1508] "Feedback" is the instruction or advice provided to the user based on a detected error.

[1509] "Means for displaying on the video frame" refers to a method for visually notifying the user of feedback.

[1510] The "means for transmitting data to the server" is a method for transmitting the user's joint coordinate data to the server.

[1511] The "means for receiving feedback" is a method for receiving feedback sent from the server.

[1512] A "pose estimation model" is a neural network model used to analyze the user's joint coordinate data.

[1513] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home or in a physical training facility. The system uses an image capture device, an image recognition device, a standard behavior database, and a feedback device.

[1514] 1. System Configuration

[1515] 1. Image Capture Device

[1516] The image capture device is a device that captures the user's movements in real time, and a camera on a smartphone or smart glasses is used.

[1517] 2. Image Recognition Device

[1518] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each user's joints using a pose estimation model (TensorFlow, Keras).

[1519] 3. Standard Behavior Database

[1520] The coordinate data of correct training movements is stored in a standard behavior database, and errors are detected by comparing with this database.

[1521] 4. Feedback Devices

[1522] The feedback device is a device that provides appropriate feedback to the user when an error is detected. This feedback is provided to the user in the form of voice or text and is displayed on the video frame.

[1523] 2. Program operation overview

[1524] Device behavior

[1525] 1. The user launches a fitness app on their smartphone or smart glasses.

[1526] 2. Start training movements in front of the camera, which will capture them in real time.

[1527] 3. The acquired video data is temporarily stored and preprocessed (resizing, normalization).

[1528] Server Operation

[1529] 1. Analyzed joint coordinate data is sent from the device to the server.

[1530] 2. The server receives this data and compares it with data in a standard behavior database.

[1531] 3. If the comparison reveals any errors, they are identified and feedback is generated.

[1532] User Feedback

[1533] 1. The device receives the feedback sent from the server and provides it to the user in voice or text format.

[1534] 2. Feedback is also displayed on the video frame, allowing the user to refer to it and correct their training movements.

[1535] 3. Specific Examples

[1536] For squat training

[1537] 1. User action: The user performs squats in front of the smartphone camera.

[1538] 2. Device movement: The camera captures the user's movements and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[1539] 3. Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[1540] 4. Device operation: The feedback sent from the server, "Please bend your knees to 90 degrees," is notified to the user via voice or text.

[1541] 5. User Action: The user follows the instructions to correct the knee angle and continue squatting.

[1542] 4. Example prompts for generative AI models

[1543] TXT

[1544] Imagine a user performing a squat in a physical store. Design an application that analyzes the joint angles in real time based on the video captured by a camera and provides appropriate feedback. The hardware used is a smartphone camera, and the software uses a pose estimation model built with TensorFlow.

[1545] This embodiment allows the user to receive accurate feedback in real time while training safely and effectively.

[1546] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1547] Step 1:

[1548] The user launches a fitness app on their smartphone or smart glasses and selects a training menu.

[1549] Input: Operation instructions from the user

[1550] Output: Training menu selection status

[1551] Specific operation: The user taps the screen of their smartphone or smart glasses to select the desired training menu (e.g., squats).

[1552] Step 2:

[1553] The device activates the camera and captures the user's training movements in real time.

[1554] Input: User action

[1555] Output: Real-time video data

[1556] Specific operation: The device's camera captures the user's movements and obtains the video as data in real time.

[1557] Step 3:

[1558] The device preprocesses (resizes and normalizes) the acquired video data and inputs it into the pose estimation model.

[1559] Input: Real-time video data

[1560] Output: Preprocessed video data, joint coordinate data

[1561] Specific operation: The device resizes and normalizes the video data, then inputs the preprocessed data into a posture estimation model (TensorFlow, Keras) to generate coordinate data for each of the user's joints.

[1562] Step 4:

[1563] The terminal transmits the generated joint coordinate data to the server.

[1564] Input: joint coordinate data

[1565] Output: Data transfer to the server

[1566] Specific operation: The device sends joint coordinate data to the server using an appropriate communication protocol (e.g., HTTP request).

[1567] Step 5:

[1568] The server compares the received joint coordinate data with a standard behavior database to detect errors.

[1569] Input: joint coordinate data, standard action data

[1570] Output: Error detection result

[1571] Specific operation: The server compares the data in the standard behavior database with the joint coordinate data and detects errors (e.g., a shallow knee angle).

[1572] Step 6:

[1573] The server generates feedback based on the detected errors and sends it to the terminal.

[1574] Input: Error detection result

[1575] Output: Feedback data

[1576] Specific operation: Based on the error, the server generates appropriate feedback (e.g., "Bend your knees to 90 degrees") and sends it to the device.

[1577] Step 7:

[1578] The terminal displays the feedback received from the server on the video frame and notifies the user in voice or text format.

[1579] Input: Feedback data

[1580] Output: Audio or text feedback, feedback displayed on the video frame

[1581] What it does: The device overlays feedback text and images on the video frame and simultaneously plays an audio notification to the user.

[1582] Step 8:

[1583] The user uses the feedback to modify their training movements.

[1584] Input: Feedback information

[1585] Output: Corrected training behavior

[1586] Specific actions: The user corrects their training actions according to the feedback provided by the device (e.g., "Bend your knees a little more").

[1587] Through these steps, users can receive appropriate feedback in real time and undergo safe and effective training.

[1588] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1589] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home, using an image capture device, an image recognition device, a standard training movement database, a feedback device, and an emotion engine.

[1590] System configuration

[1591] 1. Image Capture Device

[1592] The image capture device is a device that captures images of the user's training movements in real time, and is generally a smartphone camera.

[1593] 2. Image Recognition Device

[1594] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each joint of the user using a posture estimation model.

[1595] 3. Standard training motion database

[1596] The standard training movement database stores coordinate data of correct training movements shown by professional trainers.

[1597] 4. Feedback Devices

[1598] The feedback device is a device for comparing the data obtained from the image recognition device and the standard training motion database and providing feedback to the user when an error is detected. This feedback can be provided in the form of voice or text.

[1599] 5. Emotion Engine

[1600] The emotion engine is a device that recognizes emotions by analyzing the user's facial expressions and voice, and generates more appropriate and effective feedback based on the user's emotional state.

[1601] Program processing

[1602] User Actions

[1603] The user launches a fitness app on their smartphone, selects a training menu and goal, and then begins training in front of the camera.

[1604] Device behavior

[1605] The device captures the user's movements in real time through a camera and temporarily stores the video data. The stored data is analyzed using a posture estimation model to generate three-dimensional coordinate data for each joint. This coordinate data is then sent from the device to a server.

[1606] Server Operation

[1607] The server receives the coordinate data and compares it with data in a standard training motion database. If an error is detected as a result of the comparison, the nature of the error (e.g., knee angle is too shallow, back is bent, etc.) is identified. The server generates appropriate feedback and sends it to the device.

[1608] Emotion Engine Operation

[1609] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. For example, it can detect when the user is tired, irritated, or unable to concentrate. The emotion engine takes these emotional states into account and adjusts the content of the feedback generated by the server.

[1610] User Feedback

[1611] The device receives feedback from the server and provides it to the user in the form of voice or text. The feedback, adjusted by the emotion engine, includes encouragement and advice based on the user's emotional state. For example, specific messages such as "Great, keep it up" or "Take a short break" are provided.

[1612] Specific examples

[1613] For squat training

[1614] User action: The user performs squats in front of the smartphone camera.

[1615] Device behavior: The device camera captures the user's squatting motion, and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[1616] Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[1617] How the emotion engine works: The emotion engine analyzes the user's facial expressions and recognizes that they are tired.

[1618] Device behavior: In addition to the feedback sent from the server, "Bend your knees to 90 degrees," the device displays the message "Feeling good, take a short break" via voice or text.

[1619] User Actions: The user follows the instructions, corrects the knee angle, and takes breaks at appropriate times.

[1620] In the case of dietary management

[1621] User action: The user uploads a photo of their lunch to the app.

[1622] Device action: The photo is sent to the server.

[1623] Server operation: Analyzes the contents of lunch using image recognition and calculates nutritional value.

[1624] How the emotion engine works: The emotion engine analyzes the user's voice and recognizes that they are depressed.

[1625] What happens on the device: In addition to the nutrition report, the device will display encouraging messages based on the analysis of the emotion engine (e.g., "You're doing great! Make sure you get more vitamin C at your next meal").

[1626] In this way, by using this system, users can receive high-quality training and dietary management at home. Furthermore, the emotion engine provides feedback appropriate to the user's emotional state, increasing motivation for training and dietary management.

[1627] The processing flow will be explained below.

[1628] Step 1:

[1629] Users launch a fitness app on their smartphone and select a training menu and goals.

[1630] Step 2:

[1631] The device will perform initial setup based on the user's selections, start the camera, and notify the user that they are ready to use the camera.

[1632] Step 3:

[1633] The user stands in front of the camera and starts a training movement, for example, a squat movement.

[1634] Step 4:

[1635] The device's camera captures the user's actions in real time and temporarily stores the video data.

[1636] Step 5:

[1637] The device preprocesses the stored video data, for example changing the resolution or removing noise.

[1638] Step 6:

[1639] The device inputs the preprocessed video data into a posture estimation model, which generates 3D coordinate data for each of the user's joints.

[1640] Step 7:

[1641] The terminal transmits the generated three-dimensional coordinate data to the server.

[1642] Step 8:

[1643] The server receives the transmitted three-dimensional coordinate data and retrieves corresponding data from a standard training motion database.

[1644] Step 9:

[1645] The server compares the 3D coordinate data with standard training motion data to detect errors, such as the angle of the knees or the position of the hips.

[1646] Step 10:

[1647] The server generates appropriate feedback based on the detected error, such as instructions like "bend your knees more."

[1648] Step 11:

[1649] The server transmits the generated feedback to the terminal.

[1650] Step 12:

[1651] The device captures the user's facial expressions and voice and sends them to the emotion engine.

[1652] Step 13:

[1653] The emotion engine analyzes the user's facial expressions and voice to recognize emotions, such as fatigue, concentration, and frustration.

[1654] Step 14:

[1655] The emotion engine sends the recognized emotion data to the server.

[1656] Step 15:

[1657] The server receives data from the emotion engine and adjusts the feedback, for example adding a message to a tired user saying "Please take a break."

[1658] Step 16:

[1659] The server sends the adjusted feedback to the terminal.

[1660] Step 17:

[1661] The device will then provide the user with the feedback in the form of voice or text, for example displaying a message such as "Bend your knees to 90 degrees. Feels good, take a short break."

[1662] Step 18:

[1663] The user corrects their posture according to the feedback and continues training.

[1664] Step 19:

[1665] After training, the device stores all data locally (e.g., number of repetitions, time, posture accuracy, etc. for each set).

[1666] Step 20:

[1667] The device sends the saved data to the server as training history.

[1668] Step 21:

[1669] Based on the received data, the server updates the user's training history and analyzes long-term effects.

[1670] Step 22:

[1671] When a user uploads a photo of their meal to the app, the device sends the photo to the server.

[1672] Step 23:

[1673] The server uses image recognition to analyze the contents of the meal and calculate its nutritional value.

[1674] Step 24:

[1675] The emotion engine analyzes the user's voice and recognizes food-related emotions, such as satisfaction, stress, and appetite.

[1676] Step 25:

[1677] The server generates a nutrition report based on the analysis results and emotion engine data and sends it to the device.

[1678] Step 26:

[1679] The device displays the received nutrition report and feedback based on the emotion engine's results. For example, it provides a message like, "You're lacking in vitamin C today. Add some fruit to your next meal. You're doing a great job!"

[1680] This process allows users to receive appropriate feedback in real time during their workouts, enabling them to effectively train at home. Furthermore, the emotion engine provides encouragement and advice appropriate to the user's emotional state, helping to maintain motivation.

[1681] Example 2

[1682] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1683] Conventional training support systems only detect errors in the user's movements and provide feedback, but they have the problem of not being able to provide appropriate feedback that takes into account the user's emotional state. This can make it difficult for users to maintain their motivation and can result in ineffective training.

[1684] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1685] In this invention, the server includes means for capturing images of the user's training movements in real time using an image capture device, means for analyzing the captured training movements using an image recognition device and estimating the coordinates of each of the user's joints, means for comparing the estimated coordinate data with standard training movement data and detecting errors, means for providing feedback to the user based on the errors, and means for analyzing the user's emotional state and adjusting the content of the feedback. This makes it possible to provide appropriate feedback according to the user's emotional state, thereby achieving effective training while increasing motivation.

[1686] An "image capture device" is a device for capturing images of a user's training movements in real time, such as a camera on a mobile device.

[1687] An "image recognition device" is a device that analyzes video data captured by an image capture device and estimates the coordinates of each of the user's joints.

[1688] "Standard training movement data" refers to a database that stores coordinate data of correct training movements shown by professional trainers.

[1689] "Feedback" refers to advice or guidance provided based on errors in the user's training movements, and is provided in audio or text format.

[1690] An "emotion engine" refers to a device or system that has the function of analyzing the user's emotional state and adjusting the content of the feedback.

[1691] "Emotional state" refers to psychological states such as fatigue, concentration, and stress, which are recognized by analyzing the user's facial expressions and voice.

[1692] "Coordinate data" refers to numerical data that indicates the position of each joint of the user in three-dimensional space.

[1693] "Error" refers to the difference in the estimated coordinate data compared to the standard training motion data.

[1694] The present invention relates to a system that analyzes training movements in real time and provides appropriate feedback when a user is training at home, using an image capture device, an image recognition device, a standard training movement database, a feedback device, and an emotion engine.

[1695] System configuration

[1696] 1. Image Capture Device

[1697] The image capture device is a device that captures images of the user's training movements in real time, and is typically a camera on a mobile device.

[1698] 2. Image Recognition Device

[1699] The image recognition device receives video data captured by the image capture device and estimates the coordinates of each of the user's joints using a pose estimation model such as TensorFlow or OpenPose.

[1700] 3. Standard training motion database

[1701] The standard training movement database stores coordinate data of correct training movements shown by a professional trainer.

[1702] 4. Feedback Devices

[1703] The feedback device compares the data obtained from the image recognition device and the standard training motion database, and provides feedback to the user when an error is detected. The feedback is provided in the form of voice or text using voice recognition technology such as Google Text-to-Speech.

[1704] 5. Emotion Engine

[1705] The emotion engine uses Microsoft Azure's emotion recognition API to analyze the user's facial expressions and voice to recognize their emotional state, and adjusts the content of the feedback generated by the server based on their emotional state.

[1706] Description

[1707] The user launches a fitness app on their mobile device and selects a training menu. They then begin training in front of the camera. The device captures the user's movements in real time through the camera and temporarily stores the video data. This data is analyzed using a pose estimation model such as TensorFlow or OpenPose to generate three-dimensional coordinate data for each joint. This coordinate data is then sent from the device to a server via the Internet.

[1708] The server runs on a cloud platform and receives coordinate data sent from the device. The server compares this data with standard training movement data of professional trainers stored in a database of standard training movements, and identifies errors such as a shallow knee angle or a bent back. If an error is identified, the server generates feedback according to the error, converts it into voice or text using a speech synthesis service (e.g., Google Text-to-Speech), and sends it to the device.

[1709] The emotion engine analyzes the user's facial expressions and voice through the microphone and camera connected to the device, recognizing their emotional state, such as fatigue, concentration, and stress. This emotional data is also sent to the server and reflected in the generated feedback. For example, if the user is tired, an encouraging message such as "Take a short break" will be added.

[1710] The device receives feedback from the server and provides it to the user in the form of voice or text, allowing the user to correct their next actions. This system allows users to receive high-quality training at home and increases their motivation through appropriate feedback provided by the emotion engine.

[1711] Specific examples

[1712] For squat training

[1713] User action: The user performs squats in front of the smartphone camera.

[1714] Device behavior: The device camera captures the user's squatting motion, and analyzes the knee and hip angles through a posture estimation model. The analysis results are sent to the server.

[1715] Server operation: The server compares with the standard data and detects that the knee angle is not appropriate.

[1716] How the emotion engine works: The emotion engine analyzes the user's facial expressions and recognizes that they are tired.

[1717] Device behavior: In addition to the feedback sent from the server, "Bend your knees to 90 degrees," the device displays the message "Feeling good, take a short break" via voice or text.

[1718] User Actions: The user follows the instructions, corrects the knee angle, and takes breaks at appropriate times.

[1719] Example prompts to input to the generative AI model

[1720] "Please explain the detailed processing steps of a system that analyzes home training in real time and provides appropriate feedback. Furthermore, please provide a concrete example of how the emotion engine analyzes the user's emotional state and adjusts the feedback content."

[1721] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1722] Step 1:

[1723] User training menu selection

[1724] A user starts a fitness app on their smartphone and selects the exercise they want to do from the provided training menu. The input is the user's menu selection, and the output is the selected menu (e.g., squats, push-ups, etc.).

[1725] Step 2:

[1726] User action starts and camera captures

[1727] The user begins training in front of the camera according to the selected training menu. The device starts capturing images in real time through the camera and temporarily stores this video data. The input is the camera image, and the output is the temporarily stored video data.

[1728] Step 3:

[1729] Pose estimation and joint coordinate generation

[1730] The device analyzes the temporarily stored video data using a posture estimation model (e.g., TensorFlow, OpenPose) to generate three-dimensional coordinate data for each of the user's joints. Based on this analysis, position data for each joint is obtained. The input is the temporarily stored video data, and the output is three-dimensional coordinate data.

[1731] Step 4:

[1732] Coordinate data sent to server

[1733] The terminal sends the generated 3D coordinate data to a server via the Internet. The input is 3D coordinate data, and the output is data transmission to the server. Specifically, the terminal sends the data using the network protocol.

[1734] Step 5:

[1735] Server error detection

[1736] The server compares the received 3D coordinate data with data in a standard training motion database to detect errors, such as a knee angle that is shallower than normal. The input is the 3D coordinate data sent to the server, and the output is the content of the error.

[1737] Step 6:

[1738] Feedback Generation

[1739] The server generates appropriate feedback based on the error. This feedback is provided via a speech synthesis service (e.g., Google Text-to-Speech) or as a text message. The input is the error, and the output is the feedback (e.g., "Bend your knees a little deeper").

[1740] Step 7:

[1741] Emotional state analysis

[1742] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state (e.g., tired, unfocused). The input is the user's facial expression data and voice data, and the output is the analysis result of their emotional state.

[1743] Step 8:

[1744] Adjusting feedback

[1745] Based on the analysis results of the emotion engine, the server adjusts the feedback content. For example, if the user is tired, an encouraging message is added. The input is the analysis results of the emotion engine, and the output is the adjusted feedback content.

[1746] Step 9:

[1747] Providing feedback to users

[1748] The terminal provides the user with the feedback sent from the server in the form of voice or text. The user receives this feedback and corrects their training movements. The input is the feedback content from the server, and the output is the feedback display and audio playback for the user. Specifically, the terminal displays a message on the screen and plays audio from the speaker.

[1749] This allows users to receive high-quality training at home and increases their motivation through appropriate feedback provided by the emotion engine.

[1750] (Application example 2)

[1751] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1752] Previous fitness applications provided single, one-way feedback on training movements and nutritional management, and were unable to provide customized feedback based on the user's physical condition or emotional state. Furthermore, they lacked a system that could simultaneously provide advice that took into account the nutritional value of food and the user's emotional state. This resulted in users being unable to receive a more effective fitness experience or health management, leading to a loss of motivation and reduced effectiveness.

[1753] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1754] In this invention, the server includes a means for capturing food images and calculating nutritional values, a means for analyzing a user's emotional state and providing emotion-based feedback, and a means for analyzing training movements and providing feedback, thereby enabling the provision of comprehensive and personalized feedback that takes into account not only the user's training movements but also the nutritional information of meals and their emotional state.

[1755] An "image capture device" is a device that takes real-time images of a user's movements and meals, and generally refers to a smartphone camera.

[1756] An "image recognition device" is a device that analyzes video data captured by an image capture device and estimates the coordinates of the user's joints and food items.

[1757] "Standard training movement data" is a database that stores coordinate data of correct training movements shown by professional trainers.

[1758] The "feedback device" is a device that detects errors based on the analysis results of the image recognition device and provides feedback to the user. The feedback is provided in the form of voice or text.

[1759] An "emotion engine" is a device that analyzes a user's facial expressions and voice to recognize their emotional state and generate appropriate feedback based on that emotion.

[1760] The "nutritional value calculation means" is a means for recognizing food items from food images and calculating their nutritional information.

[1761] The "emotion analysis means" is a means for analyzing the user's emotional state, and recognizes emotions by analyzing the user's facial expressions and voice.

[1762] The "training motion analysis means" is a means for analyzing the training motion of the user and estimating the coordinates of each joint.

[1763] A "feedback providing means" is a means for providing customized feedback to a user based on errors, nutritional information, and emotional state.

[1764] "User" refers to a person who uses the system to train and manage their diet.

[1765] The present invention relates to a system for managing training movements and dietary intake. A specific embodiment for realizing this system will be described below.

[1766] First, the user uses the smartphone camera to record their training movements in real time. An image capture device captures images of the user's movements and meals. The hardware used in this process is the smartphone's built-in camera. The captured images are analyzed by an image recognition device. The image recognition device uses a posture estimation model to estimate the coordinates of each of the user's joints.

[1767] The server compares these coordinate data with a database of standard training movements to detect errors. If an error is detected, the feedback device provides specific advice to the user. This feedback can be provided in the form of voice or text.

[1768] The system also includes a function to calculate the nutritional value of a user's meal. When a user captures an image of the meal using a camera, the image is sent to a server, which uses a food identification model to analyze the meal and calculate the nutritional value.

[1769] Furthermore, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. The results of the emotion analysis are used to customize the feedback. For example, if the user is recognized as tired, an encouraging message such as "Take a short break" will be added to the feedback.

[1770] As a concrete example, when a user is doing squat training, the movement is filmed with a smartphone camera and the angles of the knees and hips are analyzed. If the analysis results differ from the standard data, feedback such as "Bend your knees more" is provided. At the same time, if the user takes a photo of "salad" and "grilled chicken" for lunch, the nutritional information is calculated and advice is provided such as "Salad contains vitamins A and C. You seem tired, so try to consume more vitamins at your next meal."

[1771] Feedback can be further personalized using generative AI models, with specific prompt examples like:

[1772] "Generate a nutritional report and feedback based on the user's emotional state based on the following meal: Meal: Salad, grilled chicken. User's emotional state: Tired."

[1773] This allows the user to receive more effective and personalized support in their training movements and dietary management.

[1774] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1775] Step 1:

[1776] The user uses the smartphone camera to capture images of their workouts and meals. The input is image data captured by the smartphone camera, and the output is a saved image file. Specifically, the user launches a fitness app and takes photos of their workouts and meals with the smartphone camera.

[1777] Step 2:

[1778] The device saves image data captured through the camera. The input is the captured image data, and the output is an image file in the local storage. Specifically, the captured image is saved in the smartphone's internal storage in JPEG format or similar.

[1779] Step 3:

[1780] The device sends the stored image data to an image recognition device for analysis. The input is the stored image file, and the output is the analyzed coordinate data of each joint and food item identification data. Specifically, the image data is sent to an image recognition model on a cloud server and analyzed using a pose estimation model and a food identification model.

[1781] Step 4:

[1782] The server receives and processes the coordinate data and food item identification data sent from the image recognition device. The input is the recognized coordinate data and food item identification data, and the output is the comparison result with standard training data and nutritional information. Specifically, the server refers to a standard training motion database and compares the recognized coordinate data.

[1783] Step 5:

[1784] The server detects errors and generates feedback. The input is the coordinate data comparison result and the nutritional information of the food item, and the output is a feedback message. Specifically, the server identifies errors in the coordinate data and generates feedback such as "Bend your knees more."

[1785] Step 6:

[1786] The server uses an emotion engine to analyze the user's emotional state. The input is the user's facial expression and voice data, and the output is the recognized emotional state. Specifically, the emotion engine recognizes the user's face and analyzes their voice to identify emotional states such as fatigue or stress.

[1787] Step 7:

[1788] The device provides the user with the feedback message received from the server. The input is the feedback message sent from the server, and the output is feedback in the form of voice or text. Specifically, the feedback is presented to the user via the smartphone's speaker or screen.

[1789] Step 8:

[1790] Based on the feedback provided by the user, the system modifies training movements or adjusts diet. The input is a feedback message, and the output is modified movements or dietary choices. Specifically, the user optimizes their movements by "bending their knees more" or takes actions such as "taking more vitamins at their next meal."

[1791] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1792] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1793] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1794] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1795] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1796] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1797] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1798] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1799] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1800] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1801] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1802] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1803] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1804] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1805] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1806] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1807] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1808] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1809] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1810] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1811] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1812] The following is further disclosed regarding the above embodiment.

[1813] (Claim 1)

[1814] a means for capturing images of the user's training movements in real time using an image capture device;

[1815] means for analyzing the photographed training motion by an image recognition device and estimating the coordinates of each joint of the user;

[1816] means for comparing said estimated coordinate data with standard training motion data to detect errors;

[1817] means for providing feedback to the user based on said error.

[1818] (Claim 2)

[1819] 10. The system of claim 1, wherein the image capture device is a smartphone camera.

[1820] (Claim 3)

[1821] 10. The system of claim 1, wherein the feedback is provided to the user in audio or text format.

[1822] "Example 1"

[1823] (Claim 1)

[1824] a means for capturing images of training movements in real time by an image capture device when a user is training at home;

[1825] a means for analyzing the training motion photographed by an image recognition device and estimating the coordinates of each joint of the user;

[1826] means for comparing the estimated coordinate data with standard training motion data to detect errors;

[1827] a means of providing feedback to users based on errors;

[1828] A means for transmitting analysis data from the terminal to a server;

[1829] A system that includes a means by which feedback is provided to users in audio or text form.

[1830] (Claim 2)

[1831] 10. The system of claim 1, wherein the image capture device is a camera on a personal digital assistant.

[1832] (Claim 3)

[1833] 2. The system of claim 1, wherein the feedback is generated by a server and transmitted to the terminal via a network.

[1834] "Application Example 1"

[1835] (Claim 1)

[1836] a means for capturing images of the user's movements in real time using an image capture device;

[1837] means for analyzing the captured motion by an image recognition device and estimating the coordinates of each joint of the user;

[1838] means for comparing the estimated coordinate data with standard behavior data to detect errors;

[1839] means for providing feedback to the user based on said error;

[1840] means for displaying said feedback on a video frame;

[1841] means for transmitting said data to a server and receiving feedback;

[1842] means for including a posture estimation model used to analyze the joint coordinates;

[1843] A system including:

[1844] (Claim 2)

[1845] 10. The system of claim 1, wherein the image capture device is a camera on a personal digital assistant.

[1846] (Claim 3)

[1847] 10. The system of claim 1, wherein the feedback is provided to the user in audio or text format.

[1848] "Example 2: Combining Emotion Engines"

[1849] (Claim 1)

[1850] a means for capturing images of the user's training movements in real time using an image capture device;

[1851] means for analyzing the photographed training motion by an image recognition device and estimating the coordinates of each joint of the user;

[1852] means for comparing said estimated coordinate data with standard training motion data to detect errors;

[1853] means for providing feedback to the user based on said error;

[1854] The system includes a means for analyzing an emotional state and adjusting the content of the feedback.

[1855] (Claim 2)

[1856] 10. The system of claim 1, wherein the image capture device is a camera of a mobile device.

[1857] (Claim 3)

[1858] 10. The system of claim 1, wherein the feedback is provided to the user in audio or text format.

[1859] "Application example 2 when combining emotion engines"

[1860] (Claim 1)

[1861] a means for capturing images of the user's training movements in real time using an image capture device;

[1862] means for analyzing the photographed training motion by an image recognition device and estimating the coordinates of each joint of the user;

[1863] means for comparing said estimated coordinate data with standard training motion data to detect errors;

[1864] In addition to providing feedback to the user based on said error,

[1865] a means for capturing food images and calculating nutritional values;

[1866] A system including means for analyzing a user's emotional state and providing emotion-based feedback.

[1867] (Claim 2)

[1868] 10. The system of claim 1, wherein the image capture device is a smartphone camera.

[1869] (Claim 3)

[1870] 10. The system of claim 1, wherein the feedback is provided to the user in audio or text format. [Explanation of symbols]

[1871] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for capturing images of the user's training movements in real time using an image capture device; means for analyzing the photographed training motion by an image recognition device and estimating the coordinates of each joint of the user; means for comparing said estimated coordinate data with standard training motion data to detect errors; means for providing feedback to the user based on said error.

2. The system of claim 1 , wherein the image capture device is a smartphone camera.

3. The system of claim 1 , wherein the feedback is provided to the user in audio or text format.

Images