System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving a moving image; means for dividing the moving image into a plurality of frames; means for detecting a position of a joint or a body part of a target in each frame; means for analyzing an exercise motion based on the detected position; means for generating an exercise improvement advice based on an analysis result; and means for outputting the generated advice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the past, parents faced difficulties in providing specific and effective advice when teaching their children exercise. Parents without specialized knowledge or experience have found it particularly difficult to provide appropriate instruction to efficiently improve their children's athletic abilities. Furthermore, receiving instruction from a professional coach involves time and financial burdens. There is a need for a system that can solve these issues and support the improvement of children's skills in all sports. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including a means for receiving a video, a means for dividing the video into a plurality of frames, a means for detecting the positions of joints and body parts of an object in each frame, a means for analyzing exercise movements based on the detected positions, a means for generating exercise improvement advice based on the analysis results, and a means for outputting the generated advice. The system may further include a means for generating visual auxiliary images and a means for comparing the analyzed exercise movements with a specific technical database, allowing the user to obtain specific and effective exercise advice.

[0006] The "means for receiving video" is a function for acquiring video data shot by the user and transferring it to the server.

[0007] "Means for dividing video into multiple frames" refers to a function that breaks video data down into individual image frames and converts them into an analyzable format.

[0008] "Means for detecting the positions of joints and body parts" refers to a technology that identifies the joint positions and body parts of an object within each frame of a video and extracts this position information.

[0009] The "means for analyzing movement" is a function that analyzes the movement patterns of an object in chronological order based on detected position information, and evaluates features and problems.

[0010] The "means for generating exercise improvement advice" is a function for generating specific advice for improving the exercise movements of a subject based on the analysis results.

[0011] The "means for outputting advice" is a function for outputting the generated exercise improvement advice in the form of text, image, sound, or the like in order to provide it to the user.

[0012] The "means for generating visual auxiliary images" is a function that generates images that visually show areas for improvement in exercise, and helps the user understand the advice.

[0013] The "means for comparison with a technical database" is a function for comparing the analyzed exercise movements with an existing technical database to evaluate the exercise and identify areas for improvement. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The system of the present invention analyzes video and automatically generates specific advice to improve a child's athletic performance. An embodiment of the system will be described below.

[0036] First, the user uses their device to record their child's exercise. For example, they can use their smartphone to record their child shooting a soccer ball. The user then uploads the video to the server via a dedicated application. The application is easy to use; the user simply selects a video and taps the upload button, and the video is sent to the cloud server.

[0037] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis. Next, it splits the video into frames and prepares each frame to be sent to the generation AI.

[0038] The generative AI detects the position of the child's joints and body parts in each frame. For example, it identifies the locations of major joints such as the knees, ankles, and shoulders, and collects these positions as time-series data. It then analyzes the child's movements based on this position data and evaluates their movement patterns. During this evaluation process, it compares them with an existing technique database to match them with the correct form and technique.

[0039] Based on the analysis results, the AI ​​generates specific exercise improvement advice, such as "Your foot usage is unstable, so try to improve your balance" or "Your knee angle is shallow, so you should bend it a bit more." It can also create visual support images, which allow users to understand the advice more intuitively.

[0040] Finally, the generated advice and auxiliary images are sent from the server to the user's device. The user can then review the advice through the application and provide specific guidance to their child. In this way, the system provides specific and effective support for improving children's motor skills.

[0041] Example: Analysis of soccer shooting movements

[0042] Consider the case where a user takes a video of their child shooting a soccer ball and uploads it using a dedicated application. The server receives the video and analyzes each frame. The generating AI analyzes the foot movement, knee angle, body balance, etc. during the shot, and determines that the knees are not bent enough, resulting in a lack of shooting power. Based on the analysis results, it generates and provides specific advice such as "Bend your knees a bit more," along with a supplementary image of the correct form. The user who receives this information can then tell their child specific areas for improvement and encourage effective practice.

[0043] As described above, this system can provide specific advice to effectively improve children's motor skills, even without specialized knowledge.

[0044] The processing flow will be explained below.

[0045] Step 1:

[0046] Users record videos of their children's exercise on their own devices, then launch the dedicated application, select the video, and tap the "Upload" button.

[0047] Step 2:

[0048] The device sends the selected video to the server. When sending the video, it also performs checksum verification and other checks to ensure the integrity of the data.

[0049] Step 3:

[0050] The server receives the video data sent from the device, reconfirms the integrity of the video file, and stores it in temporary storage.

[0051] Step 4:

[0052] The server converts the received video into a standard format for analysis, converting the codec and adjusting the resolution if necessary.

[0053] Step 5:

[0054] The server splits the video into frames, extracts each frame as an individual image file, and stores the timestamp information for each frame.

[0055] Step 6:

[0056] The server sends the divided frame data to the generation AI, which receives it and detects the positions of joints and body parts in each frame.

[0057] Step 7:

[0058] Based on the detected joint position data, the generative AI analyzes the movement over time. Specifically, it tracks the movement patterns and positions of each joint and extracts the characteristic points of the movement.

[0059] Step 8:

[0060] The generative AI compares the extracted feature points with an existing technical database, and based on the comparison results, identifies any differences between the target athletic movement and correct form and technique.

[0061] Step 9:

[0062] Based on the analysis results, the generative AI generates specific advice for improving exercise, which is output in text format and as visual support images.

[0063] Step 10:

[0064] The server receives the generated advice data, associates it with the user profile, and transmits the advice data to the terminal.

[0065] Step 11:

[0066] The user checks the advice data sent from the server through the application on the device. The application displays the advice content and also provides visual support images to help the user understand it.

[0067] Step 12:

[0068] The user provides instruction to their child based on specific improvement advice provided through the application. By filming their exercise and conducting repeated analysis, the system continuously identifies areas for improvement and aims to improve athletic ability.

[0069] Example 1

[0070] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0071] With conventional exercise improvement systems, it was difficult for ordinary users without specialized knowledge to obtain specific advice to effectively improve children's exercise performance. Furthermore, using a video analysis system required advanced technology and specialized equipment, making it difficult for many users to use. This limited the opportunities for many children to learn correct form and techniques.

[0072] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0073] In this invention, the server includes means for receiving video, means for dividing the video into frames, means for detecting the positions of joints and body parts of an object in each frame, means for analyzing exercise movements based on the detected positions, means for generating exercise improvement advice based on the analysis results, means for outputting the generated exercise improvement advice and visual auxiliary images, and means for transmitting the advice and auxiliary images from the server to a user's terminal. This enables even general users without specialized knowledge to easily obtain specific advice for effectively improving their children's exercise performance.

[0074] The "means for receiving video" refers to a device or software for transmitting video data taken by a user to a server and receiving the video data.

[0075] The "means for dividing a moving image into a plurality of frames" refers to a device or software that divides received moving image data into a plurality of still images (frames) in time.

[0076] The "means for detecting the positions of joints and body parts of an object in each frame" refers to a device or software for identifying the positions of an object, specifically joints and body parts, from each divided frame.

[0077] The "means for analyzing movement based on detected positions" refers to a device or software for analyzing the position data of detected joints or body parts and evaluating the movement of an object.

[0078] The "means for generating exercise improvement advice based on the analysis results" refers to a device or software that generates specific advice for improving the athletic ability of a subject based on the analysis results of exercise movements.

[0079] The "means for outputting the generated exercise improvement advice and the visual support image" refers to a device or software for presenting the generated advice and the accompanying visual support image to the user.

[0080] The "means for transmitting advice and auxiliary images from the server to the user's terminal" refers to a device or software that transmits the generated advice and auxiliary images to the terminal used by the user via a network.

[0081] MODE FOR CARRYING OUT THE INVENTION

[0082] The system of the present invention analyzes video and automatically generates specific advice to improve children's athletic performance. An embodiment of the system will be described below.

[0083] First, a user uses their smartphone to record their child's exercise. For example, they can record a video of their child shooting a soccer ball. The user saves the video on their smartphone and then uploads it to a server via a dedicated application.

[0084] The dedicated application is commonly available and easy to use: users simply launch the application, select the video they have taken, and tap the upload button to send the video to a cloud server.

[0085] The server receives the uploaded video and saves it. After saving, the server checks the integrity of the video file by checking the file match using file hash values, etc. Once the integrity of the video is confirmed, it is converted into a format for analysis. For example, if the video is in MP4 format, it is converted into still images of each frame. The frame images saved in a temporary folder are used as input for the generative AI model.

[0086] The server splits the video into a fixed frame rate and saves each frame as an image file. These image files are listed as inputs to the generative AI model and prepared as a JSON-formatted data packet. The server then sends this data to the generative AI model along with a prompt.

[0087] The generative AI model analyzes each frame image to identify the location of the child's major joints (knees, ankles, shoulders, etc.). The detected data is collected in chronological order and used to evaluate the child's athletic performance. The evaluation results are compared with existing technical databases to generate appropriate advice.

[0088] The generated advice is specific, such as "Your foot usage is unstable, so try to find your balance" or "Your knee angle is shallow, so you should bend it a bit more." Visual support images are also created, allowing users to understand the advice more intuitively.

[0089] Finally, the generated advice and supporting images are sent from the server to the user's device. The user can check that new advice is available through a notification in the dedicated application. The user can then review the advice through the application and provide specific guidance to their child.

[0090] Specific examples

[0091] Analysis of soccer shooting movements

[0092] Consider the case where a user records a video of their child shooting a soccer ball and uploads it using a dedicated application. The server receives the video and prepares it for frame-by-frame analysis. The generative AI model analyzes the foot movement, knee angle, body balance, etc. during the shot, and identifies that the knees are not bent enough, resulting in a lack of shooting power. Based on the analysis results, it generates and provides a supporting image of the correct form along with specific advice such as "Bend your knees a bit more." The user can receive this information and tell their child specific areas for improvement, encouraging effective practice.

[0093] Prompt Sentence Examples

[0094] "Analyze a video of a child shooting a soccer ball and generate advice on foot movement and knee angle."

[0095] In this way, the system provides specific advice to effectively improve children's motor skills without requiring specialized knowledge.

[0096] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0097] Step 1:

[0098] Video recording and uploading

[0099] The user uses their smartphone to record their child's athletic activity (e.g., a soccer shot). Once the recording is complete, they launch the dedicated application and select the video file. The user then taps the "Upload" button, and the video file is sent to the server.

[0100] Input: Video file taken by the user

[0101] Output: Video file sent to the server

[0102] Step 2:

[0103] Video reception and integrity check

[0104] The server receives the video file sent by the user. Once received, it verifies the hash value and file size to confirm the integrity of the file. Once the integrity is confirmed, the video is stored in storage.

[0105] Input: Video file submitted by the user

[0106] Output: Video file with verified integrity

[0107] Step 3:

[0108] Video format conversion for analysis

[0109] The server reads the video files stored in the storage and converts them into a format that can be analyzed (e.g., a series of still images). Specifically, it converts MP4 format videos into JPEG images for each frame.

[0110] Input: Video file with verified integrity

[0111] Output: Separate image files for each frame

[0112] Step 4:

[0113] Frame Split

[0114] The server splits the converted video into a fixed frame rate (e.g. 30fps) and saves each frame as an image file. The saved frame images are placed in a temporary folder.

[0115] Input: Video file converted to analysis format

[0116] Output: Image files for each frame

[0117] Step 5:

[0118] Sending data to the generation AI

[0119] The server prepares the image files for each frame to be sent to the generative AI model. To do this, it lists the paths of the image files and generates a JSON-formatted data packet. It then sends the generated JSON data along with the prompt to the generative AI model.

[0120] Input: Image file for each frame, prompt text

[0121] Output: JSON data packet sent to the generative AI model

[0122] Step 6:

[0123] Joint position detection and motion analysis

[0124] The generative AI model analyzes each transmitted frame and identifies the location of key joints (knees, ankles, shoulders, etc.). This position data is collected as time series data and used to evaluate the movement. During the evaluation process, it compares the data with a database to identify the optimal movement pattern.

[0125] Input: Image files for each frame, JSON data

[0126] Output: Joint position data and motion analysis results

[0127] Step 7:

[0128] Generating exercise improvement advice

[0129] The server generates exercise improvement advice based on the analysis results received from the generative AI model. Specific advice includes "bend your knees more" and "maintain your body balance." Visual support images are also created to help users intuitively understand the advice.

[0130] Input: Motion analysis results

[0131] Output: Specific exercise improvement advice, visual aid images

[0132] Step 8:

[0133] Sending advice and auxiliary images

[0134] The server sends the generated advice and auxiliary images to the user's device, and the notification function of the dedicated application is used to inform the user that new advice is available.

[0135] Input: Exercise improvement advice, visual aid images

[0136] Output: Sends advice and auxiliary images to the user's terminal

[0137] Step 9:

[0138] Confirmation of advice and guidance

[0139] The user checks the advice and supplementary images received on the device and provides specific instructions to the child on how to improve their exercise. For example, the user might be told to "bend your knees a little more," and the supplementary images would be used to demonstrate the correct form.

[0140] Input: Advice and auxiliary images sent to the user's device

[0141] Output: Specific exercise improvement guidance for children

[0142] Through the above steps, the system can provide the user with specific advice on how to improve their exercise, and support their child in improving their athletic ability.

[0143] (Application example 1)

[0144] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0145] Conventional factory robot motion analysis requires specialized knowledge, making it difficult to make efficient improvements. Furthermore, when humans observe motion and suggest areas for improvement, they may overlook subtle motion characteristics. Therefore, there is a need for a method to provide specific and accurate advice to maximize robot efficiency.

[0146] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0147] In this invention, the server

[0148] means for receiving video;

[0149] A means for dividing a video into frames;

[0150] means for detecting the position of joints and body parts of the object in each frame;

[0151] means for analyzing the movement based on the detected position;

[0152] A means for generating behavior improvement advice based on the analysis results;

[0153] means for outputting the generated advice;

[0154] A means to film and upload the robot's movements;

[0155] a means for evaluating the efficiency of the operation using the generative AI model;

[0156] A means for identifying a behavior pattern based on the evaluation results;

[0157] This makes it possible to provide specific and accurate advice to improve the efficiency of robot operations, even without specialized knowledge.

[0158] "Means for receiving video" refers to a device or function for transferring and importing video shot by a user from a device to a server.

[0159] "Means for dividing a video into multiple frames" refers to a technique or method for dividing a received video into frames and analyzing each frame individually.

[0160] "Means for detecting the positions of joints and body parts of an object in each frame" refers to a technology that identifies the positions of joints and body parts of the robot or person being analyzed for each frame.

[0161] The "means for analyzing movement based on detected positions" refers to a device or method for analyzing the movement pattern of an object using position data of detected joints and parts.

[0162] The "means for generating advice to improve movement based on the analysis results" is a technology for generating specific advice to improve movement based on the analyzed movement pattern.

[0163] The "means for outputting the generated advice" refers to a method or device for providing the generated behavior improvement advice to the user.

[0164] "Means for filming and uploading robot operations" refers to a method or technology for filming the operations of a factory robot and uploading the video data to a server.

[0165] "Means for evaluating the efficiency of movements using a generative AI model" refers to a technology that analyzes data obtained from video captured using a generative AI model and evaluates the efficiency of movements.

[0166] "Means for identifying movement patterns based on evaluation results" refers to a technology that identifies the movement patterns of a robot based on the evaluated data and extracts specific key points for improving efficiency.

[0167] The system of the present invention provides specific advice for improving operational efficiency through the analysis of the operation of a factory robot. An embodiment of the system will be described below.

[0168] First, the user records the factory robot's operations using a device such as a smartphone or head-mounted display (HMD). The user then uploads the recorded video to a cloud server via a dedicated analysis application. The application is easy to use; the user simply selects a video and taps the upload button to send the video to the cloud server.

[0169] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis. Next, it splits the video into frames and prepares each frame to be sent to the generative AI model.

[0170] The generative AI model detects the positions of the robot's joints and parts in each frame. For example, it identifies the positions of key parts such as arms, joints, and grippers, and collects these positions as time-series data. Based on the collected data, it analyzes the robot's motion and evaluates its movement patterns. During this evaluation process, it compares them with a database of existing technologies to find efficient movement patterns and technologies.

[0171] Based on the analysis results, the generative AI model generates specific advice for improving the movement. For example, "The arm's movement is slow, so you need to increase the speed" or "The joint angle is inappropriate, so you need to adjust the angle." It is also possible to create visual support images, which allows the user to understand the advice more intuitively.

[0172] Finally, the generated advice and auxiliary images are sent from the server to the user's device, where the user can review the advice and adjust the robot's operation through the application. In this way, the system can provide specific and effective support for improving the robot's operation efficiency, even without specialized knowledge.

[0173] Example: Analysis of welding motion of a robot arm

[0174] Consider a case where a user records a video of a robot arm's welding operations and uploads it using a dedicated analysis application. The server receives the video and analyzes it frame by frame. The generative AI model analyzes the arm's movements during welding, the joint angles, and areas of slow movement, and generates specific advice such as, "The welding quality is declining because the arm is moving too slowly." Based on this result, the model generates and provides specific advice such as, "Make the movement a little faster," along with a supplementary image of the correct operation. The user who receives this information can adjust the robot's behavior to improve welding quality.

[0175] Example prompt for a generative AI model:

[0176] Detect joint positions from frame images of a robot arm and generate advice to improve the efficiency of the movement. Joint position data is provided for each frame of the video below. Please provide specific advice on how to improve the movement based on this data.

[0177] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0178] Step 1:

[0179] The user records the robot's movements using a smartphone or head-mounted display. The input is the captured video. This video data is stored on the user's device.

[0180] Step 2:

[0181] Users upload the videos they have taken to the cloud server via a dedicated analysis application. The input is the taken video file, and the output is the video data saved on the cloud server. The user selects a video and taps the upload button to send the video to the cloud server.

[0182] Step 3:

[0183] The cloud server receives the uploaded video and checks the integrity of the video file. The input is the video data stored on the cloud server, and the output is the video data whose integrity has been checked. Once the integrity is confirmed, the video is saved in storage.

[0184] Step 4:

[0185] The server converts the video into a format for analysis. The input is the video data that has been validated for integrity, and the output is the converted video data in the format for analysis. This process splits the video into frames, and each frame is ready to be sent to the generative AI model.

[0186] Step 5:

[0187] The server sends each frame to the generative AI model, which detects the position of the robot's joints and body parts for each frame. The input is video data (each frame) in an analysis format, and the output is position data of the joints and body parts in each frame. The generative AI model uses a machine learning algorithm to identify the position of the joints and body parts in each frame.

[0188] Step 6:

[0189] The server analyzes the robot's movements based on joint and body part position data. The input is joint and body part position data, and the output is the analysis results of the movements. During this analysis process, it compares the data with an existing technology database and matches it with efficient movement patterns and technologies.

[0190] Step 7:

[0191] The generative AI model generates movement improvement advice based on the analysis results. The input is the analysis results of the exercise movement, and the output is movement improvement advice. During this process, visual support images are generated along with specific advice.

[0192] Step 8:

[0193] The server sends the generated advice and auxiliary image to the user's device. The input is the behavior improvement advice and auxiliary image, and the output is the advice and auxiliary image displayed on the user's device. The user can check this advice through the application.

[0194] Step 9:

[0195] The user adjusts the robot's behavior based on the behavior improvement advice. The input is the displayed advice, and the output is the adjusted robot behavior. The user adjusts the robot's settings based on specific improvements to improve operational efficiency.

[0196] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0197] The system according to the present invention not only analyzes videos and provides exercise improvement advice, but also incorporates an emotion engine that recognizes the user's emotions and adjusts the content and presentation method of the advice accordingly. A specific embodiment of the system will be described below.

[0198] First, the user uses their device to record their child's exercise. For example, they can use a smartphone to record their child practicing dribbling a basketball. The user then uploads the video to the server via a dedicated application. This application is easy to use; the user simply selects a video and presses the upload button, and the video is sent to the server.

[0199] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis. Next, it splits the video into frames and prepares each frame to be sent to the generation AI.

[0200] The generative AI detects the positions of the child's joints and body parts in each frame. For example, it identifies the positions of major joints such as elbows, wrists, and torso, and collects their positions as time-series data. It then analyzes the child's movements based on this position data and evaluates their movement patterns. During this evaluation process, it compares them with an existing technical database to identify any discrepancies from the correct form and technique.

[0201] Based on the analysis results, the AI ​​generates specific advice for improving your movements. For example, it might say, "Your wrist movements are stiff when dribbling, so try to relax them." It can also create visual aids, which can help users understand the advice more intuitively.

[0202] A distinctive feature of the present invention is the incorporation of an emotion engine. This emotion engine can recognize the user's facial expressions and voice through the device's camera and microphone and evaluate the user's emotional state. The emotion data acquired by the emotion engine is reflected in the content and presentation method of advice. For example, if the user is feeling irritated or anxious, the tone of the advice can be changed to a gentler one, encouraging the user to take a more positive approach. Also, if the user is enjoying themselves, encouraging words can be added to maintain that state.

[0203] Finally, the generated advice and auxiliary images are sent from the server to the user's device. The user can then review the advice and provide specific guidance to their child through the application. In this way, the system provides specific and effective support for improving children's motor skills, while also being able to flexibly respond to the user's emotional state.

[0204] Example: Analysis of basketball dribbling practice

[0205] Consider the case where a user films their child practicing basketball dribbling and uploads it using a dedicated application. The server receives the video and analyzes it frame by frame. The generation AI analyzes the movement of each joint in the dribbling motion and identifies stiff wrist movement. Based on the analysis results, it generates specific advice such as "Relax your wrist."

[0206] Meanwhile, the emotion engine recognizes when the user is frustrated and adjusts the tone of the advice to be gentler, for example, by saying, "It may be difficult at first, but you can improve little by little." This allows the user to provide guidance to their child in a positive manner.

[0207] As described above, this system not only helps children improve their motor skills, but also provides a flexible approach that takes into consideration the user's emotions.

[0208] The processing flow will be explained below.

[0209] Step 1:

[0210] A user uses their own device to record a video of their child exercising. For example, a user uses a smartphone to record a video of their child practicing basketball dribbling.

[0211] Step 2:

[0212] The user launches the dedicated application, selects the video they have taken, and taps the "Upload" button.

[0213] Step 3:

[0214] The device sends the selected video to the server. When the video is sent, a checksum and other verifications are performed to ensure the integrity of the data.

[0215] Step 4:

[0216] The server receives the video data sent from the device, reconfirms the integrity of the video file, and stores it in temporary storage.

[0217] Step 5:

[0218] The server converts the received video into a standard format for analysis, converting the codec and adjusting the resolution as necessary.

[0219] Step 6:

[0220] The server splits the video into frames, extracts each frame as an individual image file, and stores the timestamp information for each frame.

[0221] Step 7:

[0222] The server sends the divided frame data to the generation AI, which receives it and detects the positions of joints and body parts in each frame.

[0223] Step 8:

[0224] Based on the detected joint position data, the generative AI analyzes the movement over time, tracking the movement patterns and positions of each joint and extracting the characteristic points of the movement.

[0225] Step 9:

[0226] The generative AI compares the extracted feature points with an existing technical database, and based on the comparison results, identifies any differences between the target athletic movement and correct form and technique.

[0227] Step 10:

[0228] Based on the analysis results, the generative AI generates specific advice for improving exercise, such as "relax your wrist and move more smoothly."

[0229] Step 11:

[0230] The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotional state, for example, detecting when the user is feeling irritated.

[0231] Step 12:

[0232] Based on the emotional data recognized by the emotion engine, the generative AI adjusts the way it presents advice. For example, if a user feels frustrated, it will soften the tone of the advice and add encouraging words such as "Take your time and practice a little at a time."

[0233] Step 13:

[0234] The server transmits the generated advice and the adjusted advice data to the user's terminal.

[0235] Step 14:

[0236] The user checks the advice data sent from the server through an application on their device. The application displays the advice and provides visual support images to help the user understand the specific improvements required.

[0237] Step 15:

[0238] The user provides specific advice for improvement based on the application, and by filming the exercise again and repeating these steps, the user can continuously identify areas for improvement and improve athletic ability.

[0239] Example 2

[0240] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0241] Conventional video analysis systems can analyze exercise videos taken by users and provide exercise improvement advice, but they provide uniform advice without considering the user's emotions, making it difficult to provide appropriate guidance tailored to the user's situation.In addition, there is a lack of means to compare with technical databases to improve the accuracy of motion analysis, or to generate visual support images, making it difficult to deepen understanding of the analysis results.

[0242] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0243] In this invention, the server includes means for receiving a video, means for dividing the video into a plurality of frames, means for detecting the positions of joints and body parts of an object in each frame, means for evaluating the analyzed exercise movements by comparing with a specific database, means for generating exercise improvement advice based on the analysis results, means for recognizing the user's emotions, means for adjusting the generated advice based on the recognized emotions, and means for outputting the generated advice, thereby making it possible to provide flexible exercise improvement advice that takes the user's emotions into consideration.

[0244] The "means for receiving video" is a function for transmitting video data taken by a user to a server via a network and for the server to receive the video data.

[0245] "Means for dividing a video into multiple frames" is a function for dividing a received video into small still images (frames) along the time axis and generating data for each frame.

[0246] "Means for detecting the positions of joints and body parts of an object in each frame" refers to a technique for identifying the positions of joints and body parts relative to an object in each frame image and extracting their coordinate data.

[0247] The "means for evaluating the analyzed exercise movements by comparing with a specific database" is a function for comparing the analyzed exercise data with an existing reference database and evaluating the accuracy of the exercise and areas for improvement.

[0248] The "means for generating exercise improvement advice based on the analysis results" is a function for automatically generating specific improvement methods and instruction contents based on the evaluation results of the exercise analysis.

[0249] "Means for recognizing user emotions" refers to technology that analyzes information such as the user's facial expressions and voice to identify the user's emotional state.

[0250] The "means for adjusting the advice generated based on the recognized emotion" is a function for flexibly changing the content and tone of the exercise improvement advice generated according to the user's emotional state.

[0251] The "means for outputting generated advice" is a function for displaying or providing the generated exercise improvement advice and visual support images in an easily understandable form to the user.

[0252] MODE FOR CARRYING OUT THE INVENTION

[0253] The system according to the present invention not only analyzes videos and provides exercise improvement advice, but also incorporates an emotion recognition engine that recognizes the user's emotions and adjusts the content and presentation method of the advice accordingly. A specific embodiment of the system will be described below.

[0254] First, the user uses their device to record their child's exercise. For example, they can use a smartphone to record their child practicing dribbling a basketball. The user then uploads the video to the server via a dedicated application. This application is easy to use; the user simply selects a video and presses the upload button, and the video is sent to the server.

[0255] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis using software such as FFmpeg. Next, it splits the video into frames and prepares each frame to be sent to a generative AI (for example, using OpenPose or TensorFlow).

[0256] The generative AI detects the positions of the child's joints and body parts in each frame. For example, it identifies the positions of major joints such as elbows, wrists, and torso, and collects their positions as time-series data. It then analyzes the child's movements based on this position data and evaluates their movement patterns. During this evaluation process, it compares the child's movements with existing technical databases (e.g., Sports Motion Database) to identify any discrepancies with the correct form and technique.

[0257] Based on the analysis results, the AI ​​generates specific advice for improving your movements. For example, it might say, "Your wrist movements are stiff when dribbling, so try to relax them." It can also create visual aids, which can help users understand the advice more intuitively.

[0258] A distinctive feature of the present invention is the incorporation of an emotion engine. This emotion engine can recognize the user's facial expressions and voice through the device's camera and microphone and evaluate the user's emotional state. The emotion data acquired by the emotion engine is reflected in the content and presentation method of advice. For example, if the user is feeling irritated or anxious, the tone of the advice can be changed to a gentler one, encouraging the user to take a more positive approach. Also, if the user is enjoying themselves, encouraging words can be added to maintain that state.

[0259] Finally, the generated advice and auxiliary images are sent from the server to the user's device. The user can then review the advice and provide specific guidance to their child through the application. In this way, the system provides specific and effective support for improving children's motor skills, while also being able to flexibly respond to the user's emotional state.

[0260] Example: "Analysis of basketball dribbling practice"

[0261] Consider the case where a user films their child practicing basketball dribbling and uploads it using a dedicated application. The server receives the video and analyzes it frame by frame. The generation AI analyzes the movement of each joint in the dribbling motion and identifies stiff wrist movement. Based on the analysis results, it generates specific advice such as "Relax your wrist."

[0262] Meanwhile, the emotion engine recognizes when the user is frustrated and adjusts the tone of the advice to be gentler, for example, by saying, "It may be difficult at first, but you can improve little by little." This allows the user to provide guidance to their child in a positive manner.

[0263] Examples of prompt statements

[0264] Prompt statement:

[0265] I filmed a kid practicing dribbling a basketball. Analyze the uploaded video and identify stiff wrist movements. Then provide specific advice to relax the wrist. Also, adjust the tone of the advice to be gentler if the user is frustrated.

[0266] As described above, this system not only helps children improve their motor skills, but also provides a flexible approach that takes into consideration the user's emotions.

[0267] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0268] Step 1:

[0269] The user uses a device such as a smartphone to record their child's exercise. The user then uploads the video to a server using a dedicated application. The input is a video file, and the output is a video file stored on the server.

[0270] Step 2:

[0271] The server checks the received video and saves it in storage. At this time, it checks the integrity of the video file and checks whether it contains any invalid data. The input is the uploaded video file, and the output is the video file saved in storage whose integrity has been checked.

[0272] Step 3:

[0273] The server uses video processing software such as FFmpeg to convert the stored video into a format for analysis. The input in this step is a video file whose integrity has been confirmed, and the output is a video file converted into the format for analysis.

[0274] Step 4:

[0275] The server divides the video file for analysis into frames and generates image data for each frame. Specifically, it divides one second of video into 30 frames. The input for this step is the video file for analysis, and the output is still images for each frame (e.g., "frame0001.jpg", "frame0002.jpg").

[0276] Step 5:

[0277] The server sends each frame image to a generative AI (e.g., OpenPose or TensorFlow) to detect the positions of the child's joints and body parts in each frame. The input is the frame image data, and the output is joint position data (e.g., coordinate data for the left elbow, right wrist, and trunk).

[0278] Step 6:

[0279] The server analyzes the motion based on the detected joint position data. The analysis results are compared with a specific technique database (e.g., Sports Motion Database) to identify deviations from the correct form and technique. The input for this step is the joint position data, and the output is the evaluation result of the motion analysis (e.g., "The wrist movement is stiff").

[0280] Step 7:

[0281] The server generates exercise improvement advice based on the analysis results. The generation AI creates specific advice text (e.g., "Try to relax your wrist") and visual support images (e.g., illustrations of the movement to relax your wrist). The input is the evaluation results of the movement analysis, and the output is exercise improvement advice and support images.

[0282] Step 8:

[0283] The device uses a camera and microphone to collect the user's facial expressions and voice. This data is sent to the emotion engine to evaluate the user's emotional state. The input is the user's facial and voice data, and the output is the emotion recognition result (e.g., "irritated").

[0284] Step 9:

[0285] The server adjusts the tone and content of exercise improvement advice based on the emotion recognition results obtained from the emotion engine. For example, if a user feels frustrated, the tone may be changed to a gentler one, such as "It may be difficult at first, but you can improve little by little." The input is the emotion recognition result, and the output is the adjusted exercise improvement advice.

[0286] Step 10:

[0287] The server sends the final exercise improvement advice and supplementary images to the user's device. The user can then review the advice and provide specific guidance to their child through the application. The input is the adjusted exercise improvement advice and supplementary images, and the output is the final advice displayed on the user's device.

[0288] keyword

[0289] Generative AI model, prompt sentence

[0290] (Application example 2)

[0291] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0292] Conventional exercise improvement systems can analyze a user's exercise data and provide specific advice, but they are unable to adjust the content of the advice taking the user's emotions into account. As a result, the advice is often ineffective when the user is feeling frustrated or anxious. Furthermore, while it is important for exercise improvement advice to be presented in a visually easy-to-understand format, there has been a lack of effective ways to achieve this. This has led to issues such as insufficient user satisfaction and effective exercise improvement.

[0293] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0294] In this invention, the server includes means for receiving a video, means for dividing the video into a plurality of frames, means for detecting the positions of joints and body parts of an object in each frame, means for analyzing exercise movements based on the detected positions, means for generating exercise improvement advice based on the analysis results, means for outputting the generated advice, means for recognizing a user's emotion, and means for adjusting the content and presentation method of the advice according to the recognized emotion. This makes it possible to provide effective and flexible exercise improvement advice to the user, and to improve the acceptability and satisfaction of the advice by taking the user's emotion into consideration.

[0295] The "means for receiving video" is a mechanism by which a user sends video taken by a terminal to a server, and the server receives the video.

[0296] The "means for dividing a video into a plurality of frames" is a mechanism for dividing a received video into frames along the time axis and making each frame individually analyzable.

[0297] The "means for detecting the positions of the joints and body parts of the object in each frame" is a mechanism for identifying the positions of the joints and body parts of a person in the video frames to be analyzed and acquiring that data.

[0298] The "means for analyzing movement based on detected positions" is a mechanism for evaluating and analyzing movement based on acquired joint and body position data.

[0299] The "means for generating exercise improvement advice based on the analysis results" is a mechanism for generating specific advice for improving exercise based on the analysis results of exercise movements.

[0300] The "means for outputting generated advice" is a mechanism for presenting the generated exercise improvement advice to the user and displaying it in an easy-to-understand manner.

[0301] The "means for recognizing the user's emotions" is a mechanism for analyzing the user's facial expressions and voice through a camera or microphone and recognizing the user's emotional state.

[0302] The "means for adjusting the content and presentation method of advice according to the recognized emotion" is a mechanism for appropriately changing the content and presentation method of advice based on the user's emotional state.

[0303] The system according to the present invention provides exercise improvement support by combining video analysis and emotion recognition. This system not only analyzes videos to provide exercise improvement advice, but also incorporates an emotion engine that recognizes the user's emotions and adjusts the content and presentation method of the advice accordingly. Specific aspects of this system are described below.

[0304] First, the user films the exercise using their own device. An example of this is a child using a smartphone to film their basketball practice. The user then uploads the video to the server via a dedicated application. The application is designed to be user-friendly; simply by selecting a video and pressing the upload button, the video is automatically sent to the server.

[0305] The server stores the received video in storage, verifies the integrity of the video file, and then converts the video into a format for analysis. The video is then split into frames, and each frame is prepared to be sent to a generation AI for analysis. This generation AI uses a machine learning library such as TensorFlow.

[0306] The generative AI detects the positions of the joints and body parts of the subject (in this example, a child) in each frame. For example, it identifies the positions of major joints such as the elbows, wrists, and torso, and collects their positions as time-series data. It then analyzes the movement based on this position data and evaluates the movement patterns. During this evaluation process, it compares the results with an existing technical database to identify any discrepancies from the correct form and technique.

[0307] Based on the analysis results, the AI ​​generates specific advice for improving the movement. For example, it might say, "When shooting, your elbow position is low. Try to keep it a little higher." In addition, it creates auxiliary images that are easy to understand visually.

[0308] A distinctive feature of the present invention is the incorporation of an emotion engine. This emotion engine recognizes the user's facial expressions and voice through the device's camera and microphone and evaluates the user's emotional state. This evaluation utilizes emotion analysis APIs such as Microsoft Azure Cognitive Services. The emotion data acquired by the emotion engine is reflected in the content and presentation of the generated advice. For example, if the user is feeling frustrated or anxious, the tone of the advice can be made gentler, and soft words such as "Trial and error lead to growth. Take your time and practice slowly." If the user is enjoying themselves, encouraging words such as "That's great! Keep it up!" can be added to maintain that state of mind.

[0309] Finally, the generated advice and auxiliary images are sent from the server to the user's device, where the user can check the advice and receive specific guidance through the application.

[0310] For example, consider the following prompt:

[0311] python

[0312] import requests

[0313] import cv2

[0314] def upload_video(video_path):

[0315] url = 'http: / / yourserver.com / upload'

[0316] files = {'file': open(video_path, 'rb')}

[0317] response = requests.post(url, files=files)

[0318] if response.status_code == 200:

[0319] print("Video uploaded successfully!")

[0320] else:

[0321] print("Error in video upload")

[0322] def analyze_video_frames(frames, model):

[0323] results = []

[0324] for frame in frames:

[0325] processed_frame = preprocess_frame(frame)

[0326] result = model.predict(processed_frame)

[0327] results.append(result)

[0328] return results

[0329] def generate_advice(movement_data):

[0330] Analyze movement_data and generate advice

[0331] advice = "Your elbow position is incorrect during the shot. Try to keep it higher."

[0332] return advice

[0333] def emotion_adjust_advice(user_emotion, advice):

[0334] if user_emotion == 'frustrated':

[0335] adjusted_advice = "Don't worry, improvement takes time. " + advice

[0336] elif user_emotion == 'happy':

[0337] adjusted_advice = "Great job! Keep it up. " + advice

[0338] else:

[0339] adjusted_advice = advice

[0340] return adjusted_advice

[0341] Example usage

[0342] video_path = 'child_shooting_basketball.mp4'

[0343] upload_video(video_path)

[0344] This program aims to improve athletic ability by uploading videos taken by users to a server, analyzing the videos and recognizing emotions, and generating and presenting appropriate advice.

[0345] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0346] Step 1:

[0347] Recording and uploading videos

[0348] Action: A user uses a smartphone to record themselves exercising.

[0349] Input: The captured video file.

[0350] Output: The video file imported into the application.

[0351] Specific operation: The user presses the record button to record a video of their exercise. After recording is complete, the user presses the application's upload button to send the video to the server.

[0352] Step 2:

[0353] Receiving and storing videos

[0354] How it works: The server receives and stores videos uploaded by users.

[0355] Input: The uploaded video file.

[0356] Output: Video files stored in the server's storage.

[0357] What it does: The server receives an HTTP request, saves the file to disk, checks the integrity of the saved file, and prepares it for analysis.

[0358] Step 3:

[0359] Video frame division

[0360] How it works: The server splits the stored video into frames.

[0361] Input: A saved video file.

[0362] Output: Image data per frame.

[0363] Specific operation: Using the OpenCV library, a video file is read and split into frame-by-frame still images. Each frame is then converted into the format required for subsequent analysis.

[0364] Step 4:

[0365] Position detection of joints and body parts

[0366] How it works: The server detects the positions of joints and body parts for each frame.

[0367] Input: Frame-by-frame image data.

[0368] Output: Position data of joints and body parts (time series data).

[0369] Specific operation: Using a generative AI model such as TensorFlow, joints and body parts (e.g., elbows, wrists, torso, etc.) are analyzed for each frame to obtain positional data.

[0370] Step 5:

[0371] Analysis of athletic movements

[0372] Movement: The server analyzes the movement based on the detected location data.

[0373] Input: Position data of joints and body parts.

[0374] Output: Motor behavior evaluation data.

[0375] Specific behavior: The acquired position data is analyzed as time series data, and behavior patterns are evaluated by comparing it with a technical database. Inappropriate behavior is identified and the analysis results are output.

[0376] Step 6:

[0377] Generating exercise improvement advice

[0378] Action: The server generates exercise improvement advice based on the results of the exercise motion analysis.

[0379] Input: Exercise behavior assessment data.

[0380] Output: Text and supporting images of exercise improvement advice.

[0381] Specific actions: Based on the analysis results, specific advice is generated based on the situation, such as, "When shooting, your elbow is positioned too low, so try to keep it a little higher." Supporting images are also created to make the game easier to understand visually.

[0382] Step 7:

[0383] User Emotion Recognition

[0384] How it works: The server recognizes the user's emotions.

[0385] Input: User's facial and voice data.

[0386] Output: User's emotional state data.

[0387] Specific operation: Using emotion analysis APIs such as Microsoft Azure Cognitive Services, the system analyzes facial expression data and voice data acquired through the camera to evaluate the user's emotional state (e.g., irritation, joy, calmness, etc.).

[0388] Step 8:

[0389] Advice adjustment

[0390] How it works: The server adjusts the content and presentation of advice based on the results of emotion recognition.

[0391] Input: Exercise improvement advice, user emotional state data.

[0392] Output: Tailored exercise improvement advice.

[0393] Specific actions: Change the tone of advice based on emotion recognition data, e.g., "Trial and error leads to growth. Take your time and practice slowly." Add words of encouragement according to the emotion.

[0394] Step 9:

[0395] Providing advice

[0396] How it works: The server sends the generated and tailored advice to the user's device and presents it through the application.

[0397] Enter: tailored exercise improvement advice.

[0398] Output: Advice and auxiliary images displayed on the user's terminal.

[0399] Specific operation: The generated advice and auxiliary images are sent to the user's device, and displayed through the application so that the user can check and execute them.

[0400] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0401] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0402] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0403] [Second embodiment]

[0404] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0405] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0406] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0407] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0408] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0409] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0410] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0411] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0412] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0413] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0414] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0415] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0416] The system of the present invention analyzes video and automatically generates specific advice to improve a child's athletic performance. An embodiment of the system will be described below.

[0417] First, the user uses their device to record their child's exercise. For example, they can use their smartphone to record their child shooting a soccer ball. The user then uploads the video to the server via a dedicated application. The application is easy to use; the user simply selects a video and taps the upload button, and the video is sent to the cloud server.

[0418] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis. Next, it splits the video into frames and prepares each frame to be sent to the generation AI.

[0419] The generative AI detects the position of the child's joints and body parts in each frame. For example, it identifies the locations of major joints such as the knees, ankles, and shoulders, and collects these positions as time-series data. It then analyzes the child's movements based on this position data and evaluates their movement patterns. During this evaluation process, it compares them with an existing technique database to match them with the correct form and technique.

[0420] Based on the analysis results, the AI ​​generates specific exercise improvement advice, such as "Your foot usage is unstable, so try to improve your balance" or "Your knee angle is shallow, so you should bend it a bit more." It can also create visual support images, which allow users to understand the advice more intuitively.

[0421] Finally, the generated advice and auxiliary images are sent from the server to the user's device. The user can then review the advice through the application and provide specific guidance to their child. In this way, the system provides specific and effective support for improving children's motor skills.

[0422] Example: Analysis of soccer shooting movements

[0423] Consider the case where a user takes a video of their child shooting a soccer ball and uploads it using a dedicated application. The server receives the video and analyzes each frame. The generating AI analyzes the foot movement, knee angle, body balance, etc. during the shot, and determines that the knees are not bent enough, resulting in a lack of shooting power. Based on the analysis results, it generates and provides specific advice such as "Bend your knees a bit more," along with a supplementary image of the correct form. The user who receives this information can then tell their child specific areas for improvement and encourage effective practice.

[0424] As described above, this system can provide specific advice to effectively improve children's motor skills, even without specialized knowledge.

[0425] The processing flow will be explained below.

[0426] Step 1:

[0427] Users record videos of their children's exercise on their own devices, then launch the dedicated application, select the video, and tap the "Upload" button.

[0428] Step 2:

[0429] The device sends the selected video to the server. When sending the video, it also performs checksum verification and other checks to ensure the integrity of the data.

[0430] Step 3:

[0431] The server receives the video data sent from the device, reconfirms the integrity of the video file, and stores it in temporary storage.

[0432] Step 4:

[0433] The server converts the received video into a standard format for analysis, converting the codec and adjusting the resolution if necessary.

[0434] Step 5:

[0435] The server splits the video into frames, extracts each frame as an individual image file, and stores the timestamp information for each frame.

[0436] Step 6:

[0437] The server sends the divided frame data to the generation AI, which receives it and detects the positions of joints and body parts in each frame.

[0438] Step 7:

[0439] Based on the detected joint position data, the generative AI analyzes the movement over time. Specifically, it tracks the movement patterns and positions of each joint and extracts the characteristic points of the movement.

[0440] Step 8:

[0441] The generative AI compares the extracted feature points with an existing technical database, and based on the comparison results, identifies any differences between the target athletic movement and correct form and technique.

[0442] Step 9:

[0443] Based on the analysis results, the generative AI generates specific advice for improving exercise, which is output in text format and as visual support images.

[0444] Step 10:

[0445] The server receives the generated advice data, associates it with the user profile, and transmits the advice data to the terminal.

[0446] Step 11:

[0447] The user checks the advice data sent from the server through the application on the device. The application displays the advice content and also provides visual support images to help the user understand it.

[0448] Step 12:

[0449] The user provides instruction to their child based on specific improvement advice provided through the application. By filming their exercise and conducting repeated analysis, the system continuously identifies areas for improvement and aims to improve athletic ability.

[0450] Example 1

[0451] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0452] With conventional exercise improvement systems, it was difficult for ordinary users without specialized knowledge to obtain specific advice to effectively improve children's exercise performance. Furthermore, using a video analysis system required advanced technology and specialized equipment, making it difficult for many users to use. This limited the opportunities for many children to learn correct form and techniques.

[0453] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0454] In this invention, the server includes means for receiving video, means for dividing the video into frames, means for detecting the positions of joints and body parts of an object in each frame, means for analyzing exercise movements based on the detected positions, means for generating exercise improvement advice based on the analysis results, means for outputting the generated exercise improvement advice and visual auxiliary images, and means for transmitting the advice and auxiliary images from the server to a user's terminal. This enables even general users without specialized knowledge to easily obtain specific advice for effectively improving their children's exercise performance.

[0455] The "means for receiving video" refers to a device or software for transmitting video data taken by a user to a server and receiving the video data.

[0456] The "means for dividing a moving image into a plurality of frames" refers to a device or software that divides received moving image data into a plurality of still images (frames) in time.

[0457] The "means for detecting the positions of joints and body parts of an object in each frame" refers to a device or software for identifying the positions of an object, specifically joints and body parts, from each divided frame.

[0458] The "means for analyzing movement based on detected positions" refers to a device or software for analyzing the position data of detected joints or body parts and evaluating the movement of an object.

[0459] The "means for generating exercise improvement advice based on the analysis results" refers to a device or software that generates specific advice for improving the athletic ability of a subject based on the analysis results of exercise movements.

[0460] The "means for outputting the generated exercise improvement advice and the visual support image" refers to a device or software for presenting the generated advice and the accompanying visual support image to the user.

[0461] The "means for transmitting advice and auxiliary images from the server to the user's terminal" refers to a device or software that transmits the generated advice and auxiliary images to the terminal used by the user via a network.

[0462] MODE FOR CARRYING OUT THE INVENTION

[0463] The system of the present invention analyzes video and automatically generates specific advice to improve children's athletic performance. An embodiment of the system will be described below.

[0464] First, a user uses their smartphone to record their child's exercise. For example, they can record a video of their child shooting a soccer ball. The user saves the video on their smartphone and then uploads it to a server via a dedicated application.

[0465] The dedicated application is commonly available and easy to use: users simply launch the application, select the video they have taken, and tap the upload button to send the video to a cloud server.

[0466] The server receives the uploaded video and saves it. After saving, the server checks the integrity of the video file by checking the file match using file hash values, etc. Once the integrity of the video is confirmed, it is converted into a format for analysis. For example, if the video is in MP4 format, it is converted into still images of each frame. The frame images saved in a temporary folder are used as input for the generative AI model.

[0467] The server splits the video into a fixed frame rate and saves each frame as an image file. These image files are listed as inputs to the generative AI model and prepared as a JSON-formatted data packet. The server then sends this data to the generative AI model along with a prompt.

[0468] The generative AI model analyzes each frame image to identify the location of the child's major joints (knees, ankles, shoulders, etc.). The detected data is collected in chronological order and used to evaluate the child's athletic performance. The evaluation results are compared with existing technical databases to generate appropriate advice.

[0469] The generated advice is specific, such as "Your foot usage is unstable, so try to find your balance" or "Your knee angle is shallow, so you should bend it a bit more." Visual support images are also created, allowing users to understand the advice more intuitively.

[0470] Finally, the generated advice and supporting images are sent from the server to the user's device. The user can check that new advice is available through a notification in the dedicated application. The user can then review the advice through the application and provide specific guidance to their child.

[0471] Specific examples

[0472] Analysis of soccer shooting movements

[0473] Consider the case where a user records a video of their child shooting a soccer ball and uploads it using a dedicated application. The server receives the video and prepares it for frame-by-frame analysis. The generative AI model analyzes the foot movement, knee angle, body balance, etc. during the shot, and identifies that the knees are not bent enough, resulting in a lack of shooting power. Based on the analysis results, it generates and provides a supporting image of the correct form along with specific advice such as "Bend your knees a bit more." The user can receive this information and tell their child specific areas for improvement, encouraging effective practice.

[0474] Prompt Sentence Examples

[0475] "Analyze a video of a child shooting a soccer ball and generate advice on foot movement and knee angle."

[0476] In this way, the system provides specific advice to effectively improve children's motor skills without requiring specialized knowledge.

[0477] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0478] Step 1:

[0479] Video recording and uploading

[0480] The user uses their smartphone to record their child's athletic activity (e.g., a soccer shot). Once the recording is complete, they launch the dedicated application and select the video file. The user then taps the "Upload" button, and the video file is sent to the server.

[0481] Input: Video file taken by the user

[0482] Output: Video file sent to the server

[0483] Step 2:

[0484] Video reception and integrity check

[0485] The server receives the video file sent by the user. Once received, it verifies the hash value and file size to confirm the integrity of the file. Once the integrity is confirmed, the video is stored in storage.

[0486] Input: Video file submitted by the user

[0487] Output: Video file with verified integrity

[0488] Step 3:

[0489] Video format conversion for analysis

[0490] The server reads the video files stored in the storage and converts them into a format that can be analyzed (e.g., a series of still images). Specifically, it converts MP4 format videos into JPEG images for each frame.

[0491] Input: Video file with verified integrity

[0492] Output: Separate image files for each frame

[0493] Step 4:

[0494] Frame Split

[0495] The server splits the converted video into a fixed frame rate (e.g. 30fps) and saves each frame as an image file. The saved frame images are placed in a temporary folder.

[0496] Input: Video file converted to analysis format

[0497] Output: Image files for each frame

[0498] Step 5:

[0499] Sending data to the generation AI

[0500] The server prepares the image files for each frame to be sent to the generative AI model. To do this, it lists the paths of the image files and generates a JSON-formatted data packet. It then sends the generated JSON data along with the prompt to the generative AI model.

[0501] Input: Image file for each frame, prompt text

[0502] Output: JSON data packet sent to the generative AI model

[0503] Step 6:

[0504] Joint position detection and motion analysis

[0505] The generative AI model analyzes each transmitted frame and identifies the location of key joints (knees, ankles, shoulders, etc.). This position data is collected as time series data and used to evaluate the movement. During the evaluation process, it compares the data with a database to identify the optimal movement pattern.

[0506] Input: Image files for each frame, JSON data

[0507] Output: Joint position data and motion analysis results

[0508] Step 7:

[0509] Generating exercise improvement advice

[0510] The server generates exercise improvement advice based on the analysis results received from the generative AI model. Specific advice includes "bend your knees more" and "maintain your body balance." Visual support images are also created to help users intuitively understand the advice.

[0511] Input: Motion analysis results

[0512] Output: Specific exercise improvement advice, visual aid images

[0513] Step 8:

[0514] Sending advice and auxiliary images

[0515] The server sends the generated advice and auxiliary images to the user's device, and the notification function of the dedicated application is used to inform the user that new advice is available.

[0516] Input: Exercise improvement advice, visual aid images

[0517] Output: Sends advice and auxiliary images to the user's terminal

[0518] Step 9:

[0519] Confirmation of advice and guidance

[0520] The user checks the advice and supplementary images received on the device and provides specific instructions to the child on how to improve their exercise. For example, the user might be told to "bend your knees a little more," and the supplementary images would be used to demonstrate the correct form.

[0521] Input: Advice and auxiliary images sent to the user's device

[0522] Output: Specific exercise improvement guidance for children

[0523] Through the above steps, the system can provide the user with specific advice on how to improve their exercise, and support their child in improving their athletic ability.

[0524] (Application example 1)

[0525] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0526] Conventional factory robot motion analysis requires specialized knowledge, making it difficult to make efficient improvements. Furthermore, when humans observe motion and suggest areas for improvement, they may overlook subtle motion characteristics. Therefore, there is a need for a method to provide specific and accurate advice to maximize robot efficiency.

[0527] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0528] In this invention, the server

[0529] means for receiving video;

[0530] A means for dividing a video into frames;

[0531] means for detecting the position of joints and body parts of the object in each frame;

[0532] means for analyzing the movement based on the detected position;

[0533] A means for generating behavior improvement advice based on the analysis results;

[0534] means for outputting the generated advice;

[0535] A means to film and upload the robot's movements;

[0536] a means for evaluating the efficiency of the operation using the generative AI model;

[0537] A means for identifying a behavior pattern based on the evaluation results;

[0538] This makes it possible to provide specific and accurate advice to improve the efficiency of robot operations, even without specialized knowledge.

[0539] "Means for receiving video" refers to a device or function for transferring and importing video shot by a user from a device to a server.

[0540] "Means for dividing a video into multiple frames" refers to a technique or method for dividing a received video into frames and analyzing each frame individually.

[0541] "Means for detecting the positions of joints and body parts of an object in each frame" refers to a technology that identifies the positions of joints and body parts of the robot or person being analyzed for each frame.

[0542] The "means for analyzing movement based on detected positions" refers to a device or method for analyzing the movement pattern of an object using position data of detected joints and parts.

[0543] The "means for generating advice to improve movement based on the analysis results" is a technology for generating specific advice to improve movement based on the analyzed movement pattern.

[0544] The "means for outputting the generated advice" refers to a method or device for providing the generated behavior improvement advice to the user.

[0545] "Means for filming and uploading robot operations" refers to a method or technology for filming the operations of a factory robot and uploading the video data to a server.

[0546] "Means for evaluating the efficiency of movements using a generative AI model" refers to a technology that analyzes data obtained from video captured using a generative AI model and evaluates the efficiency of movements.

[0547] "Means for identifying movement patterns based on evaluation results" refers to a technology that identifies the movement patterns of a robot based on the evaluated data and extracts specific key points for improving efficiency.

[0548] The system of the present invention provides specific advice for improving operational efficiency through the analysis of the operation of a factory robot. An embodiment of the system will be described below.

[0549] First, the user records the factory robot's operations using a device such as a smartphone or head-mounted display (HMD). The user then uploads the recorded video to a cloud server via a dedicated analysis application. The application is easy to use; the user simply selects a video and taps the upload button to send the video to the cloud server.

[0550] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis. Next, it splits the video into frames and prepares each frame to be sent to the generative AI model.

[0551] The generative AI model detects the positions of the robot's joints and parts in each frame. For example, it identifies the positions of key parts such as arms, joints, and grippers, and collects these positions as time-series data. Based on the collected data, it analyzes the robot's motion and evaluates its movement patterns. During this evaluation process, it compares them with a database of existing technologies to find efficient movement patterns and technologies.

[0552] Based on the analysis results, the generative AI model generates specific advice for improving the movement. For example, "The arm's movement is slow, so you need to increase the speed" or "The joint angle is inappropriate, so you need to adjust the angle." It is also possible to create visual support images, which allows the user to understand the advice more intuitively.

[0553] Finally, the generated advice and auxiliary images are sent from the server to the user's device, where the user can review the advice and adjust the robot's operation through the application. In this way, the system can provide specific and effective support for improving the robot's operation efficiency, even without specialized knowledge.

[0554] Example: Analysis of welding motion of a robot arm

[0555] Consider a case where a user records a video of a robot arm's welding operations and uploads it using a dedicated analysis application. The server receives the video and analyzes it frame by frame. The generative AI model analyzes the arm's movements during welding, the joint angles, and areas of slow movement, and generates specific advice such as, "The welding quality is declining because the arm is moving too slowly." Based on this result, the model generates and provides specific advice such as, "Make the movement a little faster," along with a supplementary image of the correct operation. The user who receives this information can adjust the robot's behavior to improve welding quality.

[0556] Example prompt for a generative AI model:

[0557] Detect joint positions from frame images of a robot arm and generate advice to improve the efficiency of the movement. Joint position data is provided for each frame of the video below. Please provide specific advice on how to improve the movement based on this data.

[0558] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0559] Step 1:

[0560] The user records the robot's movements using a smartphone or head-mounted display. The input is the captured video. This video data is stored on the user's device.

[0561] Step 2:

[0562] Users upload the videos they have taken to the cloud server via a dedicated analysis application. The input is the taken video file, and the output is the video data saved on the cloud server. The user selects a video and taps the upload button to send the video to the cloud server.

[0563] Step 3:

[0564] The cloud server receives the uploaded video and checks the integrity of the video file. The input is the video data stored on the cloud server, and the output is the video data whose integrity has been checked. Once the integrity is confirmed, the video is saved in storage.

[0565] Step 4:

[0566] The server converts the video into a format for analysis. The input is the video data that has been validated for integrity, and the output is the converted video data in the format for analysis. This process splits the video into frames, and each frame is ready to be sent to the generative AI model.

[0567] Step 5:

[0568] The server sends each frame to the generative AI model, which detects the position of the robot's joints and body parts for each frame. The input is video data (each frame) in an analysis format, and the output is position data of the joints and body parts in each frame. The generative AI model uses a machine learning algorithm to identify the position of the joints and body parts in each frame.

[0569] Step 6:

[0570] The server analyzes the robot's movements based on joint and body part position data. The input is joint and body part position data, and the output is the analysis results of the movements. During this analysis process, it compares the data with an existing technology database and matches it with efficient movement patterns and technologies.

[0571] Step 7:

[0572] The generative AI model generates movement improvement advice based on the analysis results. The input is the analysis results of the exercise movement, and the output is movement improvement advice. During this process, visual support images are generated along with specific advice.

[0573] Step 8:

[0574] The server sends the generated advice and auxiliary image to the user's device. The input is the behavior improvement advice and auxiliary image, and the output is the advice and auxiliary image displayed on the user's device. The user can check this advice through the application.

[0575] Step 9:

[0576] The user adjusts the robot's behavior based on the behavior improvement advice. The input is the displayed advice, and the output is the adjusted robot behavior. The user adjusts the robot's settings based on specific improvements to improve operational efficiency.

[0577] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0578] The system according to the present invention not only analyzes videos and provides exercise improvement advice, but also incorporates an emotion engine that recognizes the user's emotions and adjusts the content and presentation method of the advice accordingly. A specific embodiment of the system will be described below.

[0579] First, the user uses their device to record their child's exercise. For example, they can use a smartphone to record their child practicing dribbling a basketball. The user then uploads the video to the server via a dedicated application. This application is easy to use; the user simply selects a video and presses the upload button, and the video is sent to the server.

[0580] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis. Next, it splits the video into frames and prepares each frame to be sent to the generation AI.

[0581] The generative AI detects the positions of the child's joints and body parts in each frame. For example, it identifies the positions of major joints such as elbows, wrists, and torso, and collects their positions as time-series data. It then analyzes the child's movements based on this position data and evaluates their movement patterns. During this evaluation process, it compares them with an existing technical database to identify any discrepancies from the correct form and technique.

[0582] Based on the analysis results, the AI ​​generates specific advice for improving your movements. For example, it might say, "Your wrist movements are stiff when dribbling, so try to relax them." It can also create visual aids, which can help users understand the advice more intuitively.

[0583] A distinctive feature of the present invention is the incorporation of an emotion engine. This emotion engine can recognize the user's facial expressions and voice through the device's camera and microphone and evaluate the user's emotional state. The emotion data acquired by the emotion engine is reflected in the content and presentation method of advice. For example, if the user is feeling irritated or anxious, the tone of the advice can be changed to a gentler one, encouraging the user to take a more positive approach. Also, if the user is enjoying themselves, encouraging words can be added to maintain that state.

[0584] Finally, the generated advice and auxiliary images are sent from the server to the user's device. The user can then review the advice and provide specific guidance to their child through the application. In this way, the system provides specific and effective support for improving children's motor skills, while also being able to flexibly respond to the user's emotional state.

[0585] Example: Analysis of basketball dribbling practice

[0586] Consider the case where a user films their child practicing basketball dribbling and uploads it using a dedicated application. The server receives the video and analyzes it frame by frame. The generation AI analyzes the movement of each joint in the dribbling motion and identifies stiff wrist movement. Based on the analysis results, it generates specific advice such as "Relax your wrist."

[0587] Meanwhile, the emotion engine recognizes when the user is frustrated and adjusts the tone of the advice to be gentler, for example, by saying, "It may be difficult at first, but you can improve little by little." This allows the user to provide guidance to their child in a positive manner.

[0588] As described above, this system not only helps children improve their motor skills, but also provides a flexible approach that takes into consideration the user's emotions.

[0589] The processing flow will be explained below.

[0590] Step 1:

[0591] A user uses their own device to record a video of their child exercising. For example, a user uses a smartphone to record a video of their child practicing basketball dribbling.

[0592] Step 2:

[0593] The user launches the dedicated application, selects the video they have taken, and taps the "Upload" button.

[0594] Step 3:

[0595] The device sends the selected video to the server. When the video is sent, a checksum and other verifications are performed to ensure the integrity of the data.

[0596] Step 4:

[0597] The server receives the video data sent from the device, reconfirms the integrity of the video file, and stores it in temporary storage.

[0598] Step 5:

[0599] The server converts the received video into a standard format for analysis, converting the codec and adjusting the resolution as necessary.

[0600] Step 6:

[0601] The server splits the video into frames, extracts each frame as an individual image file, and stores the timestamp information for each frame.

[0602] Step 7:

[0603] The server sends the divided frame data to the generation AI, which receives it and detects the positions of joints and body parts in each frame.

[0604] Step 8:

[0605] Based on the detected joint position data, the generative AI analyzes the movement over time, tracking the movement patterns and positions of each joint and extracting the characteristic points of the movement.

[0606] Step 9:

[0607] The generative AI compares the extracted feature points with an existing technical database, and based on the comparison results, identifies any differences between the target athletic movement and correct form and technique.

[0608] Step 10:

[0609] Based on the analysis results, the generative AI generates specific advice for improving exercise, such as "relax your wrist and move more smoothly."

[0610] Step 11:

[0611] The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotional state, for example, detecting when the user is feeling irritated.

[0612] Step 12:

[0613] Based on the emotional data recognized by the emotion engine, the generative AI adjusts the way it presents advice. For example, if a user feels frustrated, it will soften the tone of the advice and add encouraging words such as "Take your time and practice a little at a time."

[0614] Step 13:

[0615] The server transmits the generated advice and the adjusted advice data to the user's terminal.

[0616] Step 14:

[0617] The user checks the advice data sent from the server through an application on their device. The application displays the advice and provides visual support images to help the user understand the specific improvements required.

[0618] Step 15:

[0619] The user provides specific advice for improvement based on the application, and by filming the exercise again and repeating these steps, the user can continuously identify areas for improvement and improve athletic ability.

[0620] Example 2

[0621] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0622] Conventional video analysis systems can analyze exercise videos taken by users and provide exercise improvement advice, but they provide uniform advice without considering the user's emotions, making it difficult to provide appropriate guidance tailored to the user's situation.In addition, there is a lack of means to compare with technical databases to improve the accuracy of motion analysis, or to generate visual support images, making it difficult to deepen understanding of the analysis results.

[0623] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0624] In this invention, the server includes means for receiving a video, means for dividing the video into a plurality of frames, means for detecting the positions of joints and body parts of an object in each frame, means for evaluating the analyzed exercise movements by comparing with a specific database, means for generating exercise improvement advice based on the analysis results, means for recognizing the user's emotions, means for adjusting the generated advice based on the recognized emotions, and means for outputting the generated advice, thereby making it possible to provide flexible exercise improvement advice that takes the user's emotions into consideration.

[0625] The "means for receiving video" is a function for transmitting video data taken by a user to a server via a network and for the server to receive the video data.

[0626] "Means for dividing a video into multiple frames" is a function for dividing a received video into small still images (frames) along the time axis and generating data for each frame.

[0627] "Means for detecting the positions of joints and body parts of an object in each frame" refers to a technique for identifying the positions of joints and body parts relative to an object in each frame image and extracting their coordinate data.

[0628] The "means for evaluating the analyzed exercise movements by comparing with a specific database" is a function for comparing the analyzed exercise data with an existing reference database and evaluating the accuracy of the exercise and areas for improvement.

[0629] The "means for generating exercise improvement advice based on the analysis results" is a function for automatically generating specific improvement methods and instruction contents based on the evaluation results of the exercise analysis.

[0630] "Means for recognizing user emotions" refers to technology that analyzes information such as the user's facial expressions and voice to identify the user's emotional state.

[0631] The "means for adjusting the advice generated based on the recognized emotion" is a function for flexibly changing the content and tone of the exercise improvement advice generated according to the user's emotional state.

[0632] The "means for outputting generated advice" is a function for displaying or providing the generated exercise improvement advice and visual support images in an easily understandable form to the user.

[0633] MODE FOR CARRYING OUT THE INVENTION

[0634] The system according to the present invention not only analyzes videos and provides exercise improvement advice, but also incorporates an emotion recognition engine that recognizes the user's emotions and adjusts the content and presentation method of the advice accordingly. A specific embodiment of the system will be described below.

[0635] First, the user uses their device to record their child's exercise. For example, they can use a smartphone to record their child practicing dribbling a basketball. The user then uploads the video to the server via a dedicated application. This application is easy to use; the user simply selects a video and presses the upload button, and the video is sent to the server.

[0636] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis using software such as FFmpeg. Next, it splits the video into frames and prepares each frame to be sent to a generative AI (for example, using OpenPose or TensorFlow).

[0637] The generative AI detects the positions of the child's joints and body parts in each frame. For example, it identifies the positions of major joints such as elbows, wrists, and torso, and collects their positions as time-series data. It then analyzes the child's movements based on this position data and evaluates their movement patterns. During this evaluation process, it compares the child's movements with existing technical databases (e.g., Sports Motion Database) to identify any discrepancies with the correct form and technique.

[0638] Based on the analysis results, the AI ​​generates specific advice for improving your movements. For example, it might say, "Your wrist movements are stiff when dribbling, so try to relax them." It can also create visual aids, which can help users understand the advice more intuitively.

[0639] A distinctive feature of the present invention is the incorporation of an emotion engine. This emotion engine can recognize the user's facial expressions and voice through the device's camera and microphone and evaluate the user's emotional state. The emotion data acquired by the emotion engine is reflected in the content and presentation method of advice. For example, if the user is feeling irritated or anxious, the tone of the advice can be changed to a gentler one, encouraging the user to take a more positive approach. Also, if the user is enjoying themselves, encouraging words can be added to maintain that state.

[0640] Finally, the generated advice and auxiliary images are sent from the server to the user's device. The user can then review the advice and provide specific guidance to their child through the application. In this way, the system provides specific and effective support for improving children's motor skills, while also being able to flexibly respond to the user's emotional state.

[0641] Example: "Analysis of basketball dribbling practice"

[0642] Consider the case where a user films their child practicing basketball dribbling and uploads it using a dedicated application. The server receives the video and analyzes it frame by frame. The generation AI analyzes the movement of each joint in the dribbling motion and identifies stiff wrist movement. Based on the analysis results, it generates specific advice such as "Relax your wrist."

[0643] Meanwhile, the emotion engine recognizes when the user is frustrated and adjusts the tone of the advice to be gentler, for example, by saying, "It may be difficult at first, but you can improve little by little." This allows the user to provide guidance to their child in a positive manner.

[0644] Examples of prompt statements

[0645] Prompt statement:

[0646] I filmed a kid practicing dribbling a basketball. Analyze the uploaded video and identify stiff wrist movements. Then provide specific advice to relax the wrist. Also, adjust the tone of the advice to be gentler if the user is frustrated.

[0647] As described above, this system not only helps children improve their motor skills, but also provides a flexible approach that takes into consideration the user's emotions.

[0648] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0649] Step 1:

[0650] The user uses a device such as a smartphone to record their child's exercise. The user then uploads the video to a server using a dedicated application. The input is a video file, and the output is a video file stored on the server.

[0651] Step 2:

[0652] The server checks the received video and saves it in storage. At this time, it checks the integrity of the video file and checks whether it contains any invalid data. The input is the uploaded video file, and the output is the video file saved in storage whose integrity has been checked.

[0653] Step 3:

[0654] The server uses video processing software such as FFmpeg to convert the stored video into a format for analysis. The input in this step is a video file whose integrity has been confirmed, and the output is a video file converted into the format for analysis.

[0655] Step 4:

[0656] The server divides the video file for analysis into frames and generates image data for each frame. Specifically, it divides one second of video into 30 frames. The input for this step is the video file for analysis, and the output is still images for each frame (e.g., "frame0001.jpg", "frame0002.jpg").

[0657] Step 5:

[0658] The server sends each frame image to a generative AI (e.g., OpenPose or TensorFlow) to detect the positions of the child's joints and body parts in each frame. The input is the frame image data, and the output is joint position data (e.g., coordinate data for the left elbow, right wrist, and trunk).

[0659] Step 6:

[0660] The server analyzes the motion based on the detected joint position data. The analysis results are compared with a specific technique database (e.g., Sports Motion Database) to identify deviations from the correct form and technique. The input for this step is the joint position data, and the output is the evaluation result of the motion analysis (e.g., "The wrist movement is stiff").

[0661] Step 7:

[0662] The server generates exercise improvement advice based on the analysis results. The generation AI creates specific advice text (e.g., "Try to relax your wrist") and visual support images (e.g., illustrations of the movement to relax your wrist). The input is the evaluation results of the movement analysis, and the output is exercise improvement advice and support images.

[0663] Step 8:

[0664] The device uses a camera and microphone to collect the user's facial expressions and voice. This data is sent to the emotion engine to evaluate the user's emotional state. The input is the user's facial and voice data, and the output is the emotion recognition result (e.g., "irritated").

[0665] Step 9:

[0666] The server adjusts the tone and content of exercise improvement advice based on the emotion recognition results obtained from the emotion engine. For example, if a user feels frustrated, the tone may be changed to a gentler one, such as "It may be difficult at first, but you can improve little by little." The input is the emotion recognition result, and the output is the adjusted exercise improvement advice.

[0667] Step 10:

[0668] The server sends the final exercise improvement advice and supplementary images to the user's device. The user can then review the advice and provide specific guidance to their child through the application. The input is the adjusted exercise improvement advice and supplementary images, and the output is the final advice displayed on the user's device.

[0669] keyword

[0670] Generative AI model, prompt sentence

[0671] (Application example 2)

[0672] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0673] Conventional exercise improvement systems can analyze a user's exercise data and provide specific advice, but they are unable to adjust the content of the advice taking the user's emotions into account. As a result, the advice is often ineffective when the user is feeling frustrated or anxious. Furthermore, while it is important for exercise improvement advice to be presented in a visually easy-to-understand format, there has been a lack of effective ways to achieve this. This has led to issues such as insufficient user satisfaction and effective exercise improvement.

[0674] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0675] In this invention, the server includes means for receiving a video, means for dividing the video into a plurality of frames, means for detecting the positions of joints and body parts of an object in each frame, means for analyzing exercise movements based on the detected positions, means for generating exercise improvement advice based on the analysis results, means for outputting the generated advice, means for recognizing a user's emotion, and means for adjusting the content and presentation method of the advice according to the recognized emotion. This makes it possible to provide effective and flexible exercise improvement advice to the user, and to improve the acceptability and satisfaction of the advice by taking the user's emotion into consideration.

[0676] The "means for receiving video" is a mechanism by which a user sends video taken by a terminal to a server, and the server receives the video.

[0677] The "means for dividing a video into a plurality of frames" is a mechanism for dividing a received video into frames along the time axis and making each frame individually analyzable.

[0678] The "means for detecting the positions of the joints and body parts of the object in each frame" is a mechanism for identifying the positions of the joints and body parts of a person in the video frames to be analyzed and acquiring that data.

[0679] The "means for analyzing movement based on detected positions" is a mechanism for evaluating and analyzing movement based on acquired joint and body position data.

[0680] The "means for generating exercise improvement advice based on the analysis results" is a mechanism for generating specific advice for improving exercise based on the analysis results of exercise movements.

[0681] The "means for outputting generated advice" is a mechanism for presenting the generated exercise improvement advice to the user and displaying it in an easy-to-understand manner.

[0682] The "means for recognizing the user's emotions" is a mechanism for analyzing the user's facial expressions and voice through a camera or microphone and recognizing the user's emotional state.

[0683] The "means for adjusting the content and presentation method of advice according to the recognized emotion" is a mechanism for appropriately changing the content and presentation method of advice based on the user's emotional state.

[0684] The system according to the present invention provides exercise improvement support by combining video analysis and emotion recognition. This system not only analyzes videos to provide exercise improvement advice, but also incorporates an emotion engine that recognizes the user's emotions and adjusts the content and presentation method of the advice accordingly. Specific aspects of this system are described below.

[0685] First, the user films the exercise using their own device. An example of this is a child using a smartphone to film their basketball practice. The user then uploads the video to the server via a dedicated application. The application is designed to be user-friendly; simply by selecting a video and pressing the upload button, the video is automatically sent to the server.

[0686] The server stores the received video in storage, verifies the integrity of the video file, and then converts the video into a format for analysis. The video is then split into frames, and each frame is prepared to be sent to a generation AI for analysis. This generation AI uses a machine learning library such as TensorFlow.

[0687] The generative AI detects the positions of the joints and body parts of the subject (in this example, a child) in each frame. For example, it identifies the positions of major joints such as the elbows, wrists, and torso, and collects their positions as time-series data. It then analyzes the movement based on this position data and evaluates the movement patterns. During this evaluation process, it compares the results with an existing technical database to identify any discrepancies from the correct form and technique.

[0688] Based on the analysis results, the AI ​​generates specific advice for improving the movement. For example, it might say, "When shooting, your elbow position is low. Try to keep it a little higher." In addition, it creates auxiliary images that are easy to understand visually.

[0689] A distinctive feature of the present invention is the incorporation of an emotion engine. This emotion engine recognizes the user's facial expressions and voice through the device's camera and microphone and evaluates the user's emotional state. This evaluation utilizes emotion analysis APIs such as Microsoft Azure Cognitive Services. The emotion data acquired by the emotion engine is reflected in the content and presentation of the generated advice. For example, if the user is feeling frustrated or anxious, the tone of the advice can be made gentler, and soft words such as "Trial and error lead to growth. Take your time and practice slowly." If the user is enjoying themselves, encouraging words such as "That's great! Keep it up!" can be added to maintain that state of mind.

[0690] Finally, the generated advice and auxiliary images are sent from the server to the user's device, where the user can check the advice and receive specific guidance through the application.

[0691] For example, consider the following prompt:

[0692] python

[0693] import requests

[0694] import cv2

[0695] def upload_video(video_path):

[0696] url = 'http: / / yourserver.com / upload'

[0697] files = {'file': open(video_path, 'rb')}

[0698] response = requests.post(url, files=files)

[0699] if response.status_code == 200:

[0700] print("Video uploaded successfully!")

[0701] else:

[0702] print("Error in video upload")

[0703] def analyze_video_frames(frames, model):

[0704] results = []

[0705] for frame in frames:

[0706] processed_frame = preprocess_frame(frame)

[0707] result = model.predict(processed_frame)

[0708] results.append(result)

[0709] return results

[0710] def generate_advice(movement_data):

[0711] Analyze movement_data and generate advice

[0712] advice = "Your elbow position is incorrect during the shot. Try to keep it higher."

[0713] return advice

[0714] def emotion_adjust_advice(user_emotion, advice):

[0715] if user_emotion == 'frustrated':

[0716] adjusted_advice = "Don't worry, improvement takes time. " + advice

[0717] elif user_emotion == 'happy':

[0718] adjusted_advice = "Great job! Keep it up. " + advice

[0719] else:

[0720] adjusted_advice = advice

[0721] return adjusted_advice

[0722] Example usage

[0723] video_path = 'child_shooting_basketball.mp4'

[0724] upload_video(video_path)

[0725] This program aims to improve athletic ability by uploading videos taken by users to a server, analyzing the videos and recognizing emotions, and generating and presenting appropriate advice.

[0726] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0727] Step 1:

[0728] Recording and uploading videos

[0729] Action: A user uses a smartphone to record themselves exercising.

[0730] Input: The captured video file.

[0731] Output: The video file imported into the application.

[0732] Specific operation: The user presses the record button to record a video of their exercise. After recording is complete, the user presses the application's upload button to send the video to the server.

[0733] Step 2:

[0734] Receiving and storing videos

[0735] How it works: The server receives and stores videos uploaded by users.

[0736] Input: The uploaded video file.

[0737] Output: Video files stored in the server's storage.

[0738] What it does: The server receives an HTTP request, saves the file to disk, checks the integrity of the saved file, and prepares it for analysis.

[0739] Step 3:

[0740] Video frame division

[0741] How it works: The server splits the stored video into frames.

[0742] Input: A saved video file.

[0743] Output: Image data per frame.

[0744] Specific operation: Using the OpenCV library, a video file is read and split into frame-by-frame still images. Each frame is then converted into the format required for subsequent analysis.

[0745] Step 4:

[0746] Position detection of joints and body parts

[0747] How it works: The server detects the positions of joints and body parts for each frame.

[0748] Input: Frame-by-frame image data.

[0749] Output: Position data of joints and body parts (time series data).

[0750] Specific operation: Using a generative AI model such as TensorFlow, joints and body parts (e.g., elbows, wrists, torso, etc.) are analyzed for each frame to obtain positional data.

[0751] Step 5:

[0752] Analysis of athletic movements

[0753] Movement: The server analyzes the movement based on the detected location data.

[0754] Input: Position data of joints and body parts.

[0755] Output: Motor behavior evaluation data.

[0756] Specific behavior: The acquired position data is analyzed as time series data, and behavior patterns are evaluated by comparing it with a technical database. Inappropriate behavior is identified and the analysis results are output.

[0757] Step 6:

[0758] Generating exercise improvement advice

[0759] Action: The server generates exercise improvement advice based on the results of the exercise motion analysis.

[0760] Input: Exercise behavior assessment data.

[0761] Output: Text and supporting images of exercise improvement advice.

[0762] Specific actions: Based on the analysis results, specific advice is generated based on the situation, such as, "When shooting, your elbow is positioned too low, so try to keep it a little higher." Supporting images are also created to make the game easier to understand visually.

[0763] Step 7:

[0764] User Emotion Recognition

[0765] How it works: The server recognizes the user's emotions.

[0766] Input: User's facial and voice data.

[0767] Output: User's emotional state data.

[0768] Specific operation: Using emotion analysis APIs such as Microsoft Azure Cognitive Services, the system analyzes facial expression data and voice data acquired through the camera to evaluate the user's emotional state (e.g., irritation, joy, calmness, etc.).

[0769] Step 8:

[0770] Advice adjustment

[0771] How it works: The server adjusts the content and presentation of advice based on the results of emotion recognition.

[0772] Input: Exercise improvement advice, user emotional state data.

[0773] Output: Tailored exercise improvement advice.

[0774] Specific actions: Change the tone of advice based on emotion recognition data, e.g., "Trial and error leads to growth. Take your time and practice slowly." Add words of encouragement according to the emotion.

[0775] Step 9:

[0776] Providing advice

[0777] How it works: The server sends the generated and tailored advice to the user's device and presents it through the application.

[0778] Enter: tailored exercise improvement advice.

[0779] Output: Advice and auxiliary images displayed on the user's terminal.

[0780] Specific operation: The generated advice and auxiliary images are sent to the user's device, and displayed through the application so that the user can check and execute them.

[0781] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0782] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0783] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0784] [Third embodiment]

[0785] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0786] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0787] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0788] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0789] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0790] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0791] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0792] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0793] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0794] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0795] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0796] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0797] The system of the present invention analyzes video and automatically generates specific advice to improve a child's athletic performance. An embodiment of the system will be described below.

[0798] First, the user uses their device to record their child's exercise. For example, they can use their smartphone to record their child shooting a soccer ball. The user then uploads the video to the server via a dedicated application. The application is easy to use; the user simply selects a video and taps the upload button, and the video is sent to the cloud server.

[0799] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis. Next, it splits the video into frames and prepares each frame to be sent to the generation AI.

[0800] The generative AI detects the position of the child's joints and body parts in each frame. For example, it identifies the locations of major joints such as the knees, ankles, and shoulders, and collects these positions as time-series data. It then analyzes the child's movements based on this position data and evaluates their movement patterns. During this evaluation process, it compares them with an existing technique database to match them with the correct form and technique.

[0801] Based on the analysis results, the AI ​​generates specific exercise improvement advice, such as "Your foot usage is unstable, so try to improve your balance" or "Your knee angle is shallow, so you should bend it a bit more." It can also create visual support images, which allow users to understand the advice more intuitively.

[0802] Finally, the generated advice and auxiliary images are sent from the server to the user's device. The user can then review the advice through the application and provide specific guidance to their child. In this way, the system provides specific and effective support for improving children's motor skills.

[0803] Example: Analysis of soccer shooting movements

[0804] Consider the case where a user takes a video of their child shooting a soccer ball and uploads it using a dedicated application. The server receives the video and analyzes each frame. The generating AI analyzes the foot movement, knee angle, body balance, etc. during the shot, and determines that the knees are not bent enough, resulting in a lack of shooting power. Based on the analysis results, it generates and provides specific advice such as "Bend your knees a bit more," along with a supplementary image of the correct form. The user who receives this information can then tell their child specific areas for improvement and encourage effective practice.

[0805] As described above, this system can provide specific advice to effectively improve children's motor skills, even without specialized knowledge.

[0806] The processing flow will be explained below.

[0807] Step 1:

[0808] Users record videos of their children's exercise on their own devices, then launch the dedicated application, select the video, and tap the "Upload" button.

[0809] Step 2:

[0810] The device sends the selected video to the server. When sending the video, it also performs checksum verification and other checks to ensure the integrity of the data.

[0811] Step 3:

[0812] The server receives the video data sent from the device, reconfirms the integrity of the video file, and stores it in temporary storage.

[0813] Step 4:

[0814] The server converts the received video into a standard format for analysis, converting the codec and adjusting the resolution if necessary.

[0815] Step 5:

[0816] The server splits the video into frames, extracts each frame as an individual image file, and stores the timestamp information for each frame.

[0817] Step 6:

[0818] The server sends the divided frame data to the generation AI, which receives it and detects the positions of joints and body parts in each frame.

[0819] Step 7:

[0820] Based on the detected joint position data, the generative AI analyzes the movement over time. Specifically, it tracks the movement patterns and positions of each joint and extracts the characteristic points of the movement.

[0821] Step 8:

[0822] The generative AI compares the extracted feature points with an existing technical database, and based on the comparison results, identifies any differences between the target athletic movement and correct form and technique.

[0823] Step 9:

[0824] Based on the analysis results, the generative AI generates specific advice for improving exercise, which is output in text format and as visual support images.

[0825] Step 10:

[0826] The server receives the generated advice data, associates it with the user profile, and transmits the advice data to the terminal.

[0827] Step 11:

[0828] The user checks the advice data sent from the server through the application on the device. The application displays the advice content and also provides visual support images to help the user understand it.

[0829] Step 12:

[0830] The user provides instruction to their child based on specific improvement advice provided through the application. By filming their exercise and conducting repeated analysis, the system continuously identifies areas for improvement and aims to improve athletic ability.

[0831] Example 1

[0832] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0833] With conventional exercise improvement systems, it was difficult for ordinary users without specialized knowledge to obtain specific advice to effectively improve children's exercise performance. Furthermore, using a video analysis system required advanced technology and specialized equipment, making it difficult for many users to use. This limited the opportunities for many children to learn correct form and techniques.

[0834] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0835] In this invention, the server includes means for receiving video, means for dividing the video into frames, means for detecting the positions of joints and body parts of an object in each frame, means for analyzing exercise movements based on the detected positions, means for generating exercise improvement advice based on the analysis results, means for outputting the generated exercise improvement advice and visual auxiliary images, and means for transmitting the advice and auxiliary images from the server to a user's terminal. This enables even general users without specialized knowledge to easily obtain specific advice for effectively improving their children's exercise performance.

[0836] The "means for receiving video" refers to a device or software for transmitting video data taken by a user to a server and receiving the video data.

[0837] The "means for dividing a moving image into a plurality of frames" refers to a device or software that divides received moving image data into a plurality of still images (frames) in time.

[0838] The "means for detecting the positions of joints and body parts of an object in each frame" refers to a device or software for identifying the positions of an object, specifically joints and body parts, from each divided frame.

[0839] The "means for analyzing movement based on detected positions" refers to a device or software for analyzing the position data of detected joints or body parts and evaluating the movement of an object.

[0840] The "means for generating exercise improvement advice based on the analysis results" refers to a device or software that generates specific advice for improving the athletic ability of a subject based on the analysis results of exercise movements.

[0841] The "means for outputting the generated exercise improvement advice and the visual support image" refers to a device or software for presenting the generated advice and the accompanying visual support image to the user.

[0842] The "means for transmitting advice and auxiliary images from the server to the user's terminal" refers to a device or software that transmits the generated advice and auxiliary images to the terminal used by the user via a network.

[0843] MODE FOR CARRYING OUT THE INVENTION

[0844] The system of the present invention analyzes video and automatically generates specific advice to improve children's athletic performance. An embodiment of the system will be described below.

[0845] First, a user uses their smartphone to record their child's exercise. For example, they can record a video of their child shooting a soccer ball. The user saves the video on their smartphone and then uploads it to a server via a dedicated application.

[0846] The dedicated application is commonly available and easy to use: users simply launch the application, select the video they have taken, and tap the upload button to send the video to a cloud server.

[0847] The server receives the uploaded video and saves it. After saving, the server checks the integrity of the video file by checking the file match using file hash values, etc. Once the integrity of the video is confirmed, it is converted into a format for analysis. For example, if the video is in MP4 format, it is converted into still images of each frame. The frame images saved in a temporary folder are used as input for the generative AI model.

[0848] The server splits the video into a fixed frame rate and saves each frame as an image file. These image files are listed as inputs to the generative AI model and prepared as a JSON-formatted data packet. The server then sends this data to the generative AI model along with a prompt.

[0849] The generative AI model analyzes each frame image to identify the location of the child's major joints (knees, ankles, shoulders, etc.). The detected data is collected in chronological order and used to evaluate the child's athletic performance. The evaluation results are compared with existing technical databases to generate appropriate advice.

[0850] The generated advice is specific, such as "Your foot usage is unstable, so try to find your balance" or "Your knee angle is shallow, so you should bend it a bit more." Visual support images are also created, allowing users to understand the advice more intuitively.

[0851] Finally, the generated advice and supporting images are sent from the server to the user's device. The user can check that new advice is available through a notification in the dedicated application. The user can then review the advice through the application and provide specific guidance to their child.

[0852] Specific examples

[0853] Analysis of soccer shooting movements

[0854] Consider the case where a user records a video of their child shooting a soccer ball and uploads it using a dedicated application. The server receives the video and prepares it for frame-by-frame analysis. The generative AI model analyzes the foot movement, knee angle, body balance, etc. during the shot, and identifies that the knees are not bent enough, resulting in a lack of shooting power. Based on the analysis results, it generates and provides a supporting image of the correct form along with specific advice such as "Bend your knees a bit more." The user can receive this information and tell their child specific areas for improvement, encouraging effective practice.

[0855] Prompt Sentence Examples

[0856] "Analyze a video of a child shooting a soccer ball and generate advice on foot movement and knee angle."

[0857] In this way, the system provides specific advice to effectively improve children's motor skills without requiring specialized knowledge.

[0858] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0859] Step 1:

[0860] Video recording and uploading

[0861] The user uses their smartphone to record their child's athletic activity (e.g., a soccer shot). Once the recording is complete, they launch the dedicated application and select the video file. The user then taps the "Upload" button, and the video file is sent to the server.

[0862] Input: Video file taken by the user

[0863] Output: Video file sent to the server

[0864] Step 2:

[0865] Video reception and integrity check

[0866] The server receives the video file sent by the user. Once received, it verifies the hash value and file size to confirm the integrity of the file. Once the integrity is confirmed, the video is stored in storage.

[0867] Input: Video file submitted by the user

[0868] Output: Video file with verified integrity

[0869] Step 3:

[0870] Video format conversion for analysis

[0871] The server reads the video files stored in the storage and converts them into a format that can be analyzed (e.g., a series of still images). Specifically, it converts MP4 format videos into JPEG images for each frame.

[0872] Input: Video file with verified integrity

[0873] Output: Separate image files for each frame

[0874] Step 4:

[0875] Frame Split

[0876] The server splits the converted video into a fixed frame rate (e.g. 30fps) and saves each frame as an image file. The saved frame images are placed in a temporary folder.

[0877] Input: Video file converted to analysis format

[0878] Output: Image files for each frame

[0879] Step 5:

[0880] Sending data to the generation AI

[0881] The server prepares the image files for each frame to be sent to the generative AI model. To do this, it lists the paths of the image files and generates a JSON-formatted data packet. It then sends the generated JSON data along with the prompt to the generative AI model.

[0882] Input: Image file for each frame, prompt text

[0883] Output: JSON data packet sent to the generative AI model

[0884] Step 6:

[0885] Joint position detection and motion analysis

[0886] The generative AI model analyzes each transmitted frame and identifies the location of key joints (knees, ankles, shoulders, etc.). This position data is collected as time series data and used to evaluate the movement. During the evaluation process, it compares the data with a database to identify the optimal movement pattern.

[0887] Input: Image files for each frame, JSON data

[0888] Output: Joint position data and motion analysis results

[0889] Step 7:

[0890] Generating exercise improvement advice

[0891] The server generates exercise improvement advice based on the analysis results received from the generative AI model. Specific advice includes "bend your knees more" and "maintain your body balance." Visual support images are also created to help users intuitively understand the advice.

[0892] Input: Motion analysis results

[0893] Output: Specific exercise improvement advice, visual aid images

[0894] Step 8:

[0895] Sending advice and auxiliary images

[0896] The server sends the generated advice and auxiliary images to the user's device, and the notification function of the dedicated application is used to inform the user that new advice is available.

[0897] Input: Exercise improvement advice, visual aid images

[0898] Output: Sends advice and auxiliary images to the user's terminal

[0899] Step 9:

[0900] Confirmation of advice and guidance

[0901] The user checks the advice and supplementary images received on the device and provides specific instructions to the child on how to improve their exercise. For example, the user might be told to "bend your knees a little more," and the supplementary images would be used to demonstrate the correct form.

[0902] Input: Advice and auxiliary images sent to the user's device

[0903] Output: Specific exercise improvement guidance for children

[0904] Through the above steps, the system can provide the user with specific advice on how to improve their exercise, and support their child in improving their athletic ability.

[0905] (Application example 1)

[0906] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0907] Conventional factory robot motion analysis requires specialized knowledge, making it difficult to make efficient improvements. Furthermore, when humans observe motion and suggest areas for improvement, they may overlook subtle motion characteristics. Therefore, there is a need for a method to provide specific and accurate advice to maximize robot efficiency.

[0908] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0909] In this invention, the server

[0910] means for receiving video;

[0911] A means for dividing a video into frames;

[0912] means for detecting the position of joints and body parts of the object in each frame;

[0913] means for analyzing the movement based on the detected position;

[0914] A means for generating behavior improvement advice based on the analysis results;

[0915] means for outputting the generated advice;

[0916] A means to film and upload the robot's movements;

[0917] a means for evaluating the efficiency of the operation using the generative AI model;

[0918] A means for identifying a behavior pattern based on the evaluation results;

[0919] This makes it possible to provide specific and accurate advice to improve the efficiency of robot operations, even without specialized knowledge.

[0920] "Means for receiving video" refers to a device or function for transferring and importing video shot by a user from a device to a server.

[0921] "Means for dividing a video into multiple frames" refers to a technique or method for dividing a received video into frames and analyzing each frame individually.

[0922] "Means for detecting the positions of joints and body parts of an object in each frame" refers to a technology that identifies the positions of joints and body parts of the robot or person being analyzed for each frame.

[0923] The "means for analyzing movement based on detected positions" refers to a device or method for analyzing the movement pattern of an object using position data of detected joints and parts.

[0924] The "means for generating advice to improve movement based on the analysis results" is a technology for generating specific advice to improve movement based on the analyzed movement pattern.

[0925] The "means for outputting the generated advice" refers to a method or device for providing the generated behavior improvement advice to the user.

[0926] "Means for filming and uploading robot operations" refers to a method or technology for filming the operations of a factory robot and uploading the video data to a server.

[0927] "Means for evaluating the efficiency of movements using a generative AI model" refers to a technology that analyzes data obtained from video captured using a generative AI model and evaluates the efficiency of movements.

[0928] "Means for identifying movement patterns based on evaluation results" refers to a technology that identifies the movement patterns of a robot based on the evaluated data and extracts specific key points for improving efficiency.

[0929] The system of the present invention provides specific advice for improving operational efficiency through the analysis of the operation of a factory robot. An embodiment of the system will be described below.

[0930] First, the user records the factory robot's operations using a device such as a smartphone or head-mounted display (HMD). The user then uploads the recorded video to a cloud server via a dedicated analysis application. The application is easy to use; the user simply selects a video and taps the upload button to send the video to the cloud server.

[0931] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis. Next, it splits the video into frames and prepares each frame to be sent to the generative AI model.

[0932] The generative AI model detects the positions of the robot's joints and parts in each frame. For example, it identifies the positions of key parts such as arms, joints, and grippers, and collects these positions as time-series data. Based on the collected data, it analyzes the robot's motion and evaluates its movement patterns. During this evaluation process, it compares them with a database of existing technologies to find efficient movement patterns and technologies.

[0933] Based on the analysis results, the generative AI model generates specific advice for improving the movement. For example, "The arm's movement is slow, so you need to increase the speed" or "The joint angle is inappropriate, so you need to adjust the angle." It is also possible to create visual support images, which allows the user to understand the advice more intuitively.

[0934] Finally, the generated advice and auxiliary images are sent from the server to the user's device, where the user can review the advice and adjust the robot's operation through the application. In this way, the system can provide specific and effective support for improving the robot's operation efficiency, even without specialized knowledge.

[0935] Example: Analysis of welding motion of a robot arm

[0936] Consider a case where a user records a video of a robot arm's welding operations and uploads it using a dedicated analysis application. The server receives the video and analyzes it frame by frame. The generative AI model analyzes the arm's movements during welding, the joint angles, and areas of slow movement, and generates specific advice such as, "The welding quality is declining because the arm is moving too slowly." Based on this result, the model generates and provides specific advice such as, "Make the movement a little faster," along with a supplementary image of the correct operation. The user who receives this information can adjust the robot's behavior to improve welding quality.

[0937] Example prompt for a generative AI model:

[0938] Detect joint positions from frame images of a robot arm and generate advice to improve the efficiency of the movement. Joint position data is provided for each frame of the video below. Please provide specific advice on how to improve the movement based on this data.

[0939] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0940] Step 1:

[0941] The user records the robot's movements using a smartphone or head-mounted display. The input is the captured video. This video data is stored on the user's device.

[0942] Step 2:

[0943] Users upload the videos they have taken to the cloud server via a dedicated analysis application. The input is the taken video file, and the output is the video data saved on the cloud server. The user selects a video and taps the upload button to send the video to the cloud server.

[0944] Step 3:

[0945] The cloud server receives the uploaded video and checks the integrity of the video file. The input is the video data stored on the cloud server, and the output is the video data whose integrity has been checked. Once the integrity is confirmed, the video is saved in storage.

[0946] Step 4:

[0947] The server converts the video into a format for analysis. The input is the video data that has been validated for integrity, and the output is the converted video data in the format for analysis. This process splits the video into frames, and each frame is ready to be sent to the generative AI model.

[0948] Step 5:

[0949] The server sends each frame to the generative AI model, which detects the position of the robot's joints and body parts for each frame. The input is video data (each frame) in an analysis format, and the output is position data of the joints and body parts in each frame. The generative AI model uses a machine learning algorithm to identify the position of the joints and body parts in each frame.

[0950] Step 6:

[0951] The server analyzes the robot's movements based on joint and body part position data. The input is joint and body part position data, and the output is the analysis results of the movements. During this analysis process, it compares the data with an existing technology database and matches it with efficient movement patterns and technologies.

[0952] Step 7:

[0953] The generative AI model generates movement improvement advice based on the analysis results. The input is the analysis results of the exercise movement, and the output is movement improvement advice. During this process, visual support images are generated along with specific advice.

[0954] Step 8:

[0955] The server sends the generated advice and auxiliary image to the user's device. The input is the behavior improvement advice and auxiliary image, and the output is the advice and auxiliary image displayed on the user's device. The user can check this advice through the application.

[0956] Step 9:

[0957] The user adjusts the robot's behavior based on the behavior improvement advice. The input is the displayed advice, and the output is the adjusted robot behavior. The user adjusts the robot's settings based on specific improvements to improve operational efficiency.

[0958] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0959] The system according to the present invention not only analyzes videos and provides exercise improvement advice, but also incorporates an emotion engine that recognizes the user's emotions and adjusts the content and presentation method of the advice accordingly. A specific embodiment of the system will be described below.

[0960] First, the user uses their device to record their child's exercise. For example, they can use a smartphone to record their child practicing dribbling a basketball. The user then uploads the video to the server via a dedicated application. This application is easy to use; the user simply selects a video and presses the upload button, and the video is sent to the server.

[0961] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis. Next, it splits the video into frames and prepares each frame to be sent to the generation AI.

[0962] The generative AI detects the positions of the child's joints and body parts in each frame. For example, it identifies the positions of major joints such as elbows, wrists, and torso, and collects their positions as time-series data. It then analyzes the child's movements based on this position data and evaluates their movement patterns. During this evaluation process, it compares them with an existing technical database to identify any discrepancies from the correct form and technique.

[0963] Based on the analysis results, the AI ​​generates specific advice for improving your movements. For example, it might say, "Your wrist movements are stiff when dribbling, so try to relax them." It can also create visual aids, which can help users understand the advice more intuitively.

[0964] A distinctive feature of the present invention is the incorporation of an emotion engine. This emotion engine can recognize the user's facial expressions and voice through the device's camera and microphone and evaluate the user's emotional state. The emotion data acquired by the emotion engine is reflected in the content and presentation method of advice. For example, if the user is feeling irritated or anxious, the tone of the advice can be changed to a gentler one, encouraging the user to take a more positive approach. Also, if the user is enjoying themselves, encouraging words can be added to maintain that state.

[0965] Finally, the generated advice and auxiliary images are sent from the server to the user's device. The user can then review the advice and provide specific guidance to their child through the application. In this way, the system provides specific and effective support for improving children's motor skills, while also being able to flexibly respond to the user's emotional state.

[0966] Example: Analysis of basketball dribbling practice

[0967] Consider the case where a user films their child practicing basketball dribbling and uploads it using a dedicated application. The server receives the video and analyzes it frame by frame. The generation AI analyzes the movement of each joint in the dribbling motion and identifies stiff wrist movement. Based on the analysis results, it generates specific advice such as "Relax your wrist."

[0968] Meanwhile, the emotion engine recognizes when the user is frustrated and adjusts the tone of the advice to be gentler, for example, by saying, "It may be difficult at first, but you can improve little by little." This allows the user to provide guidance to their child in a positive manner.

[0969] As described above, this system not only helps children improve their motor skills, but also provides a flexible approach that takes into consideration the user's emotions.

[0970] The processing flow will be explained below.

[0971] Step 1:

[0972] A user uses their own device to record a video of their child exercising. For example, a user uses a smartphone to record a video of their child practicing basketball dribbling.

[0973] Step 2:

[0974] The user launches the dedicated application, selects the video they have taken, and taps the "Upload" button.

[0975] Step 3:

[0976] The device sends the selected video to the server. When the video is sent, a checksum and other verifications are performed to ensure the integrity of the data.

[0977] Step 4:

[0978] The server receives the video data sent from the device, reconfirms the integrity of the video file, and stores it in temporary storage.

[0979] Step 5:

[0980] The server converts the received video into a standard format for analysis, converting the codec and adjusting the resolution as necessary.

[0981] Step 6:

[0982] The server splits the video into frames, extracts each frame as an individual image file, and stores the timestamp information for each frame.

[0983] Step 7:

[0984] The server sends the divided frame data to the generation AI, which receives it and detects the positions of joints and body parts in each frame.

[0985] Step 8:

[0986] Based on the detected joint position data, the generative AI analyzes the movement over time, tracking the movement patterns and positions of each joint and extracting the characteristic points of the movement.

[0987] Step 9:

[0988] The generative AI compares the extracted feature points with an existing technical database, and based on the comparison results, identifies any differences between the target athletic movement and correct form and technique.

[0989] Step 10:

[0990] Based on the analysis results, the generative AI generates specific advice for improving exercise, such as "relax your wrist and move more smoothly."

[0991] Step 11:

[0992] The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotional state, for example, detecting when the user is feeling irritated.

[0993] Step 12:

[0994] Based on the emotional data recognized by the emotion engine, the generative AI adjusts the way it presents advice. For example, if a user feels frustrated, it will soften the tone of the advice and add encouraging words such as "Take your time and practice a little at a time."

[0995] Step 13:

[0996] The server transmits the generated advice and the adjusted advice data to the user's terminal.

[0997] Step 14:

[0998] The user checks the advice data sent from the server through an application on their device. The application displays the advice and provides visual support images to help the user understand the specific improvements required.

[0999] Step 15:

[1000] The user provides specific advice for improvement based on the application, and by filming the exercise again and repeating these steps, the user can continuously identify areas for improvement and improve athletic ability.

[1001] Example 2

[1002] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1003] Conventional video analysis systems can analyze exercise videos taken by users and provide exercise improvement advice, but they provide uniform advice without considering the user's emotions, making it difficult to provide appropriate guidance tailored to the user's situation.In addition, there is a lack of means to compare with technical databases to improve the accuracy of motion analysis, or to generate visual support images, making it difficult to deepen understanding of the analysis results.

[1004] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1005] In this invention, the server includes means for receiving a video, means for dividing the video into a plurality of frames, means for detecting the positions of joints and body parts of an object in each frame, means for evaluating the analyzed exercise movements by comparing with a specific database, means for generating exercise improvement advice based on the analysis results, means for recognizing the user's emotions, means for adjusting the generated advice based on the recognized emotions, and means for outputting the generated advice, thereby making it possible to provide flexible exercise improvement advice that takes the user's emotions into consideration.

[1006] The "means for receiving video" is a function for transmitting video data taken by a user to a server via a network and for the server to receive the video data.

[1007] "Means for dividing a video into multiple frames" is a function for dividing a received video into small still images (frames) along the time axis and generating data for each frame.

[1008] "Means for detecting the positions of joints and body parts of an object in each frame" refers to a technique for identifying the positions of joints and body parts relative to an object in each frame image and extracting their coordinate data.

[1009] The "means for evaluating the analyzed exercise movements by comparing with a specific database" is a function for comparing the analyzed exercise data with an existing reference database and evaluating the accuracy of the exercise and areas for improvement.

[1010] The "means for generating exercise improvement advice based on the analysis results" is a function for automatically generating specific improvement methods and instruction contents based on the evaluation results of the exercise analysis.

[1011] "Means for recognizing user emotions" refers to technology that analyzes information such as the user's facial expressions and voice to identify the user's emotional state.

[1012] The "means for adjusting the advice generated based on the recognized emotion" is a function for flexibly changing the content and tone of the exercise improvement advice generated according to the user's emotional state.

[1013] The "means for outputting generated advice" is a function for displaying or providing the generated exercise improvement advice and visual support images in an easily understandable form to the user.

[1014] MODE FOR CARRYING OUT THE INVENTION

[1015] The system according to the present invention not only analyzes videos and provides exercise improvement advice, but also incorporates an emotion recognition engine that recognizes the user's emotions and adjusts the content and presentation method of the advice accordingly. A specific embodiment of the system will be described below.

[1016] First, the user uses their device to record their child's exercise. For example, they can use a smartphone to record their child practicing dribbling a basketball. The user then uploads the video to the server via a dedicated application. This application is easy to use; the user simply selects a video and presses the upload button, and the video is sent to the server.

[1017] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis using software such as FFmpeg. Next, it splits the video into frames and prepares each frame to be sent to a generative AI (for example, using OpenPose or TensorFlow).

[1018] The generative AI detects the positions of the child's joints and body parts in each frame. For example, it identifies the positions of major joints such as elbows, wrists, and torso, and collects their positions as time-series data. It then analyzes the child's movements based on this position data and evaluates their movement patterns. During this evaluation process, it compares the child's movements with existing technical databases (e.g., Sports Motion Database) to identify any discrepancies with the correct form and technique.

[1019] Based on the analysis results, the AI ​​generates specific advice for improving your movements. For example, it might say, "Your wrist movements are stiff when dribbling, so try to relax them." It can also create visual aids, which can help users understand the advice more intuitively.

[1020] A distinctive feature of the present invention is the incorporation of an emotion engine. This emotion engine can recognize the user's facial expressions and voice through the device's camera and microphone and evaluate the user's emotional state. The emotion data acquired by the emotion engine is reflected in the content and presentation method of advice. For example, if the user is feeling irritated or anxious, the tone of the advice can be changed to a gentler one, encouraging the user to take a more positive approach. Also, if the user is enjoying themselves, encouraging words can be added to maintain that state.

[1021] Finally, the generated advice and auxiliary images are sent from the server to the user's device. The user can then review the advice and provide specific guidance to their child through the application. In this way, the system provides specific and effective support for improving children's motor skills, while also being able to flexibly respond to the user's emotional state.

[1022] Example: "Analysis of basketball dribbling practice"

[1023] Consider the case where a user films their child practicing basketball dribbling and uploads it using a dedicated application. The server receives the video and analyzes it frame by frame. The generation AI analyzes the movement of each joint in the dribbling motion and identifies stiff wrist movement. Based on the analysis results, it generates specific advice such as "Relax your wrist."

[1024] Meanwhile, the emotion engine recognizes when the user is frustrated and adjusts the tone of the advice to be gentler, for example, by saying, "It may be difficult at first, but you can improve little by little." This allows the user to provide guidance to their child in a positive manner.

[1025] Examples of prompt statements

[1026] Prompt statement:

[1027] I filmed a kid practicing dribbling a basketball. Analyze the uploaded video and identify stiff wrist movements. Then provide specific advice to relax the wrist. Also, adjust the tone of the advice to be gentler if the user is frustrated.

[1028] As described above, this system not only helps children improve their motor skills, but also provides a flexible approach that takes into consideration the user's emotions.

[1029] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1030] Step 1:

[1031] The user uses a device such as a smartphone to record their child's exercise. The user then uploads the video to a server using a dedicated application. The input is a video file, and the output is a video file stored on the server.

[1032] Step 2:

[1033] The server checks the received video and saves it in storage. At this time, it checks the integrity of the video file and checks whether it contains any invalid data. The input is the uploaded video file, and the output is the video file saved in storage whose integrity has been checked.

[1034] Step 3:

[1035] The server uses video processing software such as FFmpeg to convert the stored video into a format for analysis. The input in this step is a video file whose integrity has been confirmed, and the output is a video file converted into the format for analysis.

[1036] Step 4:

[1037] The server divides the video file for analysis into frames and generates image data for each frame. Specifically, it divides one second of video into 30 frames. The input for this step is the video file for analysis, and the output is still images for each frame (e.g., "frame0001.jpg", "frame0002.jpg").

[1038] Step 5:

[1039] The server sends each frame image to a generative AI (e.g., OpenPose or TensorFlow) to detect the positions of the child's joints and body parts in each frame. The input is the frame image data, and the output is joint position data (e.g., coordinate data for the left elbow, right wrist, and trunk).

[1040] Step 6:

[1041] The server analyzes the motion based on the detected joint position data. The analysis results are compared with a specific technique database (e.g., Sports Motion Database) to identify deviations from the correct form and technique. The input for this step is the joint position data, and the output is the evaluation result of the motion analysis (e.g., "The wrist movement is stiff").

[1042] Step 7:

[1043] The server generates exercise improvement advice based on the analysis results. The generation AI creates specific advice text (e.g., "Try to relax your wrist") and visual support images (e.g., illustrations of the movement to relax your wrist). The input is the evaluation results of the movement analysis, and the output is exercise improvement advice and support images.

[1044] Step 8:

[1045] The device uses a camera and microphone to collect the user's facial expressions and voice. This data is sent to the emotion engine to evaluate the user's emotional state. The input is the user's facial and voice data, and the output is the emotion recognition result (e.g., "irritated").

[1046] Step 9:

[1047] The server adjusts the tone and content of exercise improvement advice based on the emotion recognition results obtained from the emotion engine. For example, if a user feels frustrated, the tone may be changed to a gentler one, such as "It may be difficult at first, but you can improve little by little." The input is the emotion recognition result, and the output is the adjusted exercise improvement advice.

[1048] Step 10:

[1049] The server sends the final exercise improvement advice and supplementary images to the user's device. The user can then review the advice and provide specific guidance to their child through the application. The input is the adjusted exercise improvement advice and supplementary images, and the output is the final advice displayed on the user's device.

[1050] keyword

[1051] Generative AI model, prompt sentence

[1052] (Application example 2)

[1053] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1054] Conventional exercise improvement systems can analyze a user's exercise data and provide specific advice, but they are unable to adjust the content of the advice taking the user's emotions into account. As a result, the advice is often ineffective when the user is feeling frustrated or anxious. Furthermore, while it is important for exercise improvement advice to be presented in a visually easy-to-understand format, there has been a lack of effective ways to achieve this. This has led to issues such as insufficient user satisfaction and effective exercise improvement.

[1055] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1056] In this invention, the server includes means for receiving a video, means for dividing the video into a plurality of frames, means for detecting the positions of joints and body parts of an object in each frame, means for analyzing exercise movements based on the detected positions, means for generating exercise improvement advice based on the analysis results, means for outputting the generated advice, means for recognizing a user's emotion, and means for adjusting the content and presentation method of the advice according to the recognized emotion. This makes it possible to provide effective and flexible exercise improvement advice to the user, and to improve the acceptability and satisfaction of the advice by taking the user's emotion into consideration.

[1057] The "means for receiving video" is a mechanism by which a user sends video taken by a terminal to a server, and the server receives the video.

[1058] The "means for dividing a video into a plurality of frames" is a mechanism for dividing a received video into frames along the time axis and making each frame individually analyzable.

[1059] The "means for detecting the positions of the joints and body parts of the object in each frame" is a mechanism for identifying the positions of the joints and body parts of a person in the video frames to be analyzed and acquiring that data.

[1060] The "means for analyzing movement based on detected positions" is a mechanism for evaluating and analyzing movement based on acquired joint and body position data.

[1061] The "means for generating exercise improvement advice based on the analysis results" is a mechanism for generating specific advice for improving exercise based on the analysis results of exercise movements.

[1062] The "means for outputting generated advice" is a mechanism for presenting the generated exercise improvement advice to the user and displaying it in an easy-to-understand manner.

[1063] The "means for recognizing the user's emotions" is a mechanism for analyzing the user's facial expressions and voice through a camera or microphone and recognizing the user's emotional state.

[1064] The "means for adjusting the content and presentation method of advice according to the recognized emotion" is a mechanism for appropriately changing the content and presentation method of advice based on the user's emotional state.

[1065] The system according to the present invention provides exercise improvement support by combining video analysis and emotion recognition. This system not only analyzes videos to provide exercise improvement advice, but also incorporates an emotion engine that recognizes the user's emotions and adjusts the content and presentation method of the advice accordingly. Specific aspects of this system are described below.

[1066] First, the user films the exercise using their own device. An example of this is a child using a smartphone to film their basketball practice. The user then uploads the video to the server via a dedicated application. The application is designed to be user-friendly; simply by selecting a video and pressing the upload button, the video is automatically sent to the server.

[1067] The server stores the received video in storage, verifies the integrity of the video file, and then converts the video into a format for analysis. The video is then split into frames, and each frame is prepared to be sent to a generation AI for analysis. This generation AI uses a machine learning library such as TensorFlow.

[1068] The generative AI detects the positions of the joints and body parts of the subject (in this example, a child) in each frame. For example, it identifies the positions of major joints such as the elbows, wrists, and torso, and collects their positions as time-series data. It then analyzes the movement based on this position data and evaluates the movement patterns. During this evaluation process, it compares the results with an existing technical database to identify any discrepancies from the correct form and technique.

[1069] Based on the analysis results, the AI ​​generates specific advice for improving the movement. For example, it might say, "When shooting, your elbow position is low. Try to keep it a little higher." In addition, it creates auxiliary images that are easy to understand visually.

[1070] A distinctive feature of the present invention is the incorporation of an emotion engine. This emotion engine recognizes the user's facial expressions and voice through the device's camera and microphone and evaluates the user's emotional state. This evaluation utilizes emotion analysis APIs such as Microsoft Azure Cognitive Services. The emotion data acquired by the emotion engine is reflected in the content and presentation of the generated advice. For example, if the user is feeling frustrated or anxious, the tone of the advice can be made gentler, and soft words such as "Trial and error lead to growth. Take your time and practice slowly." If the user is enjoying themselves, encouraging words such as "That's great! Keep it up!" can be added to maintain that state of mind.

[1071] Finally, the generated advice and auxiliary images are sent from the server to the user's device, where the user can check the advice and receive specific guidance through the application.

[1072] For example, consider the following prompt:

[1073] python

[1074] import requests

[1075] import cv2

[1076] def upload_video(video_path):

[1077] url = 'http: / / yourserver.com / upload'

[1078] files = {'file': open(video_path, 'rb')}

[1079] response = requests.post(url, files=files)

[1080] if response.status_code == 200:

[1081] print("Video uploaded successfully!")

[1082] else:

[1083] print("Error in video upload")

[1084] def analyze_video_frames(frames, model):

[1085] results = []

[1086] for frame in frames:

[1087] processed_frame = preprocess_frame(frame)

[1088] result = model.predict(processed_frame)

[1089] results.append(result)

[1090] return results

[1091] def generate_advice(movement_data):

[1092] Analyze movement_data and generate advice

[1093] advice = "Your elbow position is incorrect during the shot. Try to keep it higher."

[1094] return advice

[1095] def emotion_adjust_advice(user_emotion, advice):

[1096] if user_emotion == 'frustrated':

[1097] adjusted_advice = "Don't worry, improvement takes time. " + advice

[1098] elif user_emotion == 'happy':

[1099] adjusted_advice = "Great job! Keep it up. " + advice

[1100] else:

[1101] adjusted_advice = advice

[1102] return adjusted_advice

[1103] Example usage

[1104] video_path = 'child_shooting_basketball.mp4'

[1105] upload_video(video_path)

[1106] This program aims to improve athletic ability by uploading videos taken by users to a server, analyzing the videos and recognizing emotions, and generating and presenting appropriate advice.

[1107] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1108] Step 1:

[1109] Recording and uploading videos

[1110] Action: A user uses a smartphone to record themselves exercising.

[1111] Input: The captured video file.

[1112] Output: The video file imported into the application.

[1113] Specific operation: The user presses the record button to record a video of their exercise. After recording is complete, the user presses the application's upload button to send the video to the server.

[1114] Step 2:

[1115] Receiving and storing videos

[1116] How it works: The server receives and stores videos uploaded by users.

[1117] Input: The uploaded video file.

[1118] Output: Video files stored in the server's storage.

[1119] What it does: The server receives an HTTP request, saves the file to disk, checks the integrity of the saved file, and prepares it for analysis.

[1120] Step 3:

[1121] Video frame division

[1122] How it works: The server splits the stored video into frames.

[1123] Input: A saved video file.

[1124] Output: Image data per frame.

[1125] Specific operation: Using the OpenCV library, a video file is read and split into frame-by-frame still images. Each frame is then converted into the format required for subsequent analysis.

[1126] Step 4:

[1127] Position detection of joints and body parts

[1128] How it works: The server detects the positions of joints and body parts for each frame.

[1129] Input: Frame-by-frame image data.

[1130] Output: Position data of joints and body parts (time series data).

[1131] Specific operation: Using a generative AI model such as TensorFlow, joints and body parts (e.g., elbows, wrists, torso, etc.) are analyzed for each frame to obtain positional data.

[1132] Step 5:

[1133] Analysis of athletic movements

[1134] Movement: The server analyzes the movement based on the detected location data.

[1135] Input: Position data of joints and body parts.

[1136] Output: Motor behavior evaluation data.

[1137] Specific behavior: The acquired position data is analyzed as time series data, and behavior patterns are evaluated by comparing it with a technical database. Inappropriate behavior is identified and the analysis results are output.

[1138] Step 6:

[1139] Generating exercise improvement advice

[1140] Action: The server generates exercise improvement advice based on the results of the exercise motion analysis.

[1141] Input: Exercise behavior assessment data.

[1142] Output: Text and supporting images of exercise improvement advice.

[1143] Specific actions: Based on the analysis results, specific advice is generated based on the situation, such as, "When shooting, your elbow is positioned too low, so try to keep it a little higher." Supporting images are also created to make the game easier to understand visually.

[1144] Step 7:

[1145] User Emotion Recognition

[1146] How it works: The server recognizes the user's emotions.

[1147] Input: User's facial and voice data.

[1148] Output: User's emotional state data.

[1149] Specific operation: Using emotion analysis APIs such as Microsoft Azure Cognitive Services, the system analyzes facial expression data and voice data acquired through the camera to evaluate the user's emotional state (e.g., irritation, joy, calmness, etc.).

[1150] Step 8:

[1151] Advice adjustment

[1152] How it works: The server adjusts the content and presentation of advice based on the results of emotion recognition.

[1153] Input: Exercise improvement advice, user emotional state data.

[1154] Output: Tailored exercise improvement advice.

[1155] Specific actions: Change the tone of advice based on emotion recognition data, e.g., "Trial and error leads to growth. Take your time and practice slowly." Add words of encouragement according to the emotion.

[1156] Step 9:

[1157] Providing advice

[1158] How it works: The server sends the generated and tailored advice to the user's device and presents it through the application.

[1159] Enter: tailored exercise improvement advice.

[1160] Output: Advice and auxiliary images displayed on the user's terminal.

[1161] Specific operation: The generated advice and auxiliary images are sent to the user's device, and displayed through the application so that the user can check and execute them.

[1162] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1163] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1164] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1165] [Fourth embodiment]

[1166] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1167] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1168] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1169] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1170] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1171] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1172] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1173] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1174] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1175] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1176] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1177] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1178] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1179] The system of the present invention analyzes video and automatically generates specific advice to improve a child's athletic performance. An embodiment of the system will be described below.

[1180] First, the user uses their device to record their child's exercise. For example, they can use their smartphone to record their child shooting a soccer ball. The user then uploads the video to the server via a dedicated application. The application is easy to use; the user simply selects a video and taps the upload button, and the video is sent to the cloud server.

[1181] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis. Next, it splits the video into frames and prepares each frame to be sent to the generation AI.

[1182] The generative AI detects the position of the child's joints and body parts in each frame. For example, it identifies the locations of major joints such as the knees, ankles, and shoulders, and collects these positions as time-series data. It then analyzes the child's movements based on this position data and evaluates their movement patterns. During this evaluation process, it compares them with an existing technique database to match them with the correct form and technique.

[1183] Based on the analysis results, the AI ​​generates specific exercise improvement advice, such as "Your foot usage is unstable, so try to improve your balance" or "Your knee angle is shallow, so you should bend it a bit more." It can also create visual support images, which allow users to understand the advice more intuitively.

[1184] Finally, the generated advice and auxiliary images are sent from the server to the user's device. The user can then review the advice through the application and provide specific guidance to their child. In this way, the system provides specific and effective support for improving children's motor skills.

[1185] Example: Analysis of soccer shooting movements

[1186] Consider the case where a user takes a video of their child shooting a soccer ball and uploads it using a dedicated application. The server receives the video and analyzes each frame. The generating AI analyzes the foot movement, knee angle, body balance, etc. during the shot, and determines that the knees are not bent enough, resulting in a lack of shooting power. Based on the analysis results, it generates and provides specific advice such as "Bend your knees a bit more," along with a supplementary image of the correct form. The user who receives this information can then tell their child specific areas for improvement and encourage effective practice.

[1187] As described above, this system can provide specific advice to effectively improve children's motor skills, even without specialized knowledge.

[1188] The processing flow will be explained below.

[1189] Step 1:

[1190] Users record videos of their children's exercise on their own devices, then launch the dedicated application, select the video, and tap the "Upload" button.

[1191] Step 2:

[1192] The device sends the selected video to the server. When sending the video, it also performs checksum verification and other checks to ensure the integrity of the data.

[1193] Step 3:

[1194] The server receives the video data sent from the device, reconfirms the integrity of the video file, and stores it in temporary storage.

[1195] Step 4:

[1196] The server converts the received video into a standard format for analysis, converting the codec and adjusting the resolution if necessary.

[1197] Step 5:

[1198] The server splits the video into frames, extracts each frame as an individual image file, and stores the timestamp information for each frame.

[1199] Step 6:

[1200] The server sends the divided frame data to the generation AI, which receives it and detects the positions of joints and body parts in each frame.

[1201] Step 7:

[1202] Based on the detected joint position data, the generative AI analyzes the movement over time. Specifically, it tracks the movement patterns and positions of each joint and extracts the characteristic points of the movement.

[1203] Step 8:

[1204] The generative AI compares the extracted feature points with an existing technical database, and based on the comparison results, identifies any differences between the target athletic movement and correct form and technique.

[1205] Step 9:

[1206] Based on the analysis results, the generative AI generates specific advice for improving exercise, which is output in text format and as visual support images.

[1207] Step 10:

[1208] The server receives the generated advice data, associates it with the user profile, and transmits the advice data to the terminal.

[1209] Step 11:

[1210] The user checks the advice data sent from the server through the application on the device. The application displays the advice content and also provides visual support images to help the user understand it.

[1211] Step 12:

[1212] The user provides instruction to their child based on specific improvement advice provided through the application. By filming their exercise and conducting repeated analysis, the system continuously identifies areas for improvement and aims to improve athletic ability.

[1213] Example 1

[1214] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1215] With conventional exercise improvement systems, it was difficult for ordinary users without specialized knowledge to obtain specific advice to effectively improve children's exercise performance. Furthermore, using a video analysis system required advanced technology and specialized equipment, making it difficult for many users to use. This limited the opportunities for many children to learn correct form and techniques.

[1216] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1217] In this invention, the server includes means for receiving video, means for dividing the video into frames, means for detecting the positions of joints and body parts of an object in each frame, means for analyzing exercise movements based on the detected positions, means for generating exercise improvement advice based on the analysis results, means for outputting the generated exercise improvement advice and visual auxiliary images, and means for transmitting the advice and auxiliary images from the server to a user's terminal. This enables even general users without specialized knowledge to easily obtain specific advice for effectively improving their children's exercise performance.

[1218] The "means for receiving video" refers to a device or software for transmitting video data taken by a user to a server and receiving the video data.

[1219] The "means for dividing a moving image into a plurality of frames" refers to a device or software that divides received moving image data into a plurality of still images (frames) in time.

[1220] The "means for detecting the positions of joints and body parts of an object in each frame" refers to a device or software for identifying the positions of an object, specifically joints and body parts, from each divided frame.

[1221] The "means for analyzing movement based on detected positions" refers to a device or software for analyzing the position data of detected joints or body parts and evaluating the movement of an object.

[1222] The "means for generating exercise improvement advice based on the analysis results" refers to a device or software that generates specific advice for improving the athletic ability of a subject based on the analysis results of exercise movements.

[1223] The "means for outputting the generated exercise improvement advice and the visual support image" refers to a device or software for presenting the generated advice and the accompanying visual support image to the user.

[1224] The "means for transmitting advice and auxiliary images from the server to the user's terminal" refers to a device or software that transmits the generated advice and auxiliary images to the terminal used by the user via a network.

[1225] MODE FOR CARRYING OUT THE INVENTION

[1226] The system of the present invention analyzes video and automatically generates specific advice to improve children's athletic performance. An embodiment of the system will be described below.

[1227] First, a user uses their smartphone to record their child's exercise. For example, they can record a video of their child shooting a soccer ball. The user saves the video on their smartphone and then uploads it to a server via a dedicated application.

[1228] The dedicated application is commonly available and easy to use: users simply launch the application, select the video they have taken, and tap the upload button to send the video to a cloud server.

[1229] The server receives the uploaded video and saves it. After saving, the server checks the integrity of the video file by checking the file match using file hash values, etc. Once the integrity of the video is confirmed, it is converted into a format for analysis. For example, if the video is in MP4 format, it is converted into still images of each frame. The frame images saved in a temporary folder are used as input for the generative AI model.

[1230] The server splits the video into a fixed frame rate and saves each frame as an image file. These image files are listed as inputs to the generative AI model and prepared as a JSON-formatted data packet. The server then sends this data to the generative AI model along with a prompt.

[1231] The generative AI model analyzes each frame image to identify the location of the child's major joints (knees, ankles, shoulders, etc.). The detected data is collected in chronological order and used to evaluate the child's athletic performance. The evaluation results are compared with existing technical databases to generate appropriate advice.

[1232] The generated advice is specific, such as "Your foot usage is unstable, so try to find your balance" or "Your knee angle is shallow, so you should bend it a bit more." Visual support images are also created, allowing users to understand the advice more intuitively.

[1233] Finally, the generated advice and supporting images are sent from the server to the user's device. The user can check that new advice is available through a notification in the dedicated application. The user can then review the advice through the application and provide specific guidance to their child.

[1234] Specific examples

[1235] Analysis of soccer shooting movements

[1236] Consider the case where a user records a video of their child shooting a soccer ball and uploads it using a dedicated application. The server receives the video and prepares it for frame-by-frame analysis. The generative AI model analyzes the foot movement, knee angle, body balance, etc. during the shot, and identifies that the knees are not bent enough, resulting in a lack of shooting power. Based on the analysis results, it generates and provides a supporting image of the correct form along with specific advice such as "Bend your knees a bit more." The user can receive this information and tell their child specific areas for improvement, encouraging effective practice.

[1237] Prompt Sentence Examples

[1238] "Analyze a video of a child shooting a soccer ball and generate advice on foot movement and knee angle."

[1239] In this way, the system provides specific advice to effectively improve children's motor skills without requiring specialized knowledge.

[1240] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1241] Step 1:

[1242] Video recording and uploading

[1243] The user uses their smartphone to record their child's athletic activity (e.g., a soccer shot). Once the recording is complete, they launch the dedicated application and select the video file. The user then taps the "Upload" button, and the video file is sent to the server.

[1244] Input: Video file taken by the user

[1245] Output: Video file sent to the server

[1246] Step 2:

[1247] Video reception and integrity check

[1248] The server receives the video file sent by the user. Once received, it verifies the hash value and file size to confirm the integrity of the file. Once the integrity is confirmed, the video is stored in storage.

[1249] Input: Video file submitted by the user

[1250] Output: Video file with verified integrity

[1251] Step 3:

[1252] Video format conversion for analysis

[1253] The server reads the video files stored in the storage and converts them into a format that can be analyzed (e.g., a series of still images). Specifically, it converts MP4 format videos into JPEG images for each frame.

[1254] Input: Video file with verified integrity

[1255] Output: Separate image files for each frame

[1256] Step 4:

[1257] Frame Split

[1258] The server splits the converted video into a fixed frame rate (e.g. 30fps) and saves each frame as an image file. The saved frame images are placed in a temporary folder.

[1259] Input: Video file converted to analysis format

[1260] Output: Image files for each frame

[1261] Step 5:

[1262] Sending data to the generation AI

[1263] The server prepares the image files for each frame to be sent to the generative AI model. To do this, it lists the paths of the image files and generates a JSON-formatted data packet. It then sends the generated JSON data along with the prompt to the generative AI model.

[1264] Input: Image file for each frame, prompt text

[1265] Output: JSON data packet sent to the generative AI model

[1266] Step 6:

[1267] Joint position detection and motion analysis

[1268] The generative AI model analyzes each transmitted frame and identifies the location of key joints (knees, ankles, shoulders, etc.). This position data is collected as time series data and used to evaluate the movement. During the evaluation process, it compares the data with a database to identify the optimal movement pattern.

[1269] Input: Image files for each frame, JSON data

[1270] Output: Joint position data and motion analysis results

[1271] Step 7:

[1272] Generating exercise improvement advice

[1273] The server generates exercise improvement advice based on the analysis results received from the generative AI model. Specific advice includes "bend your knees more" and "maintain your body balance." Visual support images are also created to help users intuitively understand the advice.

[1274] Input: Motion analysis results

[1275] Output: Specific exercise improvement advice, visual aid images

[1276] Step 8:

[1277] Sending advice and auxiliary images

[1278] The server sends the generated advice and auxiliary images to the user's device, and the notification function of the dedicated application is used to inform the user that new advice is available.

[1279] Input: Exercise improvement advice, visual aid images

[1280] Output: Sends advice and auxiliary images to the user's terminal

[1281] Step 9:

[1282] Confirmation of advice and guidance

[1283] The user checks the advice and supplementary images received on the device and provides specific instructions to the child on how to improve their exercise. For example, the user might be told to "bend your knees a little more," and the supplementary images would be used to demonstrate the correct form.

[1284] Input: Advice and auxiliary images sent to the user's device

[1285] Output: Specific exercise improvement guidance for children

[1286] Through the above steps, the system can provide the user with specific advice on how to improve their exercise, and support their child in improving their athletic ability.

[1287] (Application example 1)

[1288] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1289] Conventional factory robot motion analysis requires specialized knowledge, making it difficult to make efficient improvements. Furthermore, when humans observe motion and suggest areas for improvement, they may overlook subtle motion characteristics. Therefore, there is a need for a method to provide specific and accurate advice to maximize robot efficiency.

[1290] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1291] In this invention, the server

[1292] means for receiving video;

[1293] A means for dividing a video into frames;

[1294] means for detecting the position of joints and body parts of the object in each frame;

[1295] means for analyzing the movement based on the detected position;

[1296] A means for generating behavior improvement advice based on the analysis results;

[1297] means for outputting the generated advice;

[1298] A means to film and upload the robot's movements;

[1299] a means for evaluating the efficiency of the operation using the generative AI model;

[1300] A means for identifying a behavior pattern based on the evaluation results;

[1301] This makes it possible to provide specific and accurate advice to improve the efficiency of robot operations, even without specialized knowledge.

[1302] "Means for receiving video" refers to a device or function for transferring and importing video shot by a user from a device to a server.

[1303] "Means for dividing a video into multiple frames" refers to a technique or method for dividing a received video into frames and analyzing each frame individually.

[1304] "Means for detecting the positions of joints and body parts of an object in each frame" refers to a technology that identifies the positions of joints and body parts of the robot or person being analyzed for each frame.

[1305] The "means for analyzing movement based on detected positions" refers to a device or method for analyzing the movement pattern of an object using position data of detected joints and parts.

[1306] The "means for generating advice to improve movement based on the analysis results" is a technology for generating specific advice to improve movement based on the analyzed movement pattern.

[1307] The "means for outputting the generated advice" refers to a method or device for providing the generated behavior improvement advice to the user.

[1308] "Means for filming and uploading robot operations" refers to a method or technology for filming the operations of a factory robot and uploading the video data to a server.

[1309] "Means for evaluating the efficiency of movements using a generative AI model" refers to a technology that analyzes data obtained from video captured using a generative AI model and evaluates the efficiency of movements.

[1310] "Means for identifying movement patterns based on evaluation results" refers to a technology that identifies the movement patterns of a robot based on the evaluated data and extracts specific key points for improving efficiency.

[1311] The system of the present invention provides specific advice for improving operational efficiency through the analysis of the operation of a factory robot. An embodiment of the system will be described below.

[1312] First, the user records the factory robot's operations using a device such as a smartphone or head-mounted display (HMD). The user then uploads the recorded video to a cloud server via a dedicated analysis application. The application is easy to use; the user simply selects a video and taps the upload button to send the video to the cloud server.

[1313] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis. Next, it splits the video into frames and prepares each frame to be sent to the generative AI model.

[1314] The generative AI model detects the positions of the robot's joints and parts in each frame. For example, it identifies the positions of key parts such as arms, joints, and grippers, and collects these positions as time-series data. Based on the collected data, it analyzes the robot's motion and evaluates its movement patterns. During this evaluation process, it compares them with a database of existing technologies to find efficient movement patterns and technologies.

[1315] Based on the analysis results, the generative AI model generates specific advice for improving the movement. For example, "The arm's movement is slow, so you need to increase the speed" or "The joint angle is inappropriate, so you need to adjust the angle." It is also possible to create visual support images, which allows the user to understand the advice more intuitively.

[1316] Finally, the generated advice and auxiliary images are sent from the server to the user's device, where the user can review the advice and adjust the robot's operation through the application. In this way, the system can provide specific and effective support for improving the robot's operation efficiency, even without specialized knowledge.

[1317] Example: Analysis of welding motion of a robot arm

[1318] Consider a case where a user records a video of a robot arm's welding operations and uploads it using a dedicated analysis application. The server receives the video and analyzes it frame by frame. The generative AI model analyzes the arm's movements during welding, the joint angles, and areas of slow movement, and generates specific advice such as, "The welding quality is declining because the arm is moving too slowly." Based on this result, the model generates and provides specific advice such as, "Make the movement a little faster," along with a supplementary image of the correct operation. The user who receives this information can adjust the robot's behavior to improve welding quality.

[1319] Example prompt for a generative AI model:

[1320] Detect joint positions from frame images of a robot arm and generate advice to improve the efficiency of the movement. Joint position data is provided for each frame of the video below. Please provide specific advice on how to improve the movement based on this data.

[1321] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1322] Step 1:

[1323] The user records the robot's movements using a smartphone or head-mounted display. The input is the captured video. This video data is stored on the user's device.

[1324] Step 2:

[1325] Users upload the videos they have taken to the cloud server via a dedicated analysis application. The input is the taken video file, and the output is the video data saved on the cloud server. The user selects a video and taps the upload button to send the video to the cloud server.

[1326] Step 3:

[1327] The cloud server receives the uploaded video and checks the integrity of the video file. The input is the video data stored on the cloud server, and the output is the video data whose integrity has been checked. Once the integrity is confirmed, the video is saved in storage.

[1328] Step 4:

[1329] The server converts the video into a format for analysis. The input is the video data that has been validated for integrity, and the output is the converted video data in the format for analysis. This process splits the video into frames, and each frame is ready to be sent to the generative AI model.

[1330] Step 5:

[1331] The server sends each frame to the generative AI model, which detects the position of the robot's joints and body parts for each frame. The input is video data (each frame) in an analysis format, and the output is position data of the joints and body parts in each frame. The generative AI model uses a machine learning algorithm to identify the position of the joints and body parts in each frame.

[1332] Step 6:

[1333] The server analyzes the robot's movements based on joint and body part position data. The input is joint and body part position data, and the output is the analysis results of the movements. During this analysis process, it compares the data with an existing technology database and matches it with efficient movement patterns and technologies.

[1334] Step 7:

[1335] The generative AI model generates movement improvement advice based on the analysis results. The input is the analysis results of the exercise movement, and the output is movement improvement advice. During this process, visual support images are generated along with specific advice.

[1336] Step 8:

[1337] The server sends the generated advice and auxiliary image to the user's device. The input is the behavior improvement advice and auxiliary image, and the output is the advice and auxiliary image displayed on the user's device. The user can check this advice through the application.

[1338] Step 9:

[1339] The user adjusts the robot's behavior based on the behavior improvement advice. The input is the displayed advice, and the output is the adjusted robot behavior. The user adjusts the robot's settings based on specific improvements to improve operational efficiency.

[1340] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1341] The system according to the present invention not only analyzes videos and provides exercise improvement advice, but also incorporates an emotion engine that recognizes the user's emotions and adjusts the content and presentation method of the advice accordingly. A specific embodiment of the system will be described below.

[1342] First, the user uses their device to record their child's exercise. For example, they can use a smartphone to record their child practicing dribbling a basketball. The user then uploads the video to the server via a dedicated application. This application is easy to use; the user simply selects a video and presses the upload button, and the video is sent to the server.

[1343] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis. Next, it splits the video into frames and prepares each frame to be sent to the generation AI.

[1344] The generative AI detects the positions of the child's joints and body parts in each frame. For example, it identifies the positions of major joints such as elbows, wrists, and torso, and collects their positions as time-series data. It then analyzes the child's movements based on this position data and evaluates their movement patterns. During this evaluation process, it compares them with an existing technical database to identify any discrepancies from the correct form and technique.

[1345] Based on the analysis results, the AI ​​generates specific advice for improving your movements. For example, it might say, "Your wrist movements are stiff when dribbling, so try to relax them." It can also create visual aids, which can help users understand the advice more intuitively.

[1346] A distinctive feature of the present invention is the incorporation of an emotion engine. This emotion engine can recognize the user's facial expressions and voice through the device's camera and microphone and evaluate the user's emotional state. The emotion data acquired by the emotion engine is reflected in the content and presentation method of advice. For example, if the user is feeling irritated or anxious, the tone of the advice can be changed to a gentler one, encouraging the user to take a more positive approach. Also, if the user is enjoying themselves, encouraging words can be added to maintain that state.

[1347] Finally, the generated advice and auxiliary images are sent from the server to the user's device. The user can then review the advice and provide specific guidance to their child through the application. In this way, the system provides specific and effective support for improving children's motor skills, while also being able to flexibly respond to the user's emotional state.

[1348] Example: Analysis of basketball dribbling practice

[1349] Consider the case where a user films their child practicing basketball dribbling and uploads it using a dedicated application. The server receives the video and analyzes it frame by frame. The generation AI analyzes the movement of each joint in the dribbling motion and identifies stiff wrist movement. Based on the analysis results, it generates specific advice such as "Relax your wrist."

[1350] Meanwhile, the emotion engine recognizes when the user is frustrated and adjusts the tone of the advice to be gentler, for example, by saying, "It may be difficult at first, but you can improve little by little." This allows the user to provide guidance to their child in a positive manner.

[1351] As described above, this system not only helps children improve their motor skills, but also provides a flexible approach that takes into consideration the user's emotions.

[1352] The processing flow will be explained below.

[1353] Step 1:

[1354] A user uses their own device to record a video of their child exercising. For example, a user uses a smartphone to record a video of their child practicing basketball dribbling.

[1355] Step 2:

[1356] The user launches the dedicated application, selects the video they have taken, and taps the "Upload" button.

[1357] Step 3:

[1358] The device sends the selected video to the server. When the video is sent, a checksum and other verifications are performed to ensure the integrity of the data.

[1359] Step 4:

[1360] The server receives the video data sent from the device, reconfirms the integrity of the video file, and stores it in temporary storage.

[1361] Step 5:

[1362] The server converts the received video into a standard format for analysis, converting the codec and adjusting the resolution as necessary.

[1363] Step 6:

[1364] The server splits the video into frames, extracts each frame as an individual image file, and stores the timestamp information for each frame.

[1365] Step 7:

[1366] The server sends the divided frame data to the generation AI, which receives it and detects the positions of joints and body parts in each frame.

[1367] Step 8:

[1368] Based on the detected joint position data, the generative AI analyzes the movement over time, tracking the movement patterns and positions of each joint and extracting the characteristic points of the movement.

[1369] Step 9:

[1370] The generative AI compares the extracted feature points with an existing technical database, and based on the comparison results, identifies any differences between the target athletic movement and correct form and technique.

[1371] Step 10:

[1372] Based on the analysis results, the generative AI generates specific advice for improving exercise, such as "relax your wrist and move more smoothly."

[1373] Step 11:

[1374] The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice to recognize the user's emotional state, for example, detecting when the user is feeling irritated.

[1375] Step 12:

[1376] Based on the emotional data recognized by the emotion engine, the generative AI adjusts the way it presents advice. For example, if a user feels frustrated, it will soften the tone of the advice and add encouraging words such as "Take your time and practice a little at a time."

[1377] Step 13:

[1378] The server transmits the generated advice and the adjusted advice data to the user's terminal.

[1379] Step 14:

[1380] The user checks the advice data sent from the server through an application on their device. The application displays the advice and provides visual support images to help the user understand the specific improvements required.

[1381] Step 15:

[1382] The user provides specific advice for improvement based on the application, and by filming the exercise again and repeating these steps, the user can continuously identify areas for improvement and improve athletic ability.

[1383] Example 2

[1384] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1385] Conventional video analysis systems can analyze exercise videos taken by users and provide exercise improvement advice, but they provide uniform advice without considering the user's emotions, making it difficult to provide appropriate guidance tailored to the user's situation.In addition, there is a lack of means to compare with technical databases to improve the accuracy of motion analysis, or to generate visual support images, making it difficult to deepen understanding of the analysis results.

[1386] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1387] In this invention, the server includes means for receiving a video, means for dividing the video into a plurality of frames, means for detecting the positions of joints and body parts of an object in each frame, means for evaluating the analyzed exercise movements by comparing with a specific database, means for generating exercise improvement advice based on the analysis results, means for recognizing the user's emotions, means for adjusting the generated advice based on the recognized emotions, and means for outputting the generated advice, thereby making it possible to provide flexible exercise improvement advice that takes the user's emotions into consideration.

[1388] The "means for receiving video" is a function for transmitting video data taken by a user to a server via a network and for the server to receive the video data.

[1389] "Means for dividing a video into multiple frames" is a function for dividing a received video into small still images (frames) along the time axis and generating data for each frame.

[1390] "Means for detecting the positions of joints and body parts of an object in each frame" refers to a technique for identifying the positions of joints and body parts relative to an object in each frame image and extracting their coordinate data.

[1391] The "means for evaluating the analyzed exercise movements by comparing with a specific database" is a function for comparing the analyzed exercise data with an existing reference database and evaluating the accuracy of the exercise and areas for improvement.

[1392] The "means for generating exercise improvement advice based on the analysis results" is a function for automatically generating specific improvement methods and instruction contents based on the evaluation results of the exercise analysis.

[1393] "Means for recognizing user emotions" refers to technology that analyzes information such as the user's facial expressions and voice to identify the user's emotional state.

[1394] The "means for adjusting the advice generated based on the recognized emotion" is a function for flexibly changing the content and tone of the exercise improvement advice generated according to the user's emotional state.

[1395] The "means for outputting generated advice" is a function for displaying or providing the generated exercise improvement advice and visual support images in an easily understandable form to the user.

[1396] MODE FOR CARRYING OUT THE INVENTION

[1397] The system according to the present invention not only analyzes videos and provides exercise improvement advice, but also incorporates an emotion recognition engine that recognizes the user's emotions and adjusts the content and presentation method of the advice accordingly. A specific embodiment of the system will be described below.

[1398] First, the user uses their device to record their child's exercise. For example, they can use a smartphone to record their child practicing dribbling a basketball. The user then uploads the video to the server via a dedicated application. This application is easy to use; the user simply selects a video and presses the upload button, and the video is sent to the server.

[1399] The server receives the uploaded video and stores it in storage. After verifying the integrity of the video file, it converts the video into a format for analysis using software such as FFmpeg. Next, it splits the video into frames and prepares each frame to be sent to a generative AI (for example, using OpenPose or TensorFlow).

[1400] The generative AI detects the positions of the child's joints and body parts in each frame. For example, it identifies the positions of major joints such as elbows, wrists, and torso, and collects their positions as time-series data. It then analyzes the child's movements based on this position data and evaluates their movement patterns. During this evaluation process, it compares the child's movements with existing technical databases (e.g., Sports Motion Database) to identify any discrepancies with the correct form and technique.

[1401] Based on the analysis results, the AI ​​generates specific advice for improving your movements. For example, it might say, "Your wrist movements are stiff when dribbling, so try to relax them." It can also create visual aids, which can help users understand the advice more intuitively.

[1402] A distinctive feature of the present invention is the incorporation of an emotion engine. This emotion engine can recognize the user's facial expressions and voice through the device's camera and microphone and evaluate the user's emotional state. The emotion data acquired by the emotion engine is reflected in the content and presentation method of advice. For example, if the user is feeling irritated or anxious, the tone of the advice can be changed to a gentler one, encouraging the user to take a more positive approach. Also, if the user is enjoying themselves, encouraging words can be added to maintain that state.

[1403] Finally, the generated advice and auxiliary images are sent from the server to the user's device. The user can then review the advice and provide specific guidance to their child through the application. In this way, the system provides specific and effective support for improving children's motor skills, while also being able to flexibly respond to the user's emotional state.

[1404] Example: "Analysis of basketball dribbling practice"

[1405] Consider the case where a user films their child practicing basketball dribbling and uploads it using a dedicated application. The server receives the video and analyzes it frame by frame. The generation AI analyzes the movement of each joint in the dribbling motion and identifies stiff wrist movement. Based on the analysis results, it generates specific advice such as "Relax your wrist."

[1406] Meanwhile, the emotion engine recognizes when the user is frustrated and adjusts the tone of the advice to be gentler, for example, by saying, "It may be difficult at first, but you can improve little by little." This allows the user to provide guidance to their child in a positive manner.

[1407] Examples of prompt statements

[1408] Prompt statement:

[1409] I filmed a kid practicing dribbling a basketball. Analyze the uploaded video and identify stiff wrist movements. Then provide specific advice to relax the wrist. Also, adjust the tone of the advice to be gentler if the user is frustrated.

[1410] As described above, this system not only helps children improve their motor skills, but also provides a flexible approach that takes into consideration the user's emotions.

[1411] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1412] Step 1:

[1413] The user uses a device such as a smartphone to record their child's exercise. The user then uploads the video to a server using a dedicated application. The input is a video file, and the output is a video file stored on the server.

[1414] Step 2:

[1415] The server checks the received video and saves it in storage. At this time, it checks the integrity of the video file and checks whether it contains any invalid data. The input is the uploaded video file, and the output is the video file saved in storage whose integrity has been checked.

[1416] Step 3:

[1417] The server uses video processing software such as FFmpeg to convert the stored video into a format for analysis. The input in this step is a video file whose integrity has been confirmed, and the output is a video file converted into the format for analysis.

[1418] Step 4:

[1419] The server divides the video file for analysis into frames and generates image data for each frame. Specifically, it divides one second of video into 30 frames. The input for this step is the video file for analysis, and the output is still images for each frame (e.g., "frame0001.jpg", "frame0002.jpg").

[1420] Step 5:

[1421] The server sends each frame image to a generative AI (e.g., OpenPose or TensorFlow) to detect the positions of the child's joints and body parts in each frame. The input is the frame image data, and the output is joint position data (e.g., coordinate data for the left elbow, right wrist, and trunk).

[1422] Step 6:

[1423] The server analyzes the motion based on the detected joint position data. The analysis results are compared with a specific technique database (e.g., Sports Motion Database) to identify deviations from the correct form and technique. The input for this step is the joint position data, and the output is the evaluation result of the motion analysis (e.g., "The wrist movement is stiff").

[1424] Step 7:

[1425] The server generates exercise improvement advice based on the analysis results. The generation AI creates specific advice text (e.g., "Try to relax your wrist") and visual support images (e.g., illustrations of the movement to relax your wrist). The input is the evaluation results of the movement analysis, and the output is exercise improvement advice and support images.

[1426] Step 8:

[1427] The device uses a camera and microphone to collect the user's facial expressions and voice. This data is sent to the emotion engine to evaluate the user's emotional state. The input is the user's facial and voice data, and the output is the emotion recognition result (e.g., "irritated").

[1428] Step 9:

[1429] The server adjusts the tone and content of exercise improvement advice based on the emotion recognition results obtained from the emotion engine. For example, if a user feels frustrated, the tone may be changed to a gentler one, such as "It may be difficult at first, but you can improve little by little." The input is the emotion recognition result, and the output is the adjusted exercise improvement advice.

[1430] Step 10:

[1431] The server sends the final exercise improvement advice and supplementary images to the user's device. The user can then review the advice and provide specific guidance to their child through the application. The input is the adjusted exercise improvement advice and supplementary images, and the output is the final advice displayed on the user's device.

[1432] keyword

[1433] Generative AI model, prompt sentence

[1434] (Application example 2)

[1435] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1436] Conventional exercise improvement systems can analyze a user's exercise data and provide specific advice, but they are unable to adjust the content of the advice taking the user's emotions into account. As a result, the advice is often ineffective when the user is feeling frustrated or anxious. Furthermore, while it is important for exercise improvement advice to be presented in a visually easy-to-understand format, there has been a lack of effective ways to achieve this. This has led to issues such as insufficient user satisfaction and effective exercise improvement.

[1437] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1438] In this invention, the server includes means for receiving a video, means for dividing the video into a plurality of frames, means for detecting the positions of joints and body parts of an object in each frame, means for analyzing exercise movements based on the detected positions, means for generating exercise improvement advice based on the analysis results, means for outputting the generated advice, means for recognizing a user's emotion, and means for adjusting the content and presentation method of the advice according to the recognized emotion. This makes it possible to provide effective and flexible exercise improvement advice to the user, and to improve the acceptability and satisfaction of the advice by taking the user's emotion into consideration.

[1439] The "means for receiving video" is a mechanism by which a user sends video taken by a terminal to a server, and the server receives the video.

[1440] The "means for dividing a video into a plurality of frames" is a mechanism for dividing a received video into frames along the time axis and making each frame individually analyzable.

[1441] The "means for detecting the positions of the joints and body parts of the object in each frame" is a mechanism for identifying the positions of the joints and body parts of a person in the video frames to be analyzed and acquiring that data.

[1442] The "means for analyzing movement based on detected positions" is a mechanism for evaluating and analyzing movement based on acquired joint and body position data.

[1443] The "means for generating exercise improvement advice based on the analysis results" is a mechanism for generating specific advice for improving exercise based on the analysis results of exercise movements.

[1444] The "means for outputting generated advice" is a mechanism for presenting the generated exercise improvement advice to the user and displaying it in an easy-to-understand manner.

[1445] The "means for recognizing the user's emotions" is a mechanism for analyzing the user's facial expressions and voice through a camera or microphone and recognizing the user's emotional state.

[1446] The "means for adjusting the content and presentation method of advice according to the recognized emotion" is a mechanism for appropriately changing the content and presentation method of advice based on the user's emotional state.

[1447] The system according to the present invention provides exercise improvement support by combining video analysis and emotion recognition. This system not only analyzes videos to provide exercise improvement advice, but also incorporates an emotion engine that recognizes the user's emotions and adjusts the content and presentation method of the advice accordingly. Specific aspects of this system are described below.

[1448] First, the user films the exercise using their own device. An example of this is a child using a smartphone to film their basketball practice. The user then uploads the video to the server via a dedicated application. The application is designed to be user-friendly; simply by selecting a video and pressing the upload button, the video is automatically sent to the server.

[1449] The server stores the received video in storage, verifies the integrity of the video file, and then converts the video into a format for analysis. The video is then split into frames, and each frame is prepared to be sent to a generation AI for analysis. This generation AI uses a machine learning library such as TensorFlow.

[1450] The generative AI detects the positions of the joints and body parts of the subject (in this example, a child) in each frame. For example, it identifies the positions of major joints such as the elbows, wrists, and torso, and collects their positions as time-series data. It then analyzes the movement based on this position data and evaluates the movement patterns. During this evaluation process, it compares the results with an existing technical database to identify any discrepancies from the correct form and technique.

[1451] Based on the analysis results, the AI ​​generates specific advice for improving the movement. For example, it might say, "When shooting, your elbow position is low. Try to keep it a little higher." In addition, it creates auxiliary images that are easy to understand visually.

[1452] A distinctive feature of the present invention is the incorporation of an emotion engine. This emotion engine recognizes the user's facial expressions and voice through the device's camera and microphone and evaluates the user's emotional state. This evaluation utilizes emotion analysis APIs such as Microsoft Azure Cognitive Services. The emotion data acquired by the emotion engine is reflected in the content and presentation of the generated advice. For example, if the user is feeling frustrated or anxious, the tone of the advice can be made gentler, and soft words such as "Trial and error lead to growth. Take your time and practice slowly." If the user is enjoying themselves, encouraging words such as "That's great! Keep it up!" can be added to maintain that state of mind.

[1453] Finally, the generated advice and auxiliary images are sent from the server to the user's device, where the user can check the advice and receive specific guidance through the application.

[1454] For example, consider the following prompt:

[1455] python

[1456] import requests

[1457] import cv2

[1458] def upload_video(video_path):

[1459] url = 'http: / / yourserver.com / upload'

[1460] files = {'file': open(video_path, 'rb')}

[1461] response = requests.post(url, files=files)

[1462] if response.status_code == 200:

[1463] print("Video uploaded successfully!")

[1464] else:

[1465] print("Error in video upload")

[1466] def analyze_video_frames(frames, model):

[1467] results = []

[1468] for frame in frames:

[1469] processed_frame = preprocess_frame(frame)

[1470] result = model.predict(processed_frame)

[1471] results.append(result)

[1472] return results

[1473] def generate_advice(movement_data):

[1474] Analyze movement_data and generate advice

[1475] advice = "Your elbow position is incorrect during the shot. Try to keep it higher."

[1476] return advice

[1477] def emotion_adjust_advice(user_emotion, advice):

[1478] if user_emotion == 'frustrated':

[1479] adjusted_advice = "Don't worry, improvement takes time. " + advice

[1480] elif user_emotion == 'happy':

[1481] adjusted_advice = "Great job! Keep it up. " + advice

[1482] else:

[1483] adjusted_advice = advice

[1484] return adjusted_advice

[1485] Example usage

[1486] video_path = 'child_shooting_basketball.mp4'

[1487] upload_video(video_path)

[1488] This program aims to improve athletic ability by uploading videos taken by users to a server, analyzing the videos and recognizing emotions, and generating and presenting appropriate advice.

[1489] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1490] Step 1:

[1491] Recording and uploading videos

[1492] Action: A user uses a smartphone to record themselves exercising.

[1493] Input: The captured video file.

[1494] Output: The video file imported into the application.

[1495] Specific operation: The user presses the record button to record a video of their exercise. After recording is complete, the user presses the application's upload button to send the video to the server.

[1496] Step 2:

[1497] Receiving and storing videos

[1498] How it works: The server receives and stores videos uploaded by users.

[1499] Input: The uploaded video file.

[1500] Output: Video files stored in the server's storage.

[1501] What it does: The server receives an HTTP request, saves the file to disk, checks the integrity of the saved file, and prepares it for analysis.

[1502] Step 3:

[1503] Video frame division

[1504] How it works: The server splits the stored video into frames.

[1505] Input: A saved video file.

[1506] Output: Image data per frame.

[1507] Specific operation: Using the OpenCV library, a video file is read and split into frame-by-frame still images. Each frame is then converted into the format required for subsequent analysis.

[1508] Step 4:

[1509] Position detection of joints and body parts

[1510] How it works: The server detects the positions of joints and body parts for each frame.

[1511] Input: Frame-by-frame image data.

[1512] Output: Position data of joints and body parts (time series data).

[1513] Specific operation: Using a generative AI model such as TensorFlow, joints and body parts (e.g., elbows, wrists, torso, etc.) are analyzed for each frame to obtain positional data.

[1514] Step 5:

[1515] Analysis of athletic movements

[1516] Movement: The server analyzes the movement based on the detected location data.

[1517] Input: Position data of joints and body parts.

[1518] Output: Motor behavior evaluation data.

[1519] Specific behavior: The acquired position data is analyzed as time series data, and behavior patterns are evaluated by comparing it with a technical database. Inappropriate behavior is identified and the analysis results are output.

[1520] Step 6:

[1521] Generating exercise improvement advice

[1522] Action: The server generates exercise improvement advice based on the results of the exercise motion analysis.

[1523] Input: Exercise behavior assessment data.

[1524] Output: Text and supporting images of exercise improvement advice.

[1525] Specific actions: Based on the analysis results, specific advice is generated based on the situation, such as, "When shooting, your elbow is positioned too low, so try to keep it a little higher." Supporting images are also created to make the game easier to understand visually.

[1526] Step 7:

[1527] User Emotion Recognition

[1528] How it works: The server recognizes the user's emotions.

[1529] Input: User's facial and voice data.

[1530] Output: User's emotional state data.

[1531] Specific operation: Using emotion analysis APIs such as Microsoft Azure Cognitive Services, the system analyzes facial expression data and voice data acquired through the camera to evaluate the user's emotional state (e.g., irritation, joy, calmness, etc.).

[1532] Step 8:

[1533] Advice adjustment

[1534] How it works: The server adjusts the content and presentation of advice based on the results of emotion recognition.

[1535] Input: Exercise improvement advice, user emotional state data.

[1536] Output: Tailored exercise improvement advice.

[1537] Specific actions: Change the tone of advice based on emotion recognition data, e.g., "Trial and error leads to growth. Take your time and practice slowly." Add words of encouragement according to the emotion.

[1538] Step 9:

[1539] Providing advice

[1540] How it works: The server sends the generated and tailored advice to the user's device and presents it through the application.

[1541] Enter: tailored exercise improvement advice.

[1542] Output: Advice and auxiliary images displayed on the user's terminal.

[1543] Specific operation: The generated advice and auxiliary images are sent to the user's device, and displayed through the application so that the user can check and execute them.

[1544] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1545] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1546] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1547] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1548] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1549] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1550] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1551] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1552] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1553] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1554] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1555] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1556] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1557] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1558] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1559] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1560] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1561] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1562] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1563] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1564] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1565] The following is further disclosed regarding the above embodiment.

[1566] (Claim 1)

[1567] means for receiving video;

[1568] A means for dividing a video into frames;

[1569] means for detecting the position of joints and body parts of the object in each frame;

[1570] means for analyzing the movement based on the detected position;

[1571] a means for generating exercise improvement advice based on the analysis results;

[1572] means for outputting the generated advice;

[1573] A system including:

[1574] (Claim 2)

[1575] 2. The system according to claim 1, further comprising means for generating exercise improvement advice as a visual auxiliary image based on the analysis result.

[1576] (Claim 3)

[1577] 10. The system of claim 1, further comprising means for comparing the analyzed athletic performance with a specific skill database.

[1578] "Example 1"

[1579] (Claim 1)

[1580] means for receiving video;

[1581] A means for dividing a video into frames;

[1582] means for detecting the position of joints and body parts of the object in each frame;

[1583] means for analyzing the movement based on the detected position;

[1584] a means for generating exercise improvement advice based on the analysis results;

[1585] a means for outputting the generated exercise improvement advice and visual support image;

[1586] means for transmitting advice and auxiliary images from the server to the user's terminal;

[1587] A system including:

[1588] (Claim 2)

[1589] 2. The system according to claim 1, further comprising means for generating exercise improvement advice as a visual auxiliary image based on the analysis result.

[1590] (Claim 3)

[1591] 10. The system of claim 1, further comprising means for comparing the analyzed athletic performance with a specific skill database.

[1592] "Application Example 1"

[1593] (Claim 1)

[1594] means for receiving video;

[1595] A means for dividing a video into frames;

[1596] means for detecting the position of joints and body parts of the object in each frame;

[1597] means for analyzing the movement based on the detected position;

[1598] A means for generating behavior improvement advice based on the analysis results;

[1599] means for outputting the generated advice;

[1600] A means to film and upload the robot's movements;

[1601] a means for evaluating the efficiency of the operation using the generative AI model;

[1602] A means for identifying a behavior pattern based on the evaluation results;

[1603] A system including:

[1604] (Claim 2)

[1605] 2. The system according to claim 1, further comprising means for generating action improvement advice as a visual auxiliary image based on the analysis result.

[1606] (Claim 3)

[1607] 10. The system of claim 1, further comprising means for comparing the analyzed athletic performance with a specific skill database.

[1608] "Example 2: Combining Emotion Engines"

[1609] (Claim 1)

[1610] means for receiving video;

[1611] A means for dividing a video into frames;

[1612] means for detecting the position of joints and body parts of the object in each frame;

[1613] means for analyzing the movement based on the detected position;

[1614] a means for evaluating the analyzed athletic movements by comparing them with a specific database;

[1615] a means for generating exercise improvement advice based on the analysis results;

[1616] means for recognizing a user's emotion;

[1617] a means for adjusting the generated advice based on the perceived emotions;

[1618] means for outputting the generated advice;

[1619] A system including:

[1620] (Claim 2)

[1621] 2. The system according to claim 1, further comprising means for generating exercise improvement advice as a visual auxiliary image based on the analysis result.

[1622] (Claim 3)

[1623] 10. The system of claim 1, further comprising means for comparing the analyzed athletic performance with a specific skill database.

[1624] "Application example 2 when combining emotion engines"

[1625] (Claim 1)

[1626] means for receiving video;

[1627] A means for dividing a video into frames;

[1628] means for detecting the position of joints and body parts of the object in each frame;

[1629] means for analyzing the movement based on the detected position;

[1630] a means for generating exercise improvement advice based on the analysis results;

[1631] means for outputting the generated advice;

[1632] means for recognizing a user's emotion;

[1633] a means of tailoring the content and presentation of advice depending on perceived emotions;

[1634] A system including:

[1635] (Claim 2)

[1636] 2. The system according to claim 1, further comprising means for generating exercise improvement advice as a visual auxiliary image based on the analysis result.

[1637] (Claim 3)

[1638] 10. The system of claim 1, further comprising means for comparing the analyzed athletic performance with a specific skill database. [Explanation of symbols]

[1639] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving video; A means for dividing a video into frames; means for detecting the position of joints and body parts of the object in each frame; means for analyzing the movement based on the detected position; a means for generating exercise improvement advice based on the analysis results; means for outputting the generated advice; A system including:

2. The system according to claim 1 , further comprising means for generating exercise improvement advice as a visual auxiliary image based on the analysis result.

3. 10. The system of claim 1, further comprising means for comparing the analyzed athletic performance with a specific skill database.

Images