System

Abstract

A system is provided.SOLUTION: A system including means for learning motion data of a professional athlete, means for receiving a moving image of a captured sport motion, means for preprocessing the received moving image data and extracting a feature amount, means for comparing and analyzing the received moving image data with motion data of a professional athlete using the extracted feature amount, means for generating specific advice based on an analysis result, means for generating a comparison moving image in which a current form and an ideal form are superimposed, and means for outputting the generated advice and the comparison moving image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many children try sports, but parents often have no experience in the sport and are therefore unable to provide appropriate advice. Furthermore, there are few systems that can provide specific areas for improvement tailored to individual children, making it difficult to provide effective training methods. The objective of this invention is to provide a system that allows parents to provide specific advice based on professional techniques for sports in which they have no experience, thereby promoting their children's improvement in sports. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means: It provides a system that includes a means for learning the motion data of professional athletes and a means for receiving video of filmed sports movements, a means for preprocessing the received video data and extracting features, and a means for performing comparative analysis with the motion data of the professional athletes using the extracted features, and a means for generating specific advice based on the analysis results, a means for generating a comparison video that overlays the current form with an ideal form, and a means for outputting the generated advice and the comparison video.

[0006] Specifically, the system includes a means for providing users with an interface for uploading motion data used as a model by professional athletes, and sends the motion data captured by the user to a server. The server then analyzes and compares the data and generates specific advice on areas for improvement. The server then notifies the user of the advice and provides an interface for confirming the advice. This allows parents to provide specific advice based on professional techniques, even for sports in which they have no experience, helping their children improve at sports.

[0007] "Professional athlete motion data" refers to data that records the movements of highly skilled athletes in a particular sport.

[0008] "Means for receiving video" refers to the interface or technology for importing captured video data into a server or device.

[0009] "Preprocessing" refers to the process of dividing the received video data into frames and extracting features to make it easier to analyze.

[0010] "Means for extracting features" refers to technology that extracts data such as important key points and joint positions related to sports movements from video data.

[0011] "Means for comparative analysis" refers to the techniques and processes for comparing the movement data of professional athletes with the movement data of children based on the extracted features and analyzing the differences.

[0012] The "means for generating specific advice" refers to a technique for generating specific instructions or advice for improving behavior based on the results of the comparative analysis.

[0013] A "comparison video overlaying current and ideal form" is a video that allows users to visually understand the difference between a child's current movements and the ideal movements of a professional athlete by overlaying them.

[0014] "Output means" refers to an interface or system for providing the generated advice and comparison videos to the user.

[0015] An "interface for uploading motion data" refers to the functionality and technology that allows users to send video data they have taken to a server or a specific device. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The following describes in detail an embodiment of the present invention. This system uses AI to provide parents with specific advice to help their children improve their sports skills.

[0038] Overall system overview

[0039] This system learns the motion data of professional athletes, compares it with filmed footage of children's sports movements, and provides specific advice based on the results, enabling parents to provide effective instruction even if they have no experience in sports.

[0040] Program processing flow

[0041] 1. Learning the movements of professional athletes

[0042] server

[0043] 1. Receive and preprocess motion data from professional athletes. Divide the received video data into frames and extract features such as key points and joint positions.

[0044] 2. Use machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes and train the model.

[0045] 3. Save the trained model in a database.

[0046] Terminal

[0047] 1. Provides an interface for uploading operational data to a server.

[0048] 2. Display a notification to the user that the learning is complete.

[0049] User

[0050] 1. Record the motion data of a professional athlete and upload it to a server via the device.

[0051] 2. Check your learning progress and completion notifications.

[0052] 2. Filming and reading children's sports movements

[0053] Terminal

[0054] 1. Provide a camera interface for capturing children's sports activities.

[0055] 2. Display the option to upload the captured video data to the server.

[0056] User

[0057] 1. Take photos of your child's sports movements with your smartphone.

[0058] 2. Upload the captured video data to the server.

[0059] 3. Behavior analysis and advice generation phase

[0060] server

[0061] 1. Preprocess the received video data of the child, divide it into frames, and extract features.

[0062] 2. Use the saved model to compare and analyze the motion data of children and professional athletes.

[0063] 3. Based on the analysis results, specific advice (e.g., "Raise your right elbow 3 cm higher") is generated.

[0064] 4. Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[0065] 5. Send advice and comparison videos to your device.

[0066] 4. Results display and advice phase

[0067] Terminal

[0068] 1. The advice received from the server and comparison video are displayed to the user.

[0069] 2. Provide an interface for playing videos that overlay AsIs and ToBe.

[0070] User

[0071] 1. Check the advice displayed on your device and the comparison video.

[0072] 2. Work with your child to improve their sports movements using the advice provided.

[0073] 3. If necessary, re-photograph the improved behavior and re-analyze it using the same process.

[0074] Example: Improving children's baseball pitching technique

[0075] Initial setup and learning phase

[0076] Users can record the pitching motion of a top baseball pitcher and upload it to the server via their device. The server then preprocesses the received video data and learns the motion data of the professional pitcher.

[0077] Filming the pitching motion

[0078] The user continuously records the child's pitching motion with a smartphone and uploads the video data to a server.

[0079] Behavior analysis and advice generation

[0080] The server analyzes the child's pitching video and compares it with the motion data of a professional pitcher. It generates specific improvements (e.g., "Raise your right elbow 3 cm higher") and creates a video comparing AsIs and ToBe.

[0081] Displaying and Implementing Results

[0082] The user checks the advice and comparison video displayed on the device and practices together with the child. After the improvement, the movement is filmed again and uploaded to the server for reanalysis.

[0083] This system allows parents with no sports experience to provide effective advice and support their children's sports improvement.

[0084] The processing flow will be explained below.

[0085] Program processing flow

[0086] 1. Learning the movements of professional athletes

[0087] Step 1: Prepare the operation data

[0088] User: Uses a camera to record the movements of a professional athlete and saves the video file on the device.

[0089] Step 2: Uploading the behavior data

[0090] Terminal: Provides an interface for uploading operational data to the server.

[0091] User: Select the captured motion data and upload it to the server.

[0092] Device: Sends motion data to the server.

[0093] Step 3: Preprocessing the behavioral data

[0094] Server: Divides the received video file into frames and extracts key points and joint positions.

[0095] Server: Stores the preprocessed data in a database.

[0096] Step 4: Training the learning model

[0097] Server: Uses machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes.

[0098] Server: Saves the trained model in a database.

[0099] Step 5: Notification of completion of study

[0100] Server: Notifies the device that learning is complete.

[0101] Terminal: Display a message to the user that the training is complete.

[0102] 2. Filming and reading children's sports movements

[0103] Step 1: Capture the action

[0104] User: Takes photos of children's sports activities with a smartphone.

[0105] Terminal: Provides a camera interface for capturing action.

[0106] Step 2: Uploading the behavior data

[0107] On your device: Displays the option to upload the captured video file to the server.

[0108] User: Select a video file and upload it to the server.

[0109] Device: Sends the video file to the server.

[0110] 3. Behavior analysis and advice generation phase

[0111] Step 1: Preprocessing the video data

[0112] Server: Divides the received video file of the child into frames and extracts features.

[0113] Step 2: Comparative analysis of behavior

[0114] Server: Uses the trained model to compare children's motion data with that of professional athletes.

[0115] Server: Quantify the difference in each frame and identify specific areas for improvement.

[0116] Step 3: Generate Advice

[0117] Server: Generates specific advice based on the results of the comparative analysis.

[0118] Server: Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[0119] Step 4: Submit the results

[0120] Server: Sends the generated advice and comparison videos to the device.

[0121] 4. Results display and advice phase

[0122] Step 1: View the results

[0123] Terminal: Displays the advice received from the server and the comparison video to the user.

[0124] Device: Provides an interface for playing videos that overlay AsIs and ToBe.

[0125] Step 2: Operation check and improvement steps

[0126] User: Check the advice and comparison video displayed on the device.

[0127] User: Work with their child to improve their sports movements using the advice provided.

[0128] Step 3: Feedback and Retraining

[0129] User: Re-photograph the improved behavior and re-analyze it using the same process.

[0130] Device: Sends new video to the server and requests reanalysis.

[0131] As described above, a system is constructed in which the server, terminals, and users work together to support children in improving their sports movements.

[0132] Example 1

[0133] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] Conventional systems supporting children's sports improvement have the problem that it is difficult to provide effective advice to parents who lack sports experience or expertise. Furthermore, analyzing the recorded video data requires specialized expertise and is not easy to do. Furthermore, the system lacks a means to visually show specific areas for improvement and an interface for reassessment, making the improvement process difficult.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0136] In this invention, the server includes: means for learning motion data of professional athletes; means for receiving video of filmed sports movements; means for preprocessing the received video data and extracting features; means for comparative analysis with the motion data of the professional athletes using the extracted features; means for generating specific advice based on the analysis results; means for generating a comparison video that overlays the current form with an ideal form; means for outputting the generated advice and the comparison video; means for providing a camera interface for filming the child's sports movements; and means for inputting a prompt sentence for generating an analysis request to the AI ​​model. This enables parents, even without experience or knowledge of sports, to effectively analyze their children's sports movements, provide specific and visual advice, and reassess their children's sports movements.

[0137] "Professional athlete motion data" refers to data that records the motions of athletes with high levels of skill.

[0138] "Means for receiving" refers to technical means for taking in data from the outside, such as a server or storage device.

[0139] "Preprocessing" refers to the process of converting received data into a format that is easy to analyze.

[0140] A "feature" is a data point or value that contains important information in data analysis.

[0141] A "comparative analysis tool" is a tool that compares two or more data sets to analyze commonalities and differences.

[0142] "Specific advice" refers to specific, practical instructions or suggestions.

[0143] A "comparison video" is a video that displays your current form and ideal form at the same time for comparison.

[0144] "Means of output" refers to the technical means for providing data or information to the user.

[0145] A "camera interface" is a user interface for operating a camera.

[0146] An "AI model" is an algorithm trained using machine learning or deep learning.

[0147] A "prompt sentence" is an input sentence for a generative AI model, and is text that specifies an analysis request.

[0148] The following describes in detail the embodiments of the present invention. This system uses AI to provide specific advice to parents to help their children improve their sports skills. Even if the parents have no experience in sports, the system can effectively analyze their children's movements, provide specific and visual advice, and perform reassessments.

[0149] Overall system overview

[0150] This system learns the motion data of professional athletes, compares it with filmed footage of children's sports movements, and provides specific advice based on the results, allowing parents to provide effective instruction even if they have no experience in sports.

[0151] Program processing flow

[0152] 1. Learning the movements of professional athletes

[0153] server

[0154] 1. The server receives the motion data of professional athletes and performs preprocessing. This preprocessing involves dividing the video data into frames and extracting features such as key points and joint positions from each frame using open-source pose estimation technology such as OpenPose.

[0155] 2. The server uses the preprocessed feature data to train a model using machine learning algorithms such as LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network).

[0156] 3. The trained model is stored in a database by the server.

[0157] Terminal

[0158] 1. The device provides an interface for uploading professional athletes' motion data to the server, allowing users to easily select and upload videos.

[0159] 2. The device displays a learning completion notification to the user.

[0160] User

[0161] 1. The user uses their smartphone to record the movements of a professional athlete as an example, and then uploads the video data to the server via the device.

[0162] 2. The user checks the learning progress and completion notification displayed on the device.

[0163] 2. Filming and reading children's sports movements

[0164] Terminal

[0165] 1. The device provides a camera interface that allows the user to record their child's sports activities. For example, the app can activate the camera function and display a recording button to start recording.

[0166] 2. The device will display an option to upload the captured video data to the server.

[0167] User

[0168] 1. The user uses the smartphone camera to capture a video of their child's sports activity, for example, recording a video of their child pitching a baseball.

[0169] 2. The user uploads the captured video data to the server. By simply selecting the video and pressing the upload button according to the device interface, the data is sent to the server.

[0170] 3. Behavior analysis and advice generation phase

[0171] server

[0172] 1. The server preprocesses the received video data of the child, divides the video into frames, and extracts features. Specifically, it calculates the joint positions for each frame using OpenPose or similar.

[0173] 2. The server uses the stored trained model to compare and analyze the child's motion data with that of a professional athlete. This comparison evaluates the similarities and differences between the two motions based on the extracted features.

[0174] 3. The server generates specific advice based on the analysis results. For example, it may suggest to a child that they should "raise their right elbow 3cm higher" when pitching.

[0175] 4. The server generates a comparison video that overlays the AsIs (current) and ToBe (ideal) forms, making it visually easier to see which areas need improvement.

[0176] 5. The server sends the generated advice and comparison video to the device.

[0177] 4. Results display and advice phase

[0178] Terminal

[0179] 1. The device displays the advice and comparison video received from the server to the user. Specifically, the application provides a playback function for the advice text and comparison video.

[0180] 2. The device provides an interface for playing videos that overlay AsIs and ToBe, allowing users to easily play the videos and view them as many times as they like.

[0181] User

[0182] 1. The user checks the advice and comparison video displayed on the device. For example, they play the video to visually understand the suggested improvements.

[0183] 2. The user works with their child to improve their sports movements based on the provided advice. Specifically, for example, the user practices to correct the child's pitching form.

[0184] 3. If necessary, film the motion after the improvement and re-analyze it using the same process. Record the corrected pitching motion on video again, upload it to the server, and receive further advice.

[0185] Examples of prompt statements

[0186] Here are some examples of prompts that users can enter into a generative AI model:

[0187] "Please analyze my child's baseball pitching motion and let me know what needs to be improved. I have uploaded a video from the link below. I have also added a video of a professional pitcher, so you can compare it."

[0188] This allows the AI ​​model to provide appropriate analysis and advice based on the video data provided.

[0189] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0190] Step 1:

[0191] Receiving and preprocessing motion data from professional athletes

[0192] The server receives the motion data of the professional athlete, which includes video files uploaded by the user through an interface provided by the device.

[0193] Input: Video data of professional athletes' movements

[0194] The server preprocesses the received video data. This preprocessing involves dividing the video into frames and extracting key points and joint positions from each frame. Specifically, it uses open-source pose estimation technology such as OpenPose to detect the main joint points of the human body for each frame.

[0195] Data processing: Split the video into frames and extract joint points

[0196] Output: Features (keypoints and joint positions)

[0197] Step 2:

[0198] Training a machine learning model

[0199] The server uses the preprocessed feature data to train machine learning models, such as LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network). Training is necessary to capture the movement patterns of professional athletes.

[0200] Input: Preprocessed feature data

[0201] The server performs training using the given feature data and target data. During the training process, iterative parameter updates are performed using the backpropagation algorithm.

[0202] Data Computing: Model Training with Machine Learning Algorithms

[0203] Output: A trained model

[0204] Step 3:

[0205] Saving a trained model

[0206] The server stores the trained model in a database, allowing it to be quickly used in future analyses.

[0207] Input: A trained model

[0208] Data processing: Converting the model into a storage format

[0209] Output: Model saved in database

[0210] Step 4:

[0211] Filming children's sports

[0212] The user takes pictures of their child's sports activities using the smartphone camera and uses the camera interface provided by the device.

[0213] Input: Children's sports activities

[0214] The user activates the device's camera function and presses the record button to record the action. Once the recording is complete, it is saved as video data on the device.

[0215] Data generation: video data of sports movements

[0216] Output: Video data of children's sports movements

[0217] Step 5:

[0218] Uploading video data

[0219] The user uploads the captured video data to the server through the device, which displays an upload option.

[0220] Input: Video data of children's sports movements

[0221] The user selects a video from the terminal interface and presses the "upload" button to send the data to the server.

[0222] Data transfer: Uploading video data

[0223] Output: Video data of children's sports movements stored on the server

[0224] Step 6:

[0225] Preprocessing children's video data

[0226] The server preprocesses the received video data of the child by dividing the video into frames and extracting features (keypoints and joint positions) from each frame.

[0227] Input: Video data of children's sports movements stored on a server

[0228] Preprocessing includes extracting joint points using OpenPose, etc. The result of preprocessing is feature data for each frame.

[0229] Data processing: Split the video into frames and extract joint points

[0230] Output: Feature data of children's sports movements

[0231] Step 7:

[0232] Comparative analysis of motion data

[0233] The server uses the stored trained model to compare and analyze the motion data of children and that of professional athletes.

[0234] Input: feature data of children's sports movements, pre-trained model

[0235] The server evaluates the similarities and differences based on the extracted feature data. Specifically, it inputs the feature values ​​of each frame into a trained model and calculates the differences from the standard movements of professional athletes.

[0236] Data Computation: Comparative Analysis of Features

[0237] Output: Analysis results

[0238] Step 8:

[0239] Generating Specific Advice

[0240] The server generates specific advice based on the analysis results, such as "Raise your right elbow another 3 cm."

[0241] Input: Analysis results

[0242] Based on the analysis results, the server determines which parts should be improved and how, and generates advice in text format.

[0243] Data Generation: Specific Advice Text

[0244] Output: Specific advice

[0245] Step 9:

[0246] Comparison video generation

[0247] The server generates a comparison video that overlays the AsIs (current) and ToBe (ideal) forms, making it easier to visually identify areas for improvement.

[0248] Input: Video data of children's sports movements, pre-trained model

[0249] The server displays the current form and the ideal form side by side and combines the two videos to generate a single comparison video.

[0250] Data calculation: Video frame comparison and integration

[0251] Output: Comparison video

[0252] Step 10:

[0253] Advice and comparison video output

[0254] The server sends the generated advice and comparison video to the terminal, where the user can review it.

[0255] Input: Specific advice, comparison videos

[0256] The server transmits data to the terminal, and the terminal displays the received advice and video.

[0257] Data output: Sending advice and comparison videos

[0258] Output: Advice and comparison video displayed on device

[0259] (Application example 1)

[0260] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0261] Conventional sports instruction methods have the problem that it is difficult for parents and coaches to provide effective advice and instruction if they do not have specialized sports knowledge or skills. Furthermore, instruction services offered by sporting goods stores and fitness facilities are generally expensive, making them inaccessible to many people. Checking and improving sports form at physical stores is particularly time-consuming and requires a large burden on equipment and specialized staff. There is a need for a system that can solve these issues and enable ordinary parents and users to effectively improve their sports movements.

[0262] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0263] In this invention, the server includes: means for learning motion data of professional athletes; means for receiving video of filmed sports movements; means for preprocessing the received video data and extracting features; means for comparing and analyzing the extracted features with the motion data of the professional athletes; means for generating specific advice based on the analysis results; means for generating a comparison video that overlays the current form with an ideal form; means for outputting the generated advice and the comparison video; means for providing an interface that allows users at physical stores to film their sports movements and upload them to the cloud; and means for users to re-film and upload their improved movements and receive new advice to support improvement of their sports movements. This not only enables parents and physical store users with no sports experience to effectively improve their sports movements, but also enables efficient and cost-effective sports instruction services at physical stores.

[0264] "Professional athlete motion data" refers to data that records the motions of a professional athlete, and includes the feature quantities of those motions.

[0265] "Means for learning movement data" refers to the process of using machine learning algorithms to analyze the movement data of professional athletes and use it as a model for training.

[0266] "Means for receiving video" refers to the process by which the user sends the video data they have taken to the cloud or a server, and the server receives it.

[0267] "Preprocessing" refers to the process of dividing the received video data into frames and extracting important key points and joint positions.

[0268] "Features" are an important part in analyzing sports movements and include position information of key points and joints.

[0269] The "means for comparative analysis" refers to a process of comparing the motion data of a professional athlete with the photographed sports motion data using the extracted feature amounts.

[0270] "Means for generating advice" refers to the process of generating specific improvements and advice in text or video format based on the results of the comparative analysis.

[0271] "Method for generating a comparison video" refers to the process of generating a video that overlays the current (AsIs) and ideal (ToBe) forms to visually show the differences.

[0272] "Means for outputting a comparison video" refers to a process for providing the generated comparison video to a user.

[0273] "Brick-and-mortar users" refers to customers who visit physical stores, such as sporting goods stores or fitness facilities.

[0274] An "interface" is the part of the system that provides the screen and operating means for users to film their sports movements and upload them to the cloud.

[0275] "Cloud" refers to remote servers for storing, computing, and managing data over the Internet.

[0276] "Means for providing advice" refers to the process by which the generated advice is displayed to the user so that they can review it.

[0277] The following describes in detail the mode for carrying out the present invention. This system aims to support the improvement of sports movements in physical stores, and analyzes sports form and provides specific advice. By using this system, parents and general users can effectively improve their sports movements.

[0278] Overall system overview

[0279] The system learns from the motion data of professional athletes, compares and analyzes the sports motions of users filmed in physical stores, and generates and provides specific advice based on the results.

[0280] Hardware and software used

[0281] Hardware

[0282] Smartphone: A device that allows users to record and upload videos of their sports activities.

[0283] Cloud server: A remote server for storing data, training models, and performing analytical processing.

[0284] software

[0285] LSTM (Long Short-Term Memory): Used for time series analysis of behavioral data.

[0286] CNN (Convolutional Neural Network): Used to extract movement features.

[0287] OpenPose: Used to extract keypoints and joint positions from videos.

[0288] Flask / Django: Used for backend processing.

[0289] React Native: Used for the front end of smartphone applications.

[0290] Data processing and calculation

[0291] Learning from motion data of professional athletes

[0292] The server receives video data of professional athletes and performs preprocessing. During preprocessing, the video is divided into frames and key points and joint positions are extracted using OpenPose. Then, LSTM and CNN are used to learn the motion data of the professional athletes and train a model. The trained model is stored in a cloud database.

[0293] Filming and uploading users' sports activities

[0294] Users record their sports movements using their smartphones and upload them to the cloud. The server preprocesses the received video data and extracts features.

[0295] Behavior analysis and advice generation

[0296] The server uses the saved model to compare and analyze the user's motion data with that of professional athletes. Based on the analysis results, it generates specific improvement advice. It also generates a comparison video that overlays the current (AsIs) and ideal (ToBe) form.

[0297] Displaying results and providing advice

[0298] The smartphone app receives advice from the server and displays the comparison video to the user. The user can improve their sports movements based on the advice, and if necessary, record the video again and reanalyze it using the same process.

[0299] Specific examples

[0300] Example 1: Improving a child's basketball shooting technique

[0301] Parents can take a photo of their child's shooting movements on their smartphone and upload it to the app.

[0302] The app compares the movements to those of professional basketball players and advises, "Bend your knees a little."

[0303] A comparison video between AsIs and ToBe is also generated, allowing parents and children to check and practice together.

[0304] Prompt Sentence Examples

[0305] Simply record a professional basketball player's shooting technique and upload it. Then, record your child's shooting technique with your smartphone and upload it to the app. We'll provide you with relevant advice and comparison videos.

[0306] The system enables effective and cost-effective sports instruction in brick-and-mortar locations, promoting improved sports form.

[0307] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0308] Step 1:

[0309] Receiving and preprocessing motion data from professional athletes

[0310] The server receives video data of professional athletes. The received video data is divided into frames and key points and joint positions are extracted using OpenPose. As a result, the input is the video data, and the output is a set of key points and joint positions.

[0311] Specific movements: Analyze the movements of athletes frame by frame, and extract key points and joint positions for each element of the movement.

[0312] Step 2:

[0313] Learning from behavioral data and training models

[0314] The server uses the extracted keypoints and joint locations to train a machine learning model that combines LSTM and CNN. The trained model is stored in a cloud database. The input is the keypoints and joint locations obtained in step 1, and the output is the trained machine learning model.

[0315] Specific operation: Time series data is learned using LSTM, and spatial information is analyzed using CNN.

[0316] Step 3:

[0317] Filming of users' sports movements and uploading of video data

[0318] Users record their sports movements with their smartphones and upload the video data to a cloud server. The input is the video data recorded by the user, and the output is the movement data stored on the server.

[0319] Specific operation: Video is taken using the smartphone's camera function, and the video data is sent to the server via the application.

[0320] Step 4:

[0321] Video data preprocessing and feature extraction

[0322] The server preprocesses the received video data, splits it into frames, and extracts features. The input is the uploaded video data, and the output is the extracted keypoints and joint positions.

[0323] Specific operation: Divide the video into frames and extract key points and joint positions using OpenPose.

[0324] Step 5:

[0325] Comparative analysis of operational data and generation of advice

[0326] The server uses the stored model to compare and analyze the user's motion data with that of professional athletes. Specific advice is generated based on the analysis results. The input is the data after feature extraction, and the output is the analysis results and generated advice.

[0327] Specific behavior: Using machine learning models, behavior data is analyzed and specific improvement advice is generated.

[0328] Step 6:

[0329] Comparison video generation

[0330] The server generates a comparison video that overlays the current (AsIs) and ideal (ToBe) form. The input is the analyzed feature data, and the output is a comparison video of the AsIs and ToBe form.

[0331] Specific actions: Using a video editing tool, the two action data are overlaid into one video.

[0332] Step 7:

[0333] Providing advice and comparison videos

[0334] The terminal displays the advice and comparison video received from the server to the user. The input is the output data from the server, and the output is the advice and comparison video that the user can view.

[0335] Specific behavior: Provides an interface through the application that displays advice and videos.

[0336] Step 8:

[0337] Re-filming and re-analyzing the movement

[0338] The user improves their sports movements based on the advice from the server, then re-films and uploads the improved movements. The server then performs a re-analysis using the same process. The input is the improved video data, and the output is new analysis results and advice.

[0339] Specific actions: Record the video again, upload it, and provide new analysis and advice.

[0340] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0341] The following describes in detail an embodiment of the present invention. This system uses AI to provide parents with specific advice and feedback that takes into account the user's emotions, in order to help their children improve their sports skills.

[0342] Overall system overview

[0343] This system studies the motion data of professional athletes, compares it with filmed footage of children's sports movements, and provides specific advice based on the results. It also has the ability to recognize the user's emotions and adjust the advice content according to their emotional state. This allows parents, even those with no experience in sports, to provide effective instruction that takes emotions into consideration.

[0344] Program processing flow

[0345] 1. Learning the movements of professional athletes

[0346] server

[0347] 1. Receive and preprocess motion data from professional athletes. Divide the received video data into frames and extract features such as key points and joint positions.

[0348] 2. Use machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes and train the model.

[0349] 3. Save the trained model in a database.

[0350] Terminal

[0351] 1. Provides an interface for uploading operational data to a server.

[0352] 2. Display a notification to the user that the learning is complete.

[0353] User

[0354] 1. Record the motion data of a professional athlete and upload it to a server via the device.

[0355] 2. Check your learning progress and completion notifications.

[0356] 2. Filming and reading children's sports movements

[0357] Terminal

[0358] 1. Provide a camera interface for capturing children's sports activities.

[0359] 2. Display the option to upload the captured video data to the server.

[0360] User

[0361] 1. Take photos of your child's sports movements with your smartphone.

[0362] 2. Upload the captured video data to the server.

[0363] 3. Behavior analysis and advice generation phase

[0364] server

[0365] 1. Preprocess the received video data of the child, divide it into frames, and extract features.

[0366] 2. Use the saved model to compare the motion data of a child with that of a professional athlete.

[0367] 3. Based on the analysis results, specific advice (e.g., "Raise your right elbow 3 cm higher") is generated.

[0368] 4. Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[0369] 5. Send advice and comparison videos to your device.

[0370] 4. Emotion Recognition and Advice Adjustment Phase

[0371] server

[0372] 1. Using an emotion engine, input data from the device (e.g., voice, facial expressions) is analyzed to recognize the user's emotional state.

[0373] 2. Tailor the advice based on the perceived emotional state, for example adding an encouraging message if the user is feeling frustrated.

[0374] Terminal

[0375] 1. The advice received from the server, comparison videos, and emotion-sensitive feedback are displayed to the user.

[0376] 2. Provide an interface for playing videos that overlay AsIs and ToBe.

[0377] User

[0378] 1. Check the advice displayed on the device, comparison videos, and emotionally sensitive feedback.

[0379] 2. Work with your child to improve their sports movements using the advice provided.

[0380] 3. If necessary, re-photograph the improved behavior and re-analyze it using the same process.

[0381] Example: Improving children's baseball pitching technique

[0382] Initial setup and learning phase

[0383] Users can record the pitching motion of a top baseball pitcher and upload it to the server via their device. The server then preprocesses the received video data and learns the motion data of the professional pitcher.

[0384] Filming the pitching motion

[0385] The user continuously records the child's pitching motion with a smartphone and uploads the video data to a server.

[0386] Behavior analysis and advice generation

[0387] The server analyzes the child's pitching video and compares it with the motion data of a professional pitcher. It generates specific improvements (e.g., "Raise your right elbow 3 cm higher") and creates a video comparing AsIs and ToBe.

[0388] Emotion Recognition and Advice Adjustment

[0389] The server uses an emotion engine to analyze the user's emotional state and, if the user is feeling frustrated, adds an encouraging message such as, "Great progress! Now take a breather and try again."

[0390] Displaying and Implementing Results

[0391] The user checks the advice and comparison video displayed on the device, as well as the emotional feedback, and practices together with the child. After the improvement, the child's movements are filmed again and uploaded to the server for reanalysis.

[0392] This system allows parents with no prior experience in sports to provide effective and emotionally sensitive advice to help their children improve in sports.

[0393] The processing flow will be explained below.

[0394] Program processing flow

[0395] 1. Learning the movements of professional athletes

[0396] Step 1: Prepare the operation data

[0397] User: Uses a camera to record the movements of a professional athlete and saves the video file on the device.

[0398] Step 2: Uploading the behavior data

[0399] Terminal: Provides an interface for uploading operational data to the server.

[0400] User: Select the captured motion data and upload it to the server.

[0401] Device: Sends motion data to the server.

[0402] Step 3: Preprocessing the behavioral data

[0403] Server: Divides the received video file into frames and extracts key points and joint positions.

[0404] Server: Stores the preprocessed data in a database.

[0405] Step 4: Training the learning model

[0406] Server: Uses machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes.

[0407] Server: Saves the trained model in a database.

[0408] Step 5: Notification of completion of study

[0409] Server: Notifies the device that learning is complete.

[0410] Terminal: Display a message to the user that the training is complete.

[0411] 2. Filming and reading children's sports movements

[0412] Step 1: Capture the action

[0413] User: Takes photos of children's sports activities with a smartphone.

[0414] Terminal: Provides a camera interface for capturing action.

[0415] Step 2: Uploading the behavior data

[0416] On your device: Displays the option to upload the captured video file to the server.

[0417] User: Select a video file and upload it to the server.

[0418] Device: Sends the video file to the server.

[0419] 3. Behavior analysis and advice generation phase

[0420] Step 1: Preprocessing the video data

[0421] Server: Divides the received video file of the child into frames and extracts features.

[0422] Step 2: Comparative analysis of behavior

[0423] Server: Uses the trained model to compare children's motion data with that of professional athletes.

[0424] Server: Quantify the difference in each frame and identify specific areas for improvement.

[0425] Step 3: Generate Advice

[0426] Server: Generates specific advice based on the results of the comparative analysis.

[0427] Server: Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[0428] Step 4: Submit the results

[0429] Server: Sends the generated advice and comparison videos to the device.

[0430] 4. Emotion recognition and feedback regulation phase

[0431] Step 1: Enter user emotion data

[0432] User: Uses the device's camera and microphone to provide facial expressions and audio during the training session.

[0433] Device: Sends emotion data to the server.

[0434] Step 2: Sentiment Analysis

[0435] Server: Analyzes the received emotion data using the emotion engine and recognizes the user's emotional state (e.g., joy, frustration).

[0436] Server: Stores the recognized emotional state in a database.

[0437] Step 3: Adjust your advice

[0438] Server: Adjust the generated advice based on the user's emotional state. For example, if the user is feeling frustrated, add an encouraging message.

[0439] Server: Sends advice content including emotion-sensitive feedback to the device.

[0440] Step 4: View the results

[0441] Device: The advice received from the server, comparison videos, and emotional feedback are displayed to the user.

[0442] Device: Provides an interface for playing videos that overlay AsIs and ToBe.

[0443] Step 5: Check operation and make improvements

[0444] User: View advice and comparison videos displayed on the device, as well as emotionally sensitive feedback.

[0445] User: Work with their child to improve their sports movements using the advice provided.

[0446] Step 6: Feedback and Retraining

[0447] User: Re-photograph the improved behavior and re-analyze it using the same process.

[0448] Device: Sends new video to the server and requests reanalysis.

[0449] As described above, a system is constructed in which the server, terminal, and user work together to support children in improving their sports movements and provide feedback that takes emotions into consideration.

[0450] Example 2

[0451] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0452] Conventional sports coaching systems have the problem that parents lack specialized knowledge of sports, making it difficult to provide effective instruction. Furthermore, feedback is often provided that ignores the child's emotional state, resulting in a decrease in the effectiveness of instruction. To solve these problems, a system is needed that can provide effective advice while taking into consideration the parent's emotions by comparing and analyzing the child's movements based on a model trained by a professional.

[0453] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0454] In this invention, the server includes means for learning the motion data of professional athletes, means for receiving video of filmed sports movements, means for preprocessing the received video data and extracting features, means for comparative analysis using the extracted features with the motion data of the professional athletes, means for generating specific advice based on the analysis results, means for generating a comparison video that overlays the current form with an ideal form, means for outputting the generated advice and the comparison video, and means for recognizing the emotional state of the user using an emotion engine and adjusting the content of the advice based on the recognized emotional state. This enables parents to provide effective instruction based on the movements of professionals and feedback that takes into account their child's emotions, even if they do not have specialized knowledge of sports.

[0455] "Professional athlete exercise data" refers to recordings of exercises performed by skilled or expert athletes, typically stored in video format, that are used as training data for machine learning.

[0456] "Means for learning athletic data" refers to the process and technology for analyzing received athletic data of professional athletes, incorporating the features into a model, and learning the data.

[0457] "Means for receiving video" refers to an interface and protocol that allows the system to obtain video of sports exercises taken by users.

[0458] "Preprocessing" refers to the process of dividing the received video data into frames and extracting features such as key points and joint positions from each frame.

[0459] "Features" are important information extracted from movement data, and refer to data such as points and joint positions necessary for analyzing sports movements.

[0460] "Means for comparative analysis" refers to a technique that uses extracted features to compare children's movements with those of professional athletes and analyze differences and areas for improvement.

[0461] "Specific advice" refers to suggestions and instructions for improving exercise that are generated based on the analysis results and are provided to the user in an easy-to-understand format.

[0462] A "comparison video" refers to a video that displays the current (AsIs) and ideal (ToBe) exercise form superimposed on each other, allowing users to visually understand areas for improvement.

[0463] An "emotion engine" refers to technology and systems that analyze input data such as a user's voice and facial expressions to recognize their emotional state.

[0464] "Emotional state" refers to the user's current mental and emotional state and is a factor taken into account when adjusting feedback.

[0465] The present invention relates to a system that enables parents to support their children's sports improvement effectively and in a way that takes their emotions into consideration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the preferred embodiments of the present invention.

[0466] Hardware and software used

[0467] server

[0468] The server is the main component for learning motion data, data analysis, and emotion recognition. It uses the following software and frameworks:

[0469] Behavior analysis: TensorFlow, PyTorch

[0470] Emotion Recognition: Emotion analysis engines (e.g., IBM Watson or Microsoft Azure Emotion API)

[0471] Video processing: OpenPose, FFmpeg

[0472] Terminal

[0473] The terminal is the device through which the user captures video and communicates with the server. It uses the following technologies:

[0474] Video recording and uploading: Smartphone camera API (e.g., Android Camera2 API, iOS AVFoundation)

[0475] User Interface: React Native, Flutter

[0476] User

[0477] Users are primarily parents and children and perform the following actions:

[0478] Video recording and uploading

[0479] Check the advice

[0480] Providing emotional feedback

[0481] Specific processing of the system

[0482] 1. Learning from motion data of professional athletes

[0483] server

[0484] The server receives video data of professional athletes and splits it into frames. Next, OpenPose is used to extract features such as keypoints and joint positions from each frame. These features are then used to train machine learning models such as LSTM and CNN using TensorFlow or PyTorch. The trained models are stored in a database for later analysis.

[0485] Terminal

[0486] The device provides an interface for the user to upload the motion data to the server. Once the video upload is complete, the device notifies the user that the learning is complete.

[0487] User

[0488] Users can record the movements of professional athletes using their smartphones, upload the videos to the server via their devices, and then check their learning progress and receive notifications of completion.

[0489] 2. Shooting and importing children's sports movements

[0490] Terminal

[0491] The device provides a camera interface, allowing users to easily record their children's sports activities, and the captured video data is sent to the server via an upload option.

[0492] User

[0493] Users use their smartphones to record their children's sports activities and then upload the videos to a server.

[0494] 3. Behavior analysis and advice generation

[0495] server

[0496] The server divides the child's video data into frames and performs preprocessing. Next, it extracts features and compares them with the motion data of professional athletes. Based on this comparative analysis, specific advice is generated. Furthermore, a comparison video is generated that overlays the child's current and ideal form, and this is sent to the device.

[0497] 4. Emotion recognition and advice adjustment

[0498] server

[0499] The server uses an emotion engine to analyze voice and facial expression data from the device to recognize the user's emotional state. Based on the recognized emotional state, the server adjusts the content of the advice, for example, adding an encouraging message.

[0500] Terminal

[0501] The device displays the received advice, comparison videos, and emotion-sensitive feedback to the user. It also provides a video playback interface, allowing users to compare their current and ideal form.

[0502] User

[0503] The user can check the advice and feedback displayed on the device and use it to work with their child to improve their sports movements. If necessary, they can record the movements again and upload them to the server for reanalysis.

[0504] Specific examples

[0505] Improving children's baseball pitching movements

[0506] Initial setup and learning phase

[0507] Users film the pitching movements of famous baseball pitchers and upload them to the server via their devices. The server then uses this video data to train the model.

[0508] Filming the pitching motion

[0509] Users can record their child's pitching motion with their smartphone and upload the video to the server.

[0510] Behavior analysis and advice generation

[0511] The server analyzes the child's pitching video, generates specific improvements (e.g., "Raise your right elbow 3cm higher"), and creates a comparison video.

[0512] Emotion Recognition and Advice Adjustment

[0513] The server uses an emotion engine to analyze the user's emotions and add appropriate feedback (e.g., "Great progress! Let's take a breather and try again next time").

[0514] Prompt Sentence Examples

[0515] "Please explain the system that records a child's sports movements and compares them with the movement data of professional athletes. Also, please provide a concrete example of how a user would use the system."

[0516] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0517] Step 1:

[0518] Receiving and preprocessing motion data from professional athletes

[0519] server

[0520] Input: Video data of professional athletes

[0521] How it works: The server receives video data of a professional athlete, splits it into frames, and then uses OpenPose to extract features such as keypoints and joint positions from each frame.

[0522] Output: Extracted feature data

[0523] Step 2:

[0524] Model learning for motion data

[0525] server

[0526] Input: Extracted feature data

[0527] How it works: Use TensorFlow or PyTorch to train machine learning models such as LSTMs and CNNs. The training process involves providing feature data to the model and iteratively learning it.

[0528] Output: A trained machine learning model

[0529] Step 3:

[0530] Saving the trained model

[0531] server

[0532] Input: A trained machine learning model

[0533] Behavior: Once the training is complete, the model is saved in a database and used for subsequent behavior analysis.

[0534] Output: A saved machine learning model

[0535] Step 4:

[0536] Uploading model operation data

[0537] Terminal

[0538] Input: Professional athlete's athletic data

[0539] Operation: The device provides an interface for the user to upload operation data to the server. The user takes a video using the smartphone's camera API and then selects the upload option.

[0540] Output: Video data uploaded to the server

[0541] Step 5:

[0542] Learning completion notification

[0543] Terminal

[0544] Input: Learning completion notification sent from the server

[0545] Behavior: Notify the user that their learning is complete via push notification or in-app message.

[0546] Output: User notification

[0547] Step 6:

[0548] Filming and uploading children's sports activities

[0549] Terminal

[0550] Input: Video of a child's sports activity

[0551] Actions: The device records the child's sports movements through the camera interface. It displays an option to upload the captured video data to the server. The user uses the option to upload the video to the server.

[0552] Output: Child's video data uploaded to the server

[0553] Step 7:

[0554] Preprocessing and feature extraction of children's video data

[0555] server

[0556] Input: Children's video data

[0557] How it works: The server splits the received video data of the child into frames and extracts features from each frame. As a preprocessing step, it uses OpenPose to identify key points and joint positions.

[0558] Output: Extracted feature data

[0559] Step 8:

[0560] Behavior analysis and advice generation

[0561] server

[0562] Input: Extracted child feature data, saved machine learning model

[0563] How it works: The server uses the saved model to compare the child's feature data with that of a professional athlete. Based on the analysis results, specific advice is generated. In addition, a comparison video is created that overlays the child's current and ideal form.

[0564] Output: Specific advice, comparison video

[0565] Step 9:

[0566] Submitting advice and comparison videos

[0567] server

[0568] Input: Specific advice, comparison videos

[0569] Operation: The server sends the generated advice and comparison video to the device, where the data is displayed and the user can review it.

[0570] Output: Advice and comparison video sent to your device

[0571] Step 10:

[0572] Emotion Recognition and Advice Adjustment

[0573] server

[0574] Input: Voice and facial expression data that indicates the user's emotional state

[0575] How it works: The server uses the emotion engine to analyze voice and facial expression data from the device to recognize the user's emotional state. Based on the recognized emotional state, it adjusts the advice content, for example, adding an encouraging message.

[0576] Output: Tailored advice based on perceived emotional state

[0577] Step 11:

[0578] View advice and feedback

[0579] Terminal

[0580] Input: Adjusted advice received from the server, comparison video

[0581] How it works: The device displays the received advice, comparison videos, and emotion-sensitive feedback to the user. The device also provides an interface for playing videos that overlay AsIs and ToBe.

[0582] Output: Advice and feedback displayed to the user

[0583] Step 12:

[0584] Performance improvements and reanalysis

[0585] User

[0586] Input: Advice displayed on the device, comparison video

[0587] Movement: The user works with their child to improve their sports movements based on the advice provided. If necessary, the improved movements are filmed again and uploaded to the server for reanalysis.

[0588] Output: Improved sports motion video, reanalysis results

[0589] (Application example 2)

[0590] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0591] Modern factories lack an automatic analysis and improvement feedback system to improve the efficiency and accuracy of robotic work. They also lack the ability to provide appropriate feedback based on the emotional state of workers. This makes it difficult to simultaneously improve work efficiency and maintain workers' psychological stability.

[0592] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for learning professional motion data, a means for receiving video of filmed motions, and a means for preprocessing the received video data and extracting features. This makes it possible to analyze the robot's motions and provide specific improvement advice. The server also includes a means for recognizing emotions and adjusting feedback accordingly, and a means for filming and analyzing the worker's motions. This makes it possible to provide appropriate feedback according to the worker's psychological state.

[0593] "Professional motion data" is data that records the motions of skilled people or robots in a specific task or operation.

[0594] "Means for receiving video of captured action" refers to an interface that imports video of action captured by a camera or other capturing device into the system.

[0595] "Means for preprocessing received video data and extracting features" refers to the process of analyzing video data and extracting important points of movement, joint positions, etc.

[0596] "Means for comparative analysis with professional motion data using extracted features" refers to the process of finding and evaluating differences and similarities with professional motion data based on the analyzed features.

[0597] "Means for generating specific advice based on the analysis results" refers to a process for generating specific instructions for improving behavior based on the results of behavior comparison.

[0598] "Means for generating a comparison video that overlays current and ideal movements" refers to the process of creating a video that overlays current and ideal movements to visually show the difference.

[0599] "Means for outputting generated advice and comparison videos" refers to an interface that provides users with advice and comparison videos as the analysis results.

[0600] "Means for recognizing emotions and adjusting feedback accordingly" refers to the process of analyzing the worker's emotional state and appropriately changing the advice or feedback content based on the results.

[0601] "Means for filming and analyzing the worker's movements" refers to the process of filming the worker's movements using a camera or other device, and analyzing and evaluating those movements.

[0602] The following describes in detail the mode for carrying out this invention. This system analyzes the operation of a factory robot and provides feedback for improvement. It also has the ability to recognize the user's emotions and adjust the feedback according to their emotional state. This allows for both psychological stability for the worker and improved production efficiency.

[0603] Overall system overview

[0604] This system learns professional motion data, compares it with filmed robot motion data, and provides specific advice based on the results. It also has the ability to recognize the user's emotions and adjust the advice content according to their emotional state.

[0605] Hardware and software used

[0606] Hardware: smartphone, head-mounted display, camera

[0607] Software: Python, open source libraries (OpenCV, Keras)

[0608] Program processing flow

[0609] 1. Learning the movements of professionals

[0610] The server receives professional motion data and performs preprocessing. It divides the received video data into frames and extracts features such as key points and joint positions. It then uses a machine learning algorithm (e.g., LSTM or CNN) to learn from the professional motion data and train a model. The trained model is then stored in a database.

[0611] 2. Recording and reading the robot's movements

[0612] Users use smartphones or head-mounted displays to record robot movements in the factory. The captured data is uploaded to a server, which then preprocesses the received robot video data, dividing it into frames and extracting features.

[0613] 3. Behavior analysis and advice generation phase

[0614] The server uses the saved model to compare the robot's motion data with professional motion data. Based on the results of the comparative analysis, it generates specific advice (e.g., "Delay your right elbow by two seconds"). It also generates a comparison video that overlays the AsIs (current) and ToBe (ideal) forms. The advice and comparison video are sent to the device.

[0615] 4. Emotion Recognition and Advice Adjustment Phase

[0616] The server uses an emotion engine to analyze input data (e.g., voice, facial expressions) from the device and recognize the user's emotional state. Based on the recognized emotional state, the server adjusts the advice content. For example, if the user is feeling frustrated, it adds an encouraging message. It provides feedback such as, "Great progress. Let's take a breather and try again next time."

[0617] Example: Improving the operation of a robot arm

[0618] Initial setup and learning phase

[0619] Users can record professional robot arm movements and upload them to the server via their devices. The server then preprocesses the received video data and learns the professional movement data.

[0620] Filming the robot arm in action

[0621] The user continuously records the robot arm's movements with their smartphone and uploads the video data to a server.

[0622] Behavior analysis and advice generation

[0623] The server analyzes the video of the robot arm and compares it with professional motion data, generates specific improvements (e.g., "Delay the right elbow by 2 seconds"), and creates a comparison video of the AsIs and ToBe.

[0624] Emotion Recognition and Advice Adjustment

[0625] The server uses an emotion engine to analyze the user's emotional state and, if the user is feeling frustrated, adds an encouraging message such as, "Great progress! Now take a breather and try again."

[0626] Displaying and Implementing Results

[0627] The user can then improve the robot's behavior by checking the advice, comparison videos, and emotional feedback displayed on the device. After the improvements are made, the robot's behavior is filmed again and uploaded to the server for reanalysis.

[0628] This system allows workers to efficiently improve the robot's operation, thereby increasing work quality and production efficiency.

[0629] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0630] Step 1: Learning from professional motion data

[0631] In step 1, the server receives the professional's motion data and performs preprocessing. The input is the professional's motion video data, which is split into frames and features such as key points and joint positions are extracted. A machine learning algorithm (e.g., LSTM, CNN) is used to learn the features and train a model. The trained model is stored in a database. The output of this step is the features and the trained model.

[0632] Step 2: Film the robot's movements and upload the data

[0633] In step 2, the user uses a smartphone or head-mounted display to film the robot's operation in the factory. The input is video data containing the actual robot operation. The user uploads the filmed video to the server via their device. The input video data is transferred to the server. The output of this step is video data of the robot's operation stored on the server.

[0634] Step 3: Preprocessing of motion data and feature extraction

[0635] In step 3, the server preprocesses the input robot video data and splits it into frames. This allows for detailed analysis of each stage of the movement. Next, features such as key points and joint positions are extracted. The input is the robot's movement video data, and the output is the extracted features. Specific operations include splitting and analyzing the video frames using OpenCV.

[0636] Step 4: Behavior analysis and comparison

[0637] In step 4, the server uses the stored professional model to perform comparative analysis of the robot's motion data. The inputs are the extracted features and the professional model, and by comparing and analyzing these, differences in motion and areas for improvement are identified. Specifically, a machine learning model is used to calculate the motion match rate and descriptor. The output is the analysis results and areas for improvement.

[0638] Step 5: Generate advice and comparison videos

[0639] In step 5, the server generates specific advice based on the analysis results. For example, this may include instructions such as "Delay your right elbow for another two seconds." It also generates a comparison video that overlays the current and ideal movements. The input is the analysis results, and the output is specific advice and a comparison video. In this step, the two videos are overlaid using video editing software.

[0640] Step 6: View the results

[0641] In step 6, the server sends the generated advice and comparison video to the user's device. The input is the generated advice and comparison video, and the user receives this information through their device. Specifically, feedback is provided to the user via a notification function or interface. The output is advice and comparison video that the user can visually confirm.

[0642] Step 7: Emotion recognition and feedback regulation

[0643] In step 7, the server uses an emotion engine to analyze the user's emotional state. For example, it analyzes the user's voice and facial expression data to recognize emotions. The input is emotional data from the user, and the feedback content is adjusted based on the analysis results. For example, if the user is feeling frustrated, an encouraging message such as "Great progress! Let's take a breather and try again next time" is added. The output is the adjusted feedback.

[0644] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0645] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0646] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0647] [Second embodiment]

[0648] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0649] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0650] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0651] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0652] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0653] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0654] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0655] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0656] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0657] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0658] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0659] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0660] The following describes in detail an embodiment of the present invention. This system uses AI to provide parents with specific advice to help their children improve their sports skills.

[0661] Overall system overview

[0662] This system learns the motion data of professional athletes, compares it with filmed footage of children's sports movements, and provides specific advice based on the results, enabling parents to provide effective instruction even if they have no experience in sports.

[0663] Program processing flow

[0664] 1. Learning the movements of professional athletes

[0665] server

[0666] 1. Receive and preprocess motion data from professional athletes. Divide the received video data into frames and extract features such as key points and joint positions.

[0667] 2. Use machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes and train the model.

[0668] 3. Save the trained model in a database.

[0669] Terminal

[0670] 1. Provides an interface for uploading operational data to a server.

[0671] 2. Display a notification to the user that the learning is complete.

[0672] User

[0673] 1. Record the motion data of a professional athlete and upload it to a server via the device.

[0674] 2. Check your learning progress and completion notifications.

[0675] 2. Filming and reading children's sports movements

[0676] Terminal

[0677] 1. Provide a camera interface for capturing children's sports activities.

[0678] 2. Display the option to upload the captured video data to the server.

[0679] User

[0680] 1. Take photos of your child's sports movements with your smartphone.

[0681] 2. Upload the captured video data to the server.

[0682] 3. Behavior analysis and advice generation phase

[0683] server

[0684] 1. Preprocess the received video data of the child, divide it into frames, and extract features.

[0685] 2. Use the saved model to compare and analyze the motion data of children and professional athletes.

[0686] 3. Based on the analysis results, specific advice (e.g., "Raise your right elbow 3 cm higher") is generated.

[0687] 4. Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[0688] 5. Send advice and comparison videos to your device.

[0689] 4. Results display and advice phase

[0690] Terminal

[0691] 1. The advice received from the server and comparison video are displayed to the user.

[0692] 2. Provide an interface for playing videos that overlay AsIs and ToBe.

[0693] User

[0694] 1. Check the advice displayed on your device and the comparison video.

[0695] 2. Work with your child to improve their sports movements using the advice provided.

[0696] 3. If necessary, re-photograph the improved behavior and re-analyze it using the same process.

[0697] Example: Improving children's baseball pitching technique

[0698] Initial setup and learning phase

[0699] Users can record the pitching motion of a top baseball pitcher and upload it to the server via their device. The server then preprocesses the received video data and learns the motion data of the professional pitcher.

[0700] Filming the pitching motion

[0701] The user continuously records the child's pitching motion with a smartphone and uploads the video data to a server.

[0702] Behavior analysis and advice generation

[0703] The server analyzes the child's pitching video and compares it with the motion data of a professional pitcher. It generates specific improvements (e.g., "Raise your right elbow 3 cm higher") and creates a video comparing AsIs and ToBe.

[0704] Displaying and Implementing Results

[0705] The user checks the advice and comparison video displayed on the device and practices together with the child. After the improvement, the movement is filmed again and uploaded to the server for reanalysis.

[0706] This system allows parents with no sports experience to provide effective advice and support their children's sports improvement.

[0707] The processing flow will be explained below.

[0708] Program processing flow

[0709] 1. Learning the movements of professional athletes

[0710] Step 1: Prepare the operation data

[0711] User: Uses a camera to record the movements of a professional athlete and saves the video file on the device.

[0712] Step 2: Uploading the behavior data

[0713] Terminal: Provides an interface for uploading operational data to the server.

[0714] User: Select the captured motion data and upload it to the server.

[0715] Device: Sends motion data to the server.

[0716] Step 3: Preprocessing the behavioral data

[0717] Server: Divides the received video file into frames and extracts key points and joint positions.

[0718] Server: Stores the preprocessed data in a database.

[0719] Step 4: Training the learning model

[0720] Server: Uses machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes.

[0721] Server: Saves the trained model in a database.

[0722] Step 5: Notification of completion of study

[0723] Server: Notifies the device that learning is complete.

[0724] Terminal: Display a message to the user that the training is complete.

[0725] 2. Filming and reading children's sports movements

[0726] Step 1: Capture the action

[0727] User: Takes photos of children's sports activities with a smartphone.

[0728] Terminal: Provides a camera interface for capturing action.

[0729] Step 2: Uploading the behavior data

[0730] On your device: Displays the option to upload the captured video file to the server.

[0731] User: Select a video file and upload it to the server.

[0732] Device: Sends the video file to the server.

[0733] 3. Behavior analysis and advice generation phase

[0734] Step 1: Preprocessing the video data

[0735] Server: Divides the received video file of the child into frames and extracts features.

[0736] Step 2: Comparative analysis of behavior

[0737] Server: Uses the trained model to compare children's motion data with that of professional athletes.

[0738] Server: Quantify the difference in each frame and identify specific areas for improvement.

[0739] Step 3: Generate Advice

[0740] Server: Generates specific advice based on the results of the comparative analysis.

[0741] Server: Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[0742] Step 4: Submit the results

[0743] Server: Sends the generated advice and comparison videos to the device.

[0744] 4. Results display and advice phase

[0745] Step 1: View the results

[0746] Terminal: Displays the advice received from the server and the comparison video to the user.

[0747] Device: Provides an interface for playing videos that overlay AsIs and ToBe.

[0748] Step 2: Operation check and improvement steps

[0749] User: Check the advice and comparison video displayed on the device.

[0750] User: Work with their child to improve their sports movements using the advice provided.

[0751] Step 3: Feedback and Retraining

[0752] User: Re-photograph the improved behavior and re-analyze it using the same process.

[0753] Device: Sends new video to the server and requests reanalysis.

[0754] As described above, a system is constructed in which the server, terminals, and users work together to support children in improving their sports movements.

[0755] Example 1

[0756] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0757] Conventional systems supporting children's sports improvement have the problem that it is difficult to provide effective advice to parents who lack sports experience or expertise. Furthermore, analyzing the recorded video data requires specialized expertise and is not easy to do. Furthermore, the system lacks a means to visually show specific areas for improvement and an interface for reassessment, making the improvement process difficult.

[0758] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0759] In this invention, the server includes: means for learning motion data of professional athletes; means for receiving video of filmed sports movements; means for preprocessing the received video data and extracting features; means for comparative analysis with the motion data of the professional athletes using the extracted features; means for generating specific advice based on the analysis results; means for generating a comparison video that overlays the current form with an ideal form; means for outputting the generated advice and the comparison video; means for providing a camera interface for filming the child's sports movements; and means for inputting a prompt sentence for generating an analysis request to the AI ​​model. This enables parents, even without experience or knowledge of sports, to effectively analyze their children's sports movements, provide specific and visual advice, and reassess their children's sports movements.

[0760] "Professional athlete motion data" refers to data that records the motions of athletes with high levels of skill.

[0761] "Means for receiving" refers to technical means for taking in data from the outside, such as a server or storage device.

[0762] "Preprocessing" refers to the process of converting received data into a format that is easy to analyze.

[0763] A "feature" is a data point or value that contains important information in data analysis.

[0764] A "comparative analysis tool" is a tool that compares two or more data sets to analyze commonalities and differences.

[0765] "Specific advice" refers to specific, practical instructions or suggestions.

[0766] A "comparison video" is a video that displays your current form and ideal form at the same time for comparison.

[0767] "Means of output" refers to the technical means for providing data or information to the user.

[0768] A "camera interface" is a user interface for operating a camera.

[0769] An "AI model" is an algorithm trained using machine learning or deep learning.

[0770] A "prompt sentence" is an input sentence for a generative AI model, and is text that specifies an analysis request.

[0771] The following describes in detail the embodiments of the present invention. This system uses AI to provide specific advice to parents to help their children improve their sports skills. Even if the parents have no experience in sports, the system can effectively analyze their children's movements, provide specific and visual advice, and perform reassessments.

[0772] Overall system overview

[0773] This system learns the motion data of professional athletes, compares it with filmed footage of children's sports movements, and provides specific advice based on the results, allowing parents to provide effective instruction even if they have no experience in sports.

[0774] Program processing flow

[0775] 1. Learning the movements of professional athletes

[0776] server

[0777] 1. The server receives the motion data of professional athletes and performs preprocessing. This preprocessing involves dividing the video data into frames and extracting features such as key points and joint positions from each frame using open-source pose estimation technology such as OpenPose.

[0778] 2. The server uses the preprocessed feature data to train a model using machine learning algorithms such as LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network).

[0779] 3. The trained model is stored in a database by the server.

[0780] Terminal

[0781] 1. The device provides an interface for uploading professional athletes' motion data to the server, allowing users to easily select and upload videos.

[0782] 2. The device displays a learning completion notification to the user.

[0783] User

[0784] 1. The user uses their smartphone to record the movements of a professional athlete as an example, and then uploads the video data to the server via the device.

[0785] 2. The user checks the learning progress and completion notification displayed on the device.

[0786] 2. Filming and reading children's sports movements

[0787] Terminal

[0788] 1. The device provides a camera interface that allows the user to record their child's sports activities. For example, the app can activate the camera function and display a recording button to start recording.

[0789] 2. The device will display an option to upload the captured video data to the server.

[0790] User

[0791] 1. The user uses the smartphone camera to capture a video of their child's sports activity, for example, recording a video of their child pitching a baseball.

[0792] 2. The user uploads the captured video data to the server. By simply selecting the video and pressing the upload button according to the device interface, the data is sent to the server.

[0793] 3. Behavior analysis and advice generation phase

[0794] server

[0795] 1. The server preprocesses the received video data of the child, divides the video into frames, and extracts features. Specifically, it calculates the joint positions for each frame using OpenPose or similar.

[0796] 2. The server uses the stored trained model to compare and analyze the child's motion data with that of a professional athlete. This comparison evaluates the similarities and differences between the two motions based on the extracted features.

[0797] 3. The server generates specific advice based on the analysis results. For example, it may suggest to a child that they should "raise their right elbow 3cm higher" when pitching.

[0798] 4. The server generates a comparison video that overlays the AsIs (current) and ToBe (ideal) forms, making it visually easier to see which areas need improvement.

[0799] 5. The server sends the generated advice and comparison video to the device.

[0800] 4. Results display and advice phase

[0801] Terminal

[0802] 1. The device displays the advice and comparison video received from the server to the user. Specifically, the application provides a playback function for the advice text and comparison video.

[0803] 2. The device provides an interface for playing videos that overlay AsIs and ToBe, allowing users to easily play the videos and view them as many times as they like.

[0804] User

[0805] 1. The user checks the advice and comparison video displayed on the device. For example, they play the video to visually understand the suggested improvements.

[0806] 2. The user works with their child to improve their sports movements based on the provided advice. Specifically, for example, the user practices to correct the child's pitching form.

[0807] 3. If necessary, film the motion after the improvement and re-analyze it using the same process. Record the corrected pitching motion on video again, upload it to the server, and receive further advice.

[0808] Examples of prompt statements

[0809] Here are some examples of prompts that users can enter into a generative AI model:

[0810] "Please analyze my child's baseball pitching motion and let me know what needs to be improved. I have uploaded a video from the link below. I have also added a video of a professional pitcher, so you can compare it."

[0811] This allows the AI ​​model to provide appropriate analysis and advice based on the video data provided.

[0812] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0813] Step 1:

[0814] Receiving and preprocessing motion data from professional athletes

[0815] The server receives the motion data of the professional athlete, which includes video files uploaded by the user through an interface provided by the device.

[0816] Input: Video data of professional athletes' movements

[0817] The server preprocesses the received video data. This preprocessing involves dividing the video into frames and extracting key points and joint positions from each frame. Specifically, it uses open-source pose estimation technology such as OpenPose to detect the main joint points of the human body for each frame.

[0818] Data processing: Split the video into frames and extract joint points

[0819] Output: Features (keypoints and joint positions)

[0820] Step 2:

[0821] Training a machine learning model

[0822] The server uses the preprocessed feature data to train machine learning models, such as LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network). Training is necessary to capture the movement patterns of professional athletes.

[0823] Input: Preprocessed feature data

[0824] The server performs training using the given feature data and target data. During the training process, iterative parameter updates are performed using the backpropagation algorithm.

[0825] Data Computing: Model Training with Machine Learning Algorithms

[0826] Output: A trained model

[0827] Step 3:

[0828] Saving a trained model

[0829] The server stores the trained model in a database, allowing it to be quickly used in future analyses.

[0830] Input: A trained model

[0831] Data processing: Converting the model into a storage format

[0832] Output: Model saved in database

[0833] Step 4:

[0834] Filming children's sports

[0835] The user takes pictures of their child's sports activities using the smartphone camera and uses the camera interface provided by the device.

[0836] Input: Children's sports activities

[0837] The user activates the device's camera function and presses the record button to record the action. Once the recording is complete, it is saved as video data on the device.

[0838] Data generation: video data of sports movements

[0839] Output: Video data of children's sports movements

[0840] Step 5:

[0841] Uploading video data

[0842] The user uploads the captured video data to the server through the device, which displays an upload option.

[0843] Input: Video data of children's sports movements

[0844] The user selects a video from the terminal interface and presses the "upload" button to send the data to the server.

[0845] Data transfer: Uploading video data

[0846] Output: Video data of children's sports movements stored on the server

[0847] Step 6:

[0848] Preprocessing children's video data

[0849] The server preprocesses the received video data of the child by dividing the video into frames and extracting features (keypoints and joint positions) from each frame.

[0850] Input: Video data of children's sports movements stored on a server

[0851] Preprocessing includes extracting joint points using OpenPose, etc. The result of preprocessing is feature data for each frame.

[0852] Data processing: Split the video into frames and extract joint points

[0853] Output: Feature data of children's sports movements

[0854] Step 7:

[0855] Comparative analysis of motion data

[0856] The server uses the stored trained model to compare and analyze the motion data of children and that of professional athletes.

[0857] Input: feature data of children's sports movements, pre-trained model

[0858] The server evaluates the similarities and differences based on the extracted feature data. Specifically, it inputs the feature values ​​of each frame into a trained model and calculates the differences from the standard movements of professional athletes.

[0859] Data Computation: Comparative Analysis of Features

[0860] Output: Analysis results

[0861] Step 8:

[0862] Generating Specific Advice

[0863] The server generates specific advice based on the analysis results, such as "Raise your right elbow another 3 cm."

[0864] Input: Analysis results

[0865] Based on the analysis results, the server determines which parts should be improved and how, and generates advice in text format.

[0866] Data Generation: Specific Advice Text

[0867] Output: Specific advice

[0868] Step 9:

[0869] Comparison video generation

[0870] The server generates a comparison video that overlays the AsIs (current) and ToBe (ideal) forms, making it easier to visually identify areas for improvement.

[0871] Input: Video data of children's sports movements, pre-trained model

[0872] The server displays the current form and the ideal form side by side and combines the two videos to generate a single comparison video.

[0873] Data calculation: Video frame comparison and integration

[0874] Output: Comparison video

[0875] Step 10:

[0876] Advice and comparison video output

[0877] The server sends the generated advice and comparison video to the terminal, where the user can review it.

[0878] Input: Specific advice, comparison videos

[0879] The server transmits data to the terminal, and the terminal displays the received advice and video.

[0880] Data output: Sending advice and comparison videos

[0881] Output: Advice and comparison video displayed on device

[0882] (Application example 1)

[0883] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0884] Conventional sports instruction methods have the problem that it is difficult for parents and coaches to provide effective advice and instruction if they do not have specialized sports knowledge or skills. Furthermore, instruction services offered by sporting goods stores and fitness facilities are generally expensive, making them inaccessible to many people. Checking and improving sports form at physical stores is particularly time-consuming and requires a large burden on equipment and specialized staff. There is a need for a system that can solve these issues and enable ordinary parents and users to effectively improve their sports movements.

[0885] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0886] In this invention, the server includes: means for learning motion data of professional athletes; means for receiving video of filmed sports movements; means for preprocessing the received video data and extracting features; means for comparing and analyzing the extracted features with the motion data of the professional athletes; means for generating specific advice based on the analysis results; means for generating a comparison video that overlays the current form with an ideal form; means for outputting the generated advice and the comparison video; means for providing an interface that allows users at physical stores to film their sports movements and upload them to the cloud; and means for users to re-film and upload their improved movements and receive new advice to support improvement of their sports movements. This not only enables parents and physical store users with no sports experience to effectively improve their sports movements, but also enables efficient and cost-effective sports instruction services at physical stores.

[0887] "Professional athlete motion data" refers to data that records the motions of a professional athlete, and includes the feature quantities of those motions.

[0888] "Means for learning movement data" refers to the process of using machine learning algorithms to analyze the movement data of professional athletes and use it as a model for training.

[0889] "Means for receiving video" refers to the process by which the user sends the video data they have taken to the cloud or a server, and the server receives it.

[0890] "Preprocessing" refers to the process of dividing the received video data into frames and extracting important key points and joint positions.

[0891] "Features" are an important part in analyzing sports movements and include position information of key points and joints.

[0892] The "means for comparative analysis" refers to a process of comparing the motion data of a professional athlete with the photographed sports motion data using the extracted feature amounts.

[0893] "Means for generating advice" refers to the process of generating specific improvements and advice in text or video format based on the results of the comparative analysis.

[0894] "Method for generating a comparison video" refers to the process of generating a video that overlays the current (AsIs) and ideal (ToBe) forms to visually show the differences.

[0895] "Means for outputting a comparison video" refers to a process for providing the generated comparison video to a user.

[0896] "Brick-and-mortar users" refers to customers who visit physical stores, such as sporting goods stores or fitness facilities.

[0897] An "interface" is the part of the system that provides the screen and operating means for users to film their sports movements and upload them to the cloud.

[0898] "Cloud" refers to remote servers for storing, computing, and managing data over the Internet.

[0899] "Means for providing advice" refers to the process by which the generated advice is displayed to the user so that they can review it.

[0900] The following describes in detail the mode for carrying out the present invention. This system aims to support the improvement of sports movements in physical stores, and analyzes sports form and provides specific advice. By using this system, parents and general users can effectively improve their sports movements.

[0901] Overall system overview

[0902] The system learns from the motion data of professional athletes, compares and analyzes the sports motions of users filmed in physical stores, and generates and provides specific advice based on the results.

[0903] Hardware and software used

[0904] Hardware

[0905] Smartphone: A device that allows users to record and upload videos of their sports activities.

[0906] Cloud server: A remote server for storing data, training models, and performing analytical processing.

[0907] software

[0908] LSTM (Long Short-Term Memory): Used for time series analysis of behavioral data.

[0909] CNN (Convolutional Neural Network): Used to extract movement features.

[0910] OpenPose: Used to extract keypoints and joint positions from videos.

[0911] Flask / Django: Used for backend processing.

[0912] React Native: Used for the front end of smartphone applications.

[0913] Data processing and calculation

[0914] Learning from motion data of professional athletes

[0915] The server receives video data of professional athletes and performs preprocessing. During preprocessing, the video is divided into frames and key points and joint positions are extracted using OpenPose. Then, LSTM and CNN are used to learn the motion data of the professional athletes and train a model. The trained model is stored in a cloud database.

[0916] Filming and uploading users' sports activities

[0917] Users record their sports movements using their smartphones and upload them to the cloud. The server preprocesses the received video data and extracts features.

[0918] Behavior analysis and advice generation

[0919] The server uses the saved model to compare and analyze the user's motion data with that of professional athletes. Based on the analysis results, it generates specific improvement advice. It also generates a comparison video that overlays the current (AsIs) and ideal (ToBe) form.

[0920] Displaying results and providing advice

[0921] The smartphone app receives advice from the server and displays the comparison video to the user. The user can improve their sports movements based on the advice, and if necessary, record the video again and reanalyze it using the same process.

[0922] Specific examples

[0923] Example 1: Improving a child's basketball shooting technique

[0924] Parents can take a photo of their child's shooting movements on their smartphone and upload it to the app.

[0925] The app compares the movements to those of professional basketball players and advises, "Bend your knees a little."

[0926] A comparison video between AsIs and ToBe is also generated, allowing parents and children to check and practice together.

[0927] Prompt Sentence Examples

[0928] Simply record a professional basketball player's shooting technique and upload it. Then, record your child's shooting technique with your smartphone and upload it to the app. We'll provide you with relevant advice and comparison videos.

[0929] The system enables effective and cost-effective sports instruction in brick-and-mortar locations, promoting improved sports form.

[0930] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0931] Step 1:

[0932] Receiving and preprocessing motion data from professional athletes

[0933] The server receives video data of professional athletes. The received video data is divided into frames and key points and joint positions are extracted using OpenPose. As a result, the input is the video data, and the output is a set of key points and joint positions.

[0934] Specific movements: Analyze the movements of athletes frame by frame, and extract key points and joint positions for each element of the movement.

[0935] Step 2:

[0936] Learning from behavioral data and training models

[0937] The server uses the extracted keypoints and joint locations to train a machine learning model that combines LSTM and CNN. The trained model is stored in a cloud database. The input is the keypoints and joint locations obtained in step 1, and the output is the trained machine learning model.

[0938] Specific operation: Time series data is learned using LSTM, and spatial information is analyzed using CNN.

[0939] Step 3:

[0940] Filming of users' sports movements and uploading of video data

[0941] Users record their sports movements with their smartphones and upload the video data to a cloud server. The input is the video data recorded by the user, and the output is the movement data stored on the server.

[0942] Specific operation: Video is taken using the smartphone's camera function, and the video data is sent to the server via the application.

[0943] Step 4:

[0944] Video data preprocessing and feature extraction

[0945] The server preprocesses the received video data, splits it into frames, and extracts features. The input is the uploaded video data, and the output is the extracted keypoints and joint positions.

[0946] Specific operation: Divide the video into frames and extract key points and joint positions using OpenPose.

[0947] Step 5:

[0948] Comparative analysis of operational data and generation of advice

[0949] The server uses the stored model to compare and analyze the user's motion data with that of professional athletes. Specific advice is generated based on the analysis results. The input is the data after feature extraction, and the output is the analysis results and generated advice.

[0950] Specific behavior: Using machine learning models, behavior data is analyzed and specific improvement advice is generated.

[0951] Step 6:

[0952] Comparison video generation

[0953] The server generates a comparison video that overlays the current (AsIs) and ideal (ToBe) form. The input is the analyzed feature data, and the output is a comparison video of the AsIs and ToBe form.

[0954] Specific actions: Using a video editing tool, the two action data are overlaid into one video.

[0955] Step 7:

[0956] Providing advice and comparison videos

[0957] The terminal displays the advice and comparison video received from the server to the user. The input is the output data from the server, and the output is the advice and comparison video that the user can view.

[0958] Specific behavior: Provides an interface through the application that displays advice and videos.

[0959] Step 8:

[0960] Re-filming and re-analyzing the movement

[0961] The user improves their sports movements based on the advice from the server, then re-films and uploads the improved movements. The server then performs a re-analysis using the same process. The input is the improved video data, and the output is new analysis results and advice.

[0962] Specific actions: Record the video again, upload it, and provide new analysis and advice.

[0963] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0964] The following describes in detail an embodiment of the present invention. This system uses AI to provide parents with specific advice and feedback that takes into account the user's emotions, in order to help their children improve their sports skills.

[0965] Overall system overview

[0966] This system studies the motion data of professional athletes, compares it with filmed footage of children's sports movements, and provides specific advice based on the results. It also has the ability to recognize the user's emotions and adjust the advice content according to their emotional state. This allows parents, even those with no experience in sports, to provide effective instruction that takes emotions into consideration.

[0967] Program processing flow

[0968] 1. Learning the movements of professional athletes

[0969] server

[0970] 1. Receive and preprocess motion data from professional athletes. Divide the received video data into frames and extract features such as key points and joint positions.

[0971] 2. Use machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes and train the model.

[0972] 3. Save the trained model in a database.

[0973] Terminal

[0974] 1. Provides an interface for uploading operational data to a server.

[0975] 2. Display a notification to the user that the learning is complete.

[0976] User

[0977] 1. Record the motion data of a professional athlete and upload it to a server via the device.

[0978] 2. Check your learning progress and completion notifications.

[0979] 2. Filming and reading children's sports movements

[0980] Terminal

[0981] 1. Provide a camera interface for capturing children's sports activities.

[0982] 2. Display the option to upload the captured video data to the server.

[0983] User

[0984] 1. Take photos of your child's sports movements with your smartphone.

[0985] 2. Upload the captured video data to the server.

[0986] 3. Behavior analysis and advice generation phase

[0987] server

[0988] 1. Preprocess the received video data of the child, divide it into frames, and extract features.

[0989] 2. Use the saved model to compare the motion data of a child with that of a professional athlete.

[0990] 3. Based on the analysis results, specific advice (e.g., "Raise your right elbow 3 cm higher") is generated.

[0991] 4. Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[0992] 5. Send advice and comparison videos to your device.

[0993] 4. Emotion Recognition and Advice Adjustment Phase

[0994] server

[0995] 1. Using an emotion engine, input data from the device (e.g., voice, facial expressions) is analyzed to recognize the user's emotional state.

[0996] 2. Tailor the advice based on the perceived emotional state, for example adding an encouraging message if the user is feeling frustrated.

[0997] Terminal

[0998] 1. The advice received from the server, comparison videos, and emotion-sensitive feedback are displayed to the user.

[0999] 2. Provide an interface for playing videos that overlay AsIs and ToBe.

[1000] User

[1001] 1. Check the advice displayed on the device, comparison videos, and emotionally sensitive feedback.

[1002] 2. Work with your child to improve their sports movements using the advice provided.

[1003] 3. If necessary, re-photograph the improved behavior and re-analyze it using the same process.

[1004] Example: Improving children's baseball pitching technique

[1005] Initial setup and learning phase

[1006] Users can record the pitching motion of a top baseball pitcher and upload it to the server via their device. The server then preprocesses the received video data and learns the motion data of the professional pitcher.

[1007] Filming the pitching motion

[1008] The user continuously records the child's pitching motion with a smartphone and uploads the video data to a server.

[1009] Behavior analysis and advice generation

[1010] The server analyzes the child's pitching video and compares it with the motion data of a professional pitcher. It generates specific improvements (e.g., "Raise your right elbow 3 cm higher") and creates a video comparing AsIs and ToBe.

[1011] Emotion Recognition and Advice Adjustment

[1012] The server uses an emotion engine to analyze the user's emotional state and, if the user is feeling frustrated, adds an encouraging message such as, "Great progress! Now take a breather and try again."

[1013] Displaying and Implementing Results

[1014] The user checks the advice and comparison video displayed on the device, as well as the emotional feedback, and practices together with the child. After the improvement, the child's movements are filmed again and uploaded to the server for reanalysis.

[1015] This system allows parents with no prior experience in sports to provide effective and emotionally sensitive advice to help their children improve in sports.

[1016] The processing flow will be explained below.

[1017] Program processing flow

[1018] 1. Learning the movements of professional athletes

[1019] Step 1: Prepare the operation data

[1020] User: Uses a camera to record the movements of a professional athlete and saves the video file on the device.

[1021] Step 2: Uploading the behavior data

[1022] Terminal: Provides an interface for uploading operational data to the server.

[1023] User: Select the captured motion data and upload it to the server.

[1024] Device: Sends motion data to the server.

[1025] Step 3: Preprocessing the behavioral data

[1026] Server: Divides the received video file into frames and extracts key points and joint positions.

[1027] Server: Stores the preprocessed data in a database.

[1028] Step 4: Training the learning model

[1029] Server: Uses machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes.

[1030] Server: Saves the trained model in a database.

[1031] Step 5: Notification of completion of study

[1032] Server: Notifies the device that learning is complete.

[1033] Terminal: Display a message to the user that the training is complete.

[1034] 2. Filming and reading children's sports movements

[1035] Step 1: Capture the action

[1036] User: Takes photos of children's sports activities with a smartphone.

[1037] Terminal: Provides a camera interface for capturing action.

[1038] Step 2: Uploading the behavior data

[1039] On your device: Displays the option to upload the captured video file to the server.

[1040] User: Select a video file and upload it to the server.

[1041] Device: Sends the video file to the server.

[1042] 3. Behavior analysis and advice generation phase

[1043] Step 1: Preprocessing the video data

[1044] Server: Divides the received video file of the child into frames and extracts features.

[1045] Step 2: Comparative analysis of behavior

[1046] Server: Uses the trained model to compare children's motion data with that of professional athletes.

[1047] Server: Quantify the difference in each frame and identify specific areas for improvement.

[1048] Step 3: Generate Advice

[1049] Server: Generates specific advice based on the results of the comparative analysis.

[1050] Server: Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[1051] Step 4: Submit the results

[1052] Server: Sends the generated advice and comparison videos to the device.

[1053] 4. Emotion recognition and feedback regulation phase

[1054] Step 1: Enter user emotion data

[1055] User: Uses the device's camera and microphone to provide facial expressions and audio during the training session.

[1056] Device: Sends emotion data to the server.

[1057] Step 2: Sentiment Analysis

[1058] Server: Analyzes the received emotion data using the emotion engine and recognizes the user's emotional state (e.g., joy, frustration).

[1059] Server: Stores the recognized emotional state in a database.

[1060] Step 3: Adjust your advice

[1061] Server: Adjust the generated advice based on the user's emotional state. For example, if the user is feeling frustrated, add an encouraging message.

[1062] Server: Sends advice content including emotion-sensitive feedback to the device.

[1063] Step 4: View the results

[1064] Device: The advice received from the server, comparison videos, and emotional feedback are displayed to the user.

[1065] Device: Provides an interface for playing videos that overlay AsIs and ToBe.

[1066] Step 5: Check operation and make improvements

[1067] User: View advice and comparison videos displayed on the device, as well as emotionally sensitive feedback.

[1068] User: Work with their child to improve their sports movements using the advice provided.

[1069] Step 6: Feedback and Retraining

[1070] User: Re-photograph the improved behavior and re-analyze it using the same process.

[1071] Device: Sends new video to the server and requests reanalysis.

[1072] As described above, a system is constructed in which the server, terminal, and user work together to support children in improving their sports movements and provide feedback that takes emotions into consideration.

[1073] Example 2

[1074] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1075] Conventional sports coaching systems have the problem that parents lack specialized knowledge of sports, making it difficult to provide effective instruction. Furthermore, feedback is often provided that ignores the child's emotional state, resulting in a decrease in the effectiveness of instruction. To solve these problems, a system is needed that can provide effective advice while taking into consideration the parent's emotions by comparing and analyzing the child's movements based on a model trained by a professional.

[1076] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1077] In this invention, the server includes means for learning the motion data of professional athletes, means for receiving video of filmed sports movements, means for preprocessing the received video data and extracting features, means for comparative analysis using the extracted features with the motion data of the professional athletes, means for generating specific advice based on the analysis results, means for generating a comparison video that overlays the current form with an ideal form, means for outputting the generated advice and the comparison video, and means for recognizing the emotional state of the user using an emotion engine and adjusting the content of the advice based on the recognized emotional state. This enables parents to provide effective instruction based on the movements of professionals and feedback that takes into account their child's emotions, even if they do not have specialized knowledge of sports.

[1078] "Professional athlete exercise data" refers to recordings of exercises performed by skilled or expert athletes, typically stored in video format, that are used as training data for machine learning.

[1079] "Means for learning athletic data" refers to the process and technology for analyzing received athletic data of professional athletes, incorporating the features into a model, and learning the data.

[1080] "Means for receiving video" refers to an interface and protocol that allows the system to obtain video of sports exercises taken by users.

[1081] "Preprocessing" refers to the process of dividing the received video data into frames and extracting features such as key points and joint positions from each frame.

[1082] "Features" are important information extracted from movement data, and refer to data such as points and joint positions necessary for analyzing sports movements.

[1083] "Means for comparative analysis" refers to a technique that uses extracted features to compare children's movements with those of professional athletes and analyze differences and areas for improvement.

[1084] "Specific advice" refers to suggestions and instructions for improving exercise that are generated based on the analysis results and are provided to the user in an easy-to-understand format.

[1085] A "comparison video" refers to a video that displays the current (AsIs) and ideal (ToBe) exercise form superimposed on each other, allowing users to visually understand areas for improvement.

[1086] An "emotion engine" refers to technology and systems that analyze input data such as a user's voice and facial expressions to recognize their emotional state.

[1087] "Emotional state" refers to the user's current mental and emotional state and is a factor taken into account when adjusting feedback.

[1088] The present invention relates to a system that enables parents to support their children's sports improvement effectively and in a way that takes their emotions into consideration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the preferred embodiments of the present invention.

[1089] Hardware and software used

[1090] server

[1091] The server is the main component for learning motion data, data analysis, and emotion recognition. It uses the following software and frameworks:

[1092] Behavior analysis: TensorFlow, PyTorch

[1093] Emotion Recognition: Emotion analysis engines (e.g., IBM Watson or Microsoft Azure Emotion API)

[1094] Video processing: OpenPose, FFmpeg

[1095] Terminal

[1096] The terminal is the device through which the user captures video and communicates with the server. It uses the following technologies:

[1097] Video recording and uploading: Smartphone camera API (e.g., Android Camera2 API, iOS AVFoundation)

[1098] User Interface: React Native, Flutter

[1099] User

[1100] Users are primarily parents and children and perform the following actions:

[1101] Video recording and uploading

[1102] Check the advice

[1103] Providing emotional feedback

[1104] Specific processing of the system

[1105] 1. Learning from motion data of professional athletes

[1106] server

[1107] The server receives video data of professional athletes and splits it into frames. Next, OpenPose is used to extract features such as keypoints and joint positions from each frame. These features are then used to train machine learning models such as LSTM and CNN using TensorFlow or PyTorch. The trained models are stored in a database for later analysis.

[1108] Terminal

[1109] The device provides an interface for the user to upload the motion data to the server. Once the video upload is complete, the device notifies the user that the learning is complete.

[1110] User

[1111] Users can record the movements of professional athletes using their smartphones, upload the videos to the server via their devices, and then check their learning progress and receive notifications of completion.

[1112] 2. Shooting and importing children's sports movements

[1113] Terminal

[1114] The device provides a camera interface, allowing users to easily record their children's sports activities, and the captured video data is sent to the server via an upload option.

[1115] User

[1116] Users use their smartphones to record their children's sports activities and then upload the videos to a server.

[1117] 3. Behavior analysis and advice generation

[1118] server

[1119] The server divides the child's video data into frames and performs preprocessing. Next, it extracts features and compares them with the motion data of professional athletes. Based on this comparative analysis, specific advice is generated. Furthermore, a comparison video is generated that overlays the child's current and ideal form, and this is sent to the device.

[1120] 4. Emotion recognition and advice adjustment

[1121] server

[1122] The server uses an emotion engine to analyze voice and facial expression data from the device to recognize the user's emotional state. Based on the recognized emotional state, the server adjusts the content of the advice, for example, adding an encouraging message.

[1123] Terminal

[1124] The device displays the received advice, comparison videos, and emotion-sensitive feedback to the user. It also provides a video playback interface, allowing users to compare their current and ideal form.

[1125] User

[1126] The user can check the advice and feedback displayed on the device and use it to work with their child to improve their sports movements. If necessary, they can record the movements again and upload them to the server for reanalysis.

[1127] Specific examples

[1128] Improving children's baseball pitching movements

[1129] Initial setup and learning phase

[1130] Users film the pitching movements of famous baseball pitchers and upload them to the server via their devices. The server then uses this video data to train the model.

[1131] Filming the pitching motion

[1132] Users can record their child's pitching motion with their smartphone and upload the video to the server.

[1133] Behavior analysis and advice generation

[1134] The server analyzes the child's pitching video, generates specific improvements (e.g., "Raise your right elbow 3cm higher"), and creates a comparison video.

[1135] Emotion Recognition and Advice Adjustment

[1136] The server uses an emotion engine to analyze the user's emotions and add appropriate feedback (e.g., "Great progress! Let's take a breather and try again next time").

[1137] Prompt Sentence Examples

[1138] "Please explain the system that records a child's sports movements and compares them with the movement data of professional athletes. Also, please provide a concrete example of how a user would use the system."

[1139] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1140] Step 1:

[1141] Receiving and preprocessing motion data from professional athletes

[1142] server

[1143] Input: Video data of professional athletes

[1144] How it works: The server receives video data of a professional athlete, splits it into frames, and then uses OpenPose to extract features such as keypoints and joint positions from each frame.

[1145] Output: Extracted feature data

[1146] Step 2:

[1147] Model learning for motion data

[1148] server

[1149] Input: Extracted feature data

[1150] How it works: Use TensorFlow or PyTorch to train machine learning models such as LSTMs and CNNs. The training process involves providing feature data to the model and iteratively learning it.

[1151] Output: A trained machine learning model

[1152] Step 3:

[1153] Saving the trained model

[1154] server

[1155] Input: A trained machine learning model

[1156] Behavior: Once the training is complete, the model is saved in a database and used for subsequent behavior analysis.

[1157] Output: A saved machine learning model

[1158] Step 4:

[1159] Uploading model operation data

[1160] Terminal

[1161] Input: Professional athlete's athletic data

[1162] Operation: The device provides an interface for the user to upload operation data to the server. The user takes a video using the smartphone's camera API and then selects the upload option.

[1163] Output: Video data uploaded to the server

[1164] Step 5:

[1165] Learning completion notification

[1166] Terminal

[1167] Input: Learning completion notification sent from the server

[1168] Behavior: Notify the user that their learning is complete via push notification or in-app message.

[1169] Output: User notification

[1170] Step 6:

[1171] Filming and uploading children's sports activities

[1172] Terminal

[1173] Input: Video of a child's sports activity

[1174] Actions: The device records the child's sports movements through the camera interface. It displays an option to upload the captured video data to the server. The user uses the option to upload the video to the server.

[1175] Output: Child's video data uploaded to the server

[1176] Step 7:

[1177] Preprocessing and feature extraction of children's video data

[1178] server

[1179] Input: Children's video data

[1180] How it works: The server splits the received video data of the child into frames and extracts features from each frame. As a preprocessing step, it uses OpenPose to identify key points and joint positions.

[1181] Output: Extracted feature data

[1182] Step 8:

[1183] Behavior analysis and advice generation

[1184] server

[1185] Input: Extracted child feature data, saved machine learning model

[1186] How it works: The server uses the saved model to compare the child's feature data with that of a professional athlete. Based on the analysis results, specific advice is generated. In addition, a comparison video is created that overlays the child's current and ideal form.

[1187] Output: Specific advice, comparison video

[1188] Step 9:

[1189] Submitting advice and comparison videos

[1190] server

[1191] Input: Specific advice, comparison videos

[1192] Operation: The server sends the generated advice and comparison video to the device, where the data is displayed and the user can review it.

[1193] Output: Advice and comparison video sent to your device

[1194] Step 10:

[1195] Emotion Recognition and Advice Adjustment

[1196] server

[1197] Input: Voice and facial expression data that indicates the user's emotional state

[1198] How it works: The server uses the emotion engine to analyze voice and facial expression data from the device to recognize the user's emotional state. Based on the recognized emotional state, it adjusts the advice content, for example, adding an encouraging message.

[1199] Output: Tailored advice based on perceived emotional state

[1200] Step 11:

[1201] View advice and feedback

[1202] Terminal

[1203] Input: Adjusted advice received from the server, comparison video

[1204] How it works: The device displays the received advice, comparison videos, and emotion-sensitive feedback to the user. The device also provides an interface for playing videos that overlay AsIs and ToBe.

[1205] Output: Advice and feedback displayed to the user

[1206] Step 12:

[1207] Performance improvements and reanalysis

[1208] User

[1209] Input: Advice displayed on the device, comparison video

[1210] Movement: The user works with their child to improve their sports movements based on the advice provided. If necessary, the improved movements are filmed again and uploaded to the server for reanalysis.

[1211] Output: Improved sports motion video, reanalysis results

[1212] (Application example 2)

[1213] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1214] Modern factories lack an automatic analysis and improvement feedback system to improve the efficiency and accuracy of robotic work. They also lack the ability to provide appropriate feedback based on the emotional state of workers. This makes it difficult to simultaneously improve work efficiency and maintain workers' psychological stability.

[1215] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for learning professional motion data, a means for receiving video of filmed motions, and a means for preprocessing the received video data and extracting features. This makes it possible to analyze the robot's motions and provide specific improvement advice. The server also includes a means for recognizing emotions and adjusting feedback accordingly, and a means for filming and analyzing the worker's motions. This makes it possible to provide appropriate feedback according to the worker's psychological state.

[1216] "Professional motion data" is data that records the motions of skilled people or robots in a specific task or operation.

[1217] "Means for receiving video of captured action" refers to an interface that imports video of action captured by a camera or other capturing device into the system.

[1218] "Means for preprocessing received video data and extracting features" refers to the process of analyzing video data and extracting important points of movement, joint positions, etc.

[1219] "Means for comparative analysis with professional motion data using extracted features" refers to the process of finding and evaluating differences and similarities with professional motion data based on the analyzed features.

[1220] "Means for generating specific advice based on the analysis results" refers to a process for generating specific instructions for improving behavior based on the results of behavior comparison.

[1221] "Means for generating a comparison video that overlays current and ideal movements" refers to the process of creating a video that overlays current and ideal movements to visually show the difference.

[1222] "Means for outputting generated advice and comparison videos" refers to an interface that provides users with advice and comparison videos as the analysis results.

[1223] "Means for recognizing emotions and adjusting feedback accordingly" refers to the process of analyzing the worker's emotional state and appropriately changing the advice or feedback content based on the results.

[1224] "Means for filming and analyzing the worker's movements" refers to the process of filming the worker's movements using a camera or other device, and analyzing and evaluating those movements.

[1225] The following describes in detail the mode for carrying out this invention. This system analyzes the operation of a factory robot and provides feedback for improvement. It also has the ability to recognize the user's emotions and adjust the feedback according to their emotional state. This allows for both psychological stability for the worker and improved production efficiency.

[1226] Overall system overview

[1227] This system learns professional motion data, compares it with filmed robot motion data, and provides specific advice based on the results. It also has the ability to recognize the user's emotions and adjust the advice content according to their emotional state.

[1228] Hardware and software used

[1229] Hardware: smartphone, head-mounted display, camera

[1230] Software: Python, open source libraries (OpenCV, Keras)

[1231] Program processing flow

[1232] 1. Learning the movements of professionals

[1233] The server receives professional motion data and performs preprocessing. It divides the received video data into frames and extracts features such as key points and joint positions. It then uses a machine learning algorithm (e.g., LSTM or CNN) to learn from the professional motion data and train a model. The trained model is then stored in a database.

[1234] 2. Recording and reading the robot's movements

[1235] Users use smartphones or head-mounted displays to record robot movements in the factory. The captured data is uploaded to a server, which then preprocesses the received robot video data, dividing it into frames and extracting features.

[1236] 3. Behavior analysis and advice generation phase

[1237] The server uses the saved model to compare the robot's motion data with professional motion data. Based on the results of the comparative analysis, it generates specific advice (e.g., "Delay your right elbow by two seconds"). It also generates a comparison video that overlays the AsIs (current) and ToBe (ideal) forms. The advice and comparison video are sent to the device.

[1238] 4. Emotion Recognition and Advice Adjustment Phase

[1239] The server uses an emotion engine to analyze input data (e.g., voice, facial expressions) from the device and recognize the user's emotional state. Based on the recognized emotional state, the server adjusts the advice content. For example, if the user is feeling frustrated, it adds an encouraging message. It provides feedback such as, "Great progress. Let's take a breather and try again next time."

[1240] Example: Improving the operation of a robot arm

[1241] Initial setup and learning phase

[1242] Users can record professional robot arm movements and upload them to the server via their devices. The server then preprocesses the received video data and learns the professional movement data.

[1243] Filming the robot arm in action

[1244] The user continuously records the robot arm's movements with their smartphone and uploads the video data to a server.

[1245] Behavior analysis and advice generation

[1246] The server analyzes the video of the robot arm and compares it with professional motion data, generates specific improvements (e.g., "Delay the right elbow by 2 seconds"), and creates a comparison video of the AsIs and ToBe.

[1247] Emotion Recognition and Advice Adjustment

[1248] The server uses an emotion engine to analyze the user's emotional state and, if the user is feeling frustrated, adds an encouraging message such as, "Great progress! Now take a breather and try again."

[1249] Displaying and Implementing Results

[1250] The user can then improve the robot's behavior by checking the advice, comparison videos, and emotional feedback displayed on the device. After the improvements are made, the robot's behavior is filmed again and uploaded to the server for reanalysis.

[1251] This system allows workers to efficiently improve the robot's operation, thereby increasing work quality and production efficiency.

[1252] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1253] Step 1: Learning from professional motion data

[1254] In step 1, the server receives the professional's motion data and performs preprocessing. The input is the professional's motion video data, which is split into frames and features such as key points and joint positions are extracted. A machine learning algorithm (e.g., LSTM, CNN) is used to learn the features and train a model. The trained model is stored in a database. The output of this step is the features and the trained model.

[1255] Step 2: Film the robot's movements and upload the data

[1256] In step 2, the user uses a smartphone or head-mounted display to film the robot's operation in the factory. The input is video data containing the actual robot operation. The user uploads the filmed video to the server via their device. The input video data is transferred to the server. The output of this step is video data of the robot's operation stored on the server.

[1257] Step 3: Preprocessing of motion data and feature extraction

[1258] In step 3, the server preprocesses the input robot video data and splits it into frames. This allows for detailed analysis of each stage of the movement. Next, features such as key points and joint positions are extracted. The input is the robot's movement video data, and the output is the extracted features. Specific operations include splitting and analyzing the video frames using OpenCV.

[1259] Step 4: Behavior analysis and comparison

[1260] In step 4, the server uses the stored professional model to perform comparative analysis of the robot's motion data. The inputs are the extracted features and the professional model, and by comparing and analyzing these, differences in motion and areas for improvement are identified. Specifically, a machine learning model is used to calculate the motion match rate and descriptor. The output is the analysis results and areas for improvement.

[1261] Step 5: Generate advice and comparison videos

[1262] In step 5, the server generates specific advice based on the analysis results. For example, this may include instructions such as "Delay your right elbow for another two seconds." It also generates a comparison video that overlays the current and ideal movements. The input is the analysis results, and the output is specific advice and a comparison video. In this step, the two videos are overlaid using video editing software.

[1263] Step 6: View the results

[1264] In step 6, the server sends the generated advice and comparison video to the user's device. The input is the generated advice and comparison video, and the user receives this information through their device. Specifically, feedback is provided to the user via a notification function or interface. The output is advice and comparison video that the user can visually confirm.

[1265] Step 7: Emotion recognition and feedback regulation

[1266] In step 7, the server uses an emotion engine to analyze the user's emotional state. For example, it analyzes the user's voice and facial expression data to recognize emotions. The input is emotional data from the user, and the feedback content is adjusted based on the analysis results. For example, if the user is feeling frustrated, an encouraging message such as "Great progress! Let's take a breather and try again next time" is added. The output is the adjusted feedback.

[1267] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1268] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1269] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1270] [Third embodiment]

[1271] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1272] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1273] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1274] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1275] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1276] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1277] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1278] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1279] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1280] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1281] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1282] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1283] The following describes in detail an embodiment of the present invention. This system uses AI to provide parents with specific advice to help their children improve their sports skills.

[1284] Overall system overview

[1285] This system learns the motion data of professional athletes, compares it with filmed footage of children's sports movements, and provides specific advice based on the results, enabling parents to provide effective instruction even if they have no experience in sports.

[1286] Program processing flow

[1287] 1. Learning the movements of professional athletes

[1288] server

[1289] 1. Receive and preprocess motion data from professional athletes. Divide the received video data into frames and extract features such as key points and joint positions.

[1290] 2. Use machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes and train the model.

[1291] 3. Save the trained model in a database.

[1292] Terminal

[1293] 1. Provides an interface for uploading operational data to a server.

[1294] 2. Display a notification to the user that the learning is complete.

[1295] User

[1296] 1. Record the motion data of a professional athlete and upload it to a server via the device.

[1297] 2. Check your learning progress and completion notifications.

[1298] 2. Filming and reading children's sports movements

[1299] Terminal

[1300] 1. Provide a camera interface for capturing children's sports activities.

[1301] 2. Display the option to upload the captured video data to the server.

[1302] User

[1303] 1. Take photos of your child's sports movements with your smartphone.

[1304] 2. Upload the captured video data to the server.

[1305] 3. Behavior analysis and advice generation phase

[1306] server

[1307] 1. Preprocess the received video data of the child, divide it into frames, and extract features.

[1308] 2. Use the saved model to compare and analyze the motion data of children and professional athletes.

[1309] 3. Based on the analysis results, specific advice (e.g., "Raise your right elbow 3 cm higher") is generated.

[1310] 4. Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[1311] 5. Send advice and comparison videos to your device.

[1312] 4. Results display and advice phase

[1313] Terminal

[1314] 1. The advice received from the server and comparison video are displayed to the user.

[1315] 2. Provide an interface for playing videos that overlay AsIs and ToBe.

[1316] User

[1317] 1. Check the advice displayed on your device and the comparison video.

[1318] 2. Work with your child to improve their sports movements using the advice provided.

[1319] 3. If necessary, re-photograph the improved behavior and re-analyze it using the same process.

[1320] Example: Improving children's baseball pitching technique

[1321] Initial setup and learning phase

[1322] Users can record the pitching motion of a top baseball pitcher and upload it to the server via their device. The server then preprocesses the received video data and learns the motion data of the professional pitcher.

[1323] Filming the pitching motion

[1324] The user continuously records the child's pitching motion with a smartphone and uploads the video data to a server.

[1325] Behavior analysis and advice generation

[1326] The server analyzes the child's pitching video and compares it with the motion data of a professional pitcher. It generates specific improvements (e.g., "Raise your right elbow 3 cm higher") and creates a video comparing AsIs and ToBe.

[1327] Displaying and Implementing Results

[1328] The user checks the advice and comparison video displayed on the device and practices together with the child. After the improvement, the movement is filmed again and uploaded to the server for reanalysis.

[1329] This system allows parents with no sports experience to provide effective advice and support their children's sports improvement.

[1330] The processing flow will be explained below.

[1331] Program processing flow

[1332] 1. Learning the movements of professional athletes

[1333] Step 1: Prepare the operation data

[1334] User: Uses a camera to record the movements of a professional athlete and saves the video file on the device.

[1335] Step 2: Uploading the behavior data

[1336] Terminal: Provides an interface for uploading operational data to the server.

[1337] User: Select the captured motion data and upload it to the server.

[1338] Device: Sends motion data to the server.

[1339] Step 3: Preprocessing the behavioral data

[1340] Server: Divides the received video file into frames and extracts key points and joint positions.

[1341] Server: Stores the preprocessed data in a database.

[1342] Step 4: Training the learning model

[1343] Server: Uses machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes.

[1344] Server: Saves the trained model in a database.

[1345] Step 5: Notification of completion of study

[1346] Server: Notifies the device that learning is complete.

[1347] Terminal: Display a message to the user that the training is complete.

[1348] 2. Filming and reading children's sports movements

[1349] Step 1: Capture the action

[1350] User: Takes photos of children's sports activities with a smartphone.

[1351] Terminal: Provides a camera interface for capturing action.

[1352] Step 2: Uploading the behavior data

[1353] On your device: Displays the option to upload the captured video file to the server.

[1354] User: Select a video file and upload it to the server.

[1355] Device: Sends the video file to the server.

[1356] 3. Behavior analysis and advice generation phase

[1357] Step 1: Preprocessing the video data

[1358] Server: Divides the received video file of the child into frames and extracts features.

[1359] Step 2: Comparative analysis of behavior

[1360] Server: Uses the trained model to compare children's motion data with that of professional athletes.

[1361] Server: Quantify the difference in each frame and identify specific areas for improvement.

[1362] Step 3: Generate Advice

[1363] Server: Generates specific advice based on the results of the comparative analysis.

[1364] Server: Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[1365] Step 4: Submit the results

[1366] Server: Sends the generated advice and comparison videos to the device.

[1367] 4. Results display and advice phase

[1368] Step 1: View the results

[1369] Terminal: Displays the advice received from the server and the comparison video to the user.

[1370] Device: Provides an interface for playing videos that overlay AsIs and ToBe.

[1371] Step 2: Operation check and improvement steps

[1372] User: Check the advice and comparison video displayed on the device.

[1373] User: Work with their child to improve their sports movements using the advice provided.

[1374] Step 3: Feedback and Retraining

[1375] User: Re-photograph the improved behavior and re-analyze it using the same process.

[1376] Device: Sends new video to the server and requests reanalysis.

[1377] As described above, a system is constructed in which the server, terminals, and users work together to support children in improving their sports movements.

[1378] Example 1

[1379] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1380] Conventional systems supporting children's sports improvement have the problem that it is difficult to provide effective advice to parents who lack sports experience or expertise. Furthermore, analyzing the recorded video data requires specialized expertise and is not easy to do. Furthermore, the system lacks a means to visually show specific areas for improvement and an interface for reassessment, making the improvement process difficult.

[1381] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1382] In this invention, the server includes: means for learning motion data of professional athletes; means for receiving video of filmed sports movements; means for preprocessing the received video data and extracting features; means for comparative analysis with the motion data of the professional athletes using the extracted features; means for generating specific advice based on the analysis results; means for generating a comparison video that overlays the current form with an ideal form; means for outputting the generated advice and the comparison video; means for providing a camera interface for filming the child's sports movements; and means for inputting a prompt sentence for generating an analysis request to the AI ​​model. This enables parents, even without experience or knowledge of sports, to effectively analyze their children's sports movements, provide specific and visual advice, and reassess their children's sports movements.

[1383] "Professional athlete motion data" refers to data that records the motions of athletes with high levels of skill.

[1384] "Means for receiving" refers to technical means for taking in data from the outside, such as a server or storage device.

[1385] "Preprocessing" refers to the process of converting received data into a format that is easy to analyze.

[1386] A "feature" is a data point or value that contains important information in data analysis.

[1387] A "comparative analysis tool" is a tool that compares two or more data sets to analyze commonalities and differences.

[1388] "Specific advice" refers to specific, practical instructions or suggestions.

[1389] A "comparison video" is a video that displays your current form and ideal form at the same time for comparison.

[1390] "Means of output" refers to the technical means for providing data or information to the user.

[1391] A "camera interface" is a user interface for operating a camera.

[1392] An "AI model" is an algorithm trained using machine learning or deep learning.

[1393] A "prompt sentence" is an input sentence for a generative AI model, and is text that specifies an analysis request.

[1394] The following describes in detail the embodiments of the present invention. This system uses AI to provide specific advice to parents to help their children improve their sports skills. Even if the parents have no experience in sports, the system can effectively analyze their children's movements, provide specific and visual advice, and perform reassessments.

[1395] Overall system overview

[1396] This system learns the motion data of professional athletes, compares it with filmed footage of children's sports movements, and provides specific advice based on the results, allowing parents to provide effective instruction even if they have no experience in sports.

[1397] Program processing flow

[1398] 1. Learning the movements of professional athletes

[1399] server

[1400] 1. The server receives the motion data of professional athletes and performs preprocessing. This preprocessing involves dividing the video data into frames and extracting features such as key points and joint positions from each frame using open-source pose estimation technology such as OpenPose.

[1401] 2. The server uses the preprocessed feature data to train a model using machine learning algorithms such as LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network).

[1402] 3. The trained model is stored in a database by the server.

[1403] Terminal

[1404] 1. The device provides an interface for uploading professional athletes' motion data to the server, allowing users to easily select and upload videos.

[1405] 2. The device displays a learning completion notification to the user.

[1406] User

[1407] 1. The user uses their smartphone to record the movements of a professional athlete as an example, and then uploads the video data to the server via the device.

[1408] 2. The user checks the learning progress and completion notification displayed on the device.

[1409] 2. Filming and reading children's sports movements

[1410] Terminal

[1411] 1. The device provides a camera interface that allows the user to record their child's sports activities. For example, the app can activate the camera function and display a recording button to start recording.

[1412] 2. The device will display an option to upload the captured video data to the server.

[1413] User

[1414] 1. The user uses the smartphone camera to capture a video of their child's sports activity, for example, recording a video of their child pitching a baseball.

[1415] 2. The user uploads the captured video data to the server. By simply selecting the video and pressing the upload button according to the device interface, the data is sent to the server.

[1416] 3. Behavior analysis and advice generation phase

[1417] server

[1418] 1. The server preprocesses the received video data of the child, divides the video into frames, and extracts features. Specifically, it calculates the joint positions for each frame using OpenPose or similar.

[1419] 2. The server uses the stored trained model to compare and analyze the child's motion data with that of a professional athlete. This comparison evaluates the similarities and differences between the two motions based on the extracted features.

[1420] 3. The server generates specific advice based on the analysis results. For example, it may suggest to a child that they should "raise their right elbow 3cm higher" when pitching.

[1421] 4. The server generates a comparison video that overlays the AsIs (current) and ToBe (ideal) forms, making it visually easier to see which areas need improvement.

[1422] 5. The server sends the generated advice and comparison video to the device.

[1423] 4. Results display and advice phase

[1424] Terminal

[1425] 1. The device displays the advice and comparison video received from the server to the user. Specifically, the application provides a playback function for the advice text and comparison video.

[1426] 2. The device provides an interface for playing videos that overlay AsIs and ToBe, allowing users to easily play the videos and view them as many times as they like.

[1427] User

[1428] 1. The user checks the advice and comparison video displayed on the device. For example, they play the video to visually understand the suggested improvements.

[1429] 2. The user works with their child to improve their sports movements based on the provided advice. Specifically, for example, the user practices to correct the child's pitching form.

[1430] 3. If necessary, film the motion after the improvement and re-analyze it using the same process. Record the corrected pitching motion on video again, upload it to the server, and receive further advice.

[1431] Examples of prompt statements

[1432] Here are some examples of prompts that users can enter into a generative AI model:

[1433] "Please analyze my child's baseball pitching motion and let me know what needs to be improved. I have uploaded a video from the link below. I have also added a video of a professional pitcher, so you can compare it."

[1434] This allows the AI ​​model to provide appropriate analysis and advice based on the video data provided.

[1435] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1436] Step 1:

[1437] Receiving and preprocessing motion data from professional athletes

[1438] The server receives the motion data of the professional athlete, which includes video files uploaded by the user through an interface provided by the device.

[1439] Input: Video data of professional athletes' movements

[1440] The server preprocesses the received video data. This preprocessing involves dividing the video into frames and extracting key points and joint positions from each frame. Specifically, it uses open-source pose estimation technology such as OpenPose to detect the main joint points of the human body for each frame.

[1441] Data processing: Split the video into frames and extract joint points

[1442] Output: Features (keypoints and joint positions)

[1443] Step 2:

[1444] Training a machine learning model

[1445] The server uses the preprocessed feature data to train machine learning models, such as LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network). Training is necessary to capture the movement patterns of professional athletes.

[1446] Input: Preprocessed feature data

[1447] The server performs training using the given feature data and target data. During the training process, iterative parameter updates are performed using the backpropagation algorithm.

[1448] Data Computing: Model Training with Machine Learning Algorithms

[1449] Output: A trained model

[1450] Step 3:

[1451] Saving a trained model

[1452] The server stores the trained model in a database, allowing it to be quickly used in future analyses.

[1453] Input: A trained model

[1454] Data processing: Converting the model into a storage format

[1455] Output: Model saved in database

[1456] Step 4:

[1457] Filming children's sports

[1458] The user takes pictures of their child's sports activities using the smartphone camera and uses the camera interface provided by the device.

[1459] Input: Children's sports activities

[1460] The user activates the device's camera function and presses the record button to record the action. Once the recording is complete, it is saved as video data on the device.

[1461] Data generation: video data of sports movements

[1462] Output: Video data of children's sports movements

[1463] Step 5:

[1464] Uploading video data

[1465] The user uploads the captured video data to the server through the device, which displays an upload option.

[1466] Input: Video data of children's sports movements

[1467] The user selects a video from the terminal interface and presses the "upload" button to send the data to the server.

[1468] Data transfer: Uploading video data

[1469] Output: Video data of children's sports movements stored on the server

[1470] Step 6:

[1471] Preprocessing children's video data

[1472] The server preprocesses the received video data of the child by dividing the video into frames and extracting features (keypoints and joint positions) from each frame.

[1473] Input: Video data of children's sports movements stored on a server

[1474] Preprocessing includes extracting joint points using OpenPose, etc. The result of preprocessing is feature data for each frame.

[1475] Data processing: Split the video into frames and extract joint points

[1476] Output: Feature data of children's sports movements

[1477] Step 7:

[1478] Comparative analysis of motion data

[1479] The server uses the stored trained model to compare and analyze the motion data of children and that of professional athletes.

[1480] Input: feature data of children's sports movements, pre-trained model

[1481] The server evaluates the similarities and differences based on the extracted feature data. Specifically, it inputs the feature values ​​of each frame into a trained model and calculates the differences from the standard movements of professional athletes.

[1482] Data Computation: Comparative Analysis of Features

[1483] Output: Analysis results

[1484] Step 8:

[1485] Generating Specific Advice

[1486] The server generates specific advice based on the analysis results, such as "Raise your right elbow another 3 cm."

[1487] Input: Analysis results

[1488] Based on the analysis results, the server determines which parts should be improved and how, and generates advice in text format.

[1489] Data Generation: Specific Advice Text

[1490] Output: Specific advice

[1491] Step 9:

[1492] Comparison video generation

[1493] The server generates a comparison video that overlays the AsIs (current) and ToBe (ideal) forms, making it easier to visually identify areas for improvement.

[1494] Input: Video data of children's sports movements, pre-trained model

[1495] The server displays the current form and the ideal form side by side and combines the two videos to generate a single comparison video.

[1496] Data calculation: Video frame comparison and integration

[1497] Output: Comparison video

[1498] Step 10:

[1499] Advice and comparison video output

[1500] The server sends the generated advice and comparison video to the terminal, where the user can review it.

[1501] Input: Specific advice, comparison videos

[1502] The server transmits data to the terminal, and the terminal displays the received advice and video.

[1503] Data output: Sending advice and comparison videos

[1504] Output: Advice and comparison video displayed on device

[1505] (Application example 1)

[1506] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1507] Conventional sports instruction methods have the problem that it is difficult for parents and coaches to provide effective advice and instruction if they do not have specialized sports knowledge or skills. Furthermore, instruction services offered by sporting goods stores and fitness facilities are generally expensive, making them inaccessible to many people. Checking and improving sports form at physical stores is particularly time-consuming and requires a large burden on equipment and specialized staff. There is a need for a system that can solve these issues and enable ordinary parents and users to effectively improve their sports movements.

[1508] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1509] In this invention, the server includes: means for learning motion data of professional athletes; means for receiving video of filmed sports movements; means for preprocessing the received video data and extracting features; means for comparing and analyzing the extracted features with the motion data of the professional athletes; means for generating specific advice based on the analysis results; means for generating a comparison video that overlays the current form with an ideal form; means for outputting the generated advice and the comparison video; means for providing an interface that allows users at physical stores to film their sports movements and upload them to the cloud; and means for users to re-film and upload their improved movements and receive new advice to support improvement of their sports movements. This not only enables parents and physical store users with no sports experience to effectively improve their sports movements, but also enables efficient and cost-effective sports instruction services at physical stores.

[1510] "Professional athlete motion data" refers to data that records the motions of a professional athlete, and includes the feature quantities of those motions.

[1511] "Means for learning movement data" refers to the process of using machine learning algorithms to analyze the movement data of professional athletes and use it as a model for training.

[1512] "Means for receiving video" refers to the process by which the user sends the video data they have taken to the cloud or a server, and the server receives it.

[1513] "Preprocessing" refers to the process of dividing the received video data into frames and extracting important key points and joint positions.

[1514] "Features" are an important part in analyzing sports movements and include position information of key points and joints.

[1515] The "means for comparative analysis" refers to a process of comparing the motion data of a professional athlete with the photographed sports motion data using the extracted feature amounts.

[1516] "Means for generating advice" refers to the process of generating specific improvements and advice in text or video format based on the results of the comparative analysis.

[1517] "Method for generating a comparison video" refers to the process of generating a video that overlays the current (AsIs) and ideal (ToBe) forms to visually show the differences.

[1518] "Means for outputting a comparison video" refers to a process for providing the generated comparison video to a user.

[1519] "Brick-and-mortar users" refers to customers who visit physical stores, such as sporting goods stores or fitness facilities.

[1520] An "interface" is the part of the system that provides the screen and operating means for users to film their sports movements and upload them to the cloud.

[1521] "Cloud" refers to remote servers for storing, computing, and managing data over the Internet.

[1522] "Means for providing advice" refers to the process by which the generated advice is displayed to the user so that they can review it.

[1523] The following describes in detail the mode for carrying out the present invention. This system aims to support the improvement of sports movements in physical stores, and analyzes sports form and provides specific advice. By using this system, parents and general users can effectively improve their sports movements.

[1524] Overall system overview

[1525] The system learns from the motion data of professional athletes, compares and analyzes the sports motions of users filmed in physical stores, and generates and provides specific advice based on the results.

[1526] Hardware and software used

[1527] Hardware

[1528] Smartphone: A device that allows users to record and upload videos of their sports activities.

[1529] Cloud server: A remote server for storing data, training models, and performing analytical processing.

[1530] software

[1531] LSTM (Long Short-Term Memory): Used for time series analysis of behavioral data.

[1532] CNN (Convolutional Neural Network): Used to extract movement features.

[1533] OpenPose: Used to extract keypoints and joint positions from videos.

[1534] Flask / Django: Used for backend processing.

[1535] React Native: Used for the front end of smartphone applications.

[1536] Data processing and calculation

[1537] Learning from motion data of professional athletes

[1538] The server receives video data of professional athletes and performs preprocessing. During preprocessing, the video is divided into frames and key points and joint positions are extracted using OpenPose. Then, LSTM and CNN are used to learn the motion data of the professional athletes and train a model. The trained model is stored in a cloud database.

[1539] Filming and uploading users' sports activities

[1540] Users record their sports movements using their smartphones and upload them to the cloud. The server preprocesses the received video data and extracts features.

[1541] Behavior analysis and advice generation

[1542] The server uses the saved model to compare and analyze the user's motion data with that of professional athletes. Based on the analysis results, it generates specific improvement advice. It also generates a comparison video that overlays the current (AsIs) and ideal (ToBe) form.

[1543] Displaying results and providing advice

[1544] The smartphone app receives advice from the server and displays the comparison video to the user. The user can improve their sports movements based on the advice, and if necessary, record the video again and reanalyze it using the same process.

[1545] Specific examples

[1546] Example 1: Improving a child's basketball shooting technique

[1547] Parents can take a photo of their child's shooting movements on their smartphone and upload it to the app.

[1548] The app compares the movements to those of professional basketball players and advises, "Bend your knees a little."

[1549] A comparison video between AsIs and ToBe is also generated, allowing parents and children to check and practice together.

[1550] Prompt Sentence Examples

[1551] Simply record a professional basketball player's shooting technique and upload it. Then, record your child's shooting technique with your smartphone and upload it to the app. We'll provide you with relevant advice and comparison videos.

[1552] The system enables effective and cost-effective sports instruction in brick-and-mortar locations, promoting improved sports form.

[1553] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1554] Step 1:

[1555] Receiving and preprocessing motion data from professional athletes

[1556] The server receives video data of professional athletes. The received video data is divided into frames and key points and joint positions are extracted using OpenPose. As a result, the input is the video data, and the output is a set of key points and joint positions.

[1557] Specific movements: Analyze the movements of athletes frame by frame, and extract key points and joint positions for each element of the movement.

[1558] Step 2:

[1559] Learning from behavioral data and training models

[1560] The server uses the extracted keypoints and joint locations to train a machine learning model that combines LSTM and CNN. The trained model is stored in a cloud database. The input is the keypoints and joint locations obtained in step 1, and the output is the trained machine learning model.

[1561] Specific operation: Time series data is learned using LSTM, and spatial information is analyzed using CNN.

[1562] Step 3:

[1563] Filming of users' sports movements and uploading of video data

[1564] Users record their sports movements with their smartphones and upload the video data to a cloud server. The input is the video data recorded by the user, and the output is the movement data stored on the server.

[1565] Specific operation: Video is taken using the smartphone's camera function, and the video data is sent to the server via the application.

[1566] Step 4:

[1567] Video data preprocessing and feature extraction

[1568] The server preprocesses the received video data, splits it into frames, and extracts features. The input is the uploaded video data, and the output is the extracted keypoints and joint positions.

[1569] Specific operation: Divide the video into frames and extract key points and joint positions using OpenPose.

[1570] Step 5:

[1571] Comparative analysis of operational data and generation of advice

[1572] The server uses the stored model to compare and analyze the user's motion data with that of professional athletes. Specific advice is generated based on the analysis results. The input is the data after feature extraction, and the output is the analysis results and generated advice.

[1573] Specific behavior: Using machine learning models, behavior data is analyzed and specific improvement advice is generated.

[1574] Step 6:

[1575] Comparison video generation

[1576] The server generates a comparison video that overlays the current (AsIs) and ideal (ToBe) form. The input is the analyzed feature data, and the output is a comparison video of the AsIs and ToBe form.

[1577] Specific actions: Using a video editing tool, the two action data are overlaid into one video.

[1578] Step 7:

[1579] Providing advice and comparison videos

[1580] The terminal displays the advice and comparison video received from the server to the user. The input is the output data from the server, and the output is the advice and comparison video that the user can view.

[1581] Specific behavior: Provides an interface through the application that displays advice and videos.

[1582] Step 8:

[1583] Re-filming and re-analyzing the movement

[1584] The user improves their sports movements based on the advice from the server, then re-films and uploads the improved movements. The server then performs a re-analysis using the same process. The input is the improved video data, and the output is new analysis results and advice.

[1585] Specific actions: Record the video again, upload it, and provide new analysis and advice.

[1586] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1587] The following describes in detail an embodiment of the present invention. This system uses AI to provide parents with specific advice and feedback that takes into account the user's emotions, in order to help their children improve their sports skills.

[1588] Overall system overview

[1589] This system studies the motion data of professional athletes, compares it with filmed footage of children's sports movements, and provides specific advice based on the results. It also has the ability to recognize the user's emotions and adjust the advice content according to their emotional state. This allows parents, even those with no experience in sports, to provide effective instruction that takes emotions into consideration.

[1590] Program processing flow

[1591] 1. Learning the movements of professional athletes

[1592] server

[1593] 1. Receive and preprocess motion data from professional athletes. Divide the received video data into frames and extract features such as key points and joint positions.

[1594] 2. Use machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes and train the model.

[1595] 3. Save the trained model in a database.

[1596] Terminal

[1597] 1. Provides an interface for uploading operational data to a server.

[1598] 2. Display a notification to the user that the learning is complete.

[1599] User

[1600] 1. Record the motion data of a professional athlete and upload it to a server via the device.

[1601] 2. Check your learning progress and completion notifications.

[1602] 2. Filming and reading children's sports movements

[1603] Terminal

[1604] 1. Provide a camera interface for capturing children's sports activities.

[1605] 2. Display the option to upload the captured video data to the server.

[1606] User

[1607] 1. Take photos of your child's sports movements with your smartphone.

[1608] 2. Upload the captured video data to the server.

[1609] 3. Behavior analysis and advice generation phase

[1610] server

[1611] 1. Preprocess the received video data of the child, divide it into frames, and extract features.

[1612] 2. Use the saved model to compare the motion data of a child with that of a professional athlete.

[1613] 3. Based on the analysis results, specific advice (e.g., "Raise your right elbow 3 cm higher") is generated.

[1614] 4. Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[1615] 5. Send advice and comparison videos to your device.

[1616] 4. Emotion Recognition and Advice Adjustment Phase

[1617] server

[1618] 1. Using an emotion engine, input data from the device (e.g., voice, facial expressions) is analyzed to recognize the user's emotional state.

[1619] 2. Tailor the advice based on the perceived emotional state, for example adding an encouraging message if the user is feeling frustrated.

[1620] Terminal

[1621] 1. The advice received from the server, comparison videos, and emotion-sensitive feedback are displayed to the user.

[1622] 2. Provide an interface for playing videos that overlay AsIs and ToBe.

[1623] User

[1624] 1. Check the advice displayed on the device, comparison videos, and emotionally sensitive feedback.

[1625] 2. Work with your child to improve their sports movements using the advice provided.

[1626] 3. If necessary, re-photograph the improved behavior and re-analyze it using the same process.

[1627] Example: Improving children's baseball pitching technique

[1628] Initial setup and learning phase

[1629] Users can record the pitching motion of a top baseball pitcher and upload it to the server via their device. The server then preprocesses the received video data and learns the motion data of the professional pitcher.

[1630] Filming the pitching motion

[1631] The user continuously records the child's pitching motion with a smartphone and uploads the video data to a server.

[1632] Behavior analysis and advice generation

[1633] The server analyzes the child's pitching video and compares it with the motion data of a professional pitcher. It generates specific improvements (e.g., "Raise your right elbow 3 cm higher") and creates a video comparing AsIs and ToBe.

[1634] Emotion Recognition and Advice Adjustment

[1635] The server uses an emotion engine to analyze the user's emotional state and, if the user is feeling frustrated, adds an encouraging message such as, "Great progress! Now take a breather and try again."

[1636] Displaying and Implementing Results

[1637] The user checks the advice and comparison video displayed on the device, as well as the emotional feedback, and practices together with the child. After the improvement, the child's movements are filmed again and uploaded to the server for reanalysis.

[1638] This system allows parents with no prior experience in sports to provide effective and emotionally sensitive advice to help their children improve in sports.

[1639] The processing flow will be explained below.

[1640] Program processing flow

[1641] 1. Learning the movements of professional athletes

[1642] Step 1: Prepare the operation data

[1643] User: Uses a camera to record the movements of a professional athlete and saves the video file on the device.

[1644] Step 2: Uploading the behavior data

[1645] Terminal: Provides an interface for uploading operational data to the server.

[1646] User: Select the captured motion data and upload it to the server.

[1647] Device: Sends motion data to the server.

[1648] Step 3: Preprocessing the behavioral data

[1649] Server: Divides the received video file into frames and extracts key points and joint positions.

[1650] Server: Stores the preprocessed data in a database.

[1651] Step 4: Training the learning model

[1652] Server: Uses machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes.

[1653] Server: Saves the trained model in a database.

[1654] Step 5: Notification of completion of study

[1655] Server: Notifies the device that learning is complete.

[1656] Terminal: Display a message to the user that the training is complete.

[1657] 2. Filming and reading children's sports movements

[1658] Step 1: Capture the action

[1659] User: Takes photos of children's sports activities with a smartphone.

[1660] Terminal: Provides a camera interface for capturing action.

[1661] Step 2: Uploading the behavior data

[1662] On your device: Displays the option to upload the captured video file to the server.

[1663] User: Select a video file and upload it to the server.

[1664] Device: Sends the video file to the server.

[1665] 3. Behavior analysis and advice generation phase

[1666] Step 1: Preprocessing the video data

[1667] Server: Divides the received video file of the child into frames and extracts features.

[1668] Step 2: Comparative analysis of behavior

[1669] Server: Uses the trained model to compare children's motion data with that of professional athletes.

[1670] Server: Quantify the difference in each frame and identify specific areas for improvement.

[1671] Step 3: Generate Advice

[1672] Server: Generates specific advice based on the results of the comparative analysis.

[1673] Server: Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[1674] Step 4: Submit the results

[1675] Server: Sends the generated advice and comparison videos to the device.

[1676] 4. Emotion recognition and feedback regulation phase

[1677] Step 1: Enter user emotion data

[1678] User: Uses the device's camera and microphone to provide facial expressions and audio during the training session.

[1679] Device: Sends emotion data to the server.

[1680] Step 2: Sentiment Analysis

[1681] Server: Analyzes the received emotion data using the emotion engine and recognizes the user's emotional state (e.g., joy, frustration).

[1682] Server: Stores the recognized emotional state in a database.

[1683] Step 3: Adjust your advice

[1684] Server: Adjust the generated advice based on the user's emotional state. For example, if the user is feeling frustrated, add an encouraging message.

[1685] Server: Sends advice content including emotion-sensitive feedback to the device.

[1686] Step 4: View the results

[1687] Device: The advice received from the server, comparison videos, and emotional feedback are displayed to the user.

[1688] Device: Provides an interface for playing videos that overlay AsIs and ToBe.

[1689] Step 5: Check operation and make improvements

[1690] User: View advice and comparison videos displayed on the device, as well as emotionally sensitive feedback.

[1691] User: Work with their child to improve their sports movements using the advice provided.

[1692] Step 6: Feedback and Retraining

[1693] User: Re-photograph the improved behavior and re-analyze it using the same process.

[1694] Device: Sends new video to the server and requests reanalysis.

[1695] As described above, a system is constructed in which the server, terminal, and user work together to support children in improving their sports movements and provide feedback that takes emotions into consideration.

[1696] Example 2

[1697] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1698] Conventional sports coaching systems have the problem that parents lack specialized knowledge of sports, making it difficult to provide effective instruction. Furthermore, feedback is often provided that ignores the child's emotional state, resulting in a decrease in the effectiveness of instruction. To solve these problems, a system is needed that can provide effective advice while taking into consideration the parent's emotions by comparing and analyzing the child's movements based on a model trained by a professional.

[1699] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1700] In this invention, the server includes means for learning the motion data of professional athletes, means for receiving video of filmed sports movements, means for preprocessing the received video data and extracting features, means for comparative analysis using the extracted features with the motion data of the professional athletes, means for generating specific advice based on the analysis results, means for generating a comparison video that overlays the current form with an ideal form, means for outputting the generated advice and the comparison video, and means for recognizing the emotional state of the user using an emotion engine and adjusting the content of the advice based on the recognized emotional state. This enables parents to provide effective instruction based on the movements of professionals and feedback that takes into account their child's emotions, even if they do not have specialized knowledge of sports.

[1701] "Professional athlete exercise data" refers to recordings of exercises performed by skilled or expert athletes, typically stored in video format, that are used as training data for machine learning.

[1702] "Means for learning athletic data" refers to the process and technology for analyzing received athletic data of professional athletes, incorporating the features into a model, and learning the data.

[1703] "Means for receiving video" refers to an interface and protocol that allows the system to obtain video of sports exercises taken by users.

[1704] "Preprocessing" refers to the process of dividing the received video data into frames and extracting features such as key points and joint positions from each frame.

[1705] "Features" are important information extracted from movement data, and refer to data such as points and joint positions necessary for analyzing sports movements.

[1706] "Means for comparative analysis" refers to a technique that uses extracted features to compare children's movements with those of professional athletes and analyze differences and areas for improvement.

[1707] "Specific advice" refers to suggestions and instructions for improving exercise that are generated based on the analysis results and are provided to the user in an easy-to-understand format.

[1708] A "comparison video" refers to a video that displays the current (AsIs) and ideal (ToBe) exercise form superimposed on each other, allowing users to visually understand areas for improvement.

[1709] An "emotion engine" refers to technology and systems that analyze input data such as a user's voice and facial expressions to recognize their emotional state.

[1710] "Emotional state" refers to the user's current mental and emotional state and is a factor taken into account when adjusting feedback.

[1711] The present invention relates to a system that enables parents to support their children's sports improvement effectively and in a way that takes their emotions into consideration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the preferred embodiments of the present invention.

[1712] Hardware and software used

[1713] server

[1714] The server is the main component for learning motion data, data analysis, and emotion recognition. It uses the following software and frameworks:

[1715] Behavior analysis: TensorFlow, PyTorch

[1716] Emotion Recognition: Emotion analysis engines (e.g., IBM Watson or Microsoft Azure Emotion API)

[1717] Video processing: OpenPose, FFmpeg

[1718] Terminal

[1719] The terminal is the device through which the user captures video and communicates with the server. It uses the following technologies:

[1720] Video recording and uploading: Smartphone camera API (e.g., Android Camera2 API, iOS AVFoundation)

[1721] User Interface: React Native, Flutter

[1722] User

[1723] Users are primarily parents and children and perform the following actions:

[1724] Video recording and uploading

[1725] Check the advice

[1726] Providing emotional feedback

[1727] Specific processing of the system

[1728] 1. Learning from motion data of professional athletes

[1729] server

[1730] The server receives video data of professional athletes and splits it into frames. Next, OpenPose is used to extract features such as keypoints and joint positions from each frame. These features are then used to train machine learning models such as LSTM and CNN using TensorFlow or PyTorch. The trained models are stored in a database for later analysis.

[1731] Terminal

[1732] The device provides an interface for the user to upload the motion data to the server. Once the video upload is complete, the device notifies the user that the learning is complete.

[1733] User

[1734] Users can record the movements of professional athletes using their smartphones, upload the videos to the server via their devices, and then check their learning progress and receive notifications of completion.

[1735] 2. Shooting and importing children's sports movements

[1736] Terminal

[1737] The device provides a camera interface, allowing users to easily record their children's sports activities, and the captured video data is sent to the server via an upload option.

[1738] User

[1739] Users use their smartphones to record their children's sports activities and then upload the videos to a server.

[1740] 3. Behavior analysis and advice generation

[1741] server

[1742] The server divides the child's video data into frames and performs preprocessing. Next, it extracts features and compares them with the motion data of professional athletes. Based on this comparative analysis, specific advice is generated. Furthermore, a comparison video is generated that overlays the child's current and ideal form, and this is sent to the device.

[1743] 4. Emotion recognition and advice adjustment

[1744] server

[1745] The server uses an emotion engine to analyze voice and facial expression data from the device to recognize the user's emotional state. Based on the recognized emotional state, the server adjusts the content of the advice, for example, adding an encouraging message.

[1746] Terminal

[1747] The device displays the received advice, comparison videos, and emotion-sensitive feedback to the user. It also provides a video playback interface, allowing users to compare their current and ideal form.

[1748] User

[1749] The user can check the advice and feedback displayed on the device and use it to work with their child to improve their sports movements. If necessary, they can record the movements again and upload them to the server for reanalysis.

[1750] Specific examples

[1751] Improving children's baseball pitching movements

[1752] Initial setup and learning phase

[1753] Users film the pitching movements of famous baseball pitchers and upload them to the server via their devices. The server then uses this video data to train the model.

[1754] Filming the pitching motion

[1755] Users can record their child's pitching motion with their smartphone and upload the video to the server.

[1756] Behavior analysis and advice generation

[1757] The server analyzes the child's pitching video, generates specific improvements (e.g., "Raise your right elbow 3cm higher"), and creates a comparison video.

[1758] Emotion Recognition and Advice Adjustment

[1759] The server uses an emotion engine to analyze the user's emotions and add appropriate feedback (e.g., "Great progress! Let's take a breather and try again next time").

[1760] Prompt Sentence Examples

[1761] "Please explain the system that records a child's sports movements and compares them with the movement data of professional athletes. Also, please provide a concrete example of how a user would use the system."

[1762] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1763] Step 1:

[1764] Receiving and preprocessing motion data from professional athletes

[1765] server

[1766] Input: Video data of professional athletes

[1767] How it works: The server receives video data of a professional athlete, splits it into frames, and then uses OpenPose to extract features such as keypoints and joint positions from each frame.

[1768] Output: Extracted feature data

[1769] Step 2:

[1770] Model learning for motion data

[1771] server

[1772] Input: Extracted feature data

[1773] How it works: Use TensorFlow or PyTorch to train machine learning models such as LSTMs and CNNs. The training process involves providing feature data to the model and iteratively learning it.

[1774] Output: A trained machine learning model

[1775] Step 3:

[1776] Saving the trained model

[1777] server

[1778] Input: A trained machine learning model

[1779] Behavior: Once the training is complete, the model is saved in a database and used for subsequent behavior analysis.

[1780] Output: A saved machine learning model

[1781] Step 4:

[1782] Uploading model operation data

[1783] Terminal

[1784] Input: Professional athlete's athletic data

[1785] Operation: The device provides an interface for the user to upload operation data to the server. The user takes a video using the smartphone's camera API and then selects the upload option.

[1786] Output: Video data uploaded to the server

[1787] Step 5:

[1788] Learning completion notification

[1789] Terminal

[1790] Input: Learning completion notification sent from the server

[1791] Behavior: Notify the user that their learning is complete via push notification or in-app message.

[1792] Output: User notification

[1793] Step 6:

[1794] Filming and uploading children's sports activities

[1795] Terminal

[1796] Input: Video of a child's sports activity

[1797] Actions: The device records the child's sports movements through the camera interface. It displays an option to upload the captured video data to the server. The user uses the option to upload the video to the server.

[1798] Output: Child's video data uploaded to the server

[1799] Step 7:

[1800] Preprocessing and feature extraction of children's video data

[1801] server

[1802] Input: Children's video data

[1803] How it works: The server splits the received video data of the child into frames and extracts features from each frame. As a preprocessing step, it uses OpenPose to identify key points and joint positions.

[1804] Output: Extracted feature data

[1805] Step 8:

[1806] Behavior analysis and advice generation

[1807] server

[1808] Input: Extracted child feature data, saved machine learning model

[1809] How it works: The server uses the saved model to compare the child's feature data with that of a professional athlete. Based on the analysis results, specific advice is generated. In addition, a comparison video is created that overlays the child's current and ideal form.

[1810] Output: Specific advice, comparison video

[1811] Step 9:

[1812] Submitting advice and comparison videos

[1813] server

[1814] Input: Specific advice, comparison videos

[1815] Operation: The server sends the generated advice and comparison video to the device, where the data is displayed and the user can review it.

[1816] Output: Advice and comparison video sent to your device

[1817] Step 10:

[1818] Emotion Recognition and Advice Adjustment

[1819] server

[1820] Input: Voice and facial expression data that indicates the user's emotional state

[1821] How it works: The server uses the emotion engine to analyze voice and facial expression data from the device to recognize the user's emotional state. Based on the recognized emotional state, it adjusts the advice content, for example, adding an encouraging message.

[1822] Output: Tailored advice based on perceived emotional state

[1823] Step 11:

[1824] View advice and feedback

[1825] Terminal

[1826] Input: Adjusted advice received from the server, comparison video

[1827] How it works: The device displays the received advice, comparison videos, and emotion-sensitive feedback to the user. The device also provides an interface for playing videos that overlay AsIs and ToBe.

[1828] Output: Advice and feedback displayed to the user

[1829] Step 12:

[1830] Performance improvements and reanalysis

[1831] User

[1832] Input: Advice displayed on the device, comparison video

[1833] Movement: The user works with their child to improve their sports movements based on the advice provided. If necessary, the improved movements are filmed again and uploaded to the server for reanalysis.

[1834] Output: Improved sports motion video, reanalysis results

[1835] (Application example 2)

[1836] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1837] Modern factories lack an automatic analysis and improvement feedback system to improve the efficiency and accuracy of robotic work. They also lack the ability to provide appropriate feedback based on the emotional state of workers. This makes it difficult to simultaneously improve work efficiency and maintain workers' psychological stability.

[1838] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for learning professional motion data, a means for receiving video of filmed motions, and a means for preprocessing the received video data and extracting features. This makes it possible to analyze the robot's motions and provide specific improvement advice. The server also includes a means for recognizing emotions and adjusting feedback accordingly, and a means for filming and analyzing the worker's motions. This makes it possible to provide appropriate feedback according to the worker's psychological state.

[1839] "Professional motion data" is data that records the motions of skilled people or robots in a specific task or operation.

[1840] "Means for receiving video of captured action" refers to an interface that imports video of action captured by a camera or other capturing device into the system.

[1841] "Means for preprocessing received video data and extracting features" refers to the process of analyzing video data and extracting important points of movement, joint positions, etc.

[1842] "Means for comparative analysis with professional motion data using extracted features" refers to the process of finding and evaluating differences and similarities with professional motion data based on the analyzed features.

[1843] "Means for generating specific advice based on the analysis results" refers to a process for generating specific instructions for improving behavior based on the results of behavior comparison.

[1844] "Means for generating a comparison video that overlays current and ideal movements" refers to the process of creating a video that overlays current and ideal movements to visually show the difference.

[1845] "Means for outputting generated advice and comparison videos" refers to an interface that provides users with advice and comparison videos as the analysis results.

[1846] "Means for recognizing emotions and adjusting feedback accordingly" refers to the process of analyzing the worker's emotional state and appropriately changing the advice or feedback content based on the results.

[1847] "Means for filming and analyzing the worker's movements" refers to the process of filming the worker's movements using a camera or other device, and analyzing and evaluating those movements.

[1848] The following describes in detail the mode for carrying out this invention. This system analyzes the operation of a factory robot and provides feedback for improvement. It also has the ability to recognize the user's emotions and adjust the feedback according to their emotional state. This allows for both psychological stability for the worker and improved production efficiency.

[1849] Overall system overview

[1850] This system learns professional motion data, compares it with filmed robot motion data, and provides specific advice based on the results. It also has the ability to recognize the user's emotions and adjust the advice content according to their emotional state.

[1851] Hardware and software used

[1852] Hardware: smartphone, head-mounted display, camera

[1853] Software: Python, open source libraries (OpenCV, Keras)

[1854] Program processing flow

[1855] 1. Learning the movements of professionals

[1856] The server receives professional motion data and performs preprocessing. It divides the received video data into frames and extracts features such as key points and joint positions. It then uses a machine learning algorithm (e.g., LSTM or CNN) to learn from the professional motion data and train a model. The trained model is then stored in a database.

[1857] 2. Recording and reading the robot's movements

[1858] Users use smartphones or head-mounted displays to record robot movements in the factory. The captured data is uploaded to a server, which then preprocesses the received robot video data, dividing it into frames and extracting features.

[1859] 3. Behavior analysis and advice generation phase

[1860] The server uses the saved model to compare the robot's motion data with professional motion data. Based on the results of the comparative analysis, it generates specific advice (e.g., "Delay your right elbow by two seconds"). It also generates a comparison video that overlays the AsIs (current) and ToBe (ideal) forms. The advice and comparison video are sent to the device.

[1861] 4. Emotion Recognition and Advice Adjustment Phase

[1862] The server uses an emotion engine to analyze input data (e.g., voice, facial expressions) from the device and recognize the user's emotional state. Based on the recognized emotional state, the server adjusts the advice content. For example, if the user is feeling frustrated, it adds an encouraging message. It provides feedback such as, "Great progress. Let's take a breather and try again next time."

[1863] Example: Improving the operation of a robot arm

[1864] Initial setup and learning phase

[1865] Users can record professional robot arm movements and upload them to the server via their devices. The server then preprocesses the received video data and learns the professional movement data.

[1866] Filming the robot arm in action

[1867] The user continuously records the robot arm's movements with their smartphone and uploads the video data to a server.

[1868] Behavior analysis and advice generation

[1869] The server analyzes the video of the robot arm and compares it with professional motion data, generates specific improvements (e.g., "Delay the right elbow by 2 seconds"), and creates a comparison video of the AsIs and ToBe.

[1870] Emotion Recognition and Advice Adjustment

[1871] The server uses an emotion engine to analyze the user's emotional state and, if the user is feeling frustrated, adds an encouraging message such as, "Great progress! Now take a breather and try again."

[1872] Displaying and Implementing Results

[1873] The user can then improve the robot's behavior by checking the advice, comparison videos, and emotional feedback displayed on the device. After the improvements are made, the robot's behavior is filmed again and uploaded to the server for reanalysis.

[1874] This system allows workers to efficiently improve the robot's operation, thereby increasing work quality and production efficiency.

[1875] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1876] Step 1: Learning from professional motion data

[1877] In step 1, the server receives the professional's motion data and performs preprocessing. The input is the professional's motion video data, which is split into frames and features such as key points and joint positions are extracted. A machine learning algorithm (e.g., LSTM, CNN) is used to learn the features and train a model. The trained model is stored in a database. The output of this step is the features and the trained model.

[1878] Step 2: Film the robot's movements and upload the data

[1879] In step 2, the user uses a smartphone or head-mounted display to film the robot's operation in the factory. The input is video data containing the actual robot operation. The user uploads the filmed video to the server via their device. The input video data is transferred to the server. The output of this step is video data of the robot's operation stored on the server.

[1880] Step 3: Preprocessing of motion data and feature extraction

[1881] In step 3, the server preprocesses the input robot video data and splits it into frames. This allows for detailed analysis of each stage of the movement. Next, features such as key points and joint positions are extracted. The input is the robot's movement video data, and the output is the extracted features. Specific operations include splitting and analyzing the video frames using OpenCV.

[1882] Step 4: Behavior analysis and comparison

[1883] In step 4, the server uses the stored professional model to perform comparative analysis of the robot's motion data. The inputs are the extracted features and the professional model, and by comparing and analyzing these, differences in motion and areas for improvement are identified. Specifically, a machine learning model is used to calculate the motion match rate and descriptor. The output is the analysis results and areas for improvement.

[1884] Step 5: Generate advice and comparison videos

[1885] In step 5, the server generates specific advice based on the analysis results. For example, this may include instructions such as "Delay your right elbow for another two seconds." It also generates a comparison video that overlays the current and ideal movements. The input is the analysis results, and the output is specific advice and a comparison video. In this step, the two videos are overlaid using video editing software.

[1886] Step 6: View the results

[1887] In step 6, the server sends the generated advice and comparison video to the user's device. The input is the generated advice and comparison video, and the user receives this information through their device. Specifically, feedback is provided to the user via a notification function or interface. The output is advice and comparison video that the user can visually confirm.

[1888] Step 7: Emotion recognition and feedback regulation

[1889] In step 7, the server uses an emotion engine to analyze the user's emotional state. For example, it analyzes the user's voice and facial expression data to recognize emotions. The input is emotional data from the user, and the feedback content is adjusted based on the analysis results. For example, if the user is feeling frustrated, an encouraging message such as "Great progress! Let's take a breather and try again next time" is added. The output is the adjusted feedback.

[1890] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1891] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1892] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1893] [Fourth embodiment]

[1894] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1895] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1896] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1897] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1898] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1899] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1900] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1901] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1902] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1903] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1904] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1905] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1906] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1907] The following describes in detail an embodiment of the present invention. This system uses AI to provide parents with specific advice to help their children improve their sports skills.

[1908] Overall system overview

[1909] This system learns the motion data of professional athletes, compares it with filmed footage of children's sports movements, and provides specific advice based on the results, enabling parents to provide effective instruction even if they have no experience in sports.

[1910] Program processing flow

[1911] 1. Learning the movements of professional athletes

[1912] server

[1913] 1. Receive and preprocess motion data from professional athletes. Divide the received video data into frames and extract features such as key points and joint positions.

[1914] 2. Use machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes and train the model.

[1915] 3. Save the trained model in a database.

[1916] Terminal

[1917] 1. Provides an interface for uploading operational data to a server.

[1918] 2. Display a notification to the user that the learning is complete.

[1919] User

[1920] 1. Record the motion data of a professional athlete and upload it to a server via the device.

[1921] 2. Check your learning progress and completion notifications.

[1922] 2. Filming and reading children's sports movements

[1923] Terminal

[1924] 1. Provide a camera interface for capturing children's sports activities.

[1925] 2. Display the option to upload the captured video data to the server.

[1926] User

[1927] 1. Take photos of your child's sports movements with your smartphone.

[1928] 2. Upload the captured video data to the server.

[1929] 3. Behavior analysis and advice generation phase

[1930] server

[1931] 1. Preprocess the received video data of the child, divide it into frames, and extract features.

[1932] 2. Use the saved model to compare and analyze the motion data of children and professional athletes.

[1933] 3. Based on the analysis results, specific advice (e.g., "Raise your right elbow 3 cm higher") is generated.

[1934] 4. Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[1935] 5. Send advice and comparison videos to your device.

[1936] 4. Results display and advice phase

[1937] Terminal

[1938] 1. The advice received from the server and comparison video are displayed to the user.

[1939] 2. Provide an interface for playing videos that overlay AsIs and ToBe.

[1940] User

[1941] 1. Check the advice displayed on your device and the comparison video.

[1942] 2. Work with your child to improve their sports movements using the advice provided.

[1943] 3. If necessary, re-photograph the improved behavior and re-analyze it using the same process.

[1944] Example: Improving children's baseball pitching technique

[1945] Initial setup and learning phase

[1946] Users can record the pitching motion of a top baseball pitcher and upload it to the server via their device. The server then preprocesses the received video data and learns the motion data of the professional pitcher.

[1947] Filming the pitching motion

[1948] The user continuously records the child's pitching motion with a smartphone and uploads the video data to a server.

[1949] Behavior analysis and advice generation

[1950] The server analyzes the child's pitching video and compares it with the motion data of a professional pitcher. It generates specific improvements (e.g., "Raise your right elbow 3 cm higher") and creates a video comparing AsIs and ToBe.

[1951] Displaying and Implementing Results

[1952] The user checks the advice and comparison video displayed on the device and practices together with the child. After the improvement, the movement is filmed again and uploaded to the server for reanalysis.

[1953] This system allows parents with no sports experience to provide effective advice and support their children's sports improvement.

[1954] The processing flow will be explained below.

[1955] Program processing flow

[1956] 1. Learning the movements of professional athletes

[1957] Step 1: Prepare the operation data

[1958] User: Uses a camera to record the movements of a professional athlete and saves the video file on the device.

[1959] Step 2: Uploading the behavior data

[1960] Terminal: Provides an interface for uploading operational data to the server.

[1961] User: Select the captured motion data and upload it to the server.

[1962] Device: Sends motion data to the server.

[1963] Step 3: Preprocessing the behavioral data

[1964] Server: Divides the received video file into frames and extracts key points and joint positions.

[1965] Server: Stores the preprocessed data in a database.

[1966] Step 4: Training the learning model

[1967] Server: Uses machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes.

[1968] Server: Saves the trained model in a database.

[1969] Step 5: Notification of completion of study

[1970] Server: Notifies the device that learning is complete.

[1971] Terminal: Display a message to the user that the training is complete.

[1972] 2. Filming and reading children's sports movements

[1973] Step 1: Capture the action

[1974] User: Takes photos of children's sports activities with a smartphone.

[1975] Terminal: Provides a camera interface for capturing action.

[1976] Step 2: Uploading the behavior data

[1977] On your device: Displays the option to upload the captured video file to the server.

[1978] User: Select a video file and upload it to the server.

[1979] Device: Sends the video file to the server.

[1980] 3. Behavior analysis and advice generation phase

[1981] Step 1: Preprocessing the video data

[1982] Server: Divides the received video file of the child into frames and extracts features.

[1983] Step 2: Comparative analysis of behavior

[1984] Server: Uses the trained model to compare children's motion data with that of professional athletes.

[1985] Server: Quantify the difference in each frame and identify specific areas for improvement.

[1986] Step 3: Generate Advice

[1987] Server: Generates specific advice based on the results of the comparative analysis.

[1988] Server: Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[1989] Step 4: Submit the results

[1990] Server: Sends the generated advice and comparison videos to the device.

[1991] 4. Results display and advice phase

[1992] Step 1: View the results

[1993] Terminal: Displays the advice received from the server and the comparison video to the user.

[1994] Device: Provides an interface for playing videos that overlay AsIs and ToBe.

[1995] Step 2: Operation check and improvement steps

[1996] User: Check the advice and comparison video displayed on the device.

[1997] User: Work with their child to improve their sports movements using the advice provided.

[1998] Step 3: Feedback and Retraining

[1999] User: Re-photograph the improved behavior and re-analyze it using the same process.

[2000] Device: Sends new video to the server and requests reanalysis.

[2001] As described above, a system is constructed in which the server, terminals, and users work together to support children in improving their sports movements.

[2002] Example 1

[2003] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2004] Conventional systems supporting children's sports improvement have the problem that it is difficult to provide effective advice to parents who lack sports experience or expertise. Furthermore, analyzing the recorded video data requires specialized expertise and is not easy to do. Furthermore, the system lacks a means to visually show specific areas for improvement and an interface for reassessment, making the improvement process difficult.

[2005] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2006] In this invention, the server includes: means for learning motion data of professional athletes; means for receiving video of filmed sports movements; means for preprocessing the received video data and extracting features; means for comparative analysis with the motion data of the professional athletes using the extracted features; means for generating specific advice based on the analysis results; means for generating a comparison video that overlays the current form with an ideal form; means for outputting the generated advice and the comparison video; means for providing a camera interface for filming the child's sports movements; and means for inputting a prompt sentence for generating an analysis request to the AI ​​model. This enables parents, even without experience or knowledge of sports, to effectively analyze their children's sports movements, provide specific and visual advice, and reassess their children's sports movements.

[2007] "Professional athlete motion data" refers to data that records the motions of athletes with high levels of skill.

[2008] "Means for receiving" refers to technical means for taking in data from the outside, such as a server or storage device.

[2009] "Preprocessing" refers to the process of converting received data into a format that is easy to analyze.

[2010] A "feature" is a data point or value that contains important information in data analysis.

[2011] A "comparative analysis tool" is a tool that compares two or more data sets to analyze commonalities and differences.

[2012] "Specific advice" refers to specific, practical instructions or suggestions.

[2013] A "comparison video" is a video that displays your current form and ideal form at the same time for comparison.

[2014] "Means of output" refers to the technical means for providing data or information to the user.

[2015] A "camera interface" is a user interface for operating a camera.

[2016] An "AI model" is an algorithm trained using machine learning or deep learning.

[2017] A "prompt sentence" is an input sentence for a generative AI model, and is text that specifies an analysis request.

[2018] The following describes in detail the embodiments of the present invention. This system uses AI to provide specific advice to parents to help their children improve their sports skills. Even if the parents have no experience in sports, the system can effectively analyze their children's movements, provide specific and visual advice, and perform reassessments.

[2019] Overall system overview

[2020] This system learns the motion data of professional athletes, compares it with filmed footage of children's sports movements, and provides specific advice based on the results, allowing parents to provide effective instruction even if they have no experience in sports.

[2021] Program processing flow

[2022] 1. Learning the movements of professional athletes

[2023] server

[2024] 1. The server receives the motion data of professional athletes and performs preprocessing. This preprocessing involves dividing the video data into frames and extracting features such as key points and joint positions from each frame using open-source pose estimation technology such as OpenPose.

[2025] 2. The server uses the preprocessed feature data to train a model using machine learning algorithms such as LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network).

[2026] 3. The trained model is stored in a database by the server.

[2027] Terminal

[2028] 1. The device provides an interface for uploading professional athletes' motion data to the server, allowing users to easily select and upload videos.

[2029] 2. The device displays a learning completion notification to the user.

[2030] User

[2031] 1. The user uses their smartphone to record the movements of a professional athlete as an example, and then uploads the video data to the server via the device.

[2032] 2. The user checks the learning progress and completion notification displayed on the device.

[2033] 2. Filming and reading children's sports movements

[2034] Terminal

[2035] 1. The device provides a camera interface that allows the user to record their child's sports activities. For example, the app can activate the camera function and display a recording button to start recording.

[2036] 2. The device will display an option to upload the captured video data to the server.

[2037] User

[2038] 1. The user uses the smartphone camera to capture a video of their child's sports activity, for example, recording a video of their child pitching a baseball.

[2039] 2. The user uploads the captured video data to the server. By simply selecting the video and pressing the upload button according to the device interface, the data is sent to the server.

[2040] 3. Behavior analysis and advice generation phase

[2041] server

[2042] 1. The server preprocesses the received video data of the child, divides the video into frames, and extracts features. Specifically, it calculates the joint positions for each frame using OpenPose or similar.

[2043] 2. The server uses the stored trained model to compare and analyze the child's motion data with that of a professional athlete. This comparison evaluates the similarities and differences between the two motions based on the extracted features.

[2044] 3. The server generates specific advice based on the analysis results. For example, it may suggest to a child that they should "raise their right elbow 3cm higher" when pitching.

[2045] 4. The server generates a comparison video that overlays the AsIs (current) and ToBe (ideal) forms, making it visually easier to see which areas need improvement.

[2046] 5. The server sends the generated advice and comparison video to the device.

[2047] 4. Results display and advice phase

[2048] Terminal

[2049] 1. The device displays the advice and comparison video received from the server to the user. Specifically, the application provides a playback function for the advice text and comparison video.

[2050] 2. The device provides an interface for playing videos that overlay AsIs and ToBe, allowing users to easily play the videos and view them as many times as they like.

[2051] User

[2052] 1. The user checks the advice and comparison video displayed on the device. For example, they play the video to visually understand the suggested improvements.

[2053] 2. The user works with their child to improve their sports movements based on the provided advice. Specifically, for example, the user practices to correct the child's pitching form.

[2054] 3. If necessary, film the motion after the improvement and re-analyze it using the same process. Record the corrected pitching motion on video again, upload it to the server, and receive further advice.

[2055] Examples of prompt statements

[2056] Here are some examples of prompts that users can enter into a generative AI model:

[2057] "Please analyze my child's baseball pitching motion and let me know what needs to be improved. I have uploaded a video from the link below. I have also added a video of a professional pitcher, so you can compare it."

[2058] This allows the AI ​​model to provide appropriate analysis and advice based on the video data provided.

[2059] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2060] Step 1:

[2061] Receiving and preprocessing motion data from professional athletes

[2062] The server receives the motion data of the professional athlete, which includes video files uploaded by the user through an interface provided by the device.

[2063] Input: Video data of professional athletes' movements

[2064] The server preprocesses the received video data. This preprocessing involves dividing the video into frames and extracting key points and joint positions from each frame. Specifically, it uses open-source pose estimation technology such as OpenPose to detect the main joint points of the human body for each frame.

[2065] Data processing: Split the video into frames and extract joint points

[2066] Output: Features (keypoints and joint positions)

[2067] Step 2:

[2068] Training a machine learning model

[2069] The server uses the preprocessed feature data to train machine learning models, such as LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network). Training is necessary to capture the movement patterns of professional athletes.

[2070] Input: Preprocessed feature data

[2071] The server performs training using the given feature data and target data. During the training process, iterative parameter updates are performed using the backpropagation algorithm.

[2072] Data Computing: Model Training with Machine Learning Algorithms

[2073] Output: A trained model

[2074] Step 3:

[2075] Saving a trained model

[2076] The server stores the trained model in a database, allowing it to be quickly used in future analyses.

[2077] Input: A trained model

[2078] Data processing: Converting the model into a storage format

[2079] Output: Model saved in database

[2080] Step 4:

[2081] Filming children's sports

[2082] The user takes pictures of their child's sports activities using the smartphone camera and uses the camera interface provided by the device.

[2083] Input: Children's sports activities

[2084] The user activates the device's camera function and presses the record button to record the action. Once the recording is complete, it is saved as video data on the device.

[2085] Data generation: video data of sports movements

[2086] Output: Video data of children's sports movements

[2087] Step 5:

[2088] Uploading video data

[2089] The user uploads the captured video data to the server through the device, which displays an upload option.

[2090] Input: Video data of children's sports movements

[2091] The user selects a video from the terminal interface and presses the "upload" button to send the data to the server.

[2092] Data transfer: Uploading video data

[2093] Output: Video data of children's sports movements stored on the server

[2094] Step 6:

[2095] Preprocessing children's video data

[2096] The server preprocesses the received video data of the child by dividing the video into frames and extracting features (keypoints and joint positions) from each frame.

[2097] Input: Video data of children's sports movements stored on a server

[2098] Preprocessing includes extracting joint points using OpenPose, etc. The result of preprocessing is feature data for each frame.

[2099] Data processing: Split the video into frames and extract joint points

[2100] Output: Feature data of children's sports movements

[2101] Step 7:

[2102] Comparative analysis of motion data

[2103] The server uses the stored trained model to compare and analyze the motion data of children and that of professional athletes.

[2104] Input: feature data of children's sports movements, pre-trained model

[2105] The server evaluates the similarities and differences based on the extracted feature data. Specifically, it inputs the feature values ​​of each frame into a trained model and calculates the differences from the standard movements of professional athletes.

[2106] Data Computation: Comparative Analysis of Features

[2107] Output: Analysis results

[2108] Step 8:

[2109] Generating Specific Advice

[2110] The server generates specific advice based on the analysis results, such as "Raise your right elbow another 3 cm."

[2111] Input: Analysis results

[2112] Based on the analysis results, the server determines which parts should be improved and how, and generates advice in text format.

[2113] Data Generation: Specific Advice Text

[2114] Output: Specific advice

[2115] Step 9:

[2116] Comparison video generation

[2117] The server generates a comparison video that overlays the AsIs (current) and ToBe (ideal) forms, making it easier to visually identify areas for improvement.

[2118] Input: Video data of children's sports movements, pre-trained model

[2119] The server displays the current form and the ideal form side by side and combines the two videos to generate a single comparison video.

[2120] Data calculation: Video frame comparison and integration

[2121] Output: Comparison video

[2122] Step 10:

[2123] Advice and comparison video output

[2124] The server sends the generated advice and comparison video to the terminal, where the user can review it.

[2125] Input: Specific advice, comparison videos

[2126] The server transmits data to the terminal, and the terminal displays the received advice and video.

[2127] Data output: Sending advice and comparison videos

[2128] Output: Advice and comparison video displayed on device

[2129] (Application example 1)

[2130] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2131] Conventional sports instruction methods have the problem that it is difficult for parents and coaches to provide effective advice and instruction if they do not have specialized sports knowledge or skills. Furthermore, instruction services offered by sporting goods stores and fitness facilities are generally expensive, making them inaccessible to many people. Checking and improving sports form at physical stores is particularly time-consuming and requires a large burden on equipment and specialized staff. There is a need for a system that can solve these issues and enable ordinary parents and users to effectively improve their sports movements.

[2132] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2133] In this invention, the server includes: means for learning motion data of professional athletes; means for receiving video of filmed sports movements; means for preprocessing the received video data and extracting features; means for comparing and analyzing the extracted features with the motion data of the professional athletes; means for generating specific advice based on the analysis results; means for generating a comparison video that overlays the current form with an ideal form; means for outputting the generated advice and the comparison video; means for providing an interface that allows users at physical stores to film their sports movements and upload them to the cloud; and means for users to re-film and upload their improved movements and receive new advice to support improvement of their sports movements. This not only enables parents and physical store users with no sports experience to effectively improve their sports movements, but also enables efficient and cost-effective sports instruction services at physical stores.

[2134] "Professional athlete motion data" refers to data that records the motions of a professional athlete, and includes the feature quantities of those motions.

[2135] "Means for learning movement data" refers to the process of using machine learning algorithms to analyze the movement data of professional athletes and use it as a model for training.

[2136] "Means for receiving video" refers to the process by which the user sends the video data they have taken to the cloud or a server, and the server receives it.

[2137] "Preprocessing" refers to the process of dividing the received video data into frames and extracting important key points and joint positions.

[2138] "Features" are an important part in analyzing sports movements and include position information of key points and joints.

[2139] The "means for comparative analysis" refers to a process of comparing the motion data of a professional athlete with the photographed sports motion data using the extracted feature amounts.

[2140] "Means for generating advice" refers to the process of generating specific improvements and advice in text or video format based on the results of the comparative analysis.

[2141] "Method for generating a comparison video" refers to the process of generating a video that overlays the current (AsIs) and ideal (ToBe) forms to visually show the differences.

[2142] "Means for outputting a comparison video" refers to a process for providing the generated comparison video to a user.

[2143] "Brick-and-mortar users" refers to customers who visit physical stores, such as sporting goods stores or fitness facilities.

[2144] An "interface" is the part of the system that provides the screen and operating means for users to film their sports movements and upload them to the cloud.

[2145] "Cloud" refers to remote servers for storing, computing, and managing data over the Internet.

[2146] "Means for providing advice" refers to the process by which the generated advice is displayed to the user so that they can review it.

[2147] The following describes in detail the mode for carrying out the present invention. This system aims to support the improvement of sports movements in physical stores, and analyzes sports form and provides specific advice. By using this system, parents and general users can effectively improve their sports movements.

[2148] Overall system overview

[2149] The system learns from the motion data of professional athletes, compares and analyzes the sports motions of users filmed in physical stores, and generates and provides specific advice based on the results.

[2150] Hardware and software used

[2151] Hardware

[2152] Smartphone: A device that allows users to record and upload videos of their sports activities.

[2153] Cloud server: A remote server for storing data, training models, and performing analytical processing.

[2154] software

[2155] LSTM (Long Short-Term Memory): Used for time series analysis of behavioral data.

[2156] CNN (Convolutional Neural Network): Used to extract movement features.

[2157] OpenPose: Used to extract keypoints and joint positions from videos.

[2158] Flask / Django: Used for backend processing.

[2159] React Native: Used for the front end of smartphone applications.

[2160] Data processing and calculation

[2161] Learning from motion data of professional athletes

[2162] The server receives video data of professional athletes and performs preprocessing. During preprocessing, the video is divided into frames and key points and joint positions are extracted using OpenPose. Then, LSTM and CNN are used to learn the motion data of the professional athletes and train a model. The trained model is stored in a cloud database.

[2163] Filming and uploading users' sports activities

[2164] Users record their sports movements using their smartphones and upload them to the cloud. The server preprocesses the received video data and extracts features.

[2165] Behavior analysis and advice generation

[2166] The server uses the saved model to compare and analyze the user's motion data with that of professional athletes. Based on the analysis results, it generates specific improvement advice. It also generates a comparison video that overlays the current (AsIs) and ideal (ToBe) form.

[2167] Displaying results and providing advice

[2168] The smartphone app receives advice from the server and displays the comparison video to the user. The user can improve their sports movements based on the advice, and if necessary, record the video again and reanalyze it using the same process.

[2169] Specific examples

[2170] Example 1: Improving a child's basketball shooting technique

[2171] Parents can take a photo of their child's shooting movements on their smartphone and upload it to the app.

[2172] The app compares the movements to those of professional basketball players and advises, "Bend your knees a little."

[2173] A comparison video between AsIs and ToBe is also generated, allowing parents and children to check and practice together.

[2174] Prompt Sentence Examples

[2175] Simply record a professional basketball player's shooting technique and upload it. Then, record your child's shooting technique with your smartphone and upload it to the app. We'll provide you with relevant advice and comparison videos.

[2176] The system enables effective and cost-effective sports instruction in brick-and-mortar locations, promoting improved sports form.

[2177] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2178] Step 1:

[2179] Receiving and preprocessing motion data from professional athletes

[2180] The server receives video data of professional athletes. The received video data is divided into frames and key points and joint positions are extracted using OpenPose. As a result, the input is the video data, and the output is a set of key points and joint positions.

[2181] Specific movements: Analyze the movements of athletes frame by frame, and extract key points and joint positions for each element of the movement.

[2182] Step 2:

[2183] Learning from behavioral data and training models

[2184] The server uses the extracted keypoints and joint locations to train a machine learning model that combines LSTM and CNN. The trained model is stored in a cloud database. The input is the keypoints and joint locations obtained in step 1, and the output is the trained machine learning model.

[2185] Specific operation: Time series data is learned using LSTM, and spatial information is analyzed using CNN.

[2186] Step 3:

[2187] Filming of users' sports movements and uploading of video data

[2188] Users record their sports movements with their smartphones and upload the video data to a cloud server. The input is the video data recorded by the user, and the output is the movement data stored on the server.

[2189] Specific operation: Video is taken using the smartphone's camera function, and the video data is sent to the server via the application.

[2190] Step 4:

[2191] Video data preprocessing and feature extraction

[2192] The server preprocesses the received video data, splits it into frames, and extracts features. The input is the uploaded video data, and the output is the extracted keypoints and joint positions.

[2193] Specific operation: Divide the video into frames and extract key points and joint positions using OpenPose.

[2194] Step 5:

[2195] Comparative analysis of operational data and generation of advice

[2196] The server uses the stored model to compare and analyze the user's motion data with that of professional athletes. Specific advice is generated based on the analysis results. The input is the data after feature extraction, and the output is the analysis results and generated advice.

[2197] Specific behavior: Using machine learning models, behavior data is analyzed and specific improvement advice is generated.

[2198] Step 6:

[2199] Comparison video generation

[2200] The server generates a comparison video that overlays the current (AsIs) and ideal (ToBe) form. The input is the analyzed feature data, and the output is a comparison video of the AsIs and ToBe form.

[2201] Specific actions: Using a video editing tool, the two action data are overlaid into one video.

[2202] Step 7:

[2203] Providing advice and comparison videos

[2204] The terminal displays the advice and comparison video received from the server to the user. The input is the output data from the server, and the output is the advice and comparison video that the user can view.

[2205] Specific behavior: Provides an interface through the application that displays advice and videos.

[2206] Step 8:

[2207] Re-filming and re-analyzing the movement

[2208] The user improves their sports movements based on the advice from the server, then re-films and uploads the improved movements. The server then performs a re-analysis using the same process. The input is the improved video data, and the output is new analysis results and advice.

[2209] Specific actions: Record the video again, upload it, and provide new analysis and advice.

[2210] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2211] The following describes in detail an embodiment of the present invention. This system uses AI to provide parents with specific advice and feedback that takes into account the user's emotions, in order to help their children improve their sports skills.

[2212] Overall system overview

[2213] This system studies the motion data of professional athletes, compares it with filmed footage of children's sports movements, and provides specific advice based on the results. It also has the ability to recognize the user's emotions and adjust the advice content according to their emotional state. This allows parents, even those with no experience in sports, to provide effective instruction that takes emotions into consideration.

[2214] Program processing flow

[2215] 1. Learning the movements of professional athletes

[2216] server

[2217] 1. Receive and preprocess motion data from professional athletes. Divide the received video data into frames and extract features such as key points and joint positions.

[2218] 2. Use machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes and train the model.

[2219] 3. Save the trained model in a database.

[2220] Terminal

[2221] 1. Provides an interface for uploading operational data to a server.

[2222] 2. Display a notification to the user that the learning is complete.

[2223] User

[2224] 1. Record the motion data of a professional athlete and upload it to a server via the device.

[2225] 2. Check your learning progress and completion notifications.

[2226] 2. Filming and reading children's sports movements

[2227] Terminal

[2228] 1. Provide a camera interface for capturing children's sports activities.

[2229] 2. Display the option to upload the captured video data to the server.

[2230] User

[2231] 1. Take photos of your child's sports movements with your smartphone.

[2232] 2. Upload the captured video data to the server.

[2233] 3. Behavior analysis and advice generation phase

[2234] server

[2235] 1. Preprocess the received video data of the child, divide it into frames, and extract features.

[2236] 2. Use the saved model to compare the motion data of a child with that of a professional athlete.

[2237] 3. Based on the analysis results, specific advice (e.g., "Raise your right elbow 3 cm higher") is generated.

[2238] 4. Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[2239] 5. Send advice and comparison videos to your device.

[2240] 4. Emotion Recognition and Advice Adjustment Phase

[2241] server

[2242] 1. Using an emotion engine, input data from the device (e.g., voice, facial expressions) is analyzed to recognize the user's emotional state.

[2243] 2. Tailor the advice based on the perceived emotional state, for example adding an encouraging message if the user is feeling frustrated.

[2244] Terminal

[2245] 1. The advice received from the server, comparison videos, and emotion-sensitive feedback are displayed to the user.

[2246] 2. Provide an interface for playing videos that overlay AsIs and ToBe.

[2247] User

[2248] 1. Check the advice displayed on the device, comparison videos, and emotionally sensitive feedback.

[2249] 2. Work with your child to improve their sports movements using the advice provided.

[2250] 3. If necessary, re-photograph the improved behavior and re-analyze it using the same process.

[2251] Example: Improving children's baseball pitching technique

[2252] Initial setup and learning phase

[2253] Users can record the pitching motion of a top baseball pitcher and upload it to the server via their device. The server then preprocesses the received video data and learns the motion data of the professional pitcher.

[2254] Filming the pitching motion

[2255] The user continuously records the child's pitching motion with a smartphone and uploads the video data to a server.

[2256] Behavior analysis and advice generation

[2257] The server analyzes the child's pitching video and compares it with the motion data of a professional pitcher. It generates specific improvements (e.g., "Raise your right elbow 3 cm higher") and creates a video comparing AsIs and ToBe.

[2258] Emotion Recognition and Advice Adjustment

[2259] The server uses an emotion engine to analyze the user's emotional state and, if the user is feeling frustrated, adds an encouraging message such as, "Great progress! Now take a breather and try again."

[2260] Displaying and Implementing Results

[2261] The user checks the advice and comparison video displayed on the device, as well as the emotional feedback, and practices together with the child. After the improvement, the child's movements are filmed again and uploaded to the server for reanalysis.

[2262] This system allows parents with no prior experience in sports to provide effective and emotionally sensitive advice to help their children improve in sports.

[2263] The processing flow will be explained below.

[2264] Program processing flow

[2265] 1. Learning the movements of professional athletes

[2266] Step 1: Prepare the operation data

[2267] User: Uses a camera to record the movements of a professional athlete and saves the video file on the device.

[2268] Step 2: Uploading the behavior data

[2269] Terminal: Provides an interface for uploading operational data to the server.

[2270] User: Select the captured motion data and upload it to the server.

[2271] Device: Sends motion data to the server.

[2272] Step 3: Preprocessing the behavioral data

[2273] Server: Divides the received video file into frames and extracts key points and joint positions.

[2274] Server: Stores the preprocessed data in a database.

[2275] Step 4: Training the learning model

[2276] Server: Uses machine learning algorithms (e.g., LSTM, CNN) to learn from the motion data of professional athletes.

[2277] Server: Saves the trained model in a database.

[2278] Step 5: Notification of completion of study

[2279] Server: Notifies the device that learning is complete.

[2280] Terminal: Display a message to the user that the training is complete.

[2281] 2. Filming and reading children's sports movements

[2282] Step 1: Capture the action

[2283] User: Takes photos of children's sports activities with a smartphone.

[2284] Terminal: Provides a camera interface for capturing action.

[2285] Step 2: Uploading the behavior data

[2286] On your device: Displays the option to upload the captured video file to the server.

[2287] User: Select a video file and upload it to the server.

[2288] Device: Sends the video file to the server.

[2289] 3. Behavior analysis and advice generation phase

[2290] Step 1: Preprocessing the video data

[2291] Server: Divides the received video file of the child into frames and extracts features.

[2292] Step 2: Comparative analysis of behavior

[2293] Server: Uses the trained model to compare children's motion data with that of professional athletes.

[2294] Server: Quantify the difference in each frame and identify specific areas for improvement.

[2295] Step 3: Generate Advice

[2296] Server: Generates specific advice based on the results of the comparative analysis.

[2297] Server: Generate a comparison video that overlays the AsIs (current) and ToBe (ideal) forms.

[2298] Step 4: Submit the results

[2299] Server: Sends the generated advice and comparison videos to the device.

[2300] 4. Emotion recognition and feedback regulation phase

[2301] Step 1: Enter user emotion data

[2302] User: Uses the device's camera and microphone to provide facial expressions and audio during the training session.

[2303] Device: Sends emotion data to the server.

[2304] Step 2: Sentiment Analysis

[2305] Server: Analyzes the received emotion data using the emotion engine and recognizes the user's emotional state (e.g., joy, frustration).

[2306] Server: Stores the recognized emotional state in a database.

[2307] Step 3: Adjust your advice

[2308] Server: Adjust the generated advice based on the user's emotional state. For example, if the user is feeling frustrated, add an encouraging message.

[2309] Server: Sends advice content including emotion-sensitive feedback to the device.

[2310] Step 4: View the results

[2311] Device: The advice received from the server, comparison videos, and emotional feedback are displayed to the user.

[2312] Device: Provides an interface for playing videos that overlay AsIs and ToBe.

[2313] Step 5: Check operation and make improvements

[2314] User: View advice and comparison videos displayed on the device, as well as emotionally sensitive feedback.

[2315] User: Work with their child to improve their sports movements using the advice provided.

[2316] Step 6: Feedback and Retraining

[2317] User: Re-photograph the improved behavior and re-analyze it using the same process.

[2318] Device: Sends new video to the server and requests reanalysis.

[2319] As described above, a system is constructed in which the server, terminal, and user work together to support children in improving their sports movements and provide feedback that takes emotions into consideration.

[2320] Example 2

[2321] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2322] Conventional sports coaching systems have the problem that parents lack specialized knowledge of sports, making it difficult to provide effective instruction. Furthermore, feedback is often provided that ignores the child's emotional state, resulting in a decrease in the effectiveness of instruction. To solve these problems, a system is needed that can provide effective advice while taking into consideration the parent's emotions by comparing and analyzing the child's movements based on a model trained by a professional.

[2323] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2324] In this invention, the server includes means for learning the motion data of professional athletes, means for receiving video of filmed sports movements, means for preprocessing the received video data and extracting features, means for comparative analysis using the extracted features with the motion data of the professional athletes, means for generating specific advice based on the analysis results, means for generating a comparison video that overlays the current form with an ideal form, means for outputting the generated advice and the comparison video, and means for recognizing the emotional state of the user using an emotion engine and adjusting the content of the advice based on the recognized emotional state. This enables parents to provide effective instruction based on the movements of professionals and feedback that takes into account their child's emotions, even if they do not have specialized knowledge of sports.

[2325] "Professional athlete exercise data" refers to recordings of exercises performed by skilled or expert athletes, typically stored in video format, that are used as training data for machine learning.

[2326] "Means for learning athletic data" refers to the process and technology for analyzing received athletic data of professional athletes, incorporating the features into a model, and learning the data.

[2327] "Means for receiving video" refers to an interface and protocol that allows the system to obtain video of sports exercises taken by users.

[2328] "Preprocessing" refers to the process of dividing the received video data into frames and extracting features such as key points and joint positions from each frame.

[2329] "Features" are important information extracted from movement data, and refer to data such as points and joint positions necessary for analyzing sports movements.

[2330] "Means for comparative analysis" refers to a technique that uses extracted features to compare children's movements with those of professional athletes and analyze differences and areas for improvement.

[2331] "Specific advice" refers to suggestions and instructions for improving exercise that are generated based on the analysis results and are provided to the user in an easy-to-understand format.

[2332] A "comparison video" refers to a video that displays the current (AsIs) and ideal (ToBe) exercise form superimposed on each other, allowing users to visually understand areas for improvement.

[2333] An "emotion engine" refers to technology and systems that analyze input data such as a user's voice and facial expressions to recognize their emotional state.

[2334] "Emotional state" refers to the user's current mental and emotional state and is a factor taken into account when adjusting feedback.

[2335] The present invention relates to a system that enables parents to support their children's sports improvement effectively and in a way that takes their emotions into consideration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the preferred embodiments of the present invention.

[2336] Hardware and software used

[2337] server

[2338] The server is the main component for learning motion data, data analysis, and emotion recognition. It uses the following software and frameworks:

[2339] Behavior analysis: TensorFlow, PyTorch

[2340] Emotion Recognition: Emotion analysis engines (e.g., IBM Watson or Microsoft Azure Emotion API)

[2341] Video processing: OpenPose, FFmpeg

[2342] Terminal

[2343] The terminal is the device through which the user captures video and communicates with the server. It uses the following technologies:

[2344] Video recording and uploading: Smartphone camera API (e.g., Android Camera2 API, iOS AVFoundation)

[2345] User Interface: React Native, Flutter

[2346] User

[2347] Users are primarily parents and children and perform the following actions:

[2348] Video recording and uploading

[2349] Check the advice

[2350] Providing emotional feedback

[2351] Specific processing of the system

[2352] 1. Learning from motion data of professional athletes

[2353] server

[2354] The server receives video data of professional athletes and splits it into frames. Next, OpenPose is used to extract features such as keypoints and joint positions from each frame. These features are then used to train machine learning models such as LSTM and CNN using TensorFlow or PyTorch. The trained models are stored in a database for later analysis.

[2355] Terminal

[2356] The device provides an interface for the user to upload the motion data to the server. Once the video upload is complete, the device notifies the user that the learning is complete.

[2357] User

[2358] Users can record the movements of professional athletes using their smartphones, upload the videos to the server via their devices, and then check their learning progress and receive notifications of completion.

[2359] 2. Shooting and importing children's sports movements

[2360] Terminal

[2361] The device provides a camera interface, allowing users to easily record their children's sports activities, and the captured video data is sent to the server via an upload option.

[2362] User

[2363] Users use their smartphones to record their children's sports activities and then upload the videos to a server.

[2364] 3. Behavior analysis and advice generation

[2365] server

[2366] The server divides the child's video data into frames and performs preprocessing. Next, it extracts features and compares them with the motion data of professional athletes. Based on this comparative analysis, specific advice is generated. Furthermore, a comparison video is generated that overlays the child's current and ideal form, and this is sent to the device.

[2367] 4. Emotion recognition and advice adjustment

[2368] server

[2369] The server uses an emotion engine to analyze voice and facial expression data from the device to recognize the user's emotional state. Based on the recognized emotional state, the server adjusts the content of the advice, for example, adding an encouraging message.

[2370] Terminal

[2371] The device displays the received advice, comparison videos, and emotion-sensitive feedback to the user. It also provides a video playback interface, allowing users to compare their current and ideal form.

[2372] User

[2373] The user can check the advice and feedback displayed on the device and use it to work with their child to improve their sports movements. If necessary, they can record the movements again and upload them to the server for reanalysis.

[2374] Specific examples

[2375] Improving children's baseball pitching movements

[2376] Initial setup and learning phase

[2377] Users film the pitching movements of famous baseball pitchers and upload them to the server via their devices. The server then uses this video data to train the model.

[2378] Filming the pitching motion

[2379] Users can record their child's pitching motion with their smartphone and upload the video to the server.

[2380] Behavior analysis and advice generation

[2381] The server analyzes the child's pitching video, generates specific improvements (e.g., "Raise your right elbow 3cm higher"), and creates a comparison video.

[2382] Emotion Recognition and Advice Adjustment

[2383] The server uses an emotion engine to analyze the user's emotions and add appropriate feedback (e.g., "Great progress! Let's take a breather and try again next time").

[2384] Prompt Sentence Examples

[2385] "Please explain the system that records a child's sports movements and compares them with the movement data of professional athletes. Also, please provide a concrete example of how a user would use the system."

[2386] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2387] Step 1:

[2388] Receiving and preprocessing motion data from professional athletes

[2389] server

[2390] Input: Video data of professional athletes

[2391] How it works: The server receives video data of a professional athlete, splits it into frames, and then uses OpenPose to extract features such as keypoints and joint positions from each frame.

[2392] Output: Extracted feature data

[2393] Step 2:

[2394] Model learning for motion data

[2395] server

[2396] Input: Extracted feature data

[2397] How it works: Use TensorFlow or PyTorch to train machine learning models such as LSTMs and CNNs. The training process involves providing feature data to the model and iteratively learning it.

[2398] Output: A trained machine learning model

[2399] Step 3:

[2400] Saving the trained model

[2401] server

[2402] Input: A trained machine learning model

[2403] Behavior: Once the training is complete, the model is saved in a database and used for subsequent behavior analysis.

[2404] Output: A saved machine learning model

[2405] Step 4:

[2406] Uploading model operation data

[2407] Terminal

[2408] Input: Professional athlete's athletic data

[2409] Operation: The device provides an interface for the user to upload operation data to the server. The user takes a video using the smartphone's camera API and then selects the upload option.

[2410] Output: Video data uploaded to the server

[2411] Step 5:

[2412] Learning completion notification

[2413] Terminal

[2414] Input: Learning completion notification sent from the server

[2415] Behavior: Notify the user that their learning is complete via push notification or in-app message.

[2416] Output: User notification

[2417] Step 6:

[2418] Filming and uploading children's sports activities

[2419] Terminal

[2420] Input: Video of a child's sports activity

[2421] Actions: The device records the child's sports movements through the camera interface. It displays an option to upload the captured video data to the server. The user uses the option to upload the video to the server.

[2422] Output: Child's video data uploaded to the server

[2423] Step 7:

[2424] Preprocessing and feature extraction of children's video data

[2425] server

[2426] Input: Children's video data

[2427] How it works: The server splits the received video data of the child into frames and extracts features from each frame. As a preprocessing step, it uses OpenPose to identify key points and joint positions.

[2428] Output: Extracted feature data

[2429] Step 8:

[2430] Behavior analysis and advice generation

[2431] server

[2432] Input: Extracted child feature data, saved machine learning model

[2433] How it works: The server uses the saved model to compare the child's feature data with that of a professional athlete. Based on the analysis results, specific advice is generated. In addition, a comparison video is created that overlays the child's current and ideal form.

[2434] Output: Specific advice, comparison video

[2435] Step 9:

[2436] Submitting advice and comparison videos

[2437] server

[2438] Input: Specific advice, comparison videos

[2439] Operation: The server sends the generated advice and comparison video to the device, where the data is displayed and the user can review it.

[2440] Output: Advice and comparison video sent to your device

[2441] Step 10:

[2442] Emotion Recognition and Advice Adjustment

[2443] server

[2444] Input: Voice and facial expression data that indicates the user's emotional state

[2445] How it works: The server uses the emotion engine to analyze voice and facial expression data from the device to recognize the user's emotional state. Based on the recognized emotional state, it adjusts the advice content, for example, adding an encouraging message.

[2446] Output: Tailored advice based on perceived emotional state

[2447] Step 11:

[2448] View advice and feedback

[2449] Terminal

[2450] Input: Adjusted advice received from the server, comparison video

[2451] How it works: The device displays the received advice, comparison videos, and emotion-sensitive feedback to the user. The device also provides an interface for playing videos that overlay AsIs and ToBe.

[2452] Output: Advice and feedback displayed to the user

[2453] Step 12:

[2454] Performance improvements and reanalysis

[2455] User

[2456] Input: Advice displayed on the device, comparison video

[2457] Movement: The user works with their child to improve their sports movements based on the advice provided. If necessary, the improved movements are filmed again and uploaded to the server for reanalysis.

[2458] Output: Improved sports motion video, reanalysis results

[2459] (Application example 2)

[2460] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2461] Modern factories lack an automatic analysis and improvement feedback system to improve the efficiency and accuracy of robotic work. They also lack the ability to provide appropriate feedback based on the emotional state of workers. This makes it difficult to simultaneously improve work efficiency and maintain workers' psychological stability.

[2462] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for learning professional motion data, a means for receiving video of filmed motions, and a means for preprocessing the received video data and extracting features. This makes it possible to analyze the robot's motions and provide specific improvement advice. The server also includes a means for recognizing emotions and adjusting feedback accordingly, and a means for filming and analyzing the worker's motions. This makes it possible to provide appropriate feedback according to the worker's psychological state.

[2463] "Professional motion data" is data that records the motions of skilled people or robots in a specific task or operation.

[2464] "Means for receiving video of captured action" refers to an interface that imports video of action captured by a camera or other capturing device into the system.

[2465] "Means for preprocessing received video data and extracting features" refers to the process of analyzing video data and extracting important points of movement, joint positions, etc.

[2466] "Means for comparative analysis with professional motion data using extracted features" refers to the process of finding and evaluating differences and simil...

Claims

1. A means for learning motion data of professional athletes; A means for receiving a video of the captured sports motion; means for preprocessing the received video data and extracting features; A means for performing comparative analysis with motion data of professional athletes using the extracted features; a means for generating specific advice based on the analysis results; A way to generate a comparison video that overlays the current and ideal form, A means for outputting the generated advice and comparison video; A system including:

2. and means for providing a user with an interface for uploading motion data to be modeled on a professional athlete. The system of claim 1 .

3. means for notifying a user of the generated advice; and means for providing a user with an interface for confirming the notified advice. The system of claim 1 .

Images