System

Abstract

A system is provided.SOLUTION: A system, comprising: means for receiving game video; means for extracting frames from the received game video; means for analyzing the extracted frames and generating play improvements; means for receiving player personalized data; means for analyzing the received player mental data and providing mental care advice; means for analyzing the received player physical data and generating a training plan; means for analyzing the received player nutritional data and providing a dietary plan; and means for integrating and presenting the generated improvements, mental care advice, training plan, and dietary plan to the player.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the field of sports, hiring a personal coach is a significant financial burden, and lesser-known athletes and teams have difficulty securing sponsors. Furthermore, the daily work of coaches in club activities and local clubs is often limited by budgets and staffing, resulting in an excessive workload. This often prevents players and teams from receiving high-quality advice, hindering their technical and competitive improvement. The present invention aims to solve these problems and improve the skills of players and teams by easily providing high-quality advice regardless of budget or the number of coaches. [Means for solving the problem]

[0005] The present invention solves the problems by providing a system comprising the following means.

[0006] (1) A means for receiving game video and extracting frames from the video.

[0007] (2) A means of analyzing plays from extracted frames and generating improvements.

[0008] (3) A means of receiving individual player data (mental, physical, and nutritional).

[0009] (4) A means for analyzing received mental data and providing mental health advice.

[0010] (5) A means for analyzing the received physical data and generating an optimal training plan.

[0011] (6) A means for analyzing the received nutritional data and providing optimal meal plans.

[0012] (7) A means of integrating and presenting the improvement points, mental care advice, training plans, and dietary plans generated above to the player.

[0013] This allows players and teams to receive advanced coaching and support without relying on financial or human resources. Furthermore, generative AI technology allows these analyses and advice to be provided quickly and accurately, contributing to the improvement of players' skills and maximizing their performance.

[0014] A "game video" is a video file that records a sporting event.

[0015] A "frame" refers to each still image that makes up a video image.

[0016] "Play" refers to the specific movements and tactical actions of players in a sporting event.

[0017] "Areas for improvement" refers to areas in a player's technique or tactics that are lacking or need correction.

[0018] "Player" means an individual participant in a sporting event.

[0019] "Mental data" refers to information relating to a player's psychological or mental state.

[0020] "Physical data" refers to information about a player's physical abilities and strength.

[0021] "Nutrition Data" refers to information regarding a Player's diet and nutritional intake.

[0022] "Mental health advice" refers to specific suggestions and advice to improve a player's mental health.

[0023] "Training Plan" means a specific training plan for improving a player's physical abilities.

[0024] "Meal Plan" means specific dietary suggestions to optimize a player's nutritional intake.

[0025] "Integration" refers to bringing together multiple different pieces of information or data into one.

[0026] "Artificial intelligence technology" refers to systems and algorithms that mimic human intelligence using machine learning and data analysis techniques.

[0027] "Analysis" refers to the process of extracting specific patterns and useful information from the data received. [Brief explanation of the drawings]

[0028] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0029] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0030] First, the terms used in the following description will be explained.

[0031] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0032] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0033] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0034] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0035] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0036] [First embodiment]

[0037] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0038] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0039] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0040] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0041] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0042] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0043] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0044] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0045] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0046] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0047] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0048] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0049] The present invention provides optimal training and mental care support based on analysis of game video and individual player data. Below, the program and processing of the system for implementing the present invention are explained in natural language.

[0050] overview

[0051] Users upload game videos from their devices to the server, then input the player's individual data (mental, physical, and nutritional information). The server receives and analyzes this data, and generates and provides each player with individual improvement points, training plans, mental care advice, and meal plans.

[0052] Specific actions

[0053] Video Upload and Analysis

[0054] Users upload video files of recorded matches from their devices to the server. The server receives the video files and temporarily stores them. After storing them, the server extracts frames from the video files. This extraction process breaks down the video into frames and prepares each frame for analysis.

[0055] Getting individual player data

[0056] Users input their mental and physical state and nutritional data into their devices and send it to the server, which receives the data, classifies it into categories, and stores it.

[0057] Data analysis and advice generation

[0058] The server then uses the extracted video frames to analyze the play, using artificial intelligence technology to detect specific patterns and identify areas that need improvement, such as passing accuracy or defensive posture.

[0059] The server then analyzes the user's mental health based on the mental data received from the user, and generates specific mental care advice, such as ways to improve the player's mental health and reduce stress. For example, it can suggest mindfulness exercises.

[0060] The server then analyzes the physical data and generates an optimal training plan, such as a training plan aimed at improving muscle strength or providing a menu for strengthening specific body parts.

[0061] Finally, the server analyzes the nutritional data and generates an appropriate meal plan, which provides nutritional recommendations to support the player's performance, such as specific meal plans to increase protein intake.

[0062] Presentation of results

[0063] The server then integrates all generated improvements, mental health advice, training plans, and meal plans into a single package and provides it to the user, who can then view the package on their own device and incorporate it into their daily practice and life.

[0064] Specific examples

[0065] For example, if a user uploads a video of a soccer game, the server will analyze the video to identify areas for improvement in the player's passing. If the mental data indicates that the user is feeling stressed, the server will suggest mindfulness exercises. If the physical data indicates that the player needs to strengthen their lower body muscles, the server will provide an appropriate strength training plan. If the nutritional data indicates that the player is not consuming enough protein, the server will suggest a specific meal plan.

[0066] In this way, the present invention provides a system that comprehensively supports players in improving their skills and mental health.

[0067] The processing flow will be explained below.

[0068] Step 1:

[0069] Users upload game videos from their devices to the server.

[0070] Specifically, the user selects the game video file on the terminal and clicks the "upload" button.

[0071] Step 2:

[0072] The server temporarily stores the received video file.

[0073] Specifically, the server stores the uploaded video file in a temporary directory.

[0074] Step 3:

[0075] The server extracts frames from the video file.

[0076] Specifically, the server loads the video using cv2.VideoCapture and adds each frame to a list.

[0077] Step 4:

[0078] The server passes the extracted frames to an AI model to analyze the match.

[0079] Specifically, the frame data is passed to the generate_insights method, and the AI ​​model analyzes areas for improvement in the player's play.

[0080] Step 5:

[0081] The server receives mental, physical, and nutritional data from the user.

[0082] Specifically, the user enters this data into a form on the terminal and sends it to the server's API.

[0083] Step 6:

[0084] The server analyzes the mental data and generates mental care advice.

[0085] Specifically, the server passes the mental data to the analyze_mental_health method to generate mental care advice.

[0086] Step 7:

[0087] The server analyzes the physical data and generates a training plan.

[0088] Specifically, the server passes the physical data to the generate_training_plan method to generate a physical training plan.

[0089] Step 8:

[0090] The server analyzes the nutritional data and generates a meal plan.

[0091] Specifically, the server passes the nutritional data to the provide_meal_plan method to generate a meal plan.

[0092] Step 9:

[0093] The server then combines the generated improvements, mental health advice, training plans, and meal plans into a single package.

[0094] Specifically, all analysis results are integrated into one data set.

[0095] Step 10:

[0096] The server sends the combined package to the user.

[0097] Specifically, the server sends the analysis results and plan to the user's terminal so that the user can check them.

[0098] Step 11:

[0099] Users can check the analysis results and advice on their device and incorporate them into their daily training and lifestyle.

[0100] Specifically, the user opens the analysis results page on their device and implements the suggested improvements and plans.

[0101] Example 1

[0102] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0103] In modern sports, improving a player's skills and optimizing their performance involves a wide range of factors. In particular, there is a need for comprehensive analysis of game video and individual player data (mental, physical, and nutritional data). However, there is no system that consistently performs these tasks, and manually performing them by players and coaches requires a great deal of effort and time. As a result, it is difficult to efficiently review games, plan the next training session, or provide mental care, which can delay the improvement of a player's performance.

[0104] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0105] In this invention, the server includes means for uploading game videos from a terminal, means for receiving and temporarily storing the uploaded game videos, means for extracting frames from the stored game videos, means for analyzing the extracted frames to generate points for improving play, means for inputting individual player data from the terminal, means for receiving the input individual player data and storing it in a database, means for analyzing the received mental data of the player to generate mental care advice, means for analyzing the received physical data of the player to generate a training plan, means for analyzing the received nutritional data of the player to generate a meal plan, and means for integrating and presenting the generated points for improving play, mental care advice, training plan, and meal plan to the player. This makes it possible to consistently and efficiently perform processes from analyzing game videos to collecting individual data and providing comprehensive advice based on that data.

[0106] A "terminal" is a digital device that a user uses to enter information or upload video.

[0107] "Server" means a central processing unit for receiving, storing, analyzing, and providing results from data sent by users.

[0108] "Game video" is a digital file that records footage of a sporting event or other competition.

[0109] A "frame" is each of the consecutive still images that make up a game video.

[0110] "Areas for Improvement" are specific areas where players can improve their technique or tactics, identified through analysis of match video.

[0111] "Individual Data" refers to individual information about a player's own mental, physical and nutritional state.

[0112] "Mental data" refers to information about a player's mental health, such as their psychological state and stress level.

[0113] "Physical data" refers to information about a player's physical condition and training status.

[0114] "Nutrition Data" refers to information regarding a player's diet and nutritional intake.

[0115] "Mental care advice" is specific suggestions for maintaining psychological health that are provided based on the analysis of mental data.

[0116] A "training plan" is a specific exercise program provided based on the analysis of physical data to improve a player's physical strength and skills.

[0117] A "meal plan" is a specific meal menu provided based on the analysis of nutritional data to optimize a player's nutritional balance.

[0118] "Integration" refers to bringing together different analysis results and advice into a single package and presenting them to players in a consistent format.

[0119] This invention is a system that provides optimal training and mental care support based on analysis of game video and individual player data. Detailed modes for carrying out the invention are described below.

[0120] System Configuration

[0121] The whole system consists of terminals, servers, and databases. Terminals are devices used by users to input information and upload videos. The server is the central processing unit, responsible for receiving, storing, analyzing data, and providing results. The database is used to store individual player data and analysis results.

[0122] Hardware and software used

[0123] The server's role is to receive the game videos and individual data sent by the users, analyze them, and generate specific advice. The server uses the following software:

[0124] "ffmpeg": A video analysis tool for breaking down match videos frame by frame.

[0125] "OpenPose" and "YOLO": Artificial intelligence techniques for analyzing player movements within video frames.

[0126] "Scikit-learn" and "TENSORFLOW (registered trademark)": Machine learning libraries used to analyze mental, physical, and nutritional data.

[0127] "MySQL (registered trademark)" and "PostgreSQL": Relational databases for storing individual data and analysis results.

[0128] Processing Details

[0129] Users upload game videos from their own devices to the server. This operation is performed using a web browser or a dedicated application, and major file formats such as mp4, avi, and mov are supported. The server receives the uploaded video and temporarily stores it in storage. After saving, the video file is broken down into frames using "ffmpeg" to prepare it for analysis.

[0130] Users input individual data such as mental and physical state, nutritional data, etc. on the device screen. This includes self-assessment and data recorded in a dedicated app. For example, a user may enter a number in response to a question such as, "Please rate your stress level this week on a scale of 1 to 10." The server receives this data and stores it in a database for each category.

[0131] The server then analyzes the extracted video frames using OpenPose and YOLO to analyze the player's movements and detect specific patterns and areas for improvement, such as passing accuracy or defensive posture.

[0132] The analysis of mental data uses Scikit-learn and TensorFlow to assess a player's psychological state and generate mental care advice. For example, if the mental stress score is high, mindfulness exercises and relaxation techniques will be suggested.

[0133] The analysis of physical data evaluates the user's physical condition and generates an optimal training plan. For example, if strength training is required, the system will suggest specific exercise menus, number of sets, and number of repetitions.

[0134] The analysis of nutritional data evaluates the user's diet and nutritional intake status and generates an appropriate meal plan. For example, if the user is lacking in protein, it will suggest a high-protein meal menu.

[0135] Examples and prompts

[0136] As a specific example of use, if a user uploads a soccer game video, the server analyzes the video to identify areas for improvement in the player's passing. Next, if the user's mental data indicates that they are feeling stressed, the server will suggest mindfulness exercises. If the physical data indicates that the player needs to strengthen their lower body muscles, the server will provide an appropriate strength training plan. If the nutritional data indicates that the player's protein intake is insufficient, the server will suggest a specific meal plan. For example, it will suggest recipes using protein-rich ingredients such as chicken breast and tofu.

[0137] An example of a prompt to input to a generative AI model is:

[0138] "Please analyze a soccer game video and give me some advice on how to improve passing accuracy. Next, I'd like some advice on mental health, as this player tends to get stressed easily. I'd also like some suggestions on a training plan to strengthen his lower body muscles and a meal menu to increase his protein intake."

[0139] In this way, this system provides comprehensive support for players' technical improvement and mental care.

[0140] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0141] Step 1:

[0142] Users upload game videos.

[0143] Specifically, the user uploads a game video file (e.g., in mp4, avi, or mov format) using a web browser or dedicated application on their own device.

[0144] Input: Match video file

[0145] Output: Send video file to server

[0146] Step 2:

[0147] The server receives the match video file and temporarily stores it.

[0148] The server receives the uploaded video file and temporarily stores it in storage, along with the file name and metadata.

[0149] Input: Video file from user

[0150] Output: Video file in storage

[0151] Step 3:

[0152] The server extracts frames from the video file.

[0153] Using a video analysis tool such as ffmpeg, the match video is broken down into frames. For example, a 30 frame per second video breaks down into 1800 frames per minute.

[0154] Input: Video file in storage

[0155] Output: A collection of extracted frames

[0156] Step 4:

[0157] The user enters the individual data.

[0158] Users enter their individual data, such as mental and physical condition and nutritional data, on the device screen, including self-assessment and data recorded using a dedicated app.

[0159] Input: Mental state, physical state, nutritional data

[0160] Output: Send individual data to the server

[0161] Step 5:

[0162] The server receives the individual data and stores it in a database.

[0163] The server receives the individual data sent by the user and stores it in a database for each category. The data is stored in a relational database such as MySQL or PostgreSQL.

[0164] Input: Individual data from the user

[0165] Output: Data in the database

[0166] Step 6:

[0167] The server analyzes the video frames.

[0168] The server then analyzes the extracted video frames using machine learning models such as OpenPose and YOLO to analyze player movements and identify areas for improvement, such as passing accuracy or defensive posture.

[0169] Input: Extracted video frames

[0170] Output: Play Improvements

[0171] Step 7:

[0172] The server analyzes the mental data and generates mental care advice.

[0173] Using Scikit-learn and TensorFlow, the system analyzes mental data to assess a player's psychological state, suggesting mindfulness exercises if stress levels are high, for example.

[0174] Input: Mental data in a database

[0175] Output: Mental care advice

[0176] Step 8:

[0177] The server analyzes the physical data and generates a training plan.

[0178] The system analyzes physical data and generates training plans to improve the user's strength and technique, for example by suggesting the content, number of sets, and number of repetitions of strength training.

[0179] Input: Physical data in the database

[0180] Output: Training Plan

[0181] Step 9:

[0182] The server analyzes the nutritional data and generates a meal plan.

[0183] The system analyzes nutritional data and generates meal plans to provide users with a balanced diet. For example, if a user is lacking in protein, it will suggest high-protein meals.

[0184] Input: Nutrition data in the database

[0185] Output: Meal plan

[0186] Step 10:

[0187] The server aggregates all the results and serves them to the user.

[0188] The server integrates points for improving play, mental health advice, training plans, and meal plans, and provides them to users as a single package. Users can then check this package on their devices and incorporate it into their practice and daily life.

[0189] Input: Improvements to your game, mental health advice, training plans, meal plans

[0190] Output: Consolidated advice package

[0191] In this way, by identifying the specific inputs and outputs at each step, the flow of the entire system and the data processing and calculations at each processing stage become clear.

[0192] (Application example 1)

[0193] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0194] With conventional training systems, it was difficult to perform detailed analysis of the individual physical and mental data of players and operators and provide optimal advice and plans based on that data. Furthermore, there was a lack of systems that could comprehensively support the optimization of robot movements and the mental care of operators. This resulted in a lack of efficient training and effective mental care, limiting performance improvement.

[0195] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0196] In this invention, the server includes means for receiving game video, means for extracting frames from the received game video, means for analyzing the extracted frames and generating improvement points for play, means for receiving individual player data, means for analyzing the received mental data of the player and providing mental care advice, means for analyzing the received physical data of the player and generating a training plan, means for analyzing the received nutritional data of the player and providing a meal plan, means for integrating the generated improvement points, mental care advice, training plan, and meal plan and presenting them to the player, means for receiving operator data, analyzing the mental data of the operator, and suggesting mindfulness exercises, means for analyzing the physical data and providing a meal plan, and means for analyzing video frames and generating a robot operation strategy. This enables player skill improvement and mental care, as well as optimization of factory robot operation and mental care for operators.

[0197] "Game video" is a recorded video of a sports or game match.

[0198] A "frame" is a unit of still image that is a component of video.

[0199] "Player's individual data" is a collective term for information such as individual mental data, physical data, and nutritional data entered or held by a player.

[0200] "Mental data" refers to data such as the psychological state, emotions, and stress level of players and operators.

[0201] "Physical Data" refers to data relating to the physical health and athletic ability of a player or operator.

[0202] "Nutrition Data" means information about the food and drink consumed by a player or operator, as well as data about the nutrients consumed.

[0203] "Areas for improvement" refers to parts or elements that require correction or improvement in order to improve the performance of the player or robot.

[0204] "Mental care advice" refers to suggestions for psychological support and stress reduction methods provided based on mental data.

[0205] A "training plan" is an exercise or training plan created based on physical data.

[0206] A "meal plan" is a meal plan or nutritional intake plan suggested based on nutritional data.

[0207] "Operator data" is a general term for information including mental and physical data of people working in factories and work sites.

[0208] A "robot motion strategy" is a plan or method for achieving efficient motion of robots used in factories or work sites.

[0209] "Artificial intelligence technology" is a general term for technologies that allow computer systems to analyze and learn from data, automatically discovering patterns and making predictions.

[0210] A "mindfulness exercise" is a set of practices or activities that reduce psychological stress and improve mental well-being by focusing attention on the present moment.

[0211] This invention is a system that supports optimal training, mental health, and nutritional management based on game video analysis and individual player data. It also enables optimization of factory robot operations and supports the mental health of operators. The entire system is implemented using a server, user terminals, and analysis software.

[0212] Program Hardware and Software

[0213] The server receives game videos and robot movement videos and extracts frames. Specifically, it uses OpenCV to extract frames from the video and analyzes the video frames using TensorFlow and Keras. Based on the analysis results, it generates improvements to the player and robot's movements, provides training and diet plans to players, and suggests mental health advice to operators.

[0214] The hardware used is a server equipped with a high-performance CPU / GPU and a camera for video recording, and the software used is OpenCV, TensorFlow, and Keras.

[0215] Data analysis and advice generation

[0216] The server receives the game video and breaks it down into frames. The extracted frames are then analyzed to detect specific patterns in the movement of the play and the robot's behavior. This analysis is performed using TensorFlow's deep learning model. For example, passing accuracy and defensive posture can be identified from a soccer game video and improvements can be made.

[0217] Individual data on players and operators is also sent to the server. For example, the server receives mental data and analyzes it to suggest psychological strengthening methods and mindfulness exercises. Based on physical data, it generates an optimal training plan, providing an appropriate training menu if lower-body muscle strengthening is required. Furthermore, it analyzes nutritional data to propose meal plans, suggesting specific foods for players who are not consuming enough protein, for example.

[0218] Specific examples

[0219] Suppose a user uploads a soccer game video and the server analyzes it. If the server determines that the player's passing accuracy is insufficient, it will suggest areas for improvement. If the server determines that the user is experiencing high stress based on the user's mental data, it will suggest mindfulness exercises. If the server analyzes the user's physical data and determines that the user needs to strengthen their lower body muscles, it will provide an appropriate training plan. If the server determines that the user's protein intake is insufficient based on nutritional data, it will suggest a meal plan to increase protein.

[0220] In addition, by inputting video footage of robots moving in factories and operator data, the system generates an analysis of the video footage, suggestions for mindfulness exercises to address the operator's high stress levels, and a nutrition plan.

[0221] Prompt Sentence Examples

[0222] Example prompts to input to a generative AI model:

[0223] "After uploading and analyzing videos of factory robots in action, we discovered that certain movements were inefficient. Since the operators' stress levels were high, we suggested mindfulness exercises. We also discovered that they were lacking protein, so we proposed a specific diet plan to increase their protein intake."

[0224] In this way, the present invention provides a system that comprehensively supports the improvement of skills and mental care of players and operators.

[0225] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0226] Step 1:

[0227] Users upload game videos and robot movement videos from their devices to the server.

[0228] Input: Match video or robot movement video file

[0229] Specific operation: The user selects a video file using the upload function of the terminal and sends it to the server.

[0230] Output: The video file is saved on the server.

[0231] Step 2:

[0232] The server extracts frames from the received video file.

[0233] Input: Video file

[0234] Data processing: Decompose the video file frame by frame using OpenCV.

[0235] Specific operation: The server reads the video file and extracts specific frames (e.g., every 30th frame).

[0236] Output: A list of extracted frames

[0237] Step 3:

[0238] The server analyzes the extracted frames based on artificial intelligence techniques.

[0239] Input: A list of frames

[0240] Data computation: Using TensorFlow and Keras, we perform analysis using a CNN model for each frame.

[0241] Specific operation: The extracted frames are resized and normalized, and then input into the model to obtain analysis results.

[0242] Output: Improvements to play and specific patterns of robot movement

[0243] Step 4:

[0244] The user sends individual data of the player and operator from the terminal to the server.

[0245] Input: Mental data, physical data, nutritional data

[0246] Specific operation: The user inputs mental data, physical data, and nutritional data on the terminal and transmits the data to the server.

[0247] Output: Individual data stored on the server

[0248] Step 5:

[0249] The server analyzes the received mental data and generates mental care advice.

[0250] Input: Mental data

[0251] Data calculation: Analyzes mental data to evaluate psychological state and stress level and generate appropriate mental care advice.

[0252] Specific behavior: The server suggests mindfulness exercises when stress levels are high.

[0253] Output: Mental care advice

[0254] Step 6:

[0255] The server analyzes the received physical data and generates an optimal training plan.

[0256] Input: Physical data

[0257] Data calculation: Create an appropriate training menu based on physical data.

[0258] Specific Movement: The server generates a specific training plan, such as strengthening muscles.

[0259] Output: Training Plan

[0260] Step 7:

[0261] The server analyzes the received nutritional data and provides a meal plan.

[0262] Input: Nutrition data

[0263] Data calculation: Analyzes nutritional data to identify nutrient deficiencies and suggests necessary meal plans.

[0264] What it does: If you're lacking in protein, the server will suggest a meal plan to increase your protein intake.

[0265] Output: Meal plan

[0266] Step 8:

[0267] The server integrates the generated improvements, mental care advice, training plans, and meal plans and presents them to the user.

[0268] Input: Improvements, mental health advice, training plans, meal plans

[0269] Specific operation: The server integrates this information and sends it to the user terminal as a single package.

[0270] Output: A consolidated improvement plan that can be viewed on the user's device

[0271] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0272] This invention provides optimal training and mental care support based on game video analysis and individual player data. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to generate more accurate advice.

[0273] System Operation Overview

[0274] Users use their devices to upload game videos to the server, which breaks down the video into frames and analyzes each one. Individual player data (mental, physical, and nutritional information) is also sent to the server, and various advice is generated based on this data. The emotion engine recognizes the user's emotions and uses this data in analysis to further improve the quality of advice.

[0275] Specific processing flow

[0276] Video Upload and Analysis

[0277] Users upload game videos from their devices to the server. The server receives the video files and temporarily stores them. Next, it extracts each frame from the video file and analyzes each frame to identify areas for improvement. Specifically, it uses artificial intelligence technology to analyze each frame and evaluate the player's movements.

[0278] Acquiring and analyzing emotion data

[0279] To recognize the user's emotional state, the device is equipped with a camera and a microphone. The emotion engine analyzes the user's facial expressions and tone of voice in real time from these devices and generates emotional data. The server receives this data and evaluates the user's emotional state.

[0280] Getting individual player data

[0281] Users enter mental, physical, and nutritional data into a device and send it to a server, which receives the data, stores it by category, and creates a dataset for analysis.

[0282] Generating comprehensive advice

[0283] The server generates comprehensive advice by integrating the user's emotional, mental, physical, and nutritional data with the points for improvement found in the analyzed video frames. The mental care advice is tailored based on the emotional data. For example, if the user is feeling stressed, it can suggest mindfulness exercises to help relieve stress.

[0284] Training plans are also provided by combining physical and emotional data. For example, if a player is feeling fatigued, it can suggest light exercises.

[0285] The system also incorporates emotional data into nutritional plans, allowing it to provide plans that reflect a user's preferred ingredients and eating patterns. For example, if a user feels they are lacking in energy, it will suggest meals that are high in protein.

[0286] Specific examples

[0287] For example, a user can upload a soccer game video and input the results of a physical fitness test as their own physical data. If the emotion engine detects a state of stress from the user's facial expression, the server will suggest mental care advice (mindfulness exercises) to reduce stress. Furthermore, based on the physical data, it may determine that the user needs to strengthen their lower body muscles, and a training plan for this purpose will be provided. At the same time, nutritional data may detect a protein deficiency and suggest a meal plan to compensate for it.

[0288] In this way, the present invention realizes a system that provides more personalized and advanced support by incorporating user emotional data.

[0289] The processing flow will be explained below.

[0290] The present invention provides optimal training and mental care support based on game video analysis and individual player data. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the system generates more accurate advice. The processing flow of the system for implementing the present invention is specifically described below.

[0291] Specific processing flow of the system

[0292] Step 1:

[0293] Users upload game videos from their devices to the server.

[0294] Specifically, the user selects the game video file on the terminal and clicks the "upload" button.

[0295] Step 2:

[0296] The server temporarily stores the received video file.

[0297] Specifically, the server stores the uploaded video file in a temporary directory.

[0298] Step 3:

[0299] The server extracts frames from the video file.

[0300] Specifically, the server loads the video using cv2.VideoCapture and adds each frame to a list.

[0301] Step 4:

[0302] The server passes the extracted frames to an AI model to analyze the match.

[0303] Specifically, the frame data is passed to the generate_insights method, and the AI ​​model analyzes areas for improvement in the player's play.

[0304] Step 5:

[0305] An emotion engine is used to recognize the user's emotional state.

[0306] Specifically, it analyzes the user's facial expressions and tone of voice in real time through the camera and microphone installed on the device to generate emotional data.

[0307] Step 6:

[0308] The server receives the emotional data and evaluates the user's emotional state.

[0309] Specifically, the emotion data is sent to the server and the analysis results are saved.

[0310] Step 7:

[0311] Users enter mental, physical, and nutritional data into a terminal and send it to the server.

[0312] Specifically, the user enters this data into a form on the terminal and presses the send button.

[0313] Step 8:

[0314] The server analyzes the received mental data and provides mental care advice.

[0315] Specifically, the mental data is passed to the analyze_mental_health method to generate mental care advice.

[0316] Step 9:

[0317] The server analyzes the physical data and generates a training plan.

[0318] Specifically, the physical data is passed to the generate_training_plan method to generate the optimal training plan.

[0319] Step 10:

[0320] The server analyzes the nutritional data and provides meal plans.

[0321] Specifically, the nutritional data is passed to the provide_meal_plan method, which generates an appropriate meal plan.

[0322] Step 11:

[0323] The server then combines the generated improvements, mental health advice, training plans, and meal plans into a single package.

[0324] Specifically, all analysis results are integrated to create a package to be provided to the user.

[0325] Step 12:

[0326] The server sends the combined package to the user.

[0327] Specifically, the server sends the analysis results and plan to the user's terminal so that the user can check them.

[0328] Step 13:

[0329] Users can check the analysis results and advice on their device and incorporate them into their daily training and lifestyle.

[0330] Specifically, the user opens the analysis results page on their device and implements the suggested improvements and plans.

[0331] Specific examples

[0332] For example, if a user uploads a soccer game video and the emotion engine recognizes the user's stress level, the server will suggest mindfulness exercises to reduce stress. At the same time, physical data will determine that the user needs to strengthen their lower body muscles, and a training plan for that purpose will be provided. If nutritional data detects a protein deficiency, a specific meal plan to compensate for that will be proposed. The advice tailored by the emotion engine is then synthesized and provided to the user, improving the quality of their training.

[0333] In this way, the present invention realizes a system that provides more personalized and advanced support by incorporating user emotional data.

[0334] Example 2

[0335] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0336] To improve sports player performance, conventional systems simply analyze game videos and provide general training plans, but they lack detailed advice and support tailored to individual players. They also lack a comprehensive approach that takes into account the user's emotional state. Therefore, there is a need for systems that provide comprehensive support for players' technical development as well as their mental, physical, and nutritional needs.

[0337] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0338] In this invention, the server includes means for receiving game video, means for extracting frames from the received game video, means for analyzing the extracted frames and generating improvements to the player's movements, means for receiving individual player data, means for analyzing the received mental data of the player to provide mental health advice, means for analyzing the received physical data of the player to generate a training plan, means for analyzing the received nutritional data of the player to provide a meal plan, means for acquiring and analyzing user emotional data, means for improving the quality of the advice based on the acquired emotional data, and means for integrating and presenting the generated improvements, mental health advice, training plan, and meal plan to the player. This allows for the provision of personalized, advanced support, enabling comprehensive support not only for the player's technical growth but also for mental, physical, and nutritional aspects.

[0339] A "game video" is a video file that records a sports game.

[0340] A "frame" is an individual still image that makes up a video image.

[0341] "Movement improvement points" refer to specific points or areas that a sports player can use to improve their play during a game.

[0342] "Individual Data" means data relating to each player, including mental, physical and nutritional information.

[0343] "Mental data" is data that contains information about a player's psychological state and emotions.

[0344] "Mental Care Advice" is specific advice or suggestions to improve a player's psychological state and support their mental health.

[0345] "Physical Data" means data containing information about a player's physical condition and performance.

[0346] A "training plan" is a specific training menu and exercise plan based on a player's physical data to improve their physical strength and skills.

[0347] "Nutrition Data" means data containing information about a player's diet and nutritional intake.

[0348] A "meal plan" is a specific dietary suggestion or plan designed to improve a player's nutritional status and optimize performance.

[0349] "Emotion data" refers to data that includes information about the user's emotional state obtained by analyzing the user's facial expression and tone of voice.

[0350] "Artificial intelligence technology" is a technology that uses techniques including machine learning to analyze data and automatically extract knowledge and patterns.

[0351] "Machine learning" is a technique in which computer algorithms learn from data to recognize patterns and create predictive models.

[0352] Mindfulness exercises are mental exercises that help you focus your attention on the present moment and reduce stress and anxiety.

[0353] This system provides optimal training and mental care support based on game video analysis and individual player data. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to generate more accurate advice.

[0354] System Operation Overview

[0355] Users upload game videos from their devices to the server. The server extracts each frame from the received video and analyzes it using artificial intelligence techniques (e.g., OpenCV and TensorFlow). The extracted frames are evaluated to identify areas for improvement in the player's movements.

[0356] Users also input their mental, physical, and nutritional data into the server via their devices. This individual data is stored in a database to create a dataset for analysis. An emotion engine (e.g., Emotion API) is used to obtain end-user emotional data in real time. The server analyzes this data and generates integrated advice.

[0357] Hardware and software used

[0358] Device: The computer or smartphone you use

[0359] Server: A remote server that receives, processes, and analyzes data.

[0360] Artificial intelligence technology: Machine learning libraries such as OpenCV and TensorFlow

[0361] Emotion engine: Emotion API

[0362] Data acquisition and analysis flow

[0363] 1. Upload your video

[0364] Users upload game videos from their devices to the server, and the video files are sent to the designated server via the device application.

[0365] 2. Extracting and analyzing video frames

[0366] The server temporarily stores the received video and uses a video processing tool (e.g., FFmpeg) to extract each frame, which is then analyzed using artificial intelligence techniques.

[0367] 3. Acquiring and sending emotion data

[0368] The device's camera and microphone are used to capture the user's facial expressions and voice, which are then analyzed by the emotion engine. The emotion data analyzed in real time is sent to a server and stored in a database.

[0369] 4. Enter and submit individual data

[0370] Users use a dedicated application to input mental, physical, and nutritional data, which is then sent to a server where it is stored in a database and a data set is created for analysis.

[0371] 5. Data Integration and Advice Generation

[0372] The server integrates all the received data and uses a generative AI model to generate comprehensive advice, which is then displayed on the user's dashboard. For example, if a user uploads a game video and the emotion engine detects stress, the generative AI model will suggest mental care advice (e.g., mindfulness exercises) to reduce stress.

[0373] Specific examples

[0374] For example, a user can upload a soccer game video and enter the results of a recent physical fitness test as their physical data. If the emotion engine detects a state of stress from the user's facial expression, the server will suggest mental care advice (e.g., mindfulness exercises) to reduce stress. Furthermore, based on the physical data, it may determine that the user needs to strengthen their lower body muscles, and a training plan for this purpose will be provided. At the same time, nutritional data may detect a protein deficiency and suggest a meal plan to compensate for it.

[0375] Prompt Sentence Examples

[0376] "If a user uploads a soccer game video and the emotion engine detects stress from the user's facial expressions, what kind of mental health advice can be provided? Also, if the results of a physical fitness test are entered as the user's physical data and the results indicate that the user needs to strengthen their lower body muscles, what kind of training plan can be suggested?"

[0377] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0378] Understood. Now, I will explain the processing flow of this system's program in detail in the following format.

[0379] Step 1:

[0380] The user launches a dedicated application on their device, selects a game video file, and uploads it. When they press the upload button, the device sends the video file to the specified server endpoint (e.g., https: / / example.com / upload). The input is the video file selected by the user, and the output is the video file saved on the server.

[0381] Step 2:

[0382] The server temporarily stores the received video file and then uses a video processing tool such as FFmpeg to decompose the video into frames. A frame is an individual still image that is used in the next analysis step. The input is the received video file, and the output is the extracted frames. Each frame is a sequence of image data.

[0383] Step 3:

[0384] The server uses a deep learning model (e.g., a model using TensorFlow) to analyze each extracted frame. This analysis evaluates the player's movements and identifies areas for improvement. For example, elements of the play such as movement speed and positioning are analyzed. The input is the image data of the frame, and the output is information on areas for improvement.

[0385] Step 4:

[0386] The device uses a built-in camera and microphone to capture the user's facial expressions and voice in real time. An emotion recognition engine such as Emotion API analyzes this data and evaluates the user's emotional state. The generated emotion data is encrypted and sent to a server. The input is the captured facial and voice data, and the output is emotion data.

[0387] Step 5:

[0388] Users enter mental, physical, and nutritional data using a dedicated application. The data entered by the user is sent from the device to a server, which then stores it in a database. The input is the individual data entered by the user, and the output is a dataset for analysis.

[0389] Step 6:

[0390] The server integrates all data (video analysis data, emotional data, individual data). It uses a generative AI model to generate comprehensive advice. For example, it extracts areas for improvement in play from video analysis data, evaluates the need for mental care from emotional data, and generates a training plan from physical data. The input is the integrated dataset, and the output is advice for the user.

[0391] Step 7:

[0392] The user can view the generated advice on the dashboard of the dedicated application. The advice includes mental health advice (e.g., mindfulness exercises for stress reduction), training plans, and meal plans. The input is advice data from the generative AI model, and the output is advice information provided to the user.

[0393] This will enable the realization of an individualized and advanced sports support system.

[0394] (Application example 2)

[0395] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0396] Conventional game video analysis systems and work efficiency improvement systems were unable to provide advanced advice that incorporated the user's emotional and physical data. Furthermore, there was a lack of means to analyze emotional and physical data in real time and provide appropriate advice. This made it difficult to maximize the user's motivation and work efficiency.

[0397] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0398] In this invention, the server includes means for receiving game videos, means for extracting frames from the received game videos, means for analyzing the extracted frames and generating improvement points for play, means for receiving individual player data, means for analyzing the received mental data of the player and providing mental care advice, means for analyzing the received physical data of the player and generating a training plan, means for analyzing the received nutritional data of the player and providing a meal plan, means for integrating the generated improvement points, mental care advice, training plan, and meal plan and presenting them to the player, means for receiving work videos, means for extracting frames from the received work videos, means for analyzing the extracted frames and generating improvement points for work, means for receiving individual worker data, means for analyzing the received mental data of the worker and providing stress care advice, and means for analyzing the received physical data of the worker and generating a work plan. This makes it possible to analyze a user's emotions and physical data in real time and provide optimal advice.

[0399] 1. "Match Video" means a video file that records the match.

[0400] 2. "Work video" refers to a video file that records work being done in a factory or workshop.

[0401] 3. "Frame" means the individual still images that make up a video.

[0402] 4. "Individual Data" means data that indicates the mental, physical, nutritional, and other information of an individual player or worker.

[0403] 5. A "player" is a person who takes part in a sport or other competition.

[0404] 6. "Worker" means a person who performs work in a factory or workshop.

[0405] 7. "Mental Data" refers to data relating to an individual's state of mind or mental health.

[0406] 8. "Physical Data" means data relating to an individual's physical condition and physical strength.

[0407] 9. "Nutrition Data" refers to data relating to an individual's dietary content and nutritional intake status.

[0408] 10. "Mental health advice" means advice to maintain or improve mental health that is provided based on mental data.

[0409] 11. "Stress care advice" refers to specific guidance and advice to alleviate individual stress conditions.

[0410] 12. "Training Plan" means an exercise or training plan created based on physical data.

[0411] 13. "Work Plan" means a plan to improve worker efficiency and safety.

[0412] 14. "Meal Plan" means a meal suggestion based on nutritional data.

[0413] 15. "Artificial intelligence technology" refers to technology that uses techniques such as machine learning and deep learning to analyze data and make intelligent decisions.

[0414] The present invention is a system that analyzes work videos and game videos in real time and provides appropriate advice based on the mental, physical, or nutritional state of each individual user. This system is configured as follows.

[0415] The user uploads the video of their work to the server in real time using smart glasses or other devices. The server then breaks down the received video into frames and analyzes each frame using artificial intelligence techniques such as OpenCV, dlib, and EmotionRecognizer.

[0416] The server also receives the user's individual data (mental, physical, and nutritional information), including data the user previously inputs into the device, and uses an emotion engine to obtain the user's emotional data in real time, which is used for analysis.

[0417] The server combines the analysis results of each frame with the user's emotional and physical data to generate suggestions for improving their gameplay or work. For example, if the emotion engine detects that the user is under stress, it will provide mental care advice to help relieve stress.

[0418] The hardware includes smart glasses, a camera, a microphone, a server, etc. The software uses an analysis program using Python, and libraries such as OpenCV, dlib, and EmotionRecognizer.

[0419] As a concrete example, imagine a factory worker wearing smart glasses while working. If the worker feels stressed and tired, the server will recognize the worker's emotions in real time and provide advice such as "Take a five-minute break" or "Try some light stretching" based on the worker's emotional and physical data.

[0420] Below are some example prompts to input to a generative AI model:

[0421] "Capture sentiment data and generate real-time advice to improve work efficiency."

[0422] This invention makes it possible to analyze a user's emotions and physical data in real time and provide optimal advice, thereby providing comprehensive support for the user's mental and physical state and improving work efficiency and performance.

[0423] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0424] Step 1:

[0425] A user puts on the smart glasses and starts recording a work video. The camera in the smart glasses captures the work video and streams it in real time.

[0426] Input: User's working video

[0427] Output: Real-time captured video data

[0428] Step 2:

[0429] The server receives the video data in real time and extracts each frame, which is then decomposed into individual still images using OpenCV.

[0430] Input: Real-time captured video data

[0431] Output: Extracted frames (a sequence of still images)

[0432] Step 3:

[0433] The server analyzes the extracted frames and evaluates the worker's movements, using dlib for face detection and EmotionRecognizer to determine emotions.

[0434] Input: Extracted frames

[0435] Output: Emotion data (e.g., stress, excitement, fatigue, etc.)

[0436] Step 4:

[0437] The server receives the user's individual data (mental, physical, and nutritional information). Data entered from the device is sent to the server.

[0438] Input: Individual data (mental, physical, nutritional information)

[0439] Output: Individual data received

[0440] Step 5:

[0441] The server combines the individual data received and the analysis results (emotion data) of the extracted frames to comprehensively evaluate the user's condition, and uses an AI model to generate appropriate advice (mental care, work plans, meal plans, etc.).

[0442] Input: Emotion data, individual data

[0443] Output: Comprehensive advice (e.g., mental health advice, stress care advice, work plan, meal plan, etc.)

[0444] Step 6:

[0445] The generated advice is displayed on the user's smart glasses. For example, if the emotion data indicates stress, the advice "Take a 5-minute break" is displayed.

[0446] Input: General Advice

[0447] Output: Advice to the user

[0448] Step 7:

[0449] The user follows the advice to continue or stop working and take appropriate care, such as stretching, taking breaks, or following a meal plan.

[0450] Input: User advice

[0451] Output: Improved work or health status of the user

[0452] Through these steps, the server analyzes the user's emotional and physical data in real time and provides optimal advice, thereby providing comprehensive support for the user's mental and physical state and improving work efficiency and performance.

[0453] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0454] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0455] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0456] [Second embodiment]

[0457] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0458] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0459] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0460] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0461] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0462] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0463] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0464] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0465] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0466] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0467] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0468] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0469] The present invention provides optimal training and mental care support based on analysis of game video and individual player data. Below, the program and processing of the system for implementing the present invention are explained in natural language.

[0470] overview

[0471] Users upload game videos from their devices to the server, then input the player's individual data (mental, physical, and nutritional information). The server receives and analyzes this data, and generates and provides each player with individual improvement points, training plans, mental care advice, and meal plans.

[0472] Specific actions

[0473] Video Upload and Analysis

[0474] Users upload video files of recorded matches from their devices to the server. The server receives the video files and temporarily stores them. After storing them, the server extracts frames from the video files. This extraction process breaks down the video into frames and prepares each frame for analysis.

[0475] Getting individual player data

[0476] Users input their mental and physical state and nutritional data into their devices and send it to the server, which receives the data, classifies it into categories, and stores it.

[0477] Data analysis and advice generation

[0478] The server then uses the extracted video frames to analyze the play, using artificial intelligence technology to detect specific patterns and identify areas that need improvement, such as passing accuracy or defensive posture.

[0479] The server then analyzes the user's mental health based on the mental data received from the user, and generates specific mental care advice, such as ways to improve the player's mental health and reduce stress. For example, it can suggest mindfulness exercises.

[0480] The server then analyzes the physical data and generates an optimal training plan, such as a training plan aimed at improving muscle strength or providing a menu for strengthening specific body parts.

[0481] Finally, the server analyzes the nutritional data and generates an appropriate meal plan, which provides nutritional recommendations to support the player's performance, such as specific meal plans to increase protein intake.

[0482] Presentation of results

[0483] The server then integrates all generated improvements, mental health advice, training plans, and meal plans into a single package and provides it to the user, who can then view the package on their own device and incorporate it into their daily practice and life.

[0484] Specific examples

[0485] For example, if a user uploads a video of a soccer game, the server will analyze the video to identify areas for improvement in the player's passing. If the mental data indicates that the user is feeling stressed, the server will suggest mindfulness exercises. If the physical data indicates that the player needs to strengthen their lower body muscles, the server will provide an appropriate strength training plan. If the nutritional data indicates that the player is not consuming enough protein, the server will suggest a specific meal plan.

[0486] In this way, the present invention provides a system that comprehensively supports players in improving their skills and mental health.

[0487] The processing flow will be explained below.

[0488] Step 1:

[0489] Users upload game videos from their devices to the server.

[0490] Specifically, the user selects the game video file on the terminal and clicks the "upload" button.

[0491] Step 2:

[0492] The server temporarily stores the received video file.

[0493] Specifically, the server stores the uploaded video file in a temporary directory.

[0494] Step 3:

[0495] The server extracts frames from the video file.

[0496] Specifically, the server loads the video using cv2.VideoCapture and adds each frame to a list.

[0497] Step 4:

[0498] The server passes the extracted frames to an AI model to analyze the match.

[0499] Specifically, the frame data is passed to the generate_insights method, and the AI ​​model analyzes areas for improvement in the player's play.

[0500] Step 5:

[0501] The server receives mental, physical, and nutritional data from the user.

[0502] Specifically, the user enters this data into a form on the terminal and sends it to the server's API.

[0503] Step 6:

[0504] The server analyzes the mental data and generates mental care advice.

[0505] Specifically, the server passes the mental data to the analyze_mental_health method to generate mental care advice.

[0506] Step 7:

[0507] The server analyzes the physical data and generates a training plan.

[0508] Specifically, the server passes the physical data to the generate_training_plan method to generate a physical training plan.

[0509] Step 8:

[0510] The server analyzes the nutritional data and generates a meal plan.

[0511] Specifically, the server passes the nutritional data to the provide_meal_plan method to generate a meal plan.

[0512] Step 9:

[0513] The server then combines the generated improvements, mental health advice, training plans, and meal plans into a single package.

[0514] Specifically, all analysis results are integrated into one data set.

[0515] Step 10:

[0516] The server sends the combined package to the user.

[0517] Specifically, the server sends the analysis results and plan to the user's terminal so that the user can check them.

[0518] Step 11:

[0519] Users can check the analysis results and advice on their device and incorporate them into their daily training and lifestyle.

[0520] Specifically, the user opens the analysis results page on their device and implements the suggested improvements and plans.

[0521] Example 1

[0522] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0523] In modern sports, improving a player's skills and optimizing their performance involves a wide range of factors. In particular, there is a need for comprehensive analysis of game video and individual player data (mental, physical, and nutritional data). However, there is no system that consistently performs these tasks, and manually performing them by players and coaches requires a great deal of effort and time. As a result, it is difficult to efficiently review games, plan the next training session, or provide mental care, which can delay the improvement of a player's performance.

[0524] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0525] In this invention, the server includes means for uploading game videos from a terminal, means for receiving and temporarily storing the uploaded game videos, means for extracting frames from the stored game videos, means for analyzing the extracted frames to generate points for improving play, means for inputting individual player data from the terminal, means for receiving the input individual player data and storing it in a database, means for analyzing the received mental data of the player to generate mental care advice, means for analyzing the received physical data of the player to generate a training plan, means for analyzing the received nutritional data of the player to generate a meal plan, and means for integrating and presenting the generated points for improving play, mental care advice, training plan, and meal plan to the player. This makes it possible to consistently and efficiently perform processes from analyzing game videos to collecting individual data and providing comprehensive advice based on that data.

[0526] A "terminal" is a digital device that a user uses to enter information or upload video.

[0527] "Server" means a central processing unit for receiving, storing, analyzing, and providing results from data sent by users.

[0528] "Game video" is a digital file that records footage of a sporting event or other competition.

[0529] A "frame" is each of the consecutive still images that make up a game video.

[0530] "Areas for Improvement" are specific areas where players can improve their technique or tactics, identified through analysis of match video.

[0531] "Individual Data" refers to individual information about a player's own mental, physical and nutritional state.

[0532] "Mental data" refers to information about a player's mental health, such as their psychological state and stress level.

[0533] "Physical data" refers to information about a player's physical condition and training status.

[0534] "Nutrition Data" refers to information regarding a player's diet and nutritional intake.

[0535] "Mental care advice" is specific suggestions for maintaining psychological health that are provided based on the analysis of mental data.

[0536] A "training plan" is a specific exercise program provided based on the analysis of physical data to improve a player's physical strength and skills.

[0537] A "meal plan" is a specific meal menu provided based on the analysis of nutritional data to optimize a player's nutritional balance.

[0538] "Integration" refers to bringing together different analysis results and advice into a single package and presenting them to players in a consistent format.

[0539] This invention is a system that provides optimal training and mental care support based on analysis of game video and individual player data. Detailed modes for carrying out the invention are described below.

[0540] System Configuration

[0541] The whole system consists of terminals, servers, and databases. Terminals are devices used by users to input information and upload videos. The server is the central processing unit, responsible for receiving, storing, analyzing data, and providing results. The database is used to store individual player data and analysis results.

[0542] Hardware and software used

[0543] The server's role is to receive the game videos and individual data sent by the users, analyze them, and generate specific advice. The server uses the following software:

[0544] "ffmpeg": A video analysis tool for breaking down match videos frame by frame.

[0545] "OpenPose" and "YOLO": Artificial intelligence techniques for analyzing player movements within video frames.

[0546] "Scikit-learn" and "TensorFlow": Machine learning libraries used to analyze mental, physical, and nutritional data.

[0547] "MySQL" or "PostgreSQL": A relational database for storing individual data and analysis results.

[0548] Processing Details

[0549] Users upload game videos from their own devices to the server. This operation is performed using a web browser or a dedicated application, and major file formats such as mp4, avi, and mov are supported. The server receives the uploaded video and temporarily stores it in storage. After saving, the video file is broken down into frames using "ffmpeg" to prepare it for analysis.

[0550] Users input individual data such as mental and physical state, nutritional data, etc. on the device screen. This includes self-assessment and data recorded in a dedicated app. For example, a user may enter a number in response to a question such as, "Please rate your stress level this week on a scale of 1 to 10." The server receives this data and stores it in a database for each category.

[0551] The server then analyzes the extracted video frames using OpenPose and YOLO to analyze the player's movements and detect specific patterns and areas for improvement, such as passing accuracy or defensive posture.

[0552] The analysis of mental data uses Scikit-learn and TensorFlow to assess a player's psychological state and generate mental care advice. For example, if the mental stress score is high, mindfulness exercises and relaxation techniques will be suggested.

[0553] The analysis of physical data evaluates the user's physical condition and generates an optimal training plan. For example, if strength training is required, the system will suggest specific exercise menus, number of sets, and number of repetitions.

[0554] The analysis of nutritional data evaluates the user's diet and nutritional intake status and generates an appropriate meal plan. For example, if the user is lacking in protein, it will suggest a high-protein meal menu.

[0555] Examples and prompts

[0556] As a specific example of use, if a user uploads a soccer game video, the server analyzes the video to identify areas for improvement in the player's passing. Next, if the user's mental data indicates that they are feeling stressed, the server will suggest mindfulness exercises. If the physical data indicates that the player needs to strengthen their lower body muscles, the server will provide an appropriate strength training plan. If the nutritional data indicates that the player's protein intake is insufficient, the server will suggest a specific meal plan. For example, it will suggest recipes using protein-rich ingredients such as chicken breast and tofu.

[0557] An example of a prompt to input to a generative AI model is:

[0558] "Please analyze a soccer game video and give me some advice on how to improve passing accuracy. Next, I'd like some advice on mental health, as this player tends to get stressed easily. I'd also like some suggestions on a training plan to strengthen his lower body muscles and a meal menu to increase his protein intake."

[0559] In this way, this system provides comprehensive support for players' technical improvement and mental care.

[0560] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0561] Step 1:

[0562] Users upload game videos.

[0563] Specifically, the user uploads a game video file (e.g., in mp4, avi, or mov format) using a web browser or dedicated application on their own device.

[0564] Input: Match video file

[0565] Output: Send video file to server

[0566] Step 2:

[0567] The server receives the match video file and temporarily stores it.

[0568] The server receives the uploaded video file and temporarily stores it in storage, along with the file name and metadata.

[0569] Input: Video file from user

[0570] Output: Video file in storage

[0571] Step 3:

[0572] The server extracts frames from the video file.

[0573] Using a video analysis tool such as ffmpeg, the match video is broken down into frames. For example, a 30 frame per second video breaks down into 1800 frames per minute.

[0574] Input: Video file in storage

[0575] Output: A collection of extracted frames

[0576] Step 4:

[0577] The user enters the individual data.

[0578] Users enter their individual data, such as mental and physical condition and nutritional data, on the device screen, including self-assessment and data recorded using a dedicated app.

[0579] Input: Mental state, physical state, nutritional data

[0580] Output: Send individual data to the server

[0581] Step 5:

[0582] The server receives the individual data and stores it in a database.

[0583] The server receives the individual data sent by the user and stores it in a database for each category. The data is stored in a relational database such as MySQL or PostgreSQL.

[0584] Input: Individual data from the user

[0585] Output: Data in the database

[0586] Step 6:

[0587] The server analyzes the video frames.

[0588] The server then analyzes the extracted video frames using machine learning models such as OpenPose and YOLO to analyze player movements and identify areas for improvement, such as passing accuracy or defensive posture.

[0589] Input: Extracted video frames

[0590] Output: Play Improvements

[0591] Step 7:

[0592] The server analyzes the mental data and generates mental care advice.

[0593] Using Scikit-learn and TensorFlow, the system analyzes mental data to assess a player's psychological state, suggesting mindfulness exercises if stress levels are high, for example.

[0594] Input: Mental data in a database

[0595] Output: Mental care advice

[0596] Step 8:

[0597] The server analyzes the physical data and generates a training plan.

[0598] The system analyzes physical data and generates training plans to improve the user's strength and technique, for example by suggesting the content, number of sets, and number of repetitions of strength training.

[0599] Input: Physical data in the database

[0600] Output: Training Plan

[0601] Step 9:

[0602] The server analyzes the nutritional data and generates a meal plan.

[0603] The system analyzes nutritional data and generates meal plans to provide users with a balanced diet. For example, if a user is lacking in protein, it will suggest high-protein meals.

[0604] Input: Nutrition data in the database

[0605] Output: Meal plan

[0606] Step 10:

[0607] The server aggregates all the results and serves them to the user.

[0608] The server integrates points for improving play, mental health advice, training plans, and meal plans, and provides them to users as a single package. Users can then check this package on their devices and incorporate it into their practice and daily life.

[0609] Input: Improvements to your game, mental health advice, training plans, meal plans

[0610] Output: Consolidated advice package

[0611] In this way, by identifying the specific inputs and outputs at each step, the flow of the entire system and the data processing and calculations at each processing stage become clear.

[0612] (Application example 1)

[0613] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0614] With conventional training systems, it was difficult to perform detailed analysis of the individual physical and mental data of players and operators and provide optimal advice and plans based on that data. Furthermore, there was a lack of systems that could comprehensively support the optimization of robot movements and the mental care of operators. This resulted in a lack of efficient training and effective mental care, limiting performance improvement.

[0615] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0616] In this invention, the server includes means for receiving game video, means for extracting frames from the received game video, means for analyzing the extracted frames and generating improvement points for play, means for receiving individual player data, means for analyzing the received mental data of the player and providing mental care advice, means for analyzing the received physical data of the player and generating a training plan, means for analyzing the received nutritional data of the player and providing a meal plan, means for integrating the generated improvement points, mental care advice, training plan, and meal plan and presenting them to the player, means for receiving operator data, analyzing the mental data of the operator, and suggesting mindfulness exercises, means for analyzing the physical data and providing a meal plan, and means for analyzing video frames and generating a robot operation strategy. This enables player skill improvement and mental care, as well as optimization of factory robot operation and mental care for operators.

[0617] "Game video" is a recorded video of a sports or game match.

[0618] A "frame" is a unit of still image that is a component of video.

[0619] "Player's individual data" is a collective term for information such as individual mental data, physical data, and nutritional data entered or held by a player.

[0620] "Mental data" refers to data such as the psychological state, emotions, and stress level of players and operators.

[0621] "Physical Data" refers to data relating to the physical health and athletic ability of a player or operator.

[0622] "Nutrition Data" means information about the food and drink consumed by a player or operator, as well as data about the nutrients consumed.

[0623] "Areas for improvement" refers to parts or elements that require correction or improvement in order to improve the performance of the player or robot.

[0624] "Mental care advice" refers to suggestions for psychological support and stress reduction methods provided based on mental data.

[0625] A "training plan" is an exercise or training plan created based on physical data.

[0626] A "meal plan" is a meal plan or nutritional intake plan suggested based on nutritional data.

[0627] "Operator data" is a general term for information including mental and physical data of people working in factories and work sites.

[0628] A "robot motion strategy" is a plan or method for achieving efficient motion of robots used in factories or work sites.

[0629] "Artificial intelligence technology" is a general term for technologies that allow computer systems to analyze and learn from data, automatically discovering patterns and making predictions.

[0630] A "mindfulness exercise" is a set of practices or activities that reduce psychological stress and improve mental well-being by focusing attention on the present moment.

[0631] This invention is a system that supports optimal training, mental health, and nutritional management based on game video analysis and individual player data. It also enables optimization of factory robot operations and supports the mental health of operators. The entire system is implemented using a server, user terminals, and analysis software.

[0632] Program Hardware and Software

[0633] The server receives game videos and robot movement videos and extracts frames. Specifically, it uses OpenCV to extract frames from the video and analyzes the video frames using TensorFlow and Keras. Based on the analysis results, it generates improvements to the player and robot's movements, provides training and diet plans to players, and suggests mental health advice to operators.

[0634] The hardware used is a server equipped with a high-performance CPU / GPU and a camera for video recording, and the software used is OpenCV, TensorFlow, and Keras.

[0635] Data analysis and advice generation

[0636] The server receives the game video and breaks it down into frames. The extracted frames are then analyzed to detect specific patterns in the movement of the play and the robot's behavior. This analysis is performed using TensorFlow's deep learning model. For example, passing accuracy and defensive posture can be identified from a soccer game video and improvements can be made.

[0637] Individual data on players and operators is also sent to the server. For example, the server receives mental data and analyzes it to suggest psychological strengthening methods and mindfulness exercises. Based on physical data, it generates an optimal training plan, providing an appropriate training menu if lower-body muscle strengthening is required. Furthermore, it analyzes nutritional data to propose meal plans, suggesting specific foods for players who are not consuming enough protein, for example.

[0638] Specific examples

[0639] Suppose a user uploads a soccer game video and the server analyzes it. If the server determines that the player's passing accuracy is insufficient, it will suggest areas for improvement. If the server determines that the user is experiencing high stress based on the user's mental data, it will suggest mindfulness exercises. If the server analyzes the user's physical data and determines that the user needs to strengthen their lower body muscles, it will provide an appropriate training plan. If the server determines that the user's protein intake is insufficient based on nutritional data, it will suggest a meal plan to increase protein.

[0640] In addition, by inputting video footage of robots moving in factories and operator data, the system generates an analysis of the video footage, suggestions for mindfulness exercises to address the operator's high stress levels, and a nutrition plan.

[0641] Prompt Sentence Examples

[0642] Example prompts to input to a generative AI model:

[0643] "After uploading and analyzing videos of factory robots in action, we discovered that certain movements were inefficient. Since the operators' stress levels were high, we suggested mindfulness exercises. We also discovered that they were lacking protein, so we proposed a specific diet plan to increase their protein intake."

[0644] In this way, the present invention provides a system that comprehensively supports the improvement of skills and mental care of players and operators.

[0645] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0646] Step 1:

[0647] Users upload game videos and robot movement videos from their devices to the server.

[0648] Input: Match video or robot movement video file

[0649] Specific operation: The user selects a video file using the upload function of the terminal and sends it to the server.

[0650] Output: The video file is saved on the server.

[0651] Step 2:

[0652] The server extracts frames from the received video file.

[0653] Input: Video file

[0654] Data processing: Decompose the video file frame by frame using OpenCV.

[0655] Specific operation: The server reads the video file and extracts specific frames (e.g., every 30th frame).

[0656] Output: A list of extracted frames

[0657] Step 3:

[0658] The server analyzes the extracted frames based on artificial intelligence techniques.

[0659] Input: A list of frames

[0660] Data computation: Using TensorFlow and Keras, we perform analysis using a CNN model for each frame.

[0661] Specific operation: The extracted frames are resized and normalized, and then input into the model to obtain analysis results.

[0662] Output: Improvements to play and specific patterns of robot movement

[0663] Step 4:

[0664] The user sends individual data of the player and operator from the terminal to the server.

[0665] Input: Mental data, physical data, nutritional data

[0666] Specific operation: The user inputs mental data, physical data, and nutritional data on the terminal and transmits the data to the server.

[0667] Output: Individual data stored on the server

[0668] Step 5:

[0669] The server analyzes the received mental data and generates mental care advice.

[0670] Input: Mental data

[0671] Data calculation: Analyzes mental data to evaluate psychological state and stress level and generate appropriate mental care advice.

[0672] Specific behavior: The server suggests mindfulness exercises when stress levels are high.

[0673] Output: Mental care advice

[0674] Step 6:

[0675] The server analyzes the received physical data and generates an optimal training plan.

[0676] Input: Physical data

[0677] Data calculation: Create an appropriate training menu based on physical data.

[0678] Specific Movement: The server generates a specific training plan, such as strengthening muscles.

[0679] Output: Training Plan

[0680] Step 7:

[0681] The server analyzes the received nutritional data and provides a meal plan.

[0682] Input: Nutrition data

[0683] Data calculation: Analyzes nutritional data to identify nutrient deficiencies and suggests necessary meal plans.

[0684] What it does: If you're lacking in protein, the server will suggest a meal plan to increase your protein intake.

[0685] Output: Meal plan

[0686] Step 8:

[0687] The server integrates the generated improvements, mental care advice, training plans, and meal plans and presents them to the user.

[0688] Input: Improvements, mental health advice, training plans, meal plans

[0689] Specific operation: The server integrates this information and sends it to the user terminal as a single package.

[0690] Output: A consolidated improvement plan that can be viewed on the user's device

[0691] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0692] This invention provides optimal training and mental care support based on game video analysis and individual player data. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to generate more accurate advice.

[0693] System Operation Overview

[0694] Users use their devices to upload game videos to the server, which breaks down the video into frames and analyzes each one. Individual player data (mental, physical, and nutritional information) is also sent to the server, and various advice is generated based on this data. The emotion engine recognizes the user's emotions and uses this data in analysis to further improve the quality of advice.

[0695] Specific processing flow

[0696] Video Upload and Analysis

[0697] Users upload game videos from their devices to the server. The server receives the video files and temporarily stores them. Next, it extracts each frame from the video file and analyzes each frame to identify areas for improvement. Specifically, it uses artificial intelligence technology to analyze each frame and evaluate the player's movements.

[0698] Acquiring and analyzing emotion data

[0699] To recognize the user's emotional state, the device is equipped with a camera and a microphone. The emotion engine analyzes the user's facial expressions and tone of voice in real time from these devices and generates emotional data. The server receives this data and evaluates the user's emotional state.

[0700] Getting individual player data

[0701] Users enter mental, physical, and nutritional data into a device and send it to a server, which receives the data, stores it by category, and creates a dataset for analysis.

[0702] Generating comprehensive advice

[0703] The server generates comprehensive advice by integrating the user's emotional, mental, physical, and nutritional data with the points for improvement found in the analyzed video frames. The mental care advice is tailored based on the emotional data. For example, if the user is feeling stressed, it can suggest mindfulness exercises to help relieve stress.

[0704] Training plans are also provided by combining physical and emotional data. For example, if a player is feeling fatigued, it can suggest light exercises.

[0705] The system also incorporates emotional data into nutritional plans, allowing it to provide plans that reflect a user's preferred ingredients and eating patterns. For example, if a user feels they are lacking in energy, it will suggest meals that are high in protein.

[0706] Specific examples

[0707] For example, a user can upload a soccer game video and input the results of a physical fitness test as their own physical data. If the emotion engine detects a state of stress from the user's facial expression, the server will suggest mental care advice (mindfulness exercises) to reduce stress. Furthermore, based on the physical data, it may determine that the user needs to strengthen their lower body muscles, and a training plan for this purpose will be provided. At the same time, nutritional data may detect a protein deficiency and suggest a meal plan to compensate for it.

[0708] In this way, the present invention realizes a system that provides more personalized and advanced support by incorporating user emotional data.

[0709] The processing flow will be explained below.

[0710] The present invention provides optimal training and mental care support based on game video analysis and individual player data. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the system generates more accurate advice. The processing flow of the system for implementing the present invention is specifically described below.

[0711] Specific processing flow of the system

[0712] Step 1:

[0713] Users upload game videos from their devices to the server.

[0714] Specifically, the user selects the game video file on the terminal and clicks the "upload" button.

[0715] Step 2:

[0716] The server temporarily stores the received video file.

[0717] Specifically, the server stores the uploaded video file in a temporary directory.

[0718] Step 3:

[0719] The server extracts frames from the video file.

[0720] Specifically, the server loads the video using cv2.VideoCapture and adds each frame to a list.

[0721] Step 4:

[0722] The server passes the extracted frames to an AI model to analyze the match.

[0723] Specifically, the frame data is passed to the generate_insights method, and the AI ​​model analyzes areas for improvement in the player's play.

[0724] Step 5:

[0725] An emotion engine is used to recognize the user's emotional state.

[0726] Specifically, it analyzes the user's facial expressions and tone of voice in real time through the camera and microphone installed on the device to generate emotional data.

[0727] Step 6:

[0728] The server receives the emotional data and evaluates the user's emotional state.

[0729] Specifically, the emotion data is sent to the server and the analysis results are saved.

[0730] Step 7:

[0731] Users enter mental, physical, and nutritional data into a terminal and send it to the server.

[0732] Specifically, the user enters this data into a form on the terminal and presses the send button.

[0733] Step 8:

[0734] The server analyzes the received mental data and provides mental care advice.

[0735] Specifically, the mental data is passed to the analyze_mental_health method to generate mental care advice.

[0736] Step 9:

[0737] The server analyzes the physical data and generates a training plan.

[0738] Specifically, the physical data is passed to the generate_training_plan method to generate the optimal training plan.

[0739] Step 10:

[0740] The server analyzes the nutritional data and provides meal plans.

[0741] Specifically, the nutritional data is passed to the provide_meal_plan method, which generates an appropriate meal plan.

[0742] Step 11:

[0743] The server then combines the generated improvements, mental health advice, training plans, and meal plans into a single package.

[0744] Specifically, all analysis results are integrated to create a package to be provided to the user.

[0745] Step 12:

[0746] The server sends the combined package to the user.

[0747] Specifically, the server sends the analysis results and plan to the user's terminal so that the user can check them.

[0748] Step 13:

[0749] Users can check the analysis results and advice on their device and incorporate them into their daily training and lifestyle.

[0750] Specifically, the user opens the analysis results page on their device and implements the suggested improvements and plans.

[0751] Specific examples

[0752] For example, if a user uploads a soccer game video and the emotion engine recognizes the user's stress level, the server will suggest mindfulness exercises to reduce stress. At the same time, physical data will determine that the user needs to strengthen their lower body muscles, and a training plan for that purpose will be provided. If nutritional data detects a protein deficiency, a specific meal plan to compensate for that will be proposed. The advice tailored by the emotion engine is then synthesized and provided to the user, improving the quality of their training.

[0753] In this way, the present invention realizes a system that provides more personalized and advanced support by incorporating user emotional data.

[0754] Example 2

[0755] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0756] To improve sports player performance, conventional systems simply analyze game videos and provide general training plans, but they lack detailed advice and support tailored to individual players. They also lack a comprehensive approach that takes into account the user's emotional state. Therefore, there is a need for systems that provide comprehensive support for players' technical development as well as their mental, physical, and nutritional needs.

[0757] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0758] In this invention, the server includes means for receiving game video, means for extracting frames from the received game video, means for analyzing the extracted frames and generating improvements to the player's movements, means for receiving individual player data, means for analyzing the received mental data of the player to provide mental health advice, means for analyzing the received physical data of the player to generate a training plan, means for analyzing the received nutritional data of the player to provide a meal plan, means for acquiring and analyzing user emotional data, means for improving the quality of the advice based on the acquired emotional data, and means for integrating and presenting the generated improvements, mental health advice, training plan, and meal plan to the player. This allows for the provision of personalized, advanced support, enabling comprehensive support not only for the player's technical growth but also for mental, physical, and nutritional aspects.

[0759] A "game video" is a video file that records a sports game.

[0760] A "frame" is an individual still image that makes up a video image.

[0761] "Movement improvement points" refer to specific points or areas that a sports player can use to improve their play during a game.

[0762] "Individual Data" means data relating to each player, including mental, physical and nutritional information.

[0763] "Mental data" is data that contains information about a player's psychological state and emotions.

[0764] "Mental Care Advice" is specific advice or suggestions to improve a player's psychological state and support their mental health.

[0765] "Physical Data" means data containing information about a player's physical condition and performance.

[0766] A "training plan" is a specific training menu and exercise plan based on a player's physical data to improve their physical strength and skills.

[0767] "Nutrition Data" means data containing information about a player's diet and nutritional intake.

[0768] A "meal plan" is a specific dietary suggestion or plan designed to improve a player's nutritional status and optimize performance.

[0769] "Emotion data" refers to data that includes information about the user's emotional state obtained by analyzing the user's facial expression and tone of voice.

[0770] "Artificial intelligence technology" is a technology that uses techniques including machine learning to analyze data and automatically extract knowledge and patterns.

[0771] "Machine learning" is a technique in which computer algorithms learn from data to recognize patterns and create predictive models.

[0772] Mindfulness exercises are mental exercises that help you focus your attention on the present moment and reduce stress and anxiety.

[0773] This system provides optimal training and mental care support based on game video analysis and individual player data. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to generate more accurate advice.

[0774] System Operation Overview

[0775] Users upload game videos from their devices to the server. The server extracts each frame from the received video and analyzes it using artificial intelligence techniques (e.g., OpenCV and TensorFlow). The extracted frames are evaluated to identify areas for improvement in the player's movements.

[0776] Users also input their mental, physical, and nutritional data into the server via their devices. This individual data is stored in a database to create a dataset for analysis. An emotion engine (e.g., Emotion API) is used to obtain end-user emotional data in real time. The server analyzes this data and generates integrated advice.

[0777] Hardware and software used

[0778] Device: The computer or smartphone you use

[0779] Server: A remote server that receives, processes, and analyzes data.

[0780] Artificial intelligence technology: Machine learning libraries such as OpenCV and TensorFlow

[0781] Emotion engine: Emotion API

[0782] Data acquisition and analysis flow

[0783] 1. Upload your video

[0784] Users upload game videos from their devices to the server, and the video files are sent to the designated server via the device application.

[0785] 2. Extracting and analyzing video frames

[0786] The server temporarily stores the received video and uses a video processing tool (e.g., FFmpeg) to extract each frame, which is then analyzed using artificial intelligence techniques.

[0787] 3. Acquiring and sending emotion data

[0788] The device's camera and microphone are used to capture the user's facial expressions and voice, which are then analyzed by the emotion engine. The emotion data analyzed in real time is sent to a server and stored in a database.

[0789] 4. Enter and submit individual data

[0790] Users use a dedicated application to input mental, physical, and nutritional data, which is then sent to a server where it is stored in a database and a data set is created for analysis.

[0791] 5. Data Integration and Advice Generation

[0792] The server integrates all the received data and uses a generative AI model to generate comprehensive advice, which is then displayed on the user's dashboard. For example, if a user uploads a game video and the emotion engine detects stress, the generative AI model will suggest mental care advice (e.g., mindfulness exercises) to reduce stress.

[0793] Specific examples

[0794] For example, a user can upload a soccer game video and enter the results of a recent physical fitness test as their physical data. If the emotion engine detects a state of stress from the user's facial expression, the server will suggest mental care advice (e.g., mindfulness exercises) to reduce stress. Furthermore, based on the physical data, it may determine that the user needs to strengthen their lower body muscles, and a training plan for this purpose will be provided. At the same time, nutritional data may detect a protein deficiency and suggest a meal plan to compensate for it.

[0795] Prompt Sentence Examples

[0796] "If a user uploads a soccer game video and the emotion engine detects stress from the user's facial expressions, what kind of mental health advice can be provided? Also, if the results of a physical fitness test are entered as the user's physical data and the results indicate that the user needs to strengthen their lower body muscles, what kind of training plan can be suggested?"

[0797] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0798] Understood. Now, I will explain the processing flow of this system's program in detail in the following format.

[0799] Step 1:

[0800] The user launches a dedicated application on their device, selects a game video file, and uploads it. When they press the upload button, the device sends the video file to the specified server endpoint (e.g., https: / / example.com / upload). The input is the video file selected by the user, and the output is the video file saved on the server.

[0801] Step 2:

[0802] The server temporarily stores the received video file and then uses a video processing tool such as FFmpeg to decompose the video into frames. A frame is an individual still image that is used in the next analysis step. The input is the received video file, and the output is the extracted frames. Each frame is a sequence of image data.

[0803] Step 3:

[0804] The server uses a deep learning model (e.g., a model using TensorFlow) to analyze each extracted frame. This analysis evaluates the player's movements and identifies areas for improvement. For example, elements of the play such as movement speed and positioning are analyzed. The input is the image data of the frame, and the output is information on areas for improvement.

[0805] Step 4:

[0806] The device uses a built-in camera and microphone to capture the user's facial expressions and voice in real time. An emotion recognition engine such as Emotion API analyzes this data and evaluates the user's emotional state. The generated emotion data is encrypted and sent to a server. The input is the captured facial and voice data, and the output is emotion data.

[0807] Step 5:

[0808] Users enter mental, physical, and nutritional data using a dedicated application. The data entered by the user is sent from the device to a server, which then stores it in a database. The input is the individual data entered by the user, and the output is a dataset for analysis.

[0809] Step 6:

[0810] The server integrates all data (video analysis data, emotional data, individual data). It uses a generative AI model to generate comprehensive advice. For example, it extracts areas for improvement in play from video analysis data, evaluates the need for mental care from emotional data, and generates a training plan from physical data. The input is the integrated dataset, and the output is advice for the user.

[0811] Step 7:

[0812] The user can view the generated advice on the dashboard of the dedicated application. The advice includes mental health advice (e.g., mindfulness exercises for stress reduction), training plans, and meal plans. The input is advice data from the generative AI model, and the output is advice information provided to the user.

[0813] This will enable the realization of an individualized and advanced sports support system.

[0814] (Application example 2)

[0815] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0816] Conventional game video analysis systems and work efficiency improvement systems were unable to provide advanced advice that incorporated the user's emotional and physical data. Furthermore, there was a lack of means to analyze emotional and physical data in real time and provide appropriate advice. This made it difficult to maximize the user's motivation and work efficiency.

[0817] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0818] In this invention, the server includes means for receiving game videos, means for extracting frames from the received game videos, means for analyzing the extracted frames and generating improvement points for play, means for receiving individual player data, means for analyzing the received mental data of the player and providing mental care advice, means for analyzing the received physical data of the player and generating a training plan, means for analyzing the received nutritional data of the player and providing a meal plan, means for integrating the generated improvement points, mental care advice, training plan, and meal plan and presenting them to the player, means for receiving work videos, means for extracting frames from the received work videos, means for analyzing the extracted frames and generating improvement points for work, means for receiving individual worker data, means for analyzing the received mental data of the worker and providing stress care advice, and means for analyzing the received physical data of the worker and generating a work plan. This makes it possible to analyze a user's emotions and physical data in real time and provide optimal advice.

[0819] 1. "Match Video" means a video file that records the match.

[0820] 2. "Work video" refers to a video file that records work being done in a factory or workshop.

[0821] 3. "Frame" means the individual still images that make up a video.

[0822] 4. "Individual Data" means data that indicates the mental, physical, nutritional, and other information of an individual player or worker.

[0823] 5. A "player" is a person who takes part in a sport or other competition.

[0824] 6. "Worker" means a person who performs work in a factory or workshop.

[0825] 7. "Mental Data" refers to data relating to an individual's state of mind or mental health.

[0826] 8. "Physical Data" means data relating to an individual's physical condition and physical strength.

[0827] 9. "Nutrition Data" refers to data relating to an individual's dietary content and nutritional intake status.

[0828] 10. "Mental health advice" means advice to maintain or improve mental health that is provided based on mental data.

[0829] 11. "Stress care advice" refers to specific guidance and advice to alleviate individual stress conditions.

[0830] 12. "Training Plan" means an exercise or training plan created based on physical data.

[0831] 13. "Work Plan" means a plan to improve worker efficiency and safety.

[0832] 14. "Meal Plan" means a meal suggestion based on nutritional data.

[0833] 15. "Artificial intelligence technology" refers to technology that uses techniques such as machine learning and deep learning to analyze data and make intelligent decisions.

[0834] The present invention is a system that analyzes work videos and game videos in real time and provides appropriate advice based on the mental, physical, or nutritional state of each individual user. This system is configured as follows.

[0835] The user uploads the video of their work to the server in real time using smart glasses or other devices. The server then breaks down the received video into frames and analyzes each frame using artificial intelligence techniques such as OpenCV, dlib, and EmotionRecognizer.

[0836] The server also receives the user's individual data (mental, physical, and nutritional information), including data the user previously inputs into the device, and uses an emotion engine to obtain the user's emotional data in real time, which is used for analysis.

[0837] The server combines the analysis results of each frame with the user's emotional and physical data to generate suggestions for improving their gameplay or work. For example, if the emotion engine detects that the user is under stress, it will provide mental care advice to help relieve stress.

[0838] The hardware includes smart glasses, a camera, a microphone, a server, etc. The software uses an analysis program using Python, and libraries such as OpenCV, dlib, and EmotionRecognizer.

[0839] As a concrete example, imagine a factory worker wearing smart glasses while working. If the worker feels stressed and tired, the server will recognize the worker's emotions in real time and provide advice such as "Take a five-minute break" or "Try some light stretching" based on the worker's emotional and physical data.

[0840] Below are some example prompts to input to a generative AI model:

[0841] "Capture sentiment data and generate real-time advice to improve work efficiency."

[0842] This invention makes it possible to analyze a user's emotions and physical data in real time and provide optimal advice, thereby providing comprehensive support for the user's mental and physical state and improving work efficiency and performance.

[0843] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0844] Step 1:

[0845] A user puts on the smart glasses and starts recording a work video. The camera in the smart glasses captures the work video and streams it in real time.

[0846] Input: User's working video

[0847] Output: Real-time captured video data

[0848] Step 2:

[0849] The server receives the video data in real time and extracts each frame, which is then decomposed into individual still images using OpenCV.

[0850] Input: Real-time captured video data

[0851] Output: Extracted frames (a sequence of still images)

[0852] Step 3:

[0853] The server analyzes the extracted frames and evaluates the worker's movements, using dlib for face detection and EmotionRecognizer to determine emotions.

[0854] Input: Extracted frames

[0855] Output: Emotion data (e.g., stress, excitement, fatigue, etc.)

[0856] Step 4:

[0857] The server receives the user's individual data (mental, physical, and nutritional information). Data entered from the device is sent to the server.

[0858] Input: Individual data (mental, physical, nutritional information)

[0859] Output: Individual data received

[0860] Step 5:

[0861] The server combines the individual data received and the analysis results (emotion data) of the extracted frames to comprehensively evaluate the user's condition, and uses an AI model to generate appropriate advice (mental care, work plans, meal plans, etc.).

[0862] Input: Emotion data, individual data

[0863] Output: Comprehensive advice (e.g., mental health advice, stress care advice, work plan, meal plan, etc.)

[0864] Step 6:

[0865] The generated advice is displayed on the user's smart glasses. For example, if the emotion data indicates stress, the advice "Take a 5-minute break" is displayed.

[0866] Input: General Advice

[0867] Output: Advice to the user

[0868] Step 7:

[0869] The user follows the advice to continue or stop working and take appropriate care, such as stretching, taking breaks, or following a meal plan.

[0870] Input: User advice

[0871] Output: Improved work or health status of the user

[0872] Through these steps, the server analyzes the user's emotional and physical data in real time and provides optimal advice, thereby providing comprehensive support for the user's mental and physical state and improving work efficiency and performance.

[0873] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0874] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0875] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0876] [Third embodiment]

[0877] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0878] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0879] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0880] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0881] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0882] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0883] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0884] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0885] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0886] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0887] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0888] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0889] The present invention provides optimal training and mental care support based on analysis of game video and individual player data. Below, the program and processing of the system for implementing the present invention are explained in natural language.

[0890] overview

[0891] Users upload game videos from their devices to the server, then input the player's individual data (mental, physical, and nutritional information). The server receives and analyzes this data, and generates and provides each player with individual improvement points, training plans, mental care advice, and meal plans.

[0892] Specific actions

[0893] Video Upload and Analysis

[0894] Users upload video files of recorded matches from their devices to the server. The server receives the video files and temporarily stores them. After storing them, the server extracts frames from the video files. This extraction process breaks down the video into frames and prepares each frame for analysis.

[0895] Getting individual player data

[0896] Users input their mental and physical state and nutritional data into their devices and send it to the server, which receives the data, classifies it into categories, and stores it.

[0897] Data analysis and advice generation

[0898] The server then uses the extracted video frames to analyze the play, using artificial intelligence technology to detect specific patterns and identify areas that need improvement, such as passing accuracy or defensive posture.

[0899] The server then analyzes the user's mental health based on the mental data received from the user, and generates specific mental care advice, such as ways to improve the player's mental health and reduce stress. For example, it can suggest mindfulness exercises.

[0900] The server then analyzes the physical data and generates an optimal training plan, such as a training plan aimed at improving muscle strength or providing a menu for strengthening specific body parts.

[0901] Finally, the server analyzes the nutritional data and generates an appropriate meal plan, which provides nutritional recommendations to support the player's performance, such as specific meal plans to increase protein intake.

[0902] Presentation of results

[0903] The server then integrates all generated improvements, mental health advice, training plans, and meal plans into a single package and provides it to the user, who can then view the package on their own device and incorporate it into their daily practice and life.

[0904] Specific examples

[0905] For example, if a user uploads a video of a soccer game, the server will analyze the video to identify areas for improvement in the player's passing. If the mental data indicates that the user is feeling stressed, the server will suggest mindfulness exercises. If the physical data indicates that the player needs to strengthen their lower body muscles, the server will provide an appropriate strength training plan. If the nutritional data indicates that the player is not consuming enough protein, the server will suggest a specific meal plan.

[0906] In this way, the present invention provides a system that comprehensively supports players in improving their skills and mental health.

[0907] The processing flow will be explained below.

[0908] Step 1:

[0909] Users upload game videos from their devices to the server.

[0910] Specifically, the user selects the game video file on the terminal and clicks the "upload" button.

[0911] Step 2:

[0912] The server temporarily stores the received video file.

[0913] Specifically, the server stores the uploaded video file in a temporary directory.

[0914] Step 3:

[0915] The server extracts frames from the video file.

[0916] Specifically, the server loads the video using cv2.VideoCapture and adds each frame to a list.

[0917] Step 4:

[0918] The server passes the extracted frames to an AI model to analyze the match.

[0919] Specifically, the frame data is passed to the generate_insights method, and the AI ​​model analyzes areas for improvement in the player's play.

[0920] Step 5:

[0921] The server receives mental, physical, and nutritional data from the user.

[0922] Specifically, the user enters this data into a form on the terminal and sends it to the server's API.

[0923] Step 6:

[0924] The server analyzes the mental data and generates mental care advice.

[0925] Specifically, the server passes the mental data to the analyze_mental_health method to generate mental care advice.

[0926] Step 7:

[0927] The server analyzes the physical data and generates a training plan.

[0928] Specifically, the server passes the physical data to the generate_training_plan method to generate a physical training plan.

[0929] Step 8:

[0930] The server analyzes the nutritional data and generates a meal plan.

[0931] Specifically, the server passes the nutritional data to the provide_meal_plan method to generate a meal plan.

[0932] Step 9:

[0933] The server then combines the generated improvements, mental health advice, training plans, and meal plans into a single package.

[0934] Specifically, all analysis results are integrated into one data set.

[0935] Step 10:

[0936] The server sends the combined package to the user.

[0937] Specifically, the server sends the analysis results and plan to the user's terminal so that the user can check them.

[0938] Step 11:

[0939] Users can check the analysis results and advice on their device and incorporate them into their daily training and lifestyle.

[0940] Specifically, the user opens the analysis results page on their device and implements the suggested improvements and plans.

[0941] Example 1

[0942] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0943] In modern sports, improving a player's skills and optimizing their performance involves a wide range of factors. In particular, there is a need for comprehensive analysis of game video and individual player data (mental, physical, and nutritional data). However, there is no system that consistently performs these tasks, and manually performing them by players and coaches requires a great deal of effort and time. As a result, it is difficult to efficiently review games, plan the next training session, or provide mental care, which can delay the improvement of a player's performance.

[0944] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0945] In this invention, the server includes means for uploading game videos from a terminal, means for receiving and temporarily storing the uploaded game videos, means for extracting frames from the stored game videos, means for analyzing the extracted frames to generate points for improving play, means for inputting individual player data from the terminal, means for receiving the input individual player data and storing it in a database, means for analyzing the received mental data of the player to generate mental care advice, means for analyzing the received physical data of the player to generate a training plan, means for analyzing the received nutritional data of the player to generate a meal plan, and means for integrating and presenting the generated points for improving play, mental care advice, training plan, and meal plan to the player. This makes it possible to consistently and efficiently perform processes from analyzing game videos to collecting individual data and providing comprehensive advice based on that data.

[0946] A "terminal" is a digital device that a user uses to enter information or upload video.

[0947] "Server" means a central processing unit for receiving, storing, analyzing, and providing results from data sent by users.

[0948] "Game video" is a digital file that records footage of a sporting event or other competition.

[0949] A "frame" is each of the consecutive still images that make up a game video.

[0950] "Areas for Improvement" are specific areas where players can improve their technique or tactics, identified through analysis of match video.

[0951] "Individual Data" refers to individual information about a player's own mental, physical and nutritional state.

[0952] "Mental data" refers to information about a player's mental health, such as their psychological state and stress level.

[0953] "Physical data" refers to information about a player's physical condition and training status.

[0954] "Nutrition Data" refers to information regarding a player's diet and nutritional intake.

[0955] "Mental care advice" is specific suggestions for maintaining psychological health that are provided based on the analysis of mental data.

[0956] A "training plan" is a specific exercise program provided based on the analysis of physical data to improve a player's physical strength and skills.

[0957] A "meal plan" is a specific meal menu provided based on the analysis of nutritional data to optimize a player's nutritional balance.

[0958] "Integration" refers to bringing together different analysis results and advice into a single package and presenting them to players in a consistent format.

[0959] This invention is a system that provides optimal training and mental care support based on analysis of game video and individual player data. Detailed modes for carrying out the invention are described below.

[0960] System Configuration

[0961] The whole system consists of terminals, servers, and databases. Terminals are devices used by users to input information and upload videos. The server is the central processing unit, responsible for receiving, storing, analyzing data, and providing results. The database is used to store individual player data and analysis results.

[0962] Hardware and software used

[0963] The server's role is to receive the game videos and individual data sent by the users, analyze them, and generate specific advice. The server uses the following software:

[0964] "ffmpeg": A video analysis tool for breaking down match videos frame by frame.

[0965] "OpenPose" and "YOLO": Artificial intelligence techniques for analyzing player movements within video frames.

[0966] "Scikit-learn" and "TensorFlow": Machine learning libraries used to analyze mental, physical, and nutritional data.

[0967] "MySQL" or "PostgreSQL": A relational database for storing individual data and analysis results.

[0968] Processing Details

[0969] Users upload game videos from their own devices to the server. This operation is performed using a web browser or a dedicated application, and major file formats such as mp4, avi, and mov are supported. The server receives the uploaded video and temporarily stores it in storage. After saving, the video file is broken down into frames using "ffmpeg" to prepare it for analysis.

[0970] Users input individual data such as mental and physical state, nutritional data, etc. on the device screen. This includes self-assessment and data recorded in a dedicated app. For example, a user may enter a number in response to a question such as, "Please rate your stress level this week on a scale of 1 to 10." The server receives this data and stores it in a database for each category.

[0971] The server then analyzes the extracted video frames using OpenPose and YOLO to analyze the player's movements and detect specific patterns and areas for improvement, such as passing accuracy or defensive posture.

[0972] The analysis of mental data uses Scikit-learn and TensorFlow to assess a player's psychological state and generate mental care advice. For example, if the mental stress score is high, mindfulness exercises and relaxation techniques will be suggested.

[0973] The analysis of physical data evaluates the user's physical condition and generates an optimal training plan. For example, if strength training is required, the system will suggest specific exercise menus, number of sets, and number of repetitions.

[0974] The analysis of nutritional data evaluates the user's diet and nutritional intake status and generates an appropriate meal plan. For example, if the user is lacking in protein, it will suggest a high-protein meal menu.

[0975] Examples and prompts

[0976] As a specific example of use, if a user uploads a soccer game video, the server analyzes the video to identify areas for improvement in the player's passing. Next, if the user's mental data indicates that they are feeling stressed, the server will suggest mindfulness exercises. If the physical data indicates that the player needs to strengthen their lower body muscles, the server will provide an appropriate strength training plan. If the nutritional data indicates that the player's protein intake is insufficient, the server will suggest a specific meal plan. For example, it will suggest recipes using protein-rich ingredients such as chicken breast and tofu.

[0977] An example of a prompt to input to a generative AI model is:

[0978] "Please analyze a soccer game video and give me some advice on how to improve passing accuracy. Next, I'd like some advice on mental health, as this player tends to get stressed easily. I'd also like some suggestions on a training plan to strengthen his lower body muscles and a meal menu to increase his protein intake."

[0979] In this way, this system provides comprehensive support for players' technical improvement and mental care.

[0980] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0981] Step 1:

[0982] Users upload game videos.

[0983] Specifically, the user uploads a game video file (e.g., in mp4, avi, or mov format) using a web browser or dedicated application on their own device.

[0984] Input: Match video file

[0985] Output: Send video file to server

[0986] Step 2:

[0987] The server receives the match video file and temporarily stores it.

[0988] The server receives the uploaded video file and temporarily stores it in storage, along with the file name and metadata.

[0989] Input: Video file from user

[0990] Output: Video file in storage

[0991] Step 3:

[0992] The server extracts frames from the video file.

[0993] Using a video analysis tool such as ffmpeg, the match video is broken down into frames. For example, a 30 frame per second video breaks down into 1800 frames per minute.

[0994] Input: Video file in storage

[0995] Output: A collection of extracted frames

[0996] Step 4:

[0997] The user enters the individual data.

[0998] Users enter their individual data, such as mental and physical condition and nutritional data, on the device screen, including self-assessment and data recorded using a dedicated app.

[0999] Input: Mental state, physical state, nutritional data

[1000] Output: Send individual data to the server

[1001] Step 5:

[1002] The server receives the individual data and stores it in a database.

[1003] The server receives the individual data sent by the user and stores it in a database for each category. The data is stored in a relational database such as MySQL or PostgreSQL.

[1004] Input: Individual data from the user

[1005] Output: Data in the database

[1006] Step 6:

[1007] The server analyzes the video frames.

[1008] The server then analyzes the extracted video frames using machine learning models such as OpenPose and YOLO to analyze player movements and identify areas for improvement, such as passing accuracy or defensive posture.

[1009] Input: Extracted video frames

[1010] Output: Play Improvements

[1011] Step 7:

[1012] The server analyzes the mental data and generates mental care advice.

[1013] Using Scikit-learn and TensorFlow, the system analyzes mental data to assess a player's psychological state, suggesting mindfulness exercises if stress levels are high, for example.

[1014] Input: Mental data in a database

[1015] Output: Mental care advice

[1016] Step 8:

[1017] The server analyzes the physical data and generates a training plan.

[1018] The system analyzes physical data and generates training plans to improve the user's strength and technique, for example by suggesting the content, number of sets, and number of repetitions of strength training.

[1019] Input: Physical data in the database

[1020] Output: Training Plan

[1021] Step 9:

[1022] The server analyzes the nutritional data and generates a meal plan.

[1023] The system analyzes nutritional data and generates meal plans to provide users with a balanced diet. For example, if a user is lacking in protein, it will suggest high-protein meals.

[1024] Input: Nutrition data in the database

[1025] Output: Meal plan

[1026] Step 10:

[1027] The server aggregates all the results and serves them to the user.

[1028] The server integrates points for improving play, mental health advice, training plans, and meal plans, and provides them to users as a single package. Users can then check this package on their devices and incorporate it into their practice and daily life.

[1029] Input: Improvements to your game, mental health advice, training plans, meal plans

[1030] Output: Consolidated advice package

[1031] In this way, by identifying the specific inputs and outputs at each step, the flow of the entire system and the data processing and calculations at each processing stage become clear.

[1032] (Application example 1)

[1033] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1034] With conventional training systems, it was difficult to perform detailed analysis of the individual physical and mental data of players and operators and provide optimal advice and plans based on that data. Furthermore, there was a lack of systems that could comprehensively support the optimization of robot movements and the mental care of operators. This resulted in a lack of efficient training and effective mental care, limiting performance improvement.

[1035] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1036] In this invention, the server includes means for receiving game video, means for extracting frames from the received game video, means for analyzing the extracted frames and generating improvement points for play, means for receiving individual player data, means for analyzing the received mental data of the player and providing mental care advice, means for analyzing the received physical data of the player and generating a training plan, means for analyzing the received nutritional data of the player and providing a meal plan, means for integrating the generated improvement points, mental care advice, training plan, and meal plan and presenting them to the player, means for receiving operator data, analyzing the mental data of the operator, and suggesting mindfulness exercises, means for analyzing the physical data and providing a meal plan, and means for analyzing video frames and generating a robot operation strategy. This enables player skill improvement and mental care, as well as optimization of factory robot operation and mental care for operators.

[1037] "Game video" is a recorded video of a sports or game match.

[1038] A "frame" is a unit of still image that is a component of video.

[1039] "Player's individual data" is a collective term for information such as individual mental data, physical data, and nutritional data entered or held by a player.

[1040] "Mental data" refers to data such as the psychological state, emotions, and stress level of players and operators.

[1041] "Physical Data" refers to data relating to the physical health and athletic ability of a player or operator.

[1042] "Nutrition Data" means information about the food and drink consumed by a player or operator, as well as data about the nutrients consumed.

[1043] "Areas for improvement" refers to parts or elements that require correction or improvement in order to improve the performance of the player or robot.

[1044] "Mental care advice" refers to suggestions for psychological support and stress reduction methods provided based on mental data.

[1045] A "training plan" is an exercise or training plan created based on physical data.

[1046] A "meal plan" is a meal plan or nutritional intake plan suggested based on nutritional data.

[1047] "Operator data" is a general term for information including mental and physical data of people working in factories and work sites.

[1048] A "robot motion strategy" is a plan or method for achieving efficient motion of robots used in factories or work sites.

[1049] "Artificial intelligence technology" is a general term for technologies that allow computer systems to analyze and learn from data, automatically discovering patterns and making predictions.

[1050] A "mindfulness exercise" is a set of practices or activities that reduce psychological stress and improve mental well-being by focusing attention on the present moment.

[1051] This invention is a system that supports optimal training, mental health, and nutritional management based on game video analysis and individual player data. It also enables optimization of factory robot operations and supports the mental health of operators. The entire system is implemented using a server, user terminals, and analysis software.

[1052] Program Hardware and Software

[1053] The server receives game videos and robot movement videos and extracts frames. Specifically, it uses OpenCV to extract frames from the video and analyzes the video frames using TensorFlow and Keras. Based on the analysis results, it generates improvements to the player and robot's movements, provides training and diet plans to players, and suggests mental health advice to operators.

[1054] The hardware used is a server equipped with a high-performance CPU / GPU and a camera for video recording, and the software used is OpenCV, TensorFlow, and Keras.

[1055] Data analysis and advice generation

[1056] The server receives the game video and breaks it down into frames. The extracted frames are then analyzed to detect specific patterns in the movement of the play and the robot's behavior. This analysis is performed using TensorFlow's deep learning model. For example, passing accuracy and defensive posture can be identified from a soccer game video and improvements can be made.

[1057] Individual data on players and operators is also sent to the server. For example, the server receives mental data and analyzes it to suggest psychological strengthening methods and mindfulness exercises. Based on physical data, it generates an optimal training plan, providing an appropriate training menu if lower-body muscle strengthening is required. Furthermore, it analyzes nutritional data to propose meal plans, suggesting specific foods for players who are not consuming enough protein, for example.

[1058] Specific examples

[1059] Suppose a user uploads a soccer game video and the server analyzes it. If the server determines that the player's passing accuracy is insufficient, it will suggest areas for improvement. If the server determines that the user is experiencing high stress based on the user's mental data, it will suggest mindfulness exercises. If the server analyzes the user's physical data and determines that the user needs to strengthen their lower body muscles, it will provide an appropriate training plan. If the server determines that the user's protein intake is insufficient based on nutritional data, it will suggest a meal plan to increase protein.

[1060] In addition, by inputting video footage of robots moving in factories and operator data, the system generates an analysis of the video footage, suggestions for mindfulness exercises to address the operator's high stress levels, and a nutrition plan.

[1061] Prompt Sentence Examples

[1062] Example prompts to input to a generative AI model:

[1063] "After uploading and analyzing videos of factory robots in action, we discovered that certain movements were inefficient. Since the operators' stress levels were high, we suggested mindfulness exercises. We also discovered that they were lacking protein, so we proposed a specific diet plan to increase their protein intake."

[1064] In this way, the present invention provides a system that comprehensively supports the improvement of skills and mental care of players and operators.

[1065] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1066] Step 1:

[1067] Users upload game videos and robot movement videos from their devices to the server.

[1068] Input: Match video or robot movement video file

[1069] Specific operation: The user selects a video file using the upload function of the terminal and sends it to the server.

[1070] Output: The video file is saved on the server.

[1071] Step 2:

[1072] The server extracts frames from the received video file.

[1073] Input: Video file

[1074] Data processing: Decompose the video file frame by frame using OpenCV.

[1075] Specific operation: The server reads the video file and extracts specific frames (e.g., every 30th frame).

[1076] Output: A list of extracted frames

[1077] Step 3:

[1078] The server analyzes the extracted frames based on artificial intelligence techniques.

[1079] Input: A list of frames

[1080] Data computation: Using TensorFlow and Keras, we perform analysis using a CNN model for each frame.

[1081] Specific operation: The extracted frames are resized and normalized, and then input into the model to obtain analysis results.

[1082] Output: Improvements to play and specific patterns of robot movement

[1083] Step 4:

[1084] The user sends individual data of the player and operator from the terminal to the server.

[1085] Input: Mental data, physical data, nutritional data

[1086] Specific operation: The user inputs mental data, physical data, and nutritional data on the terminal and transmits the data to the server.

[1087] Output: Individual data stored on the server

[1088] Step 5:

[1089] The server analyzes the received mental data and generates mental care advice.

[1090] Input: Mental data

[1091] Data calculation: Analyzes mental data to evaluate psychological state and stress level and generate appropriate mental care advice.

[1092] Specific behavior: The server suggests mindfulness exercises when stress levels are high.

[1093] Output: Mental care advice

[1094] Step 6:

[1095] The server analyzes the received physical data and generates an optimal training plan.

[1096] Input: Physical data

[1097] Data calculation: Create an appropriate training menu based on physical data.

[1098] Specific Movement: The server generates a specific training plan, such as strengthening muscles.

[1099] Output: Training Plan

[1100] Step 7:

[1101] The server analyzes the received nutritional data and provides a meal plan.

[1102] Input: Nutrition data

[1103] Data calculation: Analyzes nutritional data to identify nutrient deficiencies and suggests necessary meal plans.

[1104] What it does: If you're lacking in protein, the server will suggest a meal plan to increase your protein intake.

[1105] Output: Meal plan

[1106] Step 8:

[1107] The server integrates the generated improvements, mental care advice, training plans, and meal plans and presents them to the user.

[1108] Input: Improvements, mental health advice, training plans, meal plans

[1109] Specific operation: The server integrates this information and sends it to the user terminal as a single package.

[1110] Output: A consolidated improvement plan that can be viewed on the user's device

[1111] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1112] This invention provides optimal training and mental care support based on game video analysis and individual player data. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to generate more accurate advice.

[1113] System Operation Overview

[1114] Users use their devices to upload game videos to the server, which breaks down the video into frames and analyzes each one. Individual player data (mental, physical, and nutritional information) is also sent to the server, and various advice is generated based on this data. The emotion engine recognizes the user's emotions and uses this data in analysis to further improve the quality of advice.

[1115] Specific processing flow

[1116] Video Upload and Analysis

[1117] Users upload game videos from their devices to the server. The server receives the video files and temporarily stores them. Next, it extracts each frame from the video file and analyzes each frame to identify areas for improvement. Specifically, it uses artificial intelligence technology to analyze each frame and evaluate the player's movements.

[1118] Acquiring and analyzing emotion data

[1119] To recognize the user's emotional state, the device is equipped with a camera and a microphone. The emotion engine analyzes the user's facial expressions and tone of voice in real time from these devices and generates emotional data. The server receives this data and evaluates the user's emotional state.

[1120] Getting individual player data

[1121] Users enter mental, physical, and nutritional data into a device and send it to a server, which receives the data, stores it by category, and creates a dataset for analysis.

[1122] Generating comprehensive advice

[1123] The server generates comprehensive advice by integrating the user's emotional, mental, physical, and nutritional data with the points for improvement found in the analyzed video frames. The mental care advice is tailored based on the emotional data. For example, if the user is feeling stressed, it can suggest mindfulness exercises to help relieve stress.

[1124] Training plans are also provided by combining physical and emotional data. For example, if a player is feeling fatigued, it can suggest light exercises.

[1125] The system also incorporates emotional data into nutritional plans, allowing it to provide plans that reflect a user's preferred ingredients and eating patterns. For example, if a user feels they are lacking in energy, it will suggest meals that are high in protein.

[1126] Specific examples

[1127] For example, a user can upload a soccer game video and input the results of a physical fitness test as their own physical data. If the emotion engine detects a state of stress from the user's facial expression, the server will suggest mental care advice (mindfulness exercises) to reduce stress. Furthermore, based on the physical data, it may determine that the user needs to strengthen their lower body muscles, and a training plan for this purpose will be provided. At the same time, nutritional data may detect a protein deficiency and suggest a meal plan to compensate for it.

[1128] In this way, the present invention realizes a system that provides more personalized and advanced support by incorporating user emotional data.

[1129] The processing flow will be explained below.

[1130] The present invention provides optimal training and mental care support based on game video analysis and individual player data. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the system generates more accurate advice. The processing flow of the system for implementing the present invention is specifically described below.

[1131] Specific processing flow of the system

[1132] Step 1:

[1133] Users upload game videos from their devices to the server.

[1134] Specifically, the user selects the game video file on the terminal and clicks the "upload" button.

[1135] Step 2:

[1136] The server temporarily stores the received video file.

[1137] Specifically, the server stores the uploaded video file in a temporary directory.

[1138] Step 3:

[1139] The server extracts frames from the video file.

[1140] Specifically, the server loads the video using cv2.VideoCapture and adds each frame to a list.

[1141] Step 4:

[1142] The server passes the extracted frames to an AI model to analyze the match.

[1143] Specifically, the frame data is passed to the generate_insights method, and the AI ​​model analyzes areas for improvement in the player's play.

[1144] Step 5:

[1145] An emotion engine is used to recognize the user's emotional state.

[1146] Specifically, it analyzes the user's facial expressions and tone of voice in real time through the camera and microphone installed on the device to generate emotional data.

[1147] Step 6:

[1148] The server receives the emotional data and evaluates the user's emotional state.

[1149] Specifically, the emotion data is sent to the server and the analysis results are saved.

[1150] Step 7:

[1151] Users enter mental, physical, and nutritional data into a terminal and send it to the server.

[1152] Specifically, the user enters this data into a form on the terminal and presses the send button.

[1153] Step 8:

[1154] The server analyzes the received mental data and provides mental care advice.

[1155] Specifically, the mental data is passed to the analyze_mental_health method to generate mental care advice.

[1156] Step 9:

[1157] The server analyzes the physical data and generates a training plan.

[1158] Specifically, the physical data is passed to the generate_training_plan method to generate the optimal training plan.

[1159] Step 10:

[1160] The server analyzes the nutritional data and provides meal plans.

[1161] Specifically, the nutritional data is passed to the provide_meal_plan method, which generates an appropriate meal plan.

[1162] Step 11:

[1163] The server then combines the generated improvements, mental health advice, training plans, and meal plans into a single package.

[1164] Specifically, all analysis results are integrated to create a package to be provided to the user.

[1165] Step 12:

[1166] The server sends the combined package to the user.

[1167] Specifically, the server sends the analysis results and plan to the user's terminal so that the user can check them.

[1168] Step 13:

[1169] Users can check the analysis results and advice on their device and incorporate them into their daily training and lifestyle.

[1170] Specifically, the user opens the analysis results page on their device and implements the suggested improvements and plans.

[1171] Specific examples

[1172] For example, if a user uploads a soccer game video and the emotion engine recognizes the user's stress level, the server will suggest mindfulness exercises to reduce stress. At the same time, physical data will determine that the user needs to strengthen their lower body muscles, and a training plan for that purpose will be provided. If nutritional data detects a protein deficiency, a specific meal plan to compensate for that will be proposed. The advice tailored by the emotion engine is then synthesized and provided to the user, improving the quality of their training.

[1173] In this way, the present invention realizes a system that provides more personalized and advanced support by incorporating user emotional data.

[1174] Example 2

[1175] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1176] To improve sports player performance, conventional systems simply analyze game videos and provide general training plans, but they lack detailed advice and support tailored to individual players. They also lack a comprehensive approach that takes into account the user's emotional state. Therefore, there is a need for systems that provide comprehensive support for players' technical development as well as their mental, physical, and nutritional needs.

[1177] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1178] In this invention, the server includes means for receiving game video, means for extracting frames from the received game video, means for analyzing the extracted frames and generating improvements to the player's movements, means for receiving individual player data, means for analyzing the received mental data of the player to provide mental health advice, means for analyzing the received physical data of the player to generate a training plan, means for analyzing the received nutritional data of the player to provide a meal plan, means for acquiring and analyzing user emotional data, means for improving the quality of the advice based on the acquired emotional data, and means for integrating and presenting the generated improvements, mental health advice, training plan, and meal plan to the player. This allows for the provision of personalized, advanced support, enabling comprehensive support not only for the player's technical growth but also for mental, physical, and nutritional aspects.

[1179] A "game video" is a video file that records a sports game.

[1180] A "frame" is an individual still image that makes up a video image.

[1181] "Movement improvement points" refer to specific points or areas that a sports player can use to improve their play during a game.

[1182] "Individual Data" means data relating to each player, including mental, physical and nutritional information.

[1183] "Mental data" is data that contains information about a player's psychological state and emotions.

[1184] "Mental Care Advice" is specific advice or suggestions to improve a player's psychological state and support their mental health.

[1185] "Physical Data" means data containing information about a player's physical condition and performance.

[1186] A "training plan" is a specific training menu and exercise plan based on a player's physical data to improve their physical strength and skills.

[1187] "Nutrition Data" means data containing information about a player's diet and nutritional intake.

[1188] A "meal plan" is a specific dietary suggestion or plan designed to improve a player's nutritional status and optimize performance.

[1189] "Emotion data" refers to data that includes information about the user's emotional state obtained by analyzing the user's facial expression and tone of voice.

[1190] "Artificial intelligence technology" is a technology that uses techniques including machine learning to analyze data and automatically extract knowledge and patterns.

[1191] "Machine learning" is a technique in which computer algorithms learn from data to recognize patterns and create predictive models.

[1192] Mindfulness exercises are mental exercises that help you focus your attention on the present moment and reduce stress and anxiety.

[1193] This system provides optimal training and mental care support based on game video analysis and individual player data. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to generate more accurate advice.

[1194] System Operation Overview

[1195] Users upload game videos from their devices to the server. The server extracts each frame from the received video and analyzes it using artificial intelligence techniques (e.g., OpenCV and TensorFlow). The extracted frames are evaluated to identify areas for improvement in the player's movements.

[1196] Users also input their mental, physical, and nutritional data into the server via their devices. This individual data is stored in a database to create a dataset for analysis. An emotion engine (e.g., Emotion API) is used to obtain end-user emotional data in real time. The server analyzes this data and generates integrated advice.

[1197] Hardware and software used

[1198] Device: The computer or smartphone you use

[1199] Server: A remote server that receives, processes, and analyzes data.

[1200] Artificial intelligence technology: Machine learning libraries such as OpenCV and TensorFlow

[1201] Emotion engine: Emotion API

[1202] Data acquisition and analysis flow

[1203] 1. Upload your video

[1204] Users upload game videos from their devices to the server, and the video files are sent to the designated server via the device application.

[1205] 2. Extracting and analyzing video frames

[1206] The server temporarily stores the received video and uses a video processing tool (e.g., FFmpeg) to extract each frame, which is then analyzed using artificial intelligence techniques.

[1207] 3. Acquiring and sending emotion data

[1208] The device's camera and microphone are used to capture the user's facial expressions and voice, which are then analyzed by the emotion engine. The emotion data analyzed in real time is sent to a server and stored in a database.

[1209] 4. Enter and submit individual data

[1210] Users use a dedicated application to input mental, physical, and nutritional data, which is then sent to a server where it is stored in a database and a data set is created for analysis.

[1211] 5. Data Integration and Advice Generation

[1212] The server integrates all the received data and uses a generative AI model to generate comprehensive advice, which is then displayed on the user's dashboard. For example, if a user uploads a game video and the emotion engine detects stress, the generative AI model will suggest mental care advice (e.g., mindfulness exercises) to reduce stress.

[1213] Specific examples

[1214] For example, a user can upload a soccer game video and enter the results of a recent physical fitness test as their physical data. If the emotion engine detects a state of stress from the user's facial expression, the server will suggest mental care advice (e.g., mindfulness exercises) to reduce stress. Furthermore, based on the physical data, it may determine that the user needs to strengthen their lower body muscles, and a training plan for this purpose will be provided. At the same time, nutritional data may detect a protein deficiency and suggest a meal plan to compensate for it.

[1215] Prompt Sentence Examples

[1216] "If a user uploads a soccer game video and the emotion engine detects stress from the user's facial expressions, what kind of mental health advice can be provided? Also, if the results of a physical fitness test are entered as the user's physical data and the results indicate that the user needs to strengthen their lower body muscles, what kind of training plan can be suggested?"

[1217] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1218] Understood. Now, I will explain the processing flow of this system's program in detail in the following format.

[1219] Step 1:

[1220] The user launches a dedicated application on their device, selects a game video file, and uploads it. When they press the upload button, the device sends the video file to the specified server endpoint (e.g., https: / / example.com / upload). The input is the video file selected by the user, and the output is the video file saved on the server.

[1221] Step 2:

[1222] The server temporarily stores the received video file and then uses a video processing tool such as FFmpeg to decompose the video into frames. A frame is an individual still image that is used in the next analysis step. The input is the received video file, and the output is the extracted frames. Each frame is a sequence of image data.

[1223] Step 3:

[1224] The server uses a deep learning model (e.g., a model using TensorFlow) to analyze each extracted frame. This analysis evaluates the player's movements and identifies areas for improvement. For example, elements of the play such as movement speed and positioning are analyzed. The input is the image data of the frame, and the output is information on areas for improvement.

[1225] Step 4:

[1226] The device uses a built-in camera and microphone to capture the user's facial expressions and voice in real time. An emotion recognition engine such as Emotion API analyzes this data and evaluates the user's emotional state. The generated emotion data is encrypted and sent to a server. The input is the captured facial and voice data, and the output is emotion data.

[1227] Step 5:

[1228] Users enter mental, physical, and nutritional data using a dedicated application. The data entered by the user is sent from the device to a server, which then stores it in a database. The input is the individual data entered by the user, and the output is a dataset for analysis.

[1229] Step 6:

[1230] The server integrates all data (video analysis data, emotional data, individual data). It uses a generative AI model to generate comprehensive advice. For example, it extracts areas for improvement in play from video analysis data, evaluates the need for mental care from emotional data, and generates a training plan from physical data. The input is the integrated dataset, and the output is advice for the user.

[1231] Step 7:

[1232] The user can view the generated advice on the dashboard of the dedicated application. The advice includes mental health advice (e.g., mindfulness exercises for stress reduction), training plans, and meal plans. The input is advice data from the generative AI model, and the output is advice information provided to the user.

[1233] This will enable the realization of an individualized and advanced sports support system.

[1234] (Application example 2)

[1235] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1236] Conventional game video analysis systems and work efficiency improvement systems were unable to provide advanced advice that incorporated the user's emotional and physical data. Furthermore, there was a lack of means to analyze emotional and physical data in real time and provide appropriate advice. This made it difficult to maximize the user's motivation and work efficiency.

[1237] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1238] In this invention, the server includes means for receiving game videos, means for extracting frames from the received game videos, means for analyzing the extracted frames and generating improvement points for play, means for receiving individual player data, means for analyzing the received mental data of the player and providing mental care advice, means for analyzing the received physical data of the player and generating a training plan, means for analyzing the received nutritional data of the player and providing a meal plan, means for integrating the generated improvement points, mental care advice, training plan, and meal plan and presenting them to the player, means for receiving work videos, means for extracting frames from the received work videos, means for analyzing the extracted frames and generating improvement points for work, means for receiving individual worker data, means for analyzing the received mental data of the worker and providing stress care advice, and means for analyzing the received physical data of the worker and generating a work plan. This makes it possible to analyze a user's emotions and physical data in real time and provide optimal advice.

[1239] 1. "Match Video" means a video file that records the match.

[1240] 2. "Work video" refers to a video file that records work being done in a factory or workshop.

[1241] 3. "Frame" means the individual still images that make up a video.

[1242] 4. "Individual Data" means data that indicates the mental, physical, nutritional, and other information of an individual player or worker.

[1243] 5. A "player" is a person who takes part in a sport or other competition.

[1244] 6. "Worker" means a person who performs work in a factory or workshop.

[1245] 7. "Mental Data" refers to data relating to an individual's state of mind or mental health.

[1246] 8. "Physical Data" means data relating to an individual's physical condition and physical strength.

[1247] 9. "Nutrition Data" refers to data relating to an individual's dietary content and nutritional intake status.

[1248] 10. "Mental health advice" means advice to maintain or improve mental health that is provided based on mental data.

[1249] 11. "Stress care advice" refers to specific guidance and advice to alleviate individual stress conditions.

[1250] 12. "Training Plan" means an exercise or training plan created based on physical data.

[1251] 13. "Work Plan" means a plan to improve worker efficiency and safety.

[1252] 14. "Meal Plan" means a meal suggestion based on nutritional data.

[1253] 15. "Artificial intelligence technology" refers to technology that uses techniques such as machine learning and deep learning to analyze data and make intelligent decisions.

[1254] The present invention is a system that analyzes work videos and game videos in real time and provides appropriate advice based on the mental, physical, or nutritional state of each individual user. This system is configured as follows.

[1255] The user uploads the video of their work to the server in real time using smart glasses or other devices. The server then breaks down the received video into frames and analyzes each frame using artificial intelligence techniques such as OpenCV, dlib, and EmotionRecognizer.

[1256] The server also receives the user's individual data (mental, physical, and nutritional information), including data the user previously inputs into the device, and uses an emotion engine to obtain the user's emotional data in real time, which is used for analysis.

[1257] The server combines the analysis results of each frame with the user's emotional and physical data to generate suggestions for improving their gameplay or work. For example, if the emotion engine detects that the user is under stress, it will provide mental care advice to help relieve stress.

[1258] The hardware includes smart glasses, a camera, a microphone, a server, etc. The software uses an analysis program using Python, and libraries such as OpenCV, dlib, and EmotionRecognizer.

[1259] As a concrete example, imagine a factory worker wearing smart glasses while working. If the worker feels stressed and tired, the server will recognize the worker's emotions in real time and provide advice such as "Take a five-minute break" or "Try some light stretching" based on the worker's emotional and physical data.

[1260] Below are some example prompts to input to a generative AI model:

[1261] "Capture sentiment data and generate real-time advice to improve work efficiency."

[1262] This invention makes it possible to analyze a user's emotions and physical data in real time and provide optimal advice, thereby providing comprehensive support for the user's mental and physical state and improving work efficiency and performance.

[1263] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1264] Step 1:

[1265] A user puts on the smart glasses and starts recording a work video. The camera in the smart glasses captures the work video and streams it in real time.

[1266] Input: User's working video

[1267] Output: Real-time captured video data

[1268] Step 2:

[1269] The server receives the video data in real time and extracts each frame, which is then decomposed into individual still images using OpenCV.

[1270] Input: Real-time captured video data

[1271] Output: Extracted frames (a sequence of still images)

[1272] Step 3:

[1273] The server analyzes the extracted frames and evaluates the worker's movements, using dlib for face detection and EmotionRecognizer to determine emotions.

[1274] Input: Extracted frames

[1275] Output: Emotion data (e.g., stress, excitement, fatigue, etc.)

[1276] Step 4:

[1277] The server receives the user's individual data (mental, physical, and nutritional information). Data entered from the device is sent to the server.

[1278] Input: Individual data (mental, physical, nutritional information)

[1279] Output: Individual data received

[1280] Step 5:

[1281] The server combines the individual data received and the analysis results (emotion data) of the extracted frames to comprehensively evaluate the user's condition, and uses an AI model to generate appropriate advice (mental care, work plans, meal plans, etc.).

[1282] Input: Emotion data, individual data

[1283] Output: Comprehensive advice (e.g., mental health advice, stress care advice, work plan, meal plan, etc.)

[1284] Step 6:

[1285] The generated advice is displayed on the user's smart glasses. For example, if the emotion data indicates stress, the advice "Take a 5-minute break" is displayed.

[1286] Input: General Advice

[1287] Output: Advice to the user

[1288] Step 7:

[1289] The user follows the advice to continue or stop working and take appropriate care, such as stretching, taking breaks, or following a meal plan.

[1290] Input: User advice

[1291] Output: Improved work or health status of the user

[1292] Through these steps, the server analyzes the user's emotional and physical data in real time and provides optimal advice, thereby providing comprehensive support for the user's mental and physical state and improving work efficiency and performance.

[1293] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1294] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1295] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1296] [Fourth embodiment]

[1297] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1298] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1299] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1300] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1301] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1302] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1303] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1304] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1305] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1306] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1307] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1308] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1309] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1310] The present invention provides optimal training and mental care support based on analysis of game video and individual player data. Below, the program and processing of the system for implementing the present invention are explained in natural language.

[1311] overview

[1312] Users upload game videos from their devices to the server, then input the player's individual data (mental, physical, and nutritional information). The server receives and analyzes this data, and generates and provides each player with individual improvement points, training plans, mental care advice, and meal plans.

[1313] Specific actions

[1314] Video Upload and Analysis

[1315] Users upload video files of recorded matches from their devices to the server. The server receives the video files and temporarily stores them. After storing them, the server extracts frames from the video files. This extraction process breaks down the video into frames and prepares each frame for analysis.

[1316] Getting individual player data

[1317] Users input their mental and physical state and nutritional data into their devices and send it to the server, which receives the data, classifies it into categories, and stores it.

[1318] Data analysis and advice generation

[1319] The server then uses the extracted video frames to analyze the play, using artificial intelligence technology to detect specific patterns and identify areas that need improvement, such as passing accuracy or defensive posture.

[1320] The server then analyzes the user's mental health based on the mental data received from the user, and generates specific mental care advice, such as ways to improve the player's mental health and reduce stress. For example, it can suggest mindfulness exercises.

[1321] The server then analyzes the physical data and generates an optimal training plan, such as a training plan aimed at improving muscle strength or providing a menu for strengthening specific body parts.

[1322] Finally, the server analyzes the nutritional data and generates an appropriate meal plan, which provides nutritional recommendations to support the player's performance, such as specific meal plans to increase protein intake.

[1323] Presentation of results

[1324] The server then integrates all generated improvements, mental health advice, training plans, and meal plans into a single package and provides it to the user, who can then view the package on their own device and incorporate it into their daily practice and life.

[1325] Specific examples

[1326] For example, if a user uploads a video of a soccer game, the server will analyze the video to identify areas for improvement in the player's passing. If the mental data indicates that the user is feeling stressed, the server will suggest mindfulness exercises. If the physical data indicates that the player needs to strengthen their lower body muscles, the server will provide an appropriate strength training plan. If the nutritional data indicates that the player is not consuming enough protein, the server will suggest a specific meal plan.

[1327] In this way, the present invention provides a system that comprehensively supports players in improving their skills and mental health.

[1328] The processing flow will be explained below.

[1329] Step 1:

[1330] Users upload game videos from their devices to the server.

[1331] Specifically, the user selects the game video file on the terminal and clicks the "upload" button.

[1332] Step 2:

[1333] The server temporarily stores the received video file.

[1334] Specifically, the server stores the uploaded video file in a temporary directory.

[1335] Step 3:

[1336] The server extracts frames from the video file.

[1337] Specifically, the server loads the video using cv2.VideoCapture and adds each frame to a list.

[1338] Step 4:

[1339] The server passes the extracted frames to an AI model to analyze the match.

[1340] Specifically, the frame data is passed to the generate_insights method, and the AI ​​model analyzes areas for improvement in the player's play.

[1341] Step 5:

[1342] The server receives mental, physical, and nutritional data from the user.

[1343] Specifically, the user enters this data into a form on the terminal and sends it to the server's API.

[1344] Step 6:

[1345] The server analyzes the mental data and generates mental care advice.

[1346] Specifically, the server passes the mental data to the analyze_mental_health method to generate mental care advice.

[1347] Step 7:

[1348] The server analyzes the physical data and generates a training plan.

[1349] Specifically, the server passes the physical data to the generate_training_plan method to generate a physical training plan.

[1350] Step 8:

[1351] The server analyzes the nutritional data and generates a meal plan.

[1352] Specifically, the server passes the nutritional data to the provide_meal_plan method to generate a meal plan.

[1353] Step 9:

[1354] The server then combines the generated improvements, mental health advice, training plans, and meal plans into a single package.

[1355] Specifically, all analysis results are integrated into one data set.

[1356] Step 10:

[1357] The server sends the combined package to the user.

[1358] Specifically, the server sends the analysis results and plan to the user's terminal so that the user can check them.

[1359] Step 11:

[1360] Users can check the analysis results and advice on their device and incorporate them into their daily training and lifestyle.

[1361] Specifically, the user opens the analysis results page on their device and implements the suggested improvements and plans.

[1362] Example 1

[1363] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1364] In modern sports, improving a player's skills and optimizing their performance involves a wide range of factors. In particular, there is a need for comprehensive analysis of game video and individual player data (mental, physical, and nutritional data). However, there is no system that consistently performs these tasks, and manually performing them by players and coaches requires a great deal of effort and time. As a result, it is difficult to efficiently review games, plan the next training session, or provide mental care, which can delay the improvement of a player's performance.

[1365] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1366] In this invention, the server includes means for uploading game videos from a terminal, means for receiving and temporarily storing the uploaded game videos, means for extracting frames from the stored game videos, means for analyzing the extracted frames to generate points for improving play, means for inputting individual player data from the terminal, means for receiving the input individual player data and storing it in a database, means for analyzing the received mental data of the player to generate mental care advice, means for analyzing the received physical data of the player to generate a training plan, means for analyzing the received nutritional data of the player to generate a meal plan, and means for integrating and presenting the generated points for improving play, mental care advice, training plan, and meal plan to the player. This makes it possible to consistently and efficiently perform processes from analyzing game videos to collecting individual data and providing comprehensive advice based on that data.

[1367] A "terminal" is a digital device that a user uses to enter information or upload video.

[1368] "Server" means a central processing unit for receiving, storing, analyzing, and providing results from data sent by users.

[1369] "Game video" is a digital file that records footage of a sporting event or other competition.

[1370] A "frame" is each of the consecutive still images that make up a game video.

[1371] "Areas for Improvement" are specific areas where players can improve their technique or tactics, identified through analysis of match video.

[1372] "Individual Data" refers to individual information about a player's own mental, physical and nutritional state.

[1373] "Mental data" refers to information about a player's mental health, such as their psychological state and stress level.

[1374] "Physical data" refers to information about a player's physical condition and training status.

[1375] "Nutrition Data" refers to information regarding a player's diet and nutritional intake.

[1376] "Mental care advice" is specific suggestions for maintaining psychological health that are provided based on the analysis of mental data.

[1377] A "training plan" is a specific exercise program provided based on the analysis of physical data to improve a player's physical strength and skills.

[1378] A "meal plan" is a specific meal menu provided based on the analysis of nutritional data to optimize a player's nutritional balance.

[1379] "Integration" refers to bringing together different analysis results and advice into a single package and presenting them to players in a consistent format.

[1380] This invention is a system that provides optimal training and mental care support based on analysis of game video and individual player data. Detailed modes for carrying out the invention are described below.

[1381] System Configuration

[1382] The whole system consists of terminals, servers, and databases. Terminals are devices used by users to input information and upload videos. The server is the central processing unit, responsible for receiving, storing, analyzing data, and providing results. The database is used to store individual player data and analysis results.

[1383] Hardware and software used

[1384] The server's role is to receive the game videos and individual data sent by the users, analyze them, and generate specific advice. The server uses the following software:

[1385] "ffmpeg": A video analysis tool for breaking down match videos frame by frame.

[1386] "OpenPose" and "YOLO": Artificial intelligence techniques for analyzing player movements within video frames.

[1387] "Scikit-learn" and "TensorFlow": Machine learning libraries used to analyze mental, physical, and nutritional data.

[1388] "MySQL" or "PostgreSQL": A relational database for storing individual data and analysis results.

[1389] Processing Details

[1390] Users upload game videos from their own devices to the server. This operation is performed using a web browser or a dedicated application, and major file formats such as mp4, avi, and mov are supported. The server receives the uploaded video and temporarily stores it in storage. After saving, the video file is broken down into frames using "ffmpeg" to prepare it for analysis.

[1391] Users input individual data such as mental and physical state, nutritional data, etc. on the device screen. This includes self-assessment and data recorded in a dedicated app. For example, a user may enter a number in response to a question such as, "Please rate your stress level this week on a scale of 1 to 10." The server receives this data and stores it in a database for each category.

[1392] The server then analyzes the extracted video frames using OpenPose and YOLO to analyze the player's movements and detect specific patterns and areas for improvement, such as passing accuracy or defensive posture.

[1393] The analysis of mental data uses Scikit-learn and TensorFlow to assess a player's psychological state and generate mental care advice. For example, if the mental stress score is high, mindfulness exercises and relaxation techniques will be suggested.

[1394] The analysis of physical data evaluates the user's physical condition and generates an optimal training plan. For example, if strength training is required, the system will suggest specific exercise menus, number of sets, and number of repetitions.

[1395] The analysis of nutritional data evaluates the user's diet and nutritional intake status and generates an appropriate meal plan. For example, if the user is lacking in protein, it will suggest a high-protein meal menu.

[1396] Examples and prompts

[1397] As a specific example of use, if a user uploads a soccer game video, the server analyzes the video to identify areas for improvement in the player's passing. Next, if the user's mental data indicates that they are feeling stressed, the server will suggest mindfulness exercises. If the physical data indicates that the player needs to strengthen their lower body muscles, the server will provide an appropriate strength training plan. If the nutritional data indicates that the player's protein intake is insufficient, the server will suggest a specific meal plan. For example, it will suggest recipes using protein-rich ingredients such as chicken breast and tofu.

[1398] An example of a prompt to input to a generative AI model is:

[1399] "Please analyze a soccer game video and give me some advice on how to improve passing accuracy. Next, I'd like some advice on mental health, as this player tends to get stressed easily. I'd also like some suggestions on a training plan to strengthen his lower body muscles and a meal menu to increase his protein intake."

[1400] In this way, this system provides comprehensive support for players' technical improvement and mental care.

[1401] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1402] Step 1:

[1403] Users upload game videos.

[1404] Specifically, the user uploads a game video file (e.g., in mp4, avi, or mov format) using a web browser or dedicated application on their own device.

[1405] Input: Match video file

[1406] Output: Send video file to server

[1407] Step 2:

[1408] The server receives the match video file and temporarily stores it.

[1409] The server receives the uploaded video file and temporarily stores it in storage, along with the file name and metadata.

[1410] Input: Video file from user

[1411] Output: Video file in storage

[1412] Step 3:

[1413] The server extracts frames from the video file.

[1414] Using a video analysis tool such as ffmpeg, the match video is broken down into frames. For example, a 30 frame per second video breaks down into 1800 frames per minute.

[1415] Input: Video file in storage

[1416] Output: A collection of extracted frames

[1417] Step 4:

[1418] The user enters the individual data.

[1419] Users enter their individual data, such as mental and physical condition and nutritional data, on the device screen, including self-assessment and data recorded using a dedicated app.

[1420] Input: Mental state, physical state, nutritional data

[1421] Output: Send individual data to the server

[1422] Step 5:

[1423] The server receives the individual data and stores it in a database.

[1424] The server receives the individual data sent by the user and stores it in a database for each category. The data is stored in a relational database such as MySQL or PostgreSQL.

[1425] Input: Individual data from the user

[1426] Output: Data in the database

[1427] Step 6:

[1428] The server analyzes the video frames.

[1429] The server then analyzes the extracted video frames using machine learning models such as OpenPose and YOLO to analyze player movements and identify areas for improvement, such as passing accuracy or defensive posture.

[1430] Input: Extracted video frames

[1431] Output: Play Improvements

[1432] Step 7:

[1433] The server analyzes the mental data and generates mental care advice.

[1434] Using Scikit-learn and TensorFlow, the system analyzes mental data to assess a player's psychological state, suggesting mindfulness exercises if stress levels are high, for example.

[1435] Input: Mental data in a database

[1436] Output: Mental care advice

[1437] Step 8:

[1438] The server analyzes the physical data and generates a training plan.

[1439] The system analyzes physical data and generates training plans to improve the user's strength and technique, for example by suggesting the content, number of sets, and number of repetitions of strength training.

[1440] Input: Physical data in the database

[1441] Output: Training Plan

[1442] Step 9:

[1443] The server analyzes the nutritional data and generates a meal plan.

[1444] The system analyzes nutritional data and generates meal plans to provide users with a balanced diet. For example, if a user is lacking in protein, it will suggest high-protein meals.

[1445] Input: Nutrition data in the database

[1446] Output: Meal plan

[1447] Step 10:

[1448] The server aggregates all the results and serves them to the user.

[1449] The server integrates points for improving play, mental health advice, training plans, and meal plans, and provides them to users as a single package. Users can then check this package on their devices and incorporate it into their practice and daily life.

[1450] Input: Improvements to your game, mental health advice, training plans, meal plans

[1451] Output: Consolidated advice package

[1452] In this way, by identifying the specific inputs and outputs at each step, the flow of the entire system and the data processing and calculations at each processing stage become clear.

[1453] (Application example 1)

[1454] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1455] With conventional training systems, it was difficult to perform detailed analysis of the individual physical and mental data of players and operators and provide optimal advice and plans based on that data. Furthermore, there was a lack of systems that could comprehensively support the optimization of robot movements and the mental care of operators. This resulted in a lack of efficient training and effective mental care, limiting performance improvement.

[1456] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1457] In this invention, the server includes means for receiving game video, means for extracting frames from the received game video, means for analyzing the extracted frames and generating improvement points for play, means for receiving individual player data, means for analyzing the received mental data of the player and providing mental care advice, means for analyzing the received physical data of the player and generating a training plan, means for analyzing the received nutritional data of the player and providing a meal plan, means for integrating the generated improvement points, mental care advice, training plan, and meal plan and presenting them to the player, means for receiving operator data, analyzing the mental data of the operator, and suggesting mindfulness exercises, means for analyzing the physical data and providing a meal plan, and means for analyzing video frames and generating a robot operation strategy. This enables player skill improvement and mental care, as well as optimization of factory robot operation and mental care for operators.

[1458] "Game video" is a recorded video of a sports or game match.

[1459] A "frame" is a unit of still image that is a component of video.

[1460] "Player's individual data" is a collective term for information such as individual mental data, physical data, and nutritional data entered or held by a player.

[1461] "Mental data" refers to data such as the psychological state, emotions, and stress level of players and operators.

[1462] "Physical Data" refers to data relating to the physical health and athletic ability of a player or operator.

[1463] "Nutrition Data" means information about the food and drink consumed by a player or operator, as well as data about the nutrients consumed.

[1464] "Areas for improvement" refers to parts or elements that require correction or improvement in order to improve the performance of the player or robot.

[1465] "Mental care advice" refers to suggestions for psychological support and stress reduction methods provided based on mental data.

[1466] A "training plan" is an exercise or training plan created based on physical data.

[1467] A "meal plan" is a meal plan or nutritional intake plan suggested based on nutritional data.

[1468] "Operator data" is a general term for information including mental and physical data of people working in factories and work sites.

[1469] A "robot motion strategy" is a plan or method for achieving efficient motion of robots used in factories or work sites.

[1470] "Artificial intelligence technology" is a general term for technologies that allow computer systems to analyze and learn from data, automatically discovering patterns and making predictions.

[1471] A "mindfulness exercise" is a set of practices or activities that reduce psychological stress and improve mental well-being by focusing attention on the present moment.

[1472] This invention is a system that supports optimal training, mental health, and nutritional management based on game video analysis and individual player data. It also enables optimization of factory robot operations and supports the mental health of operators. The entire system is implemented using a server, user terminals, and analysis software.

[1473] Program Hardware and Software

[1474] The server receives game videos and robot movement videos and extracts frames. Specifically, it uses OpenCV to extract frames from the video and analyzes the video frames using TensorFlow and Keras. Based on the analysis results, it generates improvements to the player and robot's movements, provides training and diet plans to players, and suggests mental health advice to operators.

[1475] The hardware used is a server equipped with a high-performance CPU / GPU and a camera for video recording, and the software used is OpenCV, TensorFlow, and Keras.

[1476] Data analysis and advice generation

[1477] The server receives the game video and breaks it down into frames. The extracted frames are then analyzed to detect specific patterns in the movement of the play and the robot's behavior. This analysis is performed using TensorFlow's deep learning model. For example, passing accuracy and defensive posture can be identified from a soccer game video and improvements can be made.

[1478] Individual data on players and operators is also sent to the server. For example, the server receives mental data and analyzes it to suggest psychological strengthening methods and mindfulness exercises. Based on physical data, it generates an optimal training plan, providing an appropriate training menu if lower-body muscle strengthening is required. Furthermore, it analyzes nutritional data to propose meal plans, suggesting specific foods for players who are not consuming enough protein, for example.

[1479] Specific examples

[1480] Suppose a user uploads a soccer game video and the server analyzes it. If the server determines that the player's passing accuracy is insufficient, it will suggest areas for improvement. If the server determines that the user is experiencing high stress based on the user's mental data, it will suggest mindfulness exercises. If the server analyzes the user's physical data and determines that the user needs to strengthen their lower body muscles, it will provide an appropriate training plan. If the server determines that the user's protein intake is insufficient based on nutritional data, it will suggest a meal plan to increase protein.

[1481] In addition, by inputting video footage of robots moving in factories and operator data, the system generates an analysis of the video footage, suggestions for mindfulness exercises to address the operator's high stress levels, and a nutrition plan.

[1482] Prompt Sentence Examples

[1483] Example prompts to input to a generative AI model:

[1484] "After uploading and analyzing videos of factory robots in action, we discovered that certain movements were inefficient. Since the operators' stress levels were high, we suggested mindfulness exercises. We also discovered that they were lacking protein, so we proposed a specific diet plan to increase their protein intake."

[1485] In this way, the present invention provides a system that comprehensively supports the improvement of skills and mental care of players and operators.

[1486] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1487] Step 1:

[1488] Users upload game videos and robot movement videos from their devices to the server.

[1489] Input: Match video or robot movement video file

[1490] Specific operation: The user selects a video file using the upload function of the terminal and sends it to the server.

[1491] Output: The video file is saved on the server.

[1492] Step 2:

[1493] The server extracts frames from the received video file.

[1494] Input: Video file

[1495] Data processing: Decompose the video file frame by frame using OpenCV.

[1496] Specific operation: The server reads the video file and extracts specific frames (e.g., every 30th frame).

[1497] Output: A list of extracted frames

[1498] Step 3:

[1499] The server analyzes the extracted frames based on artificial intelligence techniques.

[1500] Input: A list of frames

[1501] Data computation: Using TensorFlow and Keras, we perform analysis using a CNN model for each frame.

[1502] Specific operation: The extracted frames are resized and normalized, and then input into the model to obtain analysis results.

[1503] Output: Improvements to play and specific patterns of robot movement

[1504] Step 4:

[1505] The user sends individual data of the player and operator from the terminal to the server.

[1506] Input: Mental data, physical data, nutritional data

[1507] Specific operation: The user inputs mental data, physical data, and nutritional data on the terminal and transmits the data to the server.

[1508] Output: Individual data stored on the server

[1509] Step 5:

[1510] The server analyzes the received mental data and generates mental care advice.

[1511] Input: Mental data

[1512] Data calculation: Analyzes mental data to evaluate psychological state and stress level and generate appropriate mental care advice.

[1513] Specific behavior: The server suggests mindfulness exercises when stress levels are high.

[1514] Output: Mental care advice

[1515] Step 6:

[1516] The server analyzes the received physical data and generates an optimal training plan.

[1517] Input: Physical data

[1518] Data calculation: Create an appropriate training menu based on physical data.

[1519] Specific Movement: The server generates a specific training plan, such as strengthening muscles.

[1520] Output: Training Plan

[1521] Step 7:

[1522] The server analyzes the received nutritional data and provides a meal plan.

[1523] Input: Nutrition data

[1524] Data calculation: Analyzes nutritional data to identify nutrient deficiencies and suggests necessary meal plans.

[1525] What it does: If you're lacking in protein, the server will suggest a meal plan to increase your protein intake.

[1526] Output: Meal plan

[1527] Step 8:

[1528] The server integrates the generated improvements, mental care advice, training plans, and meal plans and presents them to the user.

[1529] Input: Improvements, mental health advice, training plans, meal plans

[1530] Specific operation: The server integrates this information and sends it to the user terminal as a single package.

[1531] Output: A consolidated improvement plan that can be viewed on the user's device

[1532] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1533] This invention provides optimal training and mental care support based on game video analysis and individual player data. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to generate more accurate advice.

[1534] System Operation Overview

[1535] Users use their devices to upload game videos to the server, which breaks down the video into frames and analyzes each one. Individual player data (mental, physical, and nutritional information) is also sent to the server, and various advice is generated based on this data. The emotion engine recognizes the user's emotions and uses this data in analysis to further improve the quality of advice.

[1536] Specific processing flow

[1537] Video Upload and Analysis

[1538] Users upload game videos from their devices to the server. The server receives the video files and temporarily stores them. Next, it extracts each frame from the video file and analyzes each frame to identify areas for improvement. Specifically, it uses artificial intelligence technology to analyze each frame and evaluate the player's movements.

[1539] Acquiring and analyzing emotion data

[1540] To recognize the user's emotional state, the device is equipped with a camera and a microphone. The emotion engine analyzes the user's facial expressions and tone of voice in real time from these devices and generates emotional data. The server receives this data and evaluates the user's emotional state.

[1541] Getting individual player data

[1542] Users enter mental, physical, and nutritional data into a device and send it to a server, which receives the data, stores it by category, and creates a dataset for analysis.

[1543] Generating comprehensive advice

[1544] The server generates comprehensive advice by integrating the user's emotional, mental, physical, and nutritional data with the points for improvement found in the analyzed video frames. The mental care advice is tailored based on the emotional data. For example, if the user is feeling stressed, it can suggest mindfulness exercises to help relieve stress.

[1545] Training plans are also provided by combining physical and emotional data. For example, if a player is feeling fatigued, it can suggest light exercises.

[1546] The system also incorporates emotional data into nutritional plans, allowing it to provide plans that reflect a user's preferred ingredients and eating patterns. For example, if a user feels they are lacking in energy, it will suggest meals that are high in protein.

[1547] Specific examples

[1548] For example, a user can upload a soccer game video and input the results of a physical fitness test as their own physical data. If the emotion engine detects a state of stress from the user's facial expression, the server will suggest mental care advice (mindfulness exercises) to reduce stress. Furthermore, based on the physical data, it may determine that the user needs to strengthen their lower body muscles, and a training plan for this purpose will be provided. At the same time, nutritional data may detect a protein deficiency and suggest a meal plan to compensate for it.

[1549] In this way, the present invention realizes a system that provides more personalized and advanced support by incorporating user emotional data.

[1550] The processing flow will be explained below.

[1551] The present invention provides optimal training and mental care support based on game video analysis and individual player data. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the system generates more accurate advice. The processing flow of the system for implementing the present invention is specifically described below.

[1552] Specific processing flow of the system

[1553] Step 1:

[1554] Users upload game videos from their devices to the server.

[1555] Specifically, the user selects the game video file on the terminal and clicks the "upload" button.

[1556] Step 2:

[1557] The server temporarily stores the received video file.

[1558] Specifically, the server stores the uploaded video file in a temporary directory.

[1559] Step 3:

[1560] The server extracts frames from the video file.

[1561] Specifically, the server loads the video using cv2.VideoCapture and adds each frame to a list.

[1562] Step 4:

[1563] The server passes the extracted frames to an AI model to analyze the match.

[1564] Specifically, the frame data is passed to the generate_insights method, and the AI ​​model analyzes areas for improvement in the player's play.

[1565] Step 5:

[1566] An emotion engine is used to recognize the user's emotional state.

[1567] Specifically, it analyzes the user's facial expressions and tone of voice in real time through the camera and microphone installed on the device to generate emotional data.

[1568] Step 6:

[1569] The server receives the emotional data and evaluates the user's emotional state.

[1570] Specifically, the emotion data is sent to the server and the analysis results are saved.

[1571] Step 7:

[1572] Users enter mental, physical, and nutritional data into a terminal and send it to the server.

[1573] Specifically, the user enters this data into a form on the terminal and presses the send button.

[1574] Step 8:

[1575] The server analyzes the received mental data and provides mental care advice.

[1576] Specifically, the mental data is passed to the analyze_mental_health method to generate mental care advice.

[1577] Step 9:

[1578] The server analyzes the physical data and generates a training plan.

[1579] Specifically, the physical data is passed to the generate_training_plan method to generate the optimal training plan.

[1580] Step 10:

[1581] The server analyzes the nutritional data and provides meal plans.

[1582] Specifically, the nutritional data is passed to the provide_meal_plan method, which generates an appropriate meal plan.

[1583] Step 11:

[1584] The server then combines the generated improvements, mental health advice, training plans, and meal plans into a single package.

[1585] Specifically, all analysis results are integrated to create a package to be provided to the user.

[1586] Step 12:

[1587] The server sends the combined package to the user.

[1588] Specifically, the server sends the analysis results and plan to the user's terminal so that the user can check them.

[1589] Step 13:

[1590] Users can check the analysis results and advice on their device and incorporate them into their daily training and lifestyle.

[1591] Specifically, the user opens the analysis results page on their device and implements the suggested improvements and plans.

[1592] Specific examples

[1593] For example, if a user uploads a soccer game video and the emotion engine recognizes the user's stress level, the server will suggest mindfulness exercises to reduce stress. At the same time, physical data will determine that the user needs to strengthen their lower body muscles, and a training plan for that purpose will be provided. If nutritional data detects a protein deficiency, a specific meal plan to compensate for that will be proposed. The advice tailored by the emotion engine is then synthesized and provided to the user, improving the quality of their training.

[1594] In this way, the present invention realizes a system that provides more personalized and advanced support by incorporating user emotional data.

[1595] Example 2

[1596] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1597] To improve sports player performance, conventional systems simply analyze game videos and provide general training plans, but they lack detailed advice and support tailored to individual players. They also lack a comprehensive approach that takes into account the user's emotional state. Therefore, there is a need for systems that provide comprehensive support for players' technical development as well as their mental, physical, and nutritional needs.

[1598] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1599] In this invention, the server includes means for receiving game video, means for extracting frames from the received game video, means for analyzing the extracted frames and generating improvements to the player's movements, means for receiving individual player data, means for analyzing the received mental data of the player to provide mental health advice, means for analyzing the received physical data of the player to generate a training plan, means for analyzing the received nutritional data of the player to provide a meal plan, means for acquiring and analyzing user emotional data, means for improving the quality of the advice based on the acquired emotional data, and means for integrating and presenting the generated improvements, mental health advice, training plan, and meal plan to the player. This allows for the provision of personalized, advanced support, enabling comprehensive support not only for the player's technical growth but also for mental, physical, and nutritional aspects.

[1600] A "game video" is a video file that records a sports game.

[1601] A "frame" is an individual still image that makes up a video image.

[1602] "Movement improvement points" refer to specific points or areas that a sports player can use to improve their play during a game.

[1603] "Individual Data" means data relating to each player, including mental, physical and nutritional information.

[1604] "Mental data" is data that contains information about a player's psychological state and emotions.

[1605] "Mental Care Advice" is specific advice or suggestions to improve a player's psychological state and support their mental health.

[1606] "Physical Data" means data containing information about a player's physical condition and performance.

[1607] A "training plan" is a specific training menu and exercise plan based on a player's physical data to improve their physical strength and skills.

[1608] "Nutrition Data" means data containing information about a player's diet and nutritional intake.

[1609] A "meal plan" is a specific dietary suggestion or plan designed to improve a player's nutritional status and optimize performance.

[1610] "Emotion data" refers to data that includes information about the user's emotional state obtained by analyzing the user's facial expression and tone of voice.

[1611] "Artificial intelligence technology" is a technology that uses techniques including machine learning to analyze data and automatically extract knowledge and patterns.

[1612] "Machine learning" is a technique in which computer algorithms learn from data to recognize patterns and create predictive models.

[1613] Mindfulness exercises are mental exercises that help you focus your attention on the present moment and reduce stress and anxiety.

[1614] This system provides optimal training and mental care support based on game video analysis and individual player data. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to generate more accurate advice.

[1615] System Operation Overview

[1616] Users upload game videos from their devices to the server. The server extracts each frame from the received video and analyzes it using artificial intelligence techniques (e.g., OpenCV and TensorFlow). The extracted frames are evaluated to identify areas for improvement in the player's movements.

[1617] Users also input their mental, physical, and nutritional data into the server via their devices. This individual data is stored in a database to create a dataset for analysis. An emotion engine (e.g., Emotion API) is used to obtain end-user emotional data in real time. The server analyzes this data and generates integrated advice.

[1618] Hardware and software used

[1619] Device: The computer or smartphone you use

[1620] Server: A remote server that receives, processes, and analyzes data.

[1621] Artificial intelligence technology: Machine learning libraries such as OpenCV and TensorFlow

[1622] Emotion engine: Emotion API

[1623] Data acquisition and analysis flow

[1624] 1. Upload your video

[1625] Users upload game videos from their devices to the server, and the video files are sent to the designated server via the device application.

[1626] 2. Extracting and analyzing video frames

[1627] The server temporarily stores the received video and uses a video processing tool (e.g., FFmpeg) to extract each frame, which is then analyzed using artificial intelligence techniques.

[1628] 3. Acquiring and sending emotion data

[1629] The device's camera and microphone are used to capture the user's facial expressions and voice, which are then analyzed by the emotion engine. The emotion data analyzed in real time is sent to a server and stored in a database.

[1630] 4. Enter and submit individual data

[1631] Users use a dedicated application to input mental, physical, and nutritional data, which is then sent to a server where it is stored in a database and a data set is created for analysis.

[1632] 5. Data Integration and Advice Generation

[1633] The server integrates all the received data and uses a generative AI model to generate comprehensive advice, which is then displayed on the user's dashboard. For example, if a user uploads a game video and the emotion engine detects stress, the generative AI model will suggest mental care advice (e.g., mindfulness exercises) to reduce stress.

[1634] Specific examples

[1635] For example, a user can upload a soccer game video and enter the results of a recent physical fitness test as their physical data. If the emotion engine detects a state of stress from the user's facial expression, the server will suggest mental care advice (e.g., mindfulness exercises) to reduce stress. Furthermore, based on the physical data, it may determine that the user needs to strengthen their lower body muscles, and a training plan for this purpose will be provided. At the same time, nutritional data may detect a protein deficiency and suggest a meal plan to compensate for it.

[1636] Prompt Sentence Examples

[1637] "If a user uploads a soccer game video and the emotion engine detects stress from the user's facial expressions, what kind of mental health advice can be provided? Also, if the results of a physical fitness test are entered as the user's physical data and the results indicate that the user needs to strengthen their lower body muscles, what kind of training plan can be suggested?"

[1638] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1639] Understood. Now, I will explain the processing flow of this system's program in detail in the following format.

[1640] Step 1:

[1641] The user launches a dedicated application on their device, selects a game video file, and uploads it. When they press the upload button, the device sends the video file to the specified server endpoint (e.g., https: / / example.com / upload). The input is the video file selected by the user, and the output is the video file saved on the server.

[1642] Step 2:

[1643] The server temporarily stores the received video file and then uses a video processing tool such as FFmpeg to decompose the video into frames. A frame is an individual still image that is used in the next analysis step. The input is the received video file, and the output is the extracted frames. Each frame is a sequence of image data.

[1644] Step 3:

[1645] The server uses a deep learning model (e.g., a model using TensorFlow) to analyze each extracted frame. This analysis evaluates the player's movements and identifies areas for improvement. For example, elements of the play such as movement speed and positioning are analyzed. The input is the image data of the frame, and the output is information on areas for improvement.

[1646] Step 4:

[1647] The device uses a built-in camera and microphone to capture the user's facial expressions and voice in real time. An emotion recognition engine such as Emotion API analyzes this data and evaluates the user's emotional state. The generated emotion data is encrypted and sent to a server. The input is the captured facial and voice data, and the output is emotion data.

[1648] Step 5:

[1649] Users enter mental, physical, and nutritional data using a dedicated application. The data entered by the user is sent from the device to a server, which then stores it in a database. The input is the individual data entered by the user, and the output is a dataset for analysis.

[1650] Step 6:

[1651] The server integrates all data (video analysis data, emotional data, individual data). It uses a generative AI model to generate comprehensive advice. For example, it extracts areas for improvement in play from video analysis data, evaluates the need for mental care from emotional data, and generates a training plan from physical data. The input is the integrated dataset, and the output is advice for the user.

[1652] Step 7:

[1653] The user can view the generated advice on the dashboard of the dedicated application. The advice includes mental health advice (e.g., mindfulness exercises for stress reduction), training plans, and meal plans. The input is advice data from the generative AI model, and the output is advice information provided to the user.

[1654] This will enable the realization of an individualized and advanced sports support system.

[1655] (Application example 2)

[1656] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1657] Conventional game video analysis systems and work efficiency improvement systems were unable to provide advanced advice that incorporated the user's emotional and physical data. Furthermore, there was a lack of means to analyze emotional and physical data in real time and provide appropriate advice. This made it difficult to maximize the user's motivation and work efficiency.

[1658] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1659] In this invention, the server includes means for receiving game videos, means for extracting frames from the received game videos, means for analyzing the extracted frames and generating improvement points for play, means for receiving individual player data, means for analyzing the received mental data of the player and providing mental care advice, means for analyzing the received physical data of the player and generating a training plan, means for analyzing the received nutritional data of the player and providing a meal plan, means for integrating the generated improvement points, mental care advice, training plan, and meal plan and presenting them to the player, means for receiving work videos, means for extracting frames from the received work videos, means for analyzing the extracted frames and generating improvement points for work, means for receiving individual worker data, means for analyzing the received mental data of the worker and providing stress care advice, and means for analyzing the received physical data of the worker and generating a work plan. This makes it possible to analyze a user's emotions and physical data in real time and provide optimal advice.

[1660] 1. "Match Video" means a video file that records the match.

[1661] 2. "Work video" refers to a video file that records work being done in a factory or workshop.

[1662] 3. "Frame" means the individual still images that make up a video.

[1663] 4. "Individual Data" means data that indicates the mental, physical, nutritional, and other information of an individual player or worker.

[1664] 5. A "player" is a person who takes part in a sport or other competition.

[1665] 6. "Worker" means a person who performs work in a factory or workshop.

[1666] 7. "Mental Data" refers to data relating to an individual's state of mind or mental health.

[1667] 8. "Physical Data" means data relating to an individual's physical condition and physical strength.

[1668] 9. "Nutrition Data" refers to data relating to an individual's dietary content and nutritional intake status.

[1669] 10. "Mental health advice" means advice to maintain or improve mental health that is provided based on mental data.

[1670] 11. "Stress care advice" refers to specific guidance and advice to alleviate individual stress conditions.

[1671] 12. "Training Plan" means an exercise or training plan created based on physical data.

[1672] 13. "Work Plan" means a plan to improve worker efficiency and safety.

[1673] 14. "Meal Plan" means a meal suggestion based on nutritional data.

[1674] 15. "Artificial intelligence technology" refers to technology that uses techniques such as machine learning and deep learning to analyze data and make intelligent decisions.

[1675] The present invention is a system that analyzes work videos and game videos in real time and provides appropriate advice based on the mental, physical, or nutritional state of each individual user. This system is configured as follows.

[1676] The user uploads the video of their work to the server in real time using smart glasses or other devices. The server then breaks down the received video into frames and analyzes each frame using artificial intelligence techniques such as OpenCV, dlib, and EmotionRecognizer.

[1677] The server also receives the user's individual data (mental, physical, and nutritional information), including data the user previously inputs into the device, and uses an emotion engine to obtain the user's emotional data in real time, which is used for analysis.

[1678] The server combines the analysis results of each frame with the user's emotional and physical data to generate suggestions for improving their gameplay or work. For example, if the emotion engine detects that the user is under stress, it will provide mental care advice to help relieve stress.

[1679] The hardware includes smart glasses, a camera, a microphone, a server, etc. The software uses an analysis program using Python, and libraries such as OpenCV, dlib, and EmotionRecognizer.

[1680] As a concrete example, imagine a factory worker wearing smart glasses while working. If the worker feels stressed and tired, the server will recognize the worker's emotions in real time and provide advice such as "Take a five-minute break" or "Try some light stretching" based on the worker's emotional and physical data.

[1681] Below are some example prompts to input to a generative AI model:

[1682] "Capture sentiment data and generate real-time advice to improve work efficiency."

[1683] This invention makes it possible to analyze a user's emotions and physical data in real time and provide optimal advice, thereby providing comprehensive support for the user's mental and physical state and improving work efficiency and performance.

[1684] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1685] Step 1:

[1686] A user puts on the smart glasses and starts recording a work video. The camera in the smart glasses captures the work video and streams it in real time.

[1687] Input: User's working video

[1688] Output: Real-time captured video data

[1689] Step 2:

[1690] The server receives the video data in real time and extracts each frame, which is then decomposed into individual still images using OpenCV.

[1691] Input: Real-time captured video data

[1692] Output: Extracted frames (a sequence of still images)

[1693] Step 3:

[1694] The server analyzes the extracted frames and evaluates the worker's movements, using dlib for face detection and EmotionRecognizer to determine emotions.

[1695] Input: Extracted frames

[1696] Output: Emotion data (e.g., stress, excitement, fatigue, etc.)

[1697] Step 4:

[1698] The server receives the user's individual data (mental, physical, and nutritional information). Data entered from the device is sent to the server.

[1699] Input: Individual data (mental, physical, nutritional information)

[1700] Output: Individual data received

[1701] Step 5:

[1702] The server combines the individual data received and the analysis results (emotion data) of the extracted frames to comprehensively evaluate the user's condition, and uses an AI model to generate appropriate advice (mental care, work plans, meal plans, etc.).

[1703] Input: Emotion data, individual data

[1704] Output: Comprehensive advice (e.g., mental health advice, stress care advice, work plan, meal plan, etc.)

[1705] Step 6:

[1706] The generated advice is displayed on the user's smart glasses. For example, if the emotion data indicates stress, the advice "Take a 5-minute break" is displayed.

[1707] Input: General Advice

[1708] Output: Advice to the user

[1709] Step 7:

[1710] The user follows the advice to continue or stop working and take appropriate care, such as stretching, taking breaks, or following a meal plan.

[1711] Input: User advice

[1712] Output: Improved work or health status of the user

[1713] Through these steps, the server analyzes the user's emotional and physical data in real time and provides optimal advice, thereby providing comprehensive support for the user's mental and physical state and improving work efficiency and performance.

[1714] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1715] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1716] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1717] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1718] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1719] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1720] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1721] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1722] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1723] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1724] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1725] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1726] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1727] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1728] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1729] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1730] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1731] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1732] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1733] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1734] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1735] The following is further disclosed regarding the above embodiment.

[1736] (Claim 1)

[1737] means for receiving match video;

[1738] means for extracting frames from the received game video;

[1739] means for analyzing the extracted frames and generating play improvements;

[1740] means for receiving player individual data;

[1741] A means for analyzing the received mental data of the player and providing mental care advice;

[1742] A means for analyzing the received player's physical data and generating a training plan;

[1743] a means for analyzing the received player nutritional data to provide a meal plan;

[1744] A means to consolidate and present generated improvement points, mental health advice, training plans, and diet plans to the player;

[1745] A system including:

[1746] (Claim 2)

[1747] 10. The system of claim 1, further comprising means for employing artificial intelligence techniques in analyzing the game video.

[1748] (Claim 3)

[1749] 10. The system of claim 1, wherein the mental health advice provided includes mindfulness exercises.

[1750] (Claim 4)

[1751] 10. The system of claim 1, wherein the training plan provided includes strength training.

[1752] (Claim 5)

[1753] 10. The system of claim 1, wherein the meal plan provided includes suggestions for increasing protein.

[1754] "Example 1"

[1755] (Claim 1)

[1756] A means to upload game videos from the device,

[1757] means for receiving and temporarily storing uploaded match videos;

[1758] means for extracting frames from the stored match video;

[1759] means for analyzing the extracted frames to generate play improvements;

[1760] a means for inputting individual player data from the terminal;

[1761] means for receiving and storing the entered player individual data in a database;

[1762] means for analyzing the received mental data of the player and generating mental care advice;

[1763] A means for analyzing the received player's physical data and generating a training plan;

[1764] means for analyzing the received player nutritional data to generate a meal plan;

[1765] A means to integrate and present the generated improvement points, mental care advice, training plans, and diet plans to the player;

[1766] A system including:

[1767] (Claim 2)

[1768] 10. The system of claim 1, further comprising means for using machine learning techniques in analyzing the match video.

[1769] (Claim 3)

[1770] 10. The system of claim 1, wherein the mental health advice provided includes mindfulness exercises.

[1771] "Application Example 1"

[1772] (Claim 1)

[1773] means for receiving match video;

[1774] means for extracting frames from the received game video;

[1775] means for analyzing the extracted frames and generating play improvements;

[1776] means for receiving player individual data;

[1777] A means for analyzing the received mental data of the player and providing mental care advice;

[1778] A means for analyzing the received player's physical data and generating a training plan;

[1779] a means for analyzing the received player nutritional data to provide a meal plan;

[1780] A means to consolidate and present generated improvement points, mental health advice, training plans, and diet plans to the player;

[1781] A means for receiving the operator's data, analyzing the mental data, and suggesting mindfulness exercises;

[1782] a means of analyzing physical data to provide meal plans;

[1783] means for analyzing video frames to generate a robot movement strategy;

[1784] A system including:

[1785] (Claim 2)

[1786] 10. The system of claim 1, further comprising means for using artificial intelligence techniques in analyzing the game video and the robot movement video.

[1787] (Claim 3)

[1788] 10. The system of claim 1, wherein the mental health advice provided includes mindfulness exercises.

[1789] "Example 2: Combining Emotion Engines"

[1790] (Claim 1)

[1791] means for receiving match video;

[1792] means for extracting frames from the received game video;

[1793] means for analyzing the extracted frames and generating player behavior improvements;

[1794] means for receiving player individual data;

[1795] A means for analyzing the received mental data of the player and providing mental care advice;

[1796] A means for analyzing the received player's physical data and generating a training plan;

[1797] a means for analyzing the received player nutritional data to provide a meal plan;

[1798] A means for acquiring and analyzing user emotion data;

[1799] A means for improving the quality of advice based on the acquired emotion data;

[1800] A means to consolidate and present generated improvement points, mental health advice, training plans, and diet plans to the player;

[1801] A system including:

[1802] (Claim 2)

[1803] 10. The system of claim 1, further comprising means for using machine learning techniques in analyzing the match video.

[1804] (Claim 3)

[1805] 10. The system of claim 1, wherein the mental health advice provided includes mindfulness exercises.

[1806] "Application example 2 when combining emotion engines"

[1807] (Claim 1)

[1808] means for receiving match video;

[1809] means for extracting frames from the received game video;

[1810] means for analyzing the extracted frames and generating play improvements;

[1811] means for receiving player individual data;

[1812] A means for analyzing the received mental data of the player and providing mental care advice;

[1813] A means for analyzing the received player's physical data and generating a training plan;

[1814] a means for analyzing the received player nutritional data to provide a meal plan;

[1815] A means to consolidate and present generated improvement points, mental health advice, training plans, and diet plans to the player;

[1816] means for receiving the working video;

[1817] means for extracting frames from the received working video;

[1818] means for analyzing the extracted frames and generating improvements to the work;

[1819] means for receiving individual data of the worker;

[1820] A means for analyzing the received mental data of the worker and providing stress care advice;

[1821] A means for analyzing the received physical data of the worker and generating a work plan;

[1822] A system including:

[1823] (Claim 2)

[1824] 10. The system of claim 1, including means for using artificial intelligence techniques in analyzing the game video and the work video.

[1825] (Claim 3)

[1826] 2. The system of claim 1, wherein the mental care advice and stress care advice provided includes mindfulness exercises. [Explanation of symbols]

[1827] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving match video; means for extracting frames from the received game video; means for analyzing the extracted frames and generating play improvements; means for receiving player individual data; A means for analyzing the received mental data of the player and providing mental care advice; A means for analyzing the received player's physical data and generating a training plan; a means for analyzing the received player nutritional data to provide a meal plan; A means to consolidate and present generated improvement points, mental health advice, training plans, and diet plans to the player; A system including:

2. 10. The system of claim 1, further comprising means for employing artificial intelligence techniques in analyzing the game video.

3. 10. The system of claim 1, wherein the mental health advice provided includes mindfulness exercises.

4. 10. The system of claim 1, wherein the training plan provided includes strength training.

5. 10. The system of claim 1, wherein the meal plan provided includes suggestions for increasing protein.

Images