System

Abstract

A system is provided.SOLUTION: A system comprising: means for uploading a player video from a player device; means for storing the uploaded player video in cloud storage; means for passing the player video stored in the cloud storage to a generative AI model for analysis; means for generating feedback based on a result of the analysis of the player video; and means for transmitting the generated feedback to the player device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Appropriate coaching and play analysis are essential for developing soccer players. However, providing detailed feedback to each player requires a significant amount of time and effort. This makes it difficult for many players to receive quick and accurate feedback. The present invention aims to solve this problem by utilizing a generative AI model and support the performance improvement of each player. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means.

[0006] First, a means for uploading player videos from a player terminal is provided. Next, a means for saving the uploaded player videos in cloud storage is provided. Next, a means for passing the player videos saved in cloud storage to a generative AI model for analysis is provided. Furthermore, a means for generating feedback based on the analysis results of the player videos is provided. Finally, a means for sending the generated feedback to the player terminal is provided. Furthermore, by including a means for a player to input a question about a specific play and send the question to the generative AI model, and a means for the generative AI model to create feedback based on the question, a system is realized that provides specific feedback focusing on a player's strengths and areas for improvement.

[0007] A "player device" is a device such as a mobile phone or camera that a soccer player uses to film his or her play and upload the video and questions to the server.

[0008] "Player video" refers to footage of a soccer player filming their own play, and is video data that can be used for analysis and feedback.

[0009] "Cloud storage" is a remote server for storing data via the Internet, where player videos are temporarily stored.

[0010] A "generative AI model" is an artificial intelligence model that uses machine learning and deep learning techniques to analyze player videos and generate feedback.

[0011] "Analysis" refers to the process in which the generative AI model evaluates and analyzes the player's movements and techniques based on player video.

[0012] "Feedback" is information created by the generative AI model based on the analysis results, including the player's strengths, areas for improvement, and specific training advice.

[0013] "Uploading" is the act of transmitting player video from a player terminal to a server.

[0014] "Destination URL" is an internet address that indicates the location of the player video stored in cloud storage.

[0015] "Analysis results" refers to the data and information obtained by the generative AI model when it analyzes the player's video.

[0016] "Questions" are questions or inquiries that players pose to the generative AI model regarding specific plays or techniques. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The online coaching and play analysis system of the present invention includes a player terminal, cloud storage, a generative AI model, and a server. The functions and processing flow of each component are shown below to implement the present invention.

[0039] Player device functions

[0040] A player device is a device that is primarily used by a player to record, save, and upload their play to a server. It is also the device through which the player receives feedback provided as analytical results. Specifically, a player device may be a smartphone, tablet, or camera.

[0041] Server Features

[0042] The server plays a central role in receiving, storing, and analyzing gameplay videos and question data sent from the player's device. The server provides the URL of the destination for the received video to the AI ​​model, which then analyzes the video. It also has the function of generating feedback based on the analysis results and sending it to the player's device.

[0043] Cloud storage features

[0044] Cloud storage is a place to temporarily store the received gameplay video. The server saves the video in cloud storage and passes the URL of the saved video to the generation AI model.

[0045] Generative AI model capabilities

[0046] The generative AI model is a system that analyzes gameplay videos stored in cloud storage to identify players' strengths and areas for improvement. It has the ability to generate specific feedback based on the analysis results. Furthermore, if the player inputs specific questions, it can also generate feedback based on those questions.

[0047] Specific processing of the program

[0048] The server receives the video of the player's play. For example, the player takes a video of their dribbling and uploads it through the application. The server saves the video in cloud storage and generates a URL for the destination.

[0049] The server then passes the URL to the generative AI model, asking it to analyze the video. The generative AI model analyzes the video and identifies strengths and areas for improvement in the player's dribbling skills. Based on the analysis results, the generative AI model creates feedback including specific training advice.

[0050] The server then sends the generated feedback to the player terminal, and the player terminal receives a notification, allowing the player to open the application, check the specific feedback content, and use it to improve their practice.

[0051] For example, the feedback provided may include, "Your dribbling speed is excellent, but your timing is off, making it difficult to control the ball." The advice also includes specific drills and practice menus to improve your timing. Based on this, players can then practice to improve their own play.

[0052] In this way, the system of the present invention allows players to receive efficient and accurate feedback and specific advice on how to improve their play without the need for a coach.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] A user takes a video of a game using a smartphone or camera and saves it on the device. For example, a user might take a video of a soccer player dribbling.

[0056] Step 2:

[0057] The user launches the application, selects a saved gameplay video, and uploads it to the server. Specifically, the user taps the "Upload Video" button, selects the video file, and sends an upload request.

[0058] Step 3:

[0059] The device sends the uploaded gameplay video to the server, which receives the HTTP POST request and saves the video file in cloud storage.

[0060] Step 4:

[0061] The server generates a URL for the video stored in cloud storage and sends an analysis request to the generative AI model. Specifically, the server passes the video URL to the generative AI model via a REST API.

[0062] Step 5:

[0063] The generative AI model receives the video URL and begins analyzing the video of the game. The generative AI model analyzes the player's movements and techniques in the video to identify strengths and areas for improvement.

[0064] Step 6:

[0065] A generative AI model uses the analysis to generate feedback, including the player's technical strengths and areas for improvement, as well as specific training advice.

[0066] Step 7:

[0067] The server receives the feedback obtained from the generative AI model and sends it to the player's device. The server returns the feedback data to the device via an HTTP response.

[0068] Step 8:

[0069] The device receives the feedback from the server and displays a notification to the user, for example, a push notification that says "Your feedback has been received."

[0070] Step 9:

[0071] The user opens the application and checks the received feedback. Specifically, the user can view the feedback on the application screen and identify strengths and areas for improvement.

[0072] Step 10:

[0073] Based on the feedback provided by the user, the system implements specific advice for training and improving play, such as "doing specific drills to improve dribbling timing."

[0074] Example 1

[0075] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0076] Conventional play analysis systems have issues such as low analysis accuracy, inability to provide instant feedback, and inability to obtain analysis results that satisfy users. Furthermore, when players themselves want feedback on specific questions, they often lack the ability to obtain specific advice. To address these issues, a system that improves the accuracy of gameplay video analysis and provides fast, specific feedback is needed.

[0077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0078] In this invention, the server includes a means for uploading gameplay videos from the player terminal, a means for saving the uploaded gameplay videos to cloud storage, and a means for generating a destination URL for the gameplay videos saved in cloud storage and passing it to the generation AI model. This improves the accuracy of gameplay video analysis and makes it possible to provide quick and specific feedback.

[0079] "Player Device" means a device used by a player to record, store, and upload their play to the server, such as a smartphone, tablet, or camera.

[0080] "Cloud storage" refers to a group of remote servers that store data via the Internet. In particular, in the present invention, it is used to temporarily store gameplay videos.

[0081] The "generative AI model" is an artificial intelligence model that analyzes gameplay videos stored in cloud storage and identifies players' strengths and areas for improvement. This model is responsible for generating specific feedback based on the results of gameplay analysis.

[0082] "Feedback" refers to specific advice and information on areas for improvement provided by the generative AI model based on the analysis of the gameplay video. The goal is to improve the player's skills.

[0083] The online coaching and play analysis system of the present invention includes a player terminal, cloud storage, a generative AI model, and a server. The components and processing flow of this system will be described in detail below.

[0084] Player device functions

[0085] A player device is a device used by a player to record, save, and upload their play to a server. Specifically, this includes smartphones, tablets, and cameras. It is also on the player device that the player receives feedback provided as analysis results. For example, a user may film a soccer dribble using a smartphone and upload the video through an application.

[0086] Server Features

[0087] The server plays a central role in receiving gameplay videos and question data sent from the player's device and storing them in cloud storage. Specifically, the server stores the received videos in cloud storage and generates a URL for the destination. This URL is then provided to the generative AI model, which requests it to analyze the video. The server also has the function of generating feedback based on the analysis results returned by the generative AI model and sending it to the player's device.

[0088] Cloud storage features

[0089] Cloud storage is a group of remote servers that temporarily store the received gameplay videos. The server saves the videos in cloud storage and passes the URL of the saved video to the generated AI model. Specifically, a common cloud storage service (e.g., Amazon S3) is used.

[0090] Generative AI model capabilities

[0091] The generative AI model is an artificial intelligence system that analyzes gameplay videos stored in cloud storage and identifies players' strengths and areas for improvement. This model has the ability to generate specific feedback based on the analysis results. Furthermore, if a player inputs a specific question, it can generate appropriate feedback based on that question.

[0092] Feedback example

[0093] An example of specific feedback might be, "Your dribbling speed is excellent, but your timing is off, making it difficult to control the ball." The advice also includes specific drills and practice exercises to improve your timing. This gives players a concrete action plan to improve their play.

[0094] Prompt Sentence Examples

[0095] The following prompts can be used as input to a generative AI model:

[0096] "Analyze this video for dribbling skills, identify player strengths and areas for improvement, and generate specific training advice."

[0097] The system allows players to receive efficient and accurate feedback quickly, giving them specific advice on how to improve their play without the need for a coach.

[0098] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0099] Step 1:

[0100] The user launches the application on the player's device and records a video of their gameplay. The recorded video is temporarily saved on the device. Then, when the user taps the "Upload" button in the application, the device sends the video file to the server. The input to this process is the recorded video of the gameplay, and the output is the transmission of the video file to the server. Specific operations include taking a video using the camera app, saving it as a video file, and uploading the video file from the application.

[0101] Step 2:

[0102] The server receives gameplay video data sent from the player's device. The received video is temporarily stored on the server. The server then uploads the video to cloud storage and generates a destination URL for the video. The input to this process is the gameplay video data sent from the device, and the output is the URL for the video stored in cloud storage. Specific operations include saving the video file on the server, uploading it to cloud storage, and generating a destination URL.

[0103] Step 3:

[0104] The server creates a request to provide the generative AI model with the destination URL. This request includes a prompt for the generative AI model (e.g., "Please analyze the dribbling skills in this video, identify the player's strengths and areas for improvement, and generate specific training advice."). The server sends this request to the generative AI model. The inputs to this process are the URL of the video stored in cloud storage and the prompt, and the output is a request sent to the generative AI model.

[0105] Step 4:

[0106] The generative AI model retrieves the video data from cloud storage using the received URL. The AI ​​model then analyzes the video data and evaluates the player's movements. This evaluation includes detailed analysis of the player's dribbling skills, ball control, speed, etc. The generative AI model then generates specific feedback based on the analysis results. The input of this process is the video data retrieved from cloud storage, and the output is specific feedback.

[0107] Step 5:

[0108] The server receives feedback sent from the generative AI model. The received feedback is sent to the player device. When the player device is ready to receive feedback, the server sends a notification. The input of this process is feedback from the generative AI model, and the output is sending feedback and a notification to the player device.

[0109] Step 6:

[0110] The player device receives the feedback sent from the server. The user opens the application and checks the received feedback. For example, the application may display a message such as "Your dribbling speed is excellent, but your timing is off, making it difficult to control the ball," and provide specific drills and practice menus to improve your timing. The input to this process is the feedback sent from the server, and the output is the feedback displayed to the user. The user can then use this information to create a practice plan and work to improve their skills.

[0111] (Application example 1)

[0112] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0113] Real-time data analysis and feedback are essential to improving the driving performance of autonomous vehicles. However, conventional systems lack real-time capabilities and have difficulty providing quick and accurate improvement measures. This has prevented improvements in safety and efficiency.

[0114] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0115] In this invention, the server includes means for uploading data from a user terminal, means for storing the uploaded data in a data storage device, means for passing the data stored in the data storage device to a generative AI model for analysis, means for generating feedback based on the data analysis results, and means for transmitting the generated feedback to the user terminal, thereby enabling the driving data of an autonomous vehicle to be analyzed in real time and providing prompt and accurate feedback based on the analysis results.

[0116] "User Device" means a device used by a User to collect, store, or transmit data, including a smartphone, tablet, camera, etc.

[0117] A "data storage device" is a storage system for temporarily or long-term storage of collected data, and includes cloud storage and on-premise storage servers.

[0118] A "generative AI model" is an algorithm that analyzes collected data and generates feedback or advice tailored to a specific purpose, and includes models that use machine learning and deep learning.

[0119] "Analysis results" are the results or evaluations that a generative AI model derives from input data, including strengths and areas for improvement regarding specific performance.

[0120] "Feedback" refers to specific instructions or advice provided to users based on the analysis results, including specific examples and improvement measures.

[0121] The driving analysis system for autonomous vehicles according to the present invention is a real-time driving analysis system that collects and analyzes data obtained from cameras and sensors on autonomous vehicles, identifies strengths and areas for improvement in driving performance, and provides specific instructions and feedback in real time based on the analysis results.

[0122] System Components and Functions

[0123] 1. User Device:

[0124] User terminals are devices used to collect, store, and transmit data. These include cameras and various sensors (such as LiDAR and radar) installed in autonomous vehicles, which collect surrounding situation data and driving information in real time.

[0125] 2. Data Storage Device:

[0126] A data storage device is a storage system for temporarily or long-term storage of collected data. Cloud storage such as Amazon S3 is an example of this, allowing data to be stored and quickly accessed.

[0127] 3. Generative AI Model:

[0128] Generative AI models are algorithms that analyze collected data and generate feedback and advice tailored to specific objectives, including machine learning and deep learning models like OpenAI GPT-4. Based on the analysis results, they identify strengths and areas for improvement regarding the driving performance of autonomous vehicles.

[0129] 4. Server:

[0130] The server uploads data from the user's device and stores it in a data storage device. Next, it requests the generated AI model to analyze the data by passing the URL of the destination. It also generates feedback based on the analysis results from the generated AI model and sends it to the user's device. Examples of such servers include AWS EC2.

[0131] Specific processing of the program

[0132] 1. Data Collection:

[0133] Cameras and various sensors (such as LiDAR and radar) installed in autonomous vehicles collect data on the surrounding environment and driving information, which is then uploaded to cloud storage in real time.

[0134] 2. Data Analysis:

[0135] The server saves the uploaded data to cloud storage (Amazon S3) and passes the URL of the saved data to an OpenAI GPT-4-based generative AI model, which analyzes the driving data and identifies strengths and areas for improvement in vehicle performance.

[0136] 3. Feedback Generation:

[0137] Based on the analysis results, the generative AI model generates feedback including specific instructions and improvement measures, such as "When turning right, it is recommended to turn on the right turn signal 50 meters before the intersection."

[0138] 4. Real-time control:

[0139] The server sends the generated feedback to the user's terminal, and the autonomous vehicle uses the feedback to improve its driving performance.

[0140] Examples and prompts

[0141] Examples:

[0142] When an autonomous vehicle makes a right turn at an intersection, it detects obstacles and uploads the information to cloud storage in real time. A generative AI model analyzes this data and provides feedback on how to improve the timing of the right turn. Based on this feedback, the vehicle completes the right turn smoothly.

[0143] Example prompt sentence:

[0144] Driving data analysis: Cameras detect obstacles 50 meters before an intersection. The system evaluates the timing of right turns at intersections and generates optimal improvement measures.

[0145] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0146] Step 1:

[0147] The autonomous vehicle's cameras and various sensors (LiDAR, radar, etc.) collect surrounding situation data and driving information in real time. The input is information about the surrounding environment and vehicle operation information. The output is the generation and storage of this data. Specifically, the camera captures video data, and the sensors measure information such as distance and angle.

[0148] Step 2:

[0149] The server stores the collected data in cloud storage (Amazon S3). The input is the environmental information and operation information generated in step 1. The output is a notification that the data has been saved to cloud storage and the URL of the destination. Specifically, the server converts the data into an appropriate format and uploads it to cloud storage.

[0150] Step 3:

[0151] The server passes the destination URL of the data stored in cloud storage to the generative AI model. The input is the destination URL obtained in step 2. The output is an analysis request to the generative AI model and a URL. Specifically, the server constructs a URL and sends the analysis request to the generative AI model.

[0152] Step 4:

[0153] The generative AI model retrieves data from cloud storage and analyzes it. The input is the destination URL. The output is the analysis results, specifically the strengths and areas for improvement in driving performance. Specifically, the generative AI model downloads the data and analyzes it using machine learning algorithms.

[0154] Step 5:

[0155] The generative AI model generates feedback based on the analysis results. The input is the analysis results obtained in step 4. The output is a feedback message that includes specific operational instructions and improvement measures. Specifically, the generative AI model evaluates the analysis results and generates appropriate feedback statements.

[0156] Step 6:

[0157] The server sends the generated feedback to the user terminal. The input is the feedback message obtained in step 5. The output is a feedback notification to the user terminal. Specifically, the server converts the feedback message into an appropriate format and sends it to the user terminal.

[0158] Step 7:

[0159] The user device receives the feedback and reflects it in the autonomous vehicle's control system. The input is the feedback notification obtained in step 6. The output is an instruction for the control system to operate. Specifically, the user device analyzes the feedback content and updates the autonomous vehicle's control commands based on it.

[0160] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0161] The online coaching and play analysis system related to the present invention includes a player terminal, cloud storage, a generative AI model, an emotion engine, and a server. The functions and processing flow of each component are described in detail below to implement the present invention.

[0162] Player device functions

[0163] A player device is a device that allows a player to record, save, and upload their play to a server. It also serves as a receiver for the feedback and emotion analysis results provided by the player as analytical results. Examples of such devices include smartphones, tablets, and cameras.

[0164] Server Features

[0165] The server plays a central role in receiving, storing, and analyzing gameplay videos, question data, and emotion data sent from player devices. Furthermore, the server passes the emotion analysis results from the emotion engine to the generative AI model, helping to create emotion-adaptive feedback. This allows for more personalized feedback to be provided to players.

[0166] Cloud storage features

[0167] Cloud storage is a place to temporarily store the received gameplay video. The server saves the gameplay video to cloud storage and passes the URL of the saved location to the generation AI model.

[0168] Generative AI model capabilities

[0169] The generative AI model analyzes gameplay videos stored in cloud storage to identify players' strengths and areas for improvement. Furthermore, the AI ​​model generates optimal feedback for players based on the player's questions and the emotional analysis results provided by the emotion engine.

[0170] Emotion Engine Functions

[0171] The emotion engine is a system that recognizes and analyzes the player's emotional state. It analyzes emotions based on the player's voice and facial expression data, and provides the results to a generative AI model to improve the quality of feedback.

[0172] Specific processing of the program

[0173] The server receives the video of the player's play from the player's device. The user takes a video of the dribble and uploads it through the application. The server saves the video to cloud storage and generates a URL for the destination.

[0174] The server then passes the URL to the generative AI model, asking it to analyze the video. The generative AI model analyzes the video and identifies strengths and areas for improvement in the player's dribbling skills. Based on the analysis results, the generative AI model creates feedback including specific training advice.

[0175] Additionally, an emotion engine analyzes the player's emotional state and provides the results to the generative AI model, determining, for example, whether the player is nervous or stressed.

[0176] The generative AI model generates emotion-adaptive feedback by comprehensively considering the player's play analysis results and emotional state. For example, if the emotion engine detects "disappointment" or "frustration" when a player makes a mistake, the generative AI model will provide feedback including "technical advice and emotional support to maintain confidence."

[0177] The server then sends the generated feedback and emotion analysis results to the player's device, which then receives a notification. The player can then open the application to view the specific feedback and emotion analysis results and use them for practice.

[0178] For example, feedback such as "Your dribbling speed is excellent, but you have difficulty maintaining your balance" is provided. Other examples of emotion-adaptive feedback include "breathing techniques to relieve tension" and "mental training to maintain positive thinking."

[0179] Such a system would enable players to receive more effective and personalized training guidance, and would also take into account their emotional state, potentially improving their overall performance.

[0180] The processing flow will be explained below.

[0181] Step 1:

[0182] A user takes a video of a game using a smartphone or camera and saves it on the device. For example, a user might take a video of a soccer player dribbling.

[0183] Step 2:

[0184] The user launches the application, selects a saved gameplay video, and uploads it to the server. Specifically, the user taps the "Upload Video" button, selects the video file, and sends an upload request.

[0185] Step 3:

[0186] The device sends the uploaded gameplay video to the server, which receives the HTTP POST request and saves the video file in cloud storage.

[0187] Step 4:

[0188] The server generates a URL for the video stored in cloud storage and sends an analysis request to the generative AI model. Specifically, the server passes the video URL to the generative AI model via a REST API.

[0189] Step 5:

[0190] The generative AI model receives the video URL and begins analyzing the video of the game. The generative AI model analyzes the player's movements and techniques in the video to identify strengths and areas for improvement.

[0191] Step 6:

[0192] The user opens the application's question input screen, enters a question about a specific play, and sends it to the server. For example, they might enter, "How can I improve my dribbling accuracy?"

[0193] Step 7:

[0194] The server sends the question to the generative AI model and asks it to also perform analysis based on the question.

[0195] Step 8:

[0196] The generative AI model generates feedback based on the video analysis and user questions, including dribbling strengths and areas for improvement, as well as specific training advice related to the question.

[0197] Step 9:

[0198] The user activates the emotion engine to collect facial and voice data while playing. For example, a camera or microphone records the user's emotion data while playing.

[0199] Step 10:

[0200] The device sends the collected emotion data to the server, which receives the emotion data and passes it to the emotion engine.

[0201] Step 11:

[0202] The emotion engine analyzes the emotion data to identify the player's emotional state, such as "tension" or "lack of concentration."

[0203] Step 12:

[0204] The emotion engine sends the emotion analysis results to the generative AI model, which receives the emotion data and uses it as feedback.

[0205] Step 13:

[0206] The generative AI model generates emotion-adaptive feedback based on emotion analysis, video analysis, and questions, including suggestions such as "breathing techniques to relax" and "mental training to improve concentration."

[0207] Step 14:

[0208] The server sends the generated feedback and the emotion analysis results to the player device, which receives the HTTP response and retrieves the feedback data.

[0209] Step 15:

[0210] The device receives the feedback and sentiment analysis results from the server and displays a notification to the user, for example, a push notification informing the user that "feedback has been received."

[0211] Step 16:

[0212] The user opens the app and sees detailed feedback and sentiment analysis, including specific advice on how to improve dribbling and relax.

[0213] Step 17:

[0214] Based on the feedback and emotional advice provided by the user, the system implements specific actions to improve training and play, such as "training to improve dribbling timing and methods to relax mentally."

[0215] In this way, the present invention provides feedback that takes into account the player's emotional state in addition to technical analysis, enabling more effective and personalized training.

[0216] Example 2

[0217] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0218] Conventional online coaching systems often do not take into account the user's emotional state when analyzing a player's playing video and generating feedback. As a result, optimal training advice is not provided to the player, limiting the effectiveness of training. Furthermore, even if a player inputs a specific question, it is difficult to obtain feedback in real time that responds to that question.

[0219] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for uploading player videos from a player terminal, means for saving the uploaded player videos in cloud storage, means for passing the player videos saved in the cloud storage to a generative AI model for analysis, means for generating feedback based on the analysis results and emotion analysis results of the player videos, and means for transmitting the generated feedback to the player terminal. This makes it possible to provide optimal feedback in real time that takes into account the player's emotional state. In addition, the player can input a question about a specific play, and feedback based on the question can be appropriately generated and provided.

[0220] A "terminal" is an electronic device used by a user, and includes a smartphone, tablet, camera, etc.

[0221] "Video" refers to video data that records a user's play.

[0222] "Cloud storage" is a data storage service accessible via the internet that provides server space for temporary storage of video files.

[0223] "Generative AI model" refers to an artificial intelligence system that analyzes videos stored in cloud storage and generates feedback.

[0224] "Feedback" refers to advice and instruction to improve a player's skills that is created by the generative AI model based on the results of video analysis.

[0225] "Emotion analysis result" refers to information about the user's emotional state obtained as a result of the emotion engine analyzing the user's voice and facial expression data.

[0226] "Emotion-adaptive feedback" refers to the process by which a generative AI model generates feedback optimized according to the user's emotional state based on the results of video analysis and emotion analysis.

[0227] "Server" means the central computer system that receives, stores, and analyzes data transmitted from Player Devices and generates and transmits feedback.

[0228] A "question" refers to an inquiry a user inputs into a generative AI model regarding a particular play.

[0229] "Player" means a person who practices a sport or game and uses the System to analyze video of their play and receive feedback.

[0230] The online coaching and play analysis system related to the present invention includes a player terminal, cloud storage, a generative AI model, an emotion engine, and a server. The functions and processing flow of each component are described in detail below.

[0231] Player device functions

[0232] A player device is a device that allows a player to record, save, and upload gameplay videos to a server. It also receives feedback and emotion analysis results from the player. Examples of such devices include smartphones, tablets, and cameras.

[0233] Server Features

[0234] The server plays a central role in receiving, storing, and analyzing gameplay videos, question data, and emotion data sent from the player's device. The server saves the received gameplay videos in cloud storage and passes the URL of the saved video to the generative AI model. In addition, the server passes the emotion analysis results from the emotion engine to the generative AI model to help create emotion-adaptive feedback. This allows for more personalized feedback to be provided to the player.

[0235] Cloud storage features

[0236] Cloud storage is a place to temporarily store the received gameplay video. The server saves the gameplay video in cloud storage (e.g., Amazon S3 or Google Cloud Storage) and generates a URL for the destination.

[0237] Generative AI model capabilities

[0238] The generative AI model is a system that analyzes gameplay videos stored in cloud storage and identifies players' strengths and areas for improvement. Furthermore, this generative AI model generates optimal feedback for players based on the player's questions and the emotional analysis results provided by the emotion engine. By comprehensively considering the analytical and emotional analysis results, it is possible to generate emotion-adaptive feedback.

[0239] Prompt Sentence Examples

[0240] For example, the prompt sentence to be input to the generative AI model is as follows:

[0241] "Download the video from this URL and perform a skill analysis of the player. Furthermore, create feedback to provide to the player based on the emotion analysis results obtained from the voice and facial expression data."

[0242] Emotion Engine Functions

[0243] The emotion engine is a system that recognizes and analyzes a player's emotional state. By analyzing emotions based on the player's voice and facial expression data and providing the results to a generative AI model, the quality of feedback can be improved. For example, it can determine whether a player is feeling nervous or stressed and reflect that information in the feedback.

[0244] Example

[0245] As a concrete example, consider a scenario in which a user films a dribbling play and uploads the video from their device to a server. The server saves the video file in cloud storage and passes the URL to the generative AI model. The generative AI model analyzes the video and finds that "the dribbling speed is excellent, but maintaining balance is difficult." The emotion engine also detects that the user is nervous. Based on this information, the generative AI model generates emotion-adaptive feedback, including technical advice such as "breathing techniques to relieve tension" and "mental training to maintain a positive mindset."

[0246] This provides players with a system that allows them to not only improve their technical skills, but also manage their emotional state and aim to improve their overall performance.

[0247] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0248] Step 1:

[0249] The user records a video of a dribbling play on the device. The user taps the "Record new video" button, which activates the device's camera. After recording is complete, the user saves the video file. The input is the video file of the play, and the output is the video file saved on the device.

[0250] Step 2:

[0251] A user uploads a video from their device to a server. When the user taps the "Upload" button in the application, the device sends the video file to the server. The input is the video file stored on the device, and the output is the video file uploaded to the server.

[0252] Step 3:

[0253] The server saves the received video in cloud storage and generates a destination URL. The server saves the video file in cloud storage (e.g., Amazon S3) and generates a unique destination URL. The input is the video file received by the server, and the output is the destination URL in cloud storage.

[0254] Step 4:

[0255] The server sends the destination URL to the generative AI model. The server sends the URL to the generative AI model and creates a prompt for analysis. The input is the destination URL in cloud storage, and the output is the prompt sent to the generative AI model.

[0256] Step 5:

[0257] The generative AI model analyzes the video and generates feedback. The generative AI model downloads the video from cloud storage and analyzes it. As a result of the analysis, it extracts the player's technical strengths and areas for improvement. It then generates specific training advice based on this information. The input is the video file downloaded from cloud storage, and the output is feedback based on the analysis results.

[0258] Step 6:

[0259] The emotion engine analyzes the player's emotional state and sends the results to the generative AI model. If voice and facial expression data are also recorded when the player uploads a video, the emotion engine analyzes this data. It identifies the tension or stress the player is feeling and sends the analysis results to the generative AI model. The input is the player's voice and facial expression data, and the output is the emotion analysis results.

[0260] Step 7:

[0261] The generative AI model generates emotion-adaptive feedback based on the results of video analysis and emotion analysis. The generative AI model integrates the results of video analysis with feedback that takes into account the player's emotional state. For example, if a player feels "disappointed" or "frustrated" after making a mistake, it creates feedback that includes technical advice and emotional support. The input is the results of video analysis and emotion analysis, and the output is emotion-adaptive feedback.

[0262] Step 8:

[0263] The server sends the generated feedback and emotion analysis results to the player's device. The server sends the feedback received from the generative AI model to the player's device and performs notification. The input is the generated feedback and emotion analysis results, and the output is the feedback and emotion analysis results sent to the player's device.

[0264] Step 9:

[0265] The user checks the feedback and emotion analysis results on the device and uses them for training. The user opens the application on the device to check the feedback and emotion analysis results, and improves their training and play based on the feedback they receive. The input is the feedback and emotion analysis results sent to the player device, and the output is the improvement results obtained by the user.

[0266] (Application example 2)

[0267] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0268] Previous online coaching and play analysis systems focused primarily on improving players' technical skills, but did not provide feedback or content recommendations based on users' emotions or viewing behavior. This limited the overall user experience. Given this background, there was a need to develop a system that could analyze users' viewing behavior and emotional state to provide more personalized feedback and content recommendations.

[0269] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading player videos from the player terminal, means for saving the uploaded player videos in cloud storage, means for passing the player videos saved in the cloud storage to a generative AI model for analysis, means for generating feedback based on the analysis results of the player videos, means for sending the generated feedback to the player terminal, means for analyzing the user's viewing behavior and emotional state, and means for recommending personalized content based on the analysis results. This makes it possible to not only improve the user's technical skills but also to provide optimal feedback and content recommendations according to the user's viewing behavior and emotional state.

[0270] "Player Device" means an electronic device used by a User to record, store, and upload videos of his / her gameplay.

[0271] "Player video" is video data showing the play and activities of a user that is filmed using a player terminal.

[0272] "Cloud storage" is an online storage system for storing and managing data via the Internet.

[0273] A "generative AI model" is an artificial intelligence system that analyzes users' video data and question data to generate personalized feedback.

[0274] "Analysis results" are the evaluation and comment data obtained after the generative AI model analyzes the video data and question data.

[0275] "Feedback" refers to information that includes evaluations and suggestions for improvement regarding a user's playing or viewing behavior.

[0276] "Viewing behavior" refers to the actions and patterns that users take when watching video content, and is the data related to that.

[0277] "Emotional state" is data that represents the user's emotional and psychological state when viewing.

[0278] "Content recommendation" is a function that presents the most appropriate videos and information based on the user's viewing behavior and emotional state.

[0279] The online coaching and play analysis system related to the present invention includes a player terminal, cloud storage, a generative AI model, an emotion engine, and a server. The functions of each component and the processing flow for implementing the present invention are described in detail below.

[0280] Player device functions

[0281] A player device is an electronic device that allows a user to record, save, and upload their gameplay video to a server. The user also receives feedback and sentiment analysis results. Examples of such devices include smartphones, tablets, and cameras.

[0282] Server Features

[0283] The server plays a central role in receiving, storing, and analyzing gameplay videos, viewing behavior data, and emotional data sent from player devices. Furthermore, the server passes the emotional analysis results from the emotion engine to the generative AI model, helping to create personalized feedback. This allows for more personalized content recommendations to be provided to users.

[0284] Cloud storage features

[0285] Cloud storage is a place to temporarily store received gameplay videos and viewing behavior data. The server saves the gameplay videos in cloud storage and passes the URL of the saved location to the generation AI model.

[0286] Generative AI model capabilities

[0287] The generative AI model analyzes gameplay videos stored in cloud storage and identifies users' strengths and areas for improvement. Furthermore, the AI ​​model generates optimal feedback for users based on their questions and the emotional analysis results provided by the emotion engine. It also includes a function to recommend optimal content based on the user's viewing behavior and emotional state.

[0288] Emotion Engine Functions

[0289] The emotion engine is a system that recognizes and analyzes a user's emotional state. It analyzes emotions based on the user's voice and facial expression data, and provides the results to a generative AI model to improve the quality of feedback and content recommendations.

[0290] Processing flow

[0291] The player's device records a video of the game and uploads it to the server. The server saves the video to cloud storage and generates a URL for the destination. The server then passes the URL to the generated AI model and requests it to analyze the video.

[0292] The generative AI model analyzes the video and identifies the user's skill strengths and areas for improvement. Based on the analysis results, the generative AI model creates feedback including specific training advice. At the same time, the emotion engine analyzes the user's emotional state and provides the analysis results to the generative AI model. The generative AI model then generates emotion-adaptive feedback by comprehensively considering the play analysis results and the user's emotional state.

[0293] Furthermore, it recommends optimal content based on the user's viewing behavior and emotional state. The server sends the generated feedback and content recommendation results to the player device, and the user can open the application and check them.

[0294] Examples and prompts

[0295] Examples:

[0296] Video content: User watching a fitness video

[0297] Sentiment analysis results: "Enjoyed" "Excited"

[0298] Generated feedback: "You seem to be really enjoying this exercise. As a next step, try these more advanced exercises."

[0299] Example prompt sentence:

[0300] You performed sentiment analysis on the fitness video the user was watching, and the result was that the user was enjoying it. Based on this analysis, you want to generate feedback to recommend the next advanced exercise to the user.

[0301] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0302] Step 1:

[0303] The player device takes video of the user's play. The camera function of the player device is used to record the user's play and activities. The input is actual play video, and the output is digital video data.

[0304] Step 2:

[0305] The gameplay video captured by the player's device is uploaded to the server. The video data is sent to the server via the Internet. The input is the gameplay video in digital format, and the output is a video file stored on the server.

[0306] Step 3:

[0307] The server saves the uploaded gameplay video to cloud storage. Using the file saving API for cloud storage, the video data is saved to the cloud. The input is the video file on the server, and the output is the URL of the cloud storage destination.

[0308] Step 4:

[0309] The server passes the URL of the video stored in cloud storage to the generative AI model. The URL of the video is passed to the API of the generative AI model and an analysis request is made. The input is the URL of the cloud storage destination, and the output is the analysis result of the generative AI model.

[0310] Step 5:

[0311] The generative AI model analyzes the gameplay video stored in cloud storage. The generative AI model's algorithm analyzes the video data and identifies the user's skills, strengths, and areas for improvement. The input is the cloud storage destination URL, and the output is the gameplay analysis results.

[0312] Step 6:

[0313] The generative AI model generates feedback based on the results of play analysis. It also takes into account the emotional analysis results provided by the emotion engine to create personalized feedback. The input is the results of play analysis and emotional analysis, and the output is feedback that includes specific training advice.

[0314] Step 7:

[0315] The server sends the feedback received from the generative AI model to the player device. The feedback data is notified to the player device via the Internet. The input is the generated feedback, and the output is the feedback displayed on the user's player device.

[0316] Step 8:

[0317] The player device receives the feedback and the user confirms it. The application is opened and the feedback content is displayed. The input is the feedback data sent from the server, and the output is the feedback screen that the user can see.

[0318] Step 9:

[0319] The emotion engine analyzes the user's viewing behavior and emotional state. It analyzes the video being watched and the user's reactions to identify the emotional state. The input is the user's viewing data and voice / facial expression data, and the output is the emotion analysis results.

[0320] Step 10:

[0321] The generative AI model recommends personalized content based on the results of sentiment analysis. It selects the most appropriate content based on the user's emotional state and viewing behavior, and creates a recommendation list. The input is the results of sentiment analysis and viewing behavior data, and the output is a content recommendation list.

[0322] Step 11:

[0323] The server sends the generated content recommendation list to the player terminal. The recommendation list is notified to the player terminal via the Internet. The input is the generated content recommendation list, and the output is the recommended content displayed on the user's player terminal.

[0324] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0325] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0326] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0327] [Second embodiment]

[0328] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0329] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0330] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0331] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0332] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0333] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0334] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0335] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0336] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0337] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0338] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0339] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0340] The online coaching and play analysis system of the present invention includes a player terminal, cloud storage, a generative AI model, and a server. The functions and processing flow of each component are shown below to implement the present invention.

[0341] Player device functions

[0342] A player device is a device that is primarily used by a player to record, save, and upload their play to a server. It is also the device through which the player receives feedback provided as analytical results. Specifically, a player device may be a smartphone, tablet, or camera.

[0343] Server Features

[0344] The server plays a central role in receiving, storing, and analyzing gameplay videos and question data sent from the player's device. The server provides the URL of the destination for the received video to the AI ​​model, which then analyzes the video. It also has the function of generating feedback based on the analysis results and sending it to the player's device.

[0345] Cloud storage features

[0346] Cloud storage is a place to temporarily store the received gameplay video. The server saves the video in cloud storage and passes the URL of the saved video to the generation AI model.

[0347] Generative AI model capabilities

[0348] The generative AI model is a system that analyzes gameplay videos stored in cloud storage to identify players' strengths and areas for improvement. It has the ability to generate specific feedback based on the analysis results. Furthermore, if the player inputs specific questions, it can also generate feedback based on those questions.

[0349] Specific processing of the program

[0350] The server receives the video of the player's play. For example, the player takes a video of their dribbling and uploads it through the application. The server saves the video in cloud storage and generates a URL for the destination.

[0351] The server then passes the URL to the generative AI model, asking it to analyze the video. The generative AI model analyzes the video and identifies strengths and areas for improvement in the player's dribbling skills. Based on the analysis results, the generative AI model creates feedback including specific training advice.

[0352] The server then sends the generated feedback to the player terminal, and the player terminal receives a notification, allowing the player to open the application, check the specific feedback content, and use it to improve their practice.

[0353] For example, the feedback provided may include, "Your dribbling speed is excellent, but your timing is off, making it difficult to control the ball." The advice also includes specific drills and practice menus to improve your timing. Based on this, players can then practice to improve their own play.

[0354] In this way, the system of the present invention allows players to receive efficient and accurate feedback and specific advice on how to improve their play without the need for a coach.

[0355] The processing flow will be explained below.

[0356] Step 1:

[0357] A user takes a video of a game using a smartphone or camera and saves it on the device. For example, a user might take a video of a soccer player dribbling.

[0358] Step 2:

[0359] The user launches the application, selects a saved gameplay video, and uploads it to the server. Specifically, the user taps the "Upload Video" button, selects the video file, and sends an upload request.

[0360] Step 3:

[0361] The device sends the uploaded gameplay video to the server, which receives the HTTP POST request and saves the video file in cloud storage.

[0362] Step 4:

[0363] The server generates a URL for the video stored in cloud storage and sends an analysis request to the generative AI model. Specifically, the server passes the video URL to the generative AI model via a REST API.

[0364] Step 5:

[0365] The generative AI model receives the video URL and begins analyzing the video of the game. The generative AI model analyzes the player's movements and techniques in the video to identify strengths and areas for improvement.

[0366] Step 6:

[0367] A generative AI model uses the analysis to generate feedback, including the player's technical strengths and areas for improvement, as well as specific training advice.

[0368] Step 7:

[0369] The server receives the feedback obtained from the generative AI model and sends it to the player's device. The server returns the feedback data to the device via an HTTP response.

[0370] Step 8:

[0371] The device receives the feedback from the server and displays a notification to the user, for example, a push notification that says "Your feedback has been received."

[0372] Step 9:

[0373] The user opens the application and checks the received feedback. Specifically, the user can view the feedback on the application screen and identify strengths and areas for improvement.

[0374] Step 10:

[0375] Based on the feedback provided by the user, the system implements specific advice for training and improving play, such as "doing specific drills to improve dribbling timing."

[0376] Example 1

[0377] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0378] Conventional play analysis systems have issues such as low analysis accuracy, inability to provide instant feedback, and inability to obtain analysis results that satisfy users. Furthermore, when players themselves want feedback on specific questions, they often lack the ability to obtain specific advice. To address these issues, a system that improves the accuracy of gameplay video analysis and provides fast, specific feedback is needed.

[0379] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0380] In this invention, the server includes a means for uploading gameplay videos from the player terminal, a means for saving the uploaded gameplay videos to cloud storage, and a means for generating a destination URL for the gameplay videos saved in cloud storage and passing it to the generation AI model. This improves the accuracy of gameplay video analysis and makes it possible to provide quick and specific feedback.

[0381] "Player Device" means a device used by a player to record, store, and upload their play to the server, such as a smartphone, tablet, or camera.

[0382] "Cloud storage" refers to a group of remote servers that store data via the Internet. In particular, in the present invention, it is used to temporarily store gameplay videos.

[0383] The "generative AI model" is an artificial intelligence model that analyzes gameplay videos stored in cloud storage and identifies players' strengths and areas for improvement. This model is responsible for generating specific feedback based on the results of gameplay analysis.

[0384] "Feedback" refers to specific advice and information on areas for improvement provided by the generative AI model based on the analysis of the gameplay video. The goal is to improve the player's skills.

[0385] The online coaching and play analysis system of the present invention includes a player terminal, cloud storage, a generative AI model, and a server. The components and processing flow of this system will be described in detail below.

[0386] Player device functions

[0387] A player device is a device used by a player to record, save, and upload their play to a server. Specifically, this includes smartphones, tablets, and cameras. It is also on the player device that the player receives feedback provided as analysis results. For example, a user may film a soccer dribble using a smartphone and upload the video through an application.

[0388] Server Features

[0389] The server plays a central role in receiving gameplay videos and question data sent from the player's device and storing them in cloud storage. Specifically, the server stores the received videos in cloud storage and generates a URL for the destination. This URL is then provided to the generative AI model, which requests it to analyze the video. The server also has the function of generating feedback based on the analysis results returned by the generative AI model and sending it to the player's device.

[0390] Cloud storage features

[0391] Cloud storage is a group of remote servers that temporarily store the received gameplay videos. The server saves the videos in cloud storage and passes the URL of the saved video to the generated AI model. Specifically, a common cloud storage service (e.g., Amazon S3) is used.

[0392] Generative AI model capabilities

[0393] The generative AI model is an artificial intelligence system that analyzes gameplay videos stored in cloud storage and identifies players' strengths and areas for improvement. This model has the ability to generate specific feedback based on the analysis results. Furthermore, if a player inputs a specific question, it can generate appropriate feedback based on that question.

[0394] Feedback example

[0395] An example of specific feedback might be, "Your dribbling speed is excellent, but your timing is off, making it difficult to control the ball." The advice also includes specific drills and practice exercises to improve your timing. This gives players a concrete action plan to improve their play.

[0396] Prompt Sentence Examples

[0397] The following prompts can be used as input to a generative AI model:

[0398] "Analyze this video for dribbling skills, identify player strengths and areas for improvement, and generate specific training advice."

[0399] The system allows players to receive efficient and accurate feedback quickly, giving them specific advice on how to improve their play without the need for a coach.

[0400] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0401] Step 1:

[0402] The user launches the application on the player's device and records a video of their gameplay. The recorded video is temporarily saved on the device. Then, when the user taps the "Upload" button in the application, the device sends the video file to the server. The input to this process is the recorded video of the gameplay, and the output is the transmission of the video file to the server. Specific operations include taking a video using the camera app, saving it as a video file, and uploading the video file from the application.

[0403] Step 2:

[0404] The server receives gameplay video data sent from the player's device. The received video is temporarily stored on the server. The server then uploads the video to cloud storage and generates a destination URL for the video. The input to this process is the gameplay video data sent from the device, and the output is the URL for the video stored in cloud storage. Specific operations include saving the video file on the server, uploading it to cloud storage, and generating a destination URL.

[0405] Step 3:

[0406] The server creates a request to provide the generative AI model with the destination URL. This request includes a prompt for the generative AI model (e.g., "Please analyze the dribbling skills in this video, identify the player's strengths and areas for improvement, and generate specific training advice."). The server sends this request to the generative AI model. The inputs to this process are the URL of the video stored in cloud storage and the prompt, and the output is a request sent to the generative AI model.

[0407] Step 4:

[0408] The generative AI model retrieves the video data from cloud storage using the received URL. The AI ​​model then analyzes the video data and evaluates the player's movements. This evaluation includes detailed analysis of the player's dribbling skills, ball control, speed, etc. The generative AI model then generates specific feedback based on the analysis results. The input of this process is the video data retrieved from cloud storage, and the output is specific feedback.

[0409] Step 5:

[0410] The server receives feedback sent from the generative AI model. The received feedback is sent to the player device. When the player device is ready to receive feedback, the server sends a notification. The input of this process is feedback from the generative AI model, and the output is sending feedback and a notification to the player device.

[0411] Step 6:

[0412] The player device receives the feedback sent from the server. The user opens the application and checks the received feedback. For example, the application may display a message such as "Your dribbling speed is excellent, but your timing is off, making it difficult to control the ball," and provide specific drills and practice menus to improve your timing. The input to this process is the feedback sent from the server, and the output is the feedback displayed to the user. The user can then use this information to create a practice plan and work to improve their skills.

[0413] (Application example 1)

[0414] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0415] Real-time data analysis and feedback are essential to improving the driving performance of autonomous vehicles. However, conventional systems lack real-time capabilities and have difficulty providing quick and accurate improvement measures. This has prevented improvements in safety and efficiency.

[0416] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0417] In this invention, the server includes means for uploading data from a user terminal, means for storing the uploaded data in a data storage device, means for passing the data stored in the data storage device to a generative AI model for analysis, means for generating feedback based on the data analysis results, and means for transmitting the generated feedback to the user terminal, thereby enabling the driving data of an autonomous vehicle to be analyzed in real time and providing prompt and accurate feedback based on the analysis results.

[0418] "User Device" means a device used by a User to collect, store, or transmit data, including a smartphone, tablet, camera, etc.

[0419] A "data storage device" is a storage system for temporarily or long-term storage of collected data, and includes cloud storage and on-premise storage servers.

[0420] A "generative AI model" is an algorithm that analyzes collected data and generates feedback or advice tailored to a specific purpose, and includes models that use machine learning and deep learning.

[0421] "Analysis results" are the results or evaluations that a generative AI model derives from input data, including strengths and areas for improvement regarding specific performance.

[0422] "Feedback" refers to specific instructions or advice provided to users based on the analysis results, including specific examples and improvement measures.

[0423] The driving analysis system for autonomous vehicles according to the present invention is a real-time driving analysis system that collects and analyzes data obtained from cameras and sensors on autonomous vehicles, identifies strengths and areas for improvement in driving performance, and provides specific instructions and feedback in real time based on the analysis results.

[0424] System Components and Functions

[0425] 1. User Device:

[0426] User terminals are devices used to collect, store, and transmit data. These include cameras and various sensors (such as LiDAR and radar) installed in autonomous vehicles, which collect surrounding situation data and driving information in real time.

[0427] 2. Data Storage Device:

[0428] A data storage device is a storage system for temporarily or long-term storage of collected data. Cloud storage such as Amazon S3 is an example of this, allowing data to be stored and quickly accessed.

[0429] 3. Generative AI Model:

[0430] Generative AI models are algorithms that analyze collected data and generate feedback and advice tailored to specific objectives, including machine learning and deep learning models like OpenAI GPT-4. Based on the analysis results, they identify strengths and areas for improvement regarding the driving performance of autonomous vehicles.

[0431] 4. Server:

[0432] The server uploads data from the user's device and stores it in a data storage device. Next, it requests the generated AI model to analyze the data by passing the URL of the destination. It also generates feedback based on the analysis results from the generated AI model and sends it to the user's device. Examples of such servers include AWS EC2.

[0433] Specific processing of the program

[0434] 1. Data Collection:

[0435] Cameras and various sensors (such as LiDAR and radar) installed in autonomous vehicles collect data on the surrounding environment and driving information, which is then uploaded to cloud storage in real time.

[0436] 2. Data Analysis:

[0437] The server saves the uploaded data to cloud storage (Amazon S3) and passes the URL of the saved data to an OpenAI GPT-4-based generative AI model, which analyzes the driving data and identifies strengths and areas for improvement in vehicle performance.

[0438] 3. Feedback Generation:

[0439] Based on the analysis results, the generative AI model generates feedback including specific instructions and improvement measures, such as "When turning right, it is recommended to turn on the right turn signal 50 meters before the intersection."

[0440] 4. Real-time control:

[0441] The server sends the generated feedback to the user's terminal, and the autonomous vehicle uses the feedback to improve its driving performance.

[0442] Examples and prompts

[0443] Examples:

[0444] When an autonomous vehicle makes a right turn at an intersection, it detects obstacles and uploads the information to cloud storage in real time. A generative AI model analyzes this data and provides feedback on how to improve the timing of the right turn. Based on this feedback, the vehicle completes the right turn smoothly.

[0445] Example prompt sentence:

[0446] Driving data analysis: Cameras detect obstacles 50 meters before an intersection. The system evaluates the timing of right turns at intersections and generates optimal improvement measures.

[0447] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0448] Step 1:

[0449] The autonomous vehicle's cameras and various sensors (LiDAR, radar, etc.) collect surrounding situation data and driving information in real time. The input is information about the surrounding environment and vehicle operation information. The output is the generation and storage of this data. Specifically, the camera captures video data, and the sensors measure information such as distance and angle.

[0450] Step 2:

[0451] The server stores the collected data in cloud storage (Amazon S3). The input is the environmental information and operation information generated in step 1. The output is a notification that the data has been saved to cloud storage and the URL of the destination. Specifically, the server converts the data into an appropriate format and uploads it to cloud storage.

[0452] Step 3:

[0453] The server passes the destination URL of the data stored in cloud storage to the generative AI model. The input is the destination URL obtained in step 2. The output is an analysis request to the generative AI model and a URL. Specifically, the server constructs a URL and sends the analysis request to the generative AI model.

[0454] Step 4:

[0455] The generative AI model retrieves data from cloud storage and analyzes it. The input is the destination URL. The output is the analysis results, specifically the strengths and areas for improvement in driving performance. Specifically, the generative AI model downloads the data and analyzes it using machine learning algorithms.

[0456] Step 5:

[0457] The generative AI model generates feedback based on the analysis results. The input is the analysis results obtained in step 4. The output is a feedback message that includes specific operational instructions and improvement measures. Specifically, the generative AI model evaluates the analysis results and generates appropriate feedback statements.

[0458] Step 6:

[0459] The server sends the generated feedback to the user terminal. The input is the feedback message obtained in step 5. The output is a feedback notification to the user terminal. Specifically, the server converts the feedback message into an appropriate format and sends it to the user terminal.

[0460] Step 7:

[0461] The user device receives the feedback and reflects it in the autonomous vehicle's control system. The input is the feedback notification obtained in step 6. The output is an instruction for the control system to operate. Specifically, the user device analyzes the feedback content and updates the autonomous vehicle's control commands based on it.

[0462] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0463] The online coaching and play analysis system related to the present invention includes a player terminal, cloud storage, a generative AI model, an emotion engine, and a server. The functions and processing flow of each component are described in detail below to implement the present invention.

[0464] Player device functions

[0465] A player device is a device that allows a player to record, save, and upload their play to a server. It also serves as a receiver for the feedback and emotion analysis results provided by the player as analytical results. Examples of such devices include smartphones, tablets, and cameras.

[0466] Server Features

[0467] The server plays a central role in receiving, storing, and analyzing gameplay videos, question data, and emotion data sent from player devices. Furthermore, the server passes the emotion analysis results from the emotion engine to the generative AI model, helping to create emotion-adaptive feedback. This allows for more personalized feedback to be provided to players.

[0468] Cloud storage features

[0469] Cloud storage is a place to temporarily store the received gameplay video. The server saves the gameplay video to cloud storage and passes the URL of the saved location to the generation AI model.

[0470] Generative AI model capabilities

[0471] The generative AI model analyzes gameplay videos stored in cloud storage to identify players' strengths and areas for improvement. Furthermore, the AI ​​model generates optimal feedback for players based on the player's questions and the emotional analysis results provided by the emotion engine.

[0472] Emotion Engine Functions

[0473] The emotion engine is a system that recognizes and analyzes the player's emotional state. It analyzes emotions based on the player's voice and facial expression data, and provides the results to a generative AI model to improve the quality of feedback.

[0474] Specific processing of the program

[0475] The server receives the video of the player's play from the player's device. The user takes a video of the dribble and uploads it through the application. The server saves the video to cloud storage and generates a URL for the destination.

[0476] The server then passes the URL to the generative AI model, asking it to analyze the video. The generative AI model analyzes the video and identifies strengths and areas for improvement in the player's dribbling skills. Based on the analysis results, the generative AI model creates feedback including specific training advice.

[0477] Additionally, an emotion engine analyzes the player's emotional state and provides the results to the generative AI model, determining, for example, whether the player is nervous or stressed.

[0478] The generative AI model generates emotion-adaptive feedback by comprehensively considering the player's play analysis results and emotional state. For example, if the emotion engine detects "disappointment" or "frustration" when a player makes a mistake, the generative AI model will provide feedback including "technical advice and emotional support to maintain confidence."

[0479] The server then sends the generated feedback and emotion analysis results to the player's device, which then receives a notification. The player can then open the application to view the specific feedback and emotion analysis results and use them for practice.

[0480] For example, feedback such as "Your dribbling speed is excellent, but you have difficulty maintaining your balance" is provided. Other examples of emotion-adaptive feedback include "breathing techniques to relieve tension" and "mental training to maintain positive thinking."

[0481] Such a system would enable players to receive more effective and personalized training guidance, and would also take into account their emotional state, potentially improving their overall performance.

[0482] The processing flow will be explained below.

[0483] Step 1:

[0484] A user takes a video of a game using a smartphone or camera and saves it on the device. For example, a user might take a video of a soccer player dribbling.

[0485] Step 2:

[0486] The user launches the application, selects a saved gameplay video, and uploads it to the server. Specifically, the user taps the "Upload Video" button, selects the video file, and sends an upload request.

[0487] Step 3:

[0488] The device sends the uploaded gameplay video to the server, which receives the HTTP POST request and saves the video file in cloud storage.

[0489] Step 4:

[0490] The server generates a URL for the video stored in cloud storage and sends an analysis request to the generative AI model. Specifically, the server passes the video URL to the generative AI model via a REST API.

[0491] Step 5:

[0492] The generative AI model receives the video URL and begins analyzing the video of the game. The generative AI model analyzes the player's movements and techniques in the video to identify strengths and areas for improvement.

[0493] Step 6:

[0494] The user opens the application's question input screen, enters a question about a specific play, and sends it to the server. For example, they might enter, "How can I improve my dribbling accuracy?"

[0495] Step 7:

[0496] The server sends the question to the generative AI model and asks it to also perform analysis based on the question.

[0497] Step 8:

[0498] The generative AI model generates feedback based on the video analysis and user questions, including dribbling strengths and areas for improvement, as well as specific training advice related to the question.

[0499] Step 9:

[0500] The user activates the emotion engine to collect facial and voice data while playing. For example, a camera or microphone records the user's emotion data while playing.

[0501] Step 10:

[0502] The device sends the collected emotion data to the server, which receives the emotion data and passes it to the emotion engine.

[0503] Step 11:

[0504] The emotion engine analyzes the emotion data to identify the player's emotional state, such as "tension" or "lack of concentration."

[0505] Step 12:

[0506] The emotion engine sends the emotion analysis results to the generative AI model, which receives the emotion data and uses it as feedback.

[0507] Step 13:

[0508] The generative AI model generates emotion-adaptive feedback based on emotion analysis, video analysis, and questions, including suggestions such as "breathing techniques to relax" and "mental training to improve concentration."

[0509] Step 14:

[0510] The server sends the generated feedback and the emotion analysis results to the player device, which receives the HTTP response and retrieves the feedback data.

[0511] Step 15:

[0512] The device receives the feedback and sentiment analysis results from the server and displays a notification to the user, for example, a push notification informing the user that "feedback has been received."

[0513] Step 16:

[0514] The user opens the app and sees detailed feedback and sentiment analysis, including specific advice on how to improve dribbling and relax.

[0515] Step 17:

[0516] Based on the feedback and emotional advice provided by the user, the system implements specific actions to improve training and play, such as "training to improve dribbling timing and methods to relax mentally."

[0517] In this way, the present invention provides feedback that takes into account the player's emotional state in addition to technical analysis, enabling more effective and personalized training.

[0518] Example 2

[0519] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0520] Conventional online coaching systems often do not take into account the user's emotional state when analyzing a player's playing video and generating feedback. As a result, optimal training advice is not provided to the player, limiting the effectiveness of training. Furthermore, even if a player inputs a specific question, it is difficult to obtain feedback in real time that responds to that question.

[0521] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for uploading player videos from a player terminal, means for saving the uploaded player videos in cloud storage, means for passing the player videos saved in the cloud storage to a generative AI model for analysis, means for generating feedback based on the analysis results and emotion analysis results of the player videos, and means for transmitting the generated feedback to the player terminal. This makes it possible to provide optimal feedback in real time that takes into account the player's emotional state. In addition, the player can input a question about a specific play, and feedback based on the question can be appropriately generated and provided.

[0522] A "terminal" is an electronic device used by a user, and includes a smartphone, tablet, camera, etc.

[0523] "Video" refers to video data that records a user's play.

[0524] "Cloud storage" is a data storage service accessible via the internet that provides server space for temporary storage of video files.

[0525] "Generative AI model" refers to an artificial intelligence system that analyzes videos stored in cloud storage and generates feedback.

[0526] "Feedback" refers to advice and instruction to improve a player's skills that is created by the generative AI model based on the results of video analysis.

[0527] "Emotion analysis result" refers to information about the user's emotional state obtained as a result of the emotion engine analyzing the user's voice and facial expression data.

[0528] "Emotion-adaptive feedback" refers to the process by which a generative AI model generates feedback optimized according to the user's emotional state based on the results of video analysis and emotion analysis.

[0529] "Server" means the central computer system that receives, stores, and analyzes data transmitted from Player Devices and generates and transmits feedback.

[0530] A "question" refers to an inquiry a user inputs into a generative AI model regarding a particular play.

[0531] "Player" means a person who practices a sport or game and uses the System to analyze video of their play and receive feedback.

[0532] The online coaching and play analysis system related to the present invention includes a player terminal, cloud storage, a generative AI model, an emotion engine, and a server. The functions and processing flow of each component are described in detail below.

[0533] Player device functions

[0534] A player device is a device that allows a player to record, save, and upload gameplay videos to a server. It also receives feedback and emotion analysis results from the player. Examples of such devices include smartphones, tablets, and cameras.

[0535] Server Features

[0536] The server plays a central role in receiving, storing, and analyzing gameplay videos, question data, and emotion data sent from the player's device. The server saves the received gameplay videos in cloud storage and passes the URL of the saved video to the generative AI model. In addition, the server passes the emotion analysis results from the emotion engine to the generative AI model to help create emotion-adaptive feedback. This allows for more personalized feedback to be provided to the player.

[0537] Cloud storage features

[0538] Cloud storage is a place to temporarily store the received gameplay video. The server saves the gameplay video in cloud storage (e.g., Amazon S3 or Google Cloud Storage) and generates a URL for the destination.

[0539] Generative AI model capabilities

[0540] The generative AI model is a system that analyzes gameplay videos stored in cloud storage and identifies players' strengths and areas for improvement. Furthermore, this generative AI model generates optimal feedback for players based on the player's questions and the emotional analysis results provided by the emotion engine. By comprehensively considering the analytical and emotional analysis results, it is possible to generate emotion-adaptive feedback.

[0541] Prompt Sentence Examples

[0542] For example, the prompt sentence to be input to the generative AI model is as follows:

[0543] "Download the video from this URL and perform a skill analysis of the player. Furthermore, create feedback to provide to the player based on the emotion analysis results obtained from the voice and facial expression data."

[0544] Emotion Engine Functions

[0545] The emotion engine is a system that recognizes and analyzes a player's emotional state. By analyzing emotions based on the player's voice and facial expression data and providing the results to a generative AI model, the quality of feedback can be improved. For example, it can determine whether a player is feeling nervous or stressed and reflect that information in the feedback.

[0546] Example

[0547] As a concrete example, consider a scenario in which a user films a dribbling play and uploads the video from their device to a server. The server saves the video file in cloud storage and passes the URL to the generative AI model. The generative AI model analyzes the video and finds that "the dribbling speed is excellent, but maintaining balance is difficult." The emotion engine also detects that the user is nervous. Based on this information, the generative AI model generates emotion-adaptive feedback, including technical advice such as "breathing techniques to relieve tension" and "mental training to maintain a positive mindset."

[0548] This provides players with a system that allows them to not only improve their technical skills, but also manage their emotional state and aim to improve their overall performance.

[0549] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0550] Step 1:

[0551] The user records a video of a dribbling play on the device. The user taps the "Record new video" button, which activates the device's camera. After recording is complete, the user saves the video file. The input is the video file of the play, and the output is the video file saved on the device.

[0552] Step 2:

[0553] A user uploads a video from their device to a server. When the user taps the "Upload" button in the application, the device sends the video file to the server. The input is the video file stored on the device, and the output is the video file uploaded to the server.

[0554] Step 3:

[0555] The server saves the received video in cloud storage and generates a destination URL. The server saves the video file in cloud storage (e.g., Amazon S3) and generates a unique destination URL. The input is the video file received by the server, and the output is the destination URL in cloud storage.

[0556] Step 4:

[0557] The server sends the destination URL to the generative AI model. The server sends the URL to the generative AI model and creates a prompt for analysis. The input is the destination URL in cloud storage, and the output is the prompt sent to the generative AI model.

[0558] Step 5:

[0559] The generative AI model analyzes the video and generates feedback. The generative AI model downloads the video from cloud storage and analyzes it. As a result of the analysis, it extracts the player's technical strengths and areas for improvement. It then generates specific training advice based on this information. The input is the video file downloaded from cloud storage, and the output is feedback based on the analysis results.

[0560] Step 6:

[0561] The emotion engine analyzes the player's emotional state and sends the results to the generative AI model. If voice and facial expression data are also recorded when the player uploads a video, the emotion engine analyzes this data. It identifies the tension or stress the player is feeling and sends the analysis results to the generative AI model. The input is the player's voice and facial expression data, and the output is the emotion analysis results.

[0562] Step 7:

[0563] The generative AI model generates emotion-adaptive feedback based on the results of video analysis and emotion analysis. The generative AI model integrates the results of video analysis with feedback that takes into account the player's emotional state. For example, if a player feels "disappointed" or "frustrated" after making a mistake, it creates feedback that includes technical advice and emotional support. The input is the results of video analysis and emotion analysis, and the output is emotion-adaptive feedback.

[0564] Step 8:

[0565] The server sends the generated feedback and emotion analysis results to the player's device. The server sends the feedback received from the generative AI model to the player's device and performs notification. The input is the generated feedback and emotion analysis results, and the output is the feedback and emotion analysis results sent to the player's device.

[0566] Step 9:

[0567] The user checks the feedback and emotion analysis results on the device and uses them for training. The user opens the application on the device to check the feedback and emotion analysis results, and improves their training and play based on the feedback they receive. The input is the feedback and emotion analysis results sent to the player device, and the output is the improvement results obtained by the user.

[0568] (Application example 2)

[0569] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0570] Previous online coaching and play analysis systems focused primarily on improving players' technical skills, but did not provide feedback or content recommendations based on users' emotions or viewing behavior. This limited the overall user experience. Given this background, there was a need to develop a system that could analyze users' viewing behavior and emotional state to provide more personalized feedback and content recommendations.

[0571] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading player videos from the player terminal, means for saving the uploaded player videos in cloud storage, means for passing the player videos saved in the cloud storage to a generative AI model for analysis, means for generating feedback based on the analysis results of the player videos, means for sending the generated feedback to the player terminal, means for analyzing the user's viewing behavior and emotional state, and means for recommending personalized content based on the analysis results. This makes it possible to not only improve the user's technical skills but also to provide optimal feedback and content recommendations according to the user's viewing behavior and emotional state.

[0572] "Player Device" means an electronic device used by a User to record, store, and upload videos of his / her gameplay.

[0573] "Player video" is video data showing the play and activities of a user that is filmed using a player terminal.

[0574] "Cloud storage" is an online storage system for storing and managing data via the Internet.

[0575] A "generative AI model" is an artificial intelligence system that analyzes users' video data and question data to generate personalized feedback.

[0576] "Analysis results" are the evaluation and comment data obtained after the generative AI model analyzes the video data and question data.

[0577] "Feedback" refers to information that includes evaluations and suggestions for improvement regarding a user's playing or viewing behavior.

[0578] "Viewing behavior" refers to the actions and patterns that users take when watching video content, and is the data related to that.

[0579] "Emotional state" is data that represents the user's emotional and psychological state when viewing.

[0580] "Content recommendation" is a function that presents the most appropriate videos and information based on the user's viewing behavior and emotional state.

[0581] The online coaching and play analysis system related to the present invention includes a player terminal, cloud storage, a generative AI model, an emotion engine, and a server. The functions of each component and the processing flow for implementing the present invention are described in detail below.

[0582] Player device functions

[0583] A player device is an electronic device that allows a user to record, save, and upload their gameplay video to a server. The user also receives feedback and sentiment analysis results. Examples of such devices include smartphones, tablets, and cameras.

[0584] Server Features

[0585] The server plays a central role in receiving, storing, and analyzing gameplay videos, viewing behavior data, and emotional data sent from player devices. Furthermore, the server passes the emotional analysis results from the emotion engine to the generative AI model, helping to create personalized feedback. This allows for more personalized content recommendations to be provided to users.

[0586] Cloud storage features

[0587] Cloud storage is a place to temporarily store received gameplay videos and viewing behavior data. The server saves the gameplay videos in cloud storage and passes the URL of the saved location to the generation AI model.

[0588] Generative AI model capabilities

[0589] The generative AI model analyzes gameplay videos stored in cloud storage and identifies users' strengths and areas for improvement. Furthermore, the AI ​​model generates optimal feedback for users based on their questions and the emotional analysis results provided by the emotion engine. It also includes a function to recommend optimal content based on the user's viewing behavior and emotional state.

[0590] Emotion Engine Functions

[0591] The emotion engine is a system that recognizes and analyzes a user's emotional state. It analyzes emotions based on the user's voice and facial expression data, and provides the results to a generative AI model to improve the quality of feedback and content recommendations.

[0592] Processing flow

[0593] The player's device records a video of the game and uploads it to the server. The server saves the video to cloud storage and generates a URL for the destination. The server then passes the URL to the generated AI model and requests it to analyze the video.

[0594] The generative AI model analyzes the video and identifies the user's skill strengths and areas for improvement. Based on the analysis results, the generative AI model creates feedback including specific training advice. At the same time, the emotion engine analyzes the user's emotional state and provides the analysis results to the generative AI model. The generative AI model then generates emotion-adaptive feedback by comprehensively considering the play analysis results and the user's emotional state.

[0595] Furthermore, it recommends optimal content based on the user's viewing behavior and emotional state. The server sends the generated feedback and content recommendation results to the player device, and the user can open the application and check them.

[0596] Examples and prompts

[0597] Examples:

[0598] Video content: User watching a fitness video

[0599] Sentiment analysis results: "Enjoyed" "Excited"

[0600] Generated feedback: "You seem to be really enjoying this exercise. As a next step, try these more advanced exercises."

[0601] Example prompt sentence:

[0602] You performed sentiment analysis on the fitness video the user was watching, and the result was that the user was enjoying it. Based on this analysis, you want to generate feedback to recommend the next advanced exercise to the user.

[0603] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0604] Step 1:

[0605] The player device takes video of the user's play. The camera function of the player device is used to record the user's play and activities. The input is actual play video, and the output is digital video data.

[0606] Step 2:

[0607] The gameplay video captured by the player's device is uploaded to the server. The video data is sent to the server via the Internet. The input is the gameplay video in digital format, and the output is a video file stored on the server.

[0608] Step 3:

[0609] The server saves the uploaded gameplay video to cloud storage. Using the file saving API for cloud storage, the video data is saved to the cloud. The input is the video file on the server, and the output is the URL of the cloud storage destination.

[0610] Step 4:

[0611] The server passes the URL of the video stored in cloud storage to the generative AI model. The URL of the video is passed to the API of the generative AI model and an analysis request is made. The input is the URL of the cloud storage destination, and the output is the analysis result of the generative AI model.

[0612] Step 5:

[0613] The generative AI model analyzes the gameplay video stored in cloud storage. The generative AI model's algorithm analyzes the video data and identifies the user's skills, strengths, and areas for improvement. The input is the cloud storage destination URL, and the output is the gameplay analysis results.

[0614] Step 6:

[0615] The generative AI model generates feedback based on the results of play analysis. It also takes into account the emotional analysis results provided by the emotion engine to create personalized feedback. The input is the results of play analysis and emotional analysis, and the output is feedback that includes specific training advice.

[0616] Step 7:

[0617] The server sends the feedback received from the generative AI model to the player device. The feedback data is notified to the player device via the Internet. The input is the generated feedback, and the output is the feedback displayed on the user's player device.

[0618] Step 8:

[0619] The player device receives the feedback and the user confirms it. The application is opened and the feedback content is displayed. The input is the feedback data sent from the server, and the output is the feedback screen that the user can see.

[0620] Step 9:

[0621] The emotion engine analyzes the user's viewing behavior and emotional state. It analyzes the video being watched and the user's reactions to identify the emotional state. The input is the user's viewing data and voice / facial expression data, and the output is the emotion analysis results.

[0622] Step 10:

[0623] The generative AI model recommends personalized content based on the results of sentiment analysis. It selects the most appropriate content based on the user's emotional state and viewing behavior, and creates a recommendation list. The input is the results of sentiment analysis and viewing behavior data, and the output is a content recommendation list.

[0624] Step 11:

[0625] The server sends the generated content recommendation list to the player terminal. The recommendation list is notified to the player terminal via the Internet. The input is the generated content recommendation list, and the output is the recommended content displayed on the user's player terminal.

[0626] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0627] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0628] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0629] [Third embodiment]

[0630] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0631] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0632] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0633] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0634] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0635] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0636] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0637] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0638] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0639] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0640] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0641] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0642] The online coaching and play analysis system of the present invention includes a player terminal, cloud storage, a generative AI model, and a server. The functions and processing flow of each component are shown below to implement the present invention.

[0643] Player device functions

[0644] A player device is a device that is primarily used by a player to record, save, and upload their play to a server. It is also the device through which the player receives feedback provided as analytical results. Specifically, a player device may be a smartphone, tablet, or camera.

[0645] Server Features

[0646] The server plays a central role in receiving, storing, and analyzing gameplay videos and question data sent from the player's device. The server provides the URL of the destination for the received video to the AI ​​model, which then analyzes the video. It also has the function of generating feedback based on the analysis results and sending it to the player's device.

[0647] Cloud storage features

[0648] Cloud storage is a place to temporarily store the received gameplay video. The server saves the video in cloud storage and passes the URL of the saved video to the generation AI model.

[0649] Generative AI model capabilities

[0650] The generative AI model is a system that analyzes gameplay videos stored in cloud storage to identify players' strengths and areas for improvement. It has the ability to generate specific feedback based on the analysis results. Furthermore, if the player inputs specific questions, it can also generate feedback based on those questions.

[0651] Specific processing of the program

[0652] The server receives the video of the player's play. For example, the player takes a video of their dribbling and uploads it through the application. The server saves the video in cloud storage and generates a URL for the destination.

[0653] The server then passes the URL to the generative AI model, asking it to analyze the video. The generative AI model analyzes the video and identifies strengths and areas for improvement in the player's dribbling skills. Based on the analysis results, the generative AI model creates feedback including specific training advice.

[0654] The server then sends the generated feedback to the player terminal, and the player terminal receives a notification, allowing the player to open the application, check the specific feedback content, and use it to improve their practice.

[0655] For example, the feedback provided may include, "Your dribbling speed is excellent, but your timing is off, making it difficult to control the ball." The advice also includes specific drills and practice menus to improve your timing. Based on this, players can then practice to improve their own play.

[0656] In this way, the system of the present invention allows players to receive efficient and accurate feedback and specific advice on how to improve their play without the need for a coach.

[0657] The processing flow will be explained below.

[0658] Step 1:

[0659] A user takes a video of a game using a smartphone or camera and saves it on the device. For example, a user might take a video of a soccer player dribbling.

[0660] Step 2:

[0661] The user launches the application, selects a saved gameplay video, and uploads it to the server. Specifically, the user taps the "Upload Video" button, selects the video file, and sends an upload request.

[0662] Step 3:

[0663] The device sends the uploaded gameplay video to the server, which receives the HTTP POST request and saves the video file in cloud storage.

[0664] Step 4:

[0665] The server generates a URL for the video stored in cloud storage and sends an analysis request to the generative AI model. Specifically, the server passes the video URL to the generative AI model via a REST API.

[0666] Step 5:

[0667] The generative AI model receives the video URL and begins analyzing the video of the game. The generative AI model analyzes the player's movements and techniques in the video to identify strengths and areas for improvement.

[0668] Step 6:

[0669] A generative AI model uses the analysis to generate feedback, including the player's technical strengths and areas for improvement, as well as specific training advice.

[0670] Step 7:

[0671] The server receives the feedback obtained from the generative AI model and sends it to the player's device. The server returns the feedback data to the device via an HTTP response.

[0672] Step 8:

[0673] The device receives the feedback from the server and displays a notification to the user, for example, a push notification that says "Your feedback has been received."

[0674] Step 9:

[0675] The user opens the application and checks the received feedback. Specifically, the user can view the feedback on the application screen and identify strengths and areas for improvement.

[0676] Step 10:

[0677] Based on the feedback provided by the user, the system implements specific advice for training and improving play, such as "doing specific drills to improve dribbling timing."

[0678] Example 1

[0679] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0680] Conventional play analysis systems have issues such as low analysis accuracy, inability to provide instant feedback, and inability to obtain analysis results that satisfy users. Furthermore, when players themselves want feedback on specific questions, they often lack the ability to obtain specific advice. To address these issues, a system that improves the accuracy of gameplay video analysis and provides fast, specific feedback is needed.

[0681] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0682] In this invention, the server includes a means for uploading gameplay videos from the player terminal, a means for saving the uploaded gameplay videos to cloud storage, and a means for generating a destination URL for the gameplay videos saved in cloud storage and passing it to the generation AI model. This improves the accuracy of gameplay video analysis and makes it possible to provide quick and specific feedback.

[0683] "Player Device" means a device used by a player to record, store, and upload their play to the server, such as a smartphone, tablet, or camera.

[0684] "Cloud storage" refers to a group of remote servers that store data via the Internet. In particular, in the present invention, it is used to temporarily store gameplay videos.

[0685] The "generative AI model" is an artificial intelligence model that analyzes gameplay videos stored in cloud storage and identifies players' strengths and areas for improvement. This model is responsible for generating specific feedback based on the results of gameplay analysis.

[0686] "Feedback" refers to specific advice and information on areas for improvement provided by the generative AI model based on the analysis of the gameplay video. The goal is to improve the player's skills.

[0687] The online coaching and play analysis system of the present invention includes a player terminal, cloud storage, a generative AI model, and a server. The components and processing flow of this system will be described in detail below.

[0688] Player device functions

[0689] A player device is a device used by a player to record, save, and upload their play to a server. Specifically, this includes smartphones, tablets, and cameras. It is also on the player device that the player receives feedback provided as analysis results. For example, a user may film a soccer dribble using a smartphone and upload the video through an application.

[0690] Server Features

[0691] The server plays a central role in receiving gameplay videos and question data sent from the player's device and storing them in cloud storage. Specifically, the server stores the received videos in cloud storage and generates a URL for the destination. This URL is then provided to the generative AI model, which requests it to analyze the video. The server also has the function of generating feedback based on the analysis results returned by the generative AI model and sending it to the player's device.

[0692] Cloud storage features

[0693] Cloud storage is a group of remote servers that temporarily store the received gameplay videos. The server saves the videos in cloud storage and passes the URL of the saved video to the generated AI model. Specifically, a common cloud storage service (e.g., Amazon S3) is used.

[0694] Generative AI model capabilities

[0695] The generative AI model is an artificial intelligence system that analyzes gameplay videos stored in cloud storage and identifies players' strengths and areas for improvement. This model has the ability to generate specific feedback based on the analysis results. Furthermore, if a player inputs a specific question, it can generate appropriate feedback based on that question.

[0696] Feedback example

[0697] An example of specific feedback might be, "Your dribbling speed is excellent, but your timing is off, making it difficult to control the ball." The advice also includes specific drills and practice exercises to improve your timing. This gives players a concrete action plan to improve their play.

[0698] Prompt Sentence Examples

[0699] The following prompts can be used as input to a generative AI model:

[0700] "Analyze this video for dribbling skills, identify player strengths and areas for improvement, and generate specific training advice."

[0701] The system allows players to receive efficient and accurate feedback quickly, giving them specific advice on how to improve their play without the need for a coach.

[0702] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0703] Step 1:

[0704] The user launches the application on the player's device and records a video of their gameplay. The recorded video is temporarily saved on the device. Then, when the user taps the "Upload" button in the application, the device sends the video file to the server. The input to this process is the recorded video of the gameplay, and the output is the transmission of the video file to the server. Specific operations include taking a video using the camera app, saving it as a video file, and uploading the video file from the application.

[0705] Step 2:

[0706] The server receives gameplay video data sent from the player's device. The received video is temporarily stored on the server. The server then uploads the video to cloud storage and generates a destination URL for the video. The input to this process is the gameplay video data sent from the device, and the output is the URL for the video stored in cloud storage. Specific operations include saving the video file on the server, uploading it to cloud storage, and generating a destination URL.

[0707] Step 3:

[0708] The server creates a request to provide the generative AI model with the destination URL. This request includes a prompt for the generative AI model (e.g., "Please analyze the dribbling skills in this video, identify the player's strengths and areas for improvement, and generate specific training advice."). The server sends this request to the generative AI model. The inputs to this process are the URL of the video stored in cloud storage and the prompt, and the output is a request sent to the generative AI model.

[0709] Step 4:

[0710] The generative AI model retrieves the video data from cloud storage using the received URL. The AI ​​model then analyzes the video data and evaluates the player's movements. This evaluation includes detailed analysis of the player's dribbling skills, ball control, speed, etc. The generative AI model then generates specific feedback based on the analysis results. The input of this process is the video data retrieved from cloud storage, and the output is specific feedback.

[0711] Step 5:

[0712] The server receives feedback sent from the generative AI model. The received feedback is sent to the player device. When the player device is ready to receive feedback, the server sends a notification. The input of this process is feedback from the generative AI model, and the output is sending feedback and a notification to the player device.

[0713] Step 6:

[0714] The player device receives the feedback sent from the server. The user opens the application and checks the received feedback. For example, the application may display a message such as "Your dribbling speed is excellent, but your timing is off, making it difficult to control the ball," and provide specific drills and practice menus to improve your timing. The input to this process is the feedback sent from the server, and the output is the feedback displayed to the user. The user can then use this information to create a practice plan and work to improve their skills.

[0715] (Application example 1)

[0716] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0717] Real-time data analysis and feedback are essential to improving the driving performance of autonomous vehicles. However, conventional systems lack real-time capabilities and have difficulty providing quick and accurate improvement measures. This has prevented improvements in safety and efficiency.

[0718] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0719] In this invention, the server includes means for uploading data from a user terminal, means for storing the uploaded data in a data storage device, means for passing the data stored in the data storage device to a generative AI model for analysis, means for generating feedback based on the data analysis results, and means for transmitting the generated feedback to the user terminal, thereby enabling the driving data of an autonomous vehicle to be analyzed in real time and providing prompt and accurate feedback based on the analysis results.

[0720] "User Device" means a device used by a User to collect, store, or transmit data, including a smartphone, tablet, camera, etc.

[0721] A "data storage device" is a storage system for temporarily or long-term storage of collected data, and includes cloud storage and on-premise storage servers.

[0722] A "generative AI model" is an algorithm that analyzes collected data and generates feedback or advice tailored to a specific purpose, and includes models that use machine learning and deep learning.

[0723] "Analysis results" are the results or evaluations that a generative AI model derives from input data, including strengths and areas for improvement regarding specific performance.

[0724] "Feedback" refers to specific instructions or advice provided to users based on the analysis results, including specific examples and improvement measures.

[0725] The driving analysis system for autonomous vehicles according to the present invention is a real-time driving analysis system that collects and analyzes data obtained from cameras and sensors on autonomous vehicles, identifies strengths and areas for improvement in driving performance, and provides specific instructions and feedback in real time based on the analysis results.

[0726] System Components and Functions

[0727] 1. User Device:

[0728] User terminals are devices used to collect, store, and transmit data. These include cameras and various sensors (such as LiDAR and radar) installed in autonomous vehicles, which collect surrounding situation data and driving information in real time.

[0729] 2. Data Storage Device:

[0730] A data storage device is a storage system for temporarily or long-term storage of collected data. Cloud storage such as Amazon S3 is an example of this, allowing data to be stored and quickly accessed.

[0731] 3. Generative AI Model:

[0732] Generative AI models are algorithms that analyze collected data and generate feedback and advice tailored to specific objectives, including machine learning and deep learning models like OpenAI GPT-4. Based on the analysis results, they identify strengths and areas for improvement regarding the driving performance of autonomous vehicles.

[0733] 4. Server:

[0734] The server uploads data from the user's device and stores it in a data storage device. Next, it requests the generated AI model to analyze the data by passing the URL of the destination. It also generates feedback based on the analysis results from the generated AI model and sends it to the user's device. Examples of such servers include AWS EC2.

[0735] Specific processing of the program

[0736] 1. Data Collection:

[0737] Cameras and various sensors (such as LiDAR and radar) installed in autonomous vehicles collect data on the surrounding environment and driving information, which is then uploaded to cloud storage in real time.

[0738] 2. Data Analysis:

[0739] The server saves the uploaded data to cloud storage (Amazon S3) and passes the URL of the saved data to an OpenAI GPT-4-based generative AI model, which analyzes the driving data and identifies strengths and areas for improvement in vehicle performance.

[0740] 3. Feedback Generation:

[0741] Based on the analysis results, the generative AI model generates feedback including specific instructions and improvement measures, such as "When turning right, it is recommended to turn on the right turn signal 50 meters before the intersection."

[0742] 4. Real-time control:

[0743] The server sends the generated feedback to the user's terminal, and the autonomous vehicle uses the feedback to improve its driving performance.

[0744] Examples and prompts

[0745] Examples:

[0746] When an autonomous vehicle makes a right turn at an intersection, it detects obstacles and uploads the information to cloud storage in real time. A generative AI model analyzes this data and provides feedback on how to improve the timing of the right turn. Based on this feedback, the vehicle completes the right turn smoothly.

[0747] Example prompt sentence:

[0748] Driving data analysis: Cameras detect obstacles 50 meters before an intersection. The system evaluates the timing of right turns at intersections and generates optimal improvement measures.

[0749] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0750] Step 1:

[0751] The autonomous vehicle's cameras and various sensors (LiDAR, radar, etc.) collect surrounding situation data and driving information in real time. The input is information about the surrounding environment and vehicle operation information. The output is the generation and storage of this data. Specifically, the camera captures video data, and the sensors measure information such as distance and angle.

[0752] Step 2:

[0753] The server stores the collected data in cloud storage (Amazon S3). The input is the environmental information and operation information generated in step 1. The output is a notification that the data has been saved to cloud storage and the URL of the destination. Specifically, the server converts the data into an appropriate format and uploads it to cloud storage.

[0754] Step 3:

[0755] The server passes the destination URL of the data stored in cloud storage to the generative AI model. The input is the destination URL obtained in step 2. The output is an analysis request to the generative AI model and a URL. Specifically, the server constructs a URL and sends the analysis request to the generative AI model.

[0756] Step 4:

[0757] The generative AI model retrieves data from cloud storage and analyzes it. The input is the destination URL. The output is the analysis results, specifically the strengths and areas for improvement in driving performance. Specifically, the generative AI model downloads the data and analyzes it using machine learning algorithms.

[0758] Step 5:

[0759] The generative AI model generates feedback based on the analysis results. The input is the analysis results obtained in step 4. The output is a feedback message that includes specific operational instructions and improvement measures. Specifically, the generative AI model evaluates the analysis results and generates appropriate feedback statements.

[0760] Step 6:

[0761] The server sends the generated feedback to the user terminal. The input is the feedback message obtained in step 5. The output is a feedback notification to the user terminal. Specifically, the server converts the feedback message into an appropriate format and sends it to the user terminal.

[0762] Step 7:

[0763] The user device receives the feedback and reflects it in the autonomous vehicle's control system. The input is the feedback notification obtained in step 6. The output is an instruction for the control system to operate. Specifically, the user device analyzes the feedback content and updates the autonomous vehicle's control commands based on it.

[0764] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0765] The online coaching and play analysis system related to the present invention includes a player terminal, cloud storage, a generative AI model, an emotion engine, and a server. The functions and processing flow of each component are described in detail below to implement the present invention.

[0766] Player device functions

[0767] A player device is a device that allows a player to record, save, and upload their play to a server. It also serves as a receiver for the feedback and emotion analysis results provided by the player as analytical results. Examples of such devices include smartphones, tablets, and cameras.

[0768] Server Features

[0769] The server plays a central role in receiving, storing, and analyzing gameplay videos, question data, and emotion data sent from player devices. Furthermore, the server passes the emotion analysis results from the emotion engine to the generative AI model, helping to create emotion-adaptive feedback. This allows for more personalized feedback to be provided to players.

[0770] Cloud storage features

[0771] Cloud storage is a place to temporarily store the received gameplay video. The server saves the gameplay video to cloud storage and passes the URL of the saved location to the generation AI model.

[0772] Generative AI model capabilities

[0773] The generative AI model analyzes gameplay videos stored in cloud storage to identify players' strengths and areas for improvement. Furthermore, the AI ​​model generates optimal feedback for players based on the player's questions and the emotional analysis results provided by the emotion engine.

[0774] Emotion Engine Functions

[0775] The emotion engine is a system that recognizes and analyzes the player's emotional state. It analyzes emotions based on the player's voice and facial expression data, and provides the results to a generative AI model to improve the quality of feedback.

[0776] Specific processing of the program

[0777] The server receives the video of the player's play from the player's device. The user takes a video of the dribble and uploads it through the application. The server saves the video to cloud storage and generates a URL for the destination.

[0778] The server then passes the URL to the generative AI model, asking it to analyze the video. The generative AI model analyzes the video and identifies strengths and areas for improvement in the player's dribbling skills. Based on the analysis results, the generative AI model creates feedback including specific training advice.

[0779] Additionally, an emotion engine analyzes the player's emotional state and provides the results to the generative AI model, determining, for example, whether the player is nervous or stressed.

[0780] The generative AI model generates emotion-adaptive feedback by comprehensively considering the player's play analysis results and emotional state. For example, if the emotion engine detects "disappointment" or "frustration" when a player makes a mistake, the generative AI model will provide feedback including "technical advice and emotional support to maintain confidence."

[0781] The server then sends the generated feedback and emotion analysis results to the player's device, which then receives a notification. The player can then open the application to view the specific feedback and emotion analysis results and use them for practice.

[0782] For example, feedback such as "Your dribbling speed is excellent, but you have difficulty maintaining your balance" is provided. Other examples of emotion-adaptive feedback include "breathing techniques to relieve tension" and "mental training to maintain positive thinking."

[0783] Such a system would enable players to receive more effective and personalized training guidance, and would also take into account their emotional state, potentially improving their overall performance.

[0784] The processing flow will be explained below.

[0785] Step 1:

[0786] A user takes a video of a game using a smartphone or camera and saves it on the device. For example, a user might take a video of a soccer player dribbling.

[0787] Step 2:

[0788] The user launches the application, selects a saved gameplay video, and uploads it to the server. Specifically, the user taps the "Upload Video" button, selects the video file, and sends an upload request.

[0789] Step 3:

[0790] The device sends the uploaded gameplay video to the server, which receives the HTTP POST request and saves the video file in cloud storage.

[0791] Step 4:

[0792] The server generates a URL for the video stored in cloud storage and sends an analysis request to the generative AI model. Specifically, the server passes the video URL to the generative AI model via a REST API.

[0793] Step 5:

[0794] The generative AI model receives the video URL and begins analyzing the video of the game. The generative AI model analyzes the player's movements and techniques in the video to identify strengths and areas for improvement.

[0795] Step 6:

[0796] The user opens the application's question input screen, enters a question about a specific play, and sends it to the server. For example, they might enter, "How can I improve my dribbling accuracy?"

[0797] Step 7:

[0798] The server sends the question to the generative AI model and asks it to also perform analysis based on the question.

[0799] Step 8:

[0800] The generative AI model generates feedback based on the video analysis and user questions, including dribbling strengths and areas for improvement, as well as specific training advice related to the question.

[0801] Step 9:

[0802] The user activates the emotion engine to collect facial and voice data while playing. For example, a camera or microphone records the user's emotion data while playing.

[0803] Step 10:

[0804] The device sends the collected emotion data to the server, which receives the emotion data and passes it to the emotion engine.

[0805] Step 11:

[0806] The emotion engine analyzes the emotion data to identify the player's emotional state, such as "tension" or "lack of concentration."

[0807] Step 12:

[0808] The emotion engine sends the emotion analysis results to the generative AI model, which receives the emotion data and uses it as feedback.

[0809] Step 13:

[0810] The generative AI model generates emotion-adaptive feedback based on emotion analysis, video analysis, and questions, including suggestions such as "breathing techniques to relax" and "mental training to improve concentration."

[0811] Step 14:

[0812] The server sends the generated feedback and the emotion analysis results to the player device, which receives the HTTP response and retrieves the feedback data.

[0813] Step 15:

[0814] The device receives the feedback and sentiment analysis results from the server and displays a notification to the user, for example, a push notification informing the user that "feedback has been received."

[0815] Step 16:

[0816] The user opens the app and sees detailed feedback and sentiment analysis, including specific advice on how to improve dribbling and relax.

[0817] Step 17:

[0818] Based on the feedback and emotional advice provided by the user, the system implements specific actions to improve training and play, such as "training to improve dribbling timing and methods to relax mentally."

[0819] In this way, the present invention provides feedback that takes into account the player's emotional state in addition to technical analysis, enabling more effective and personalized training.

[0820] Example 2

[0821] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0822] Conventional online coaching systems often do not take into account the user's emotional state when analyzing a player's playing video and generating feedback. As a result, optimal training advice is not provided to the player, limiting the effectiveness of training. Furthermore, even if a player inputs a specific question, it is difficult to obtain feedback in real time that responds to that question.

[0823] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for uploading player videos from a player terminal, means for saving the uploaded player videos in cloud storage, means for passing the player videos saved in the cloud storage to a generative AI model for analysis, means for generating feedback based on the analysis results and emotion analysis results of the player videos, and means for transmitting the generated feedback to the player terminal. This makes it possible to provide optimal feedback in real time that takes into account the player's emotional state. In addition, the player can input a question about a specific play, and feedback based on the question can be appropriately generated and provided.

[0824] A "terminal" is an electronic device used by a user, and includes a smartphone, tablet, camera, etc.

[0825] "Video" refers to video data that records a user's play.

[0826] "Cloud storage" is a data storage service accessible via the internet that provides server space for temporary storage of video files.

[0827] "Generative AI model" refers to an artificial intelligence system that analyzes videos stored in cloud storage and generates feedback.

[0828] "Feedback" refers to advice and instruction to improve a player's skills that is created by the generative AI model based on the results of video analysis.

[0829] "Emotion analysis result" refers to information about the user's emotional state obtained as a result of the emotion engine analyzing the user's voice and facial expression data.

[0830] "Emotion-adaptive feedback" refers to the process by which a generative AI model generates feedback optimized according to the user's emotional state based on the results of video analysis and emotion analysis.

[0831] "Server" means the central computer system that receives, stores, and analyzes data transmitted from Player Devices and generates and transmits feedback.

[0832] A "question" refers to an inquiry a user inputs into a generative AI model regarding a particular play.

[0833] "Player" means a person who practices a sport or game and uses the System to analyze video of their play and receive feedback.

[0834] The online coaching and play analysis system related to the present invention includes a player terminal, cloud storage, a generative AI model, an emotion engine, and a server. The functions and processing flow of each component are described in detail below.

[0835] Player device functions

[0836] A player device is a device that allows a player to record, save, and upload gameplay videos to a server. It also receives feedback and emotion analysis results from the player. Examples of such devices include smartphones, tablets, and cameras.

[0837] Server Features

[0838] The server plays a central role in receiving, storing, and analyzing gameplay videos, question data, and emotion data sent from the player's device. The server saves the received gameplay videos in cloud storage and passes the URL of the saved video to the generative AI model. In addition, the server passes the emotion analysis results from the emotion engine to the generative AI model to help create emotion-adaptive feedback. This allows for more personalized feedback to be provided to the player.

[0839] Cloud storage features

[0840] Cloud storage is a place to temporarily store the received gameplay video. The server saves the gameplay video in cloud storage (e.g., Amazon S3 or Google Cloud Storage) and generates a URL for the destination.

[0841] Generative AI model capabilities

[0842] The generative AI model is a system that analyzes gameplay videos stored in cloud storage and identifies players' strengths and areas for improvement. Furthermore, this generative AI model generates optimal feedback for players based on the player's questions and the emotional analysis results provided by the emotion engine. By comprehensively considering the analytical and emotional analysis results, it is possible to generate emotion-adaptive feedback.

[0843] Prompt Sentence Examples

[0844] For example, the prompt sentence to be input to the generative AI model is as follows:

[0845] "Download the video from this URL and perform a skill analysis of the player. Furthermore, create feedback to provide to the player based on the emotion analysis results obtained from the voice and facial expression data."

[0846] Emotion Engine Functions

[0847] The emotion engine is a system that recognizes and analyzes a player's emotional state. By analyzing emotions based on the player's voice and facial expression data and providing the results to a generative AI model, the quality of feedback can be improved. For example, it can determine whether a player is feeling nervous or stressed and reflect that information in the feedback.

[0848] Example

[0849] As a concrete example, consider a scenario in which a user films a dribbling play and uploads the video from their device to a server. The server saves the video file in cloud storage and passes the URL to the generative AI model. The generative AI model analyzes the video and finds that "the dribbling speed is excellent, but maintaining balance is difficult." The emotion engine also detects that the user is nervous. Based on this information, the generative AI model generates emotion-adaptive feedback, including technical advice such as "breathing techniques to relieve tension" and "mental training to maintain a positive mindset."

[0850] This provides players with a system that allows them to not only improve their technical skills, but also manage their emotional state and aim to improve their overall performance.

[0851] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0852] Step 1:

[0853] The user records a video of a dribbling play on the device. The user taps the "Record new video" button, which activates the device's camera. After recording is complete, the user saves the video file. The input is the video file of the play, and the output is the video file saved on the device.

[0854] Step 2:

[0855] A user uploads a video from their device to a server. When the user taps the "Upload" button in the application, the device sends the video file to the server. The input is the video file stored on the device, and the output is the video file uploaded to the server.

[0856] Step 3:

[0857] The server saves the received video in cloud storage and generates a destination URL. The server saves the video file in cloud storage (e.g., Amazon S3) and generates a unique destination URL. The input is the video file received by the server, and the output is the destination URL in cloud storage.

[0858] Step 4:

[0859] The server sends the destination URL to the generative AI model. The server sends the URL to the generative AI model and creates a prompt for analysis. The input is the destination URL in cloud storage, and the output is the prompt sent to the generative AI model.

[0860] Step 5:

[0861] The generative AI model analyzes the video and generates feedback. The generative AI model downloads the video from cloud storage and analyzes it. As a result of the analysis, it extracts the player's technical strengths and areas for improvement. It then generates specific training advice based on this information. The input is the video file downloaded from cloud storage, and the output is feedback based on the analysis results.

[0862] Step 6:

[0863] The emotion engine analyzes the player's emotional state and sends the results to the generative AI model. If voice and facial expression data are also recorded when the player uploads a video, the emotion engine analyzes this data. It identifies the tension or stress the player is feeling and sends the analysis results to the generative AI model. The input is the player's voice and facial expression data, and the output is the emotion analysis results.

[0864] Step 7:

[0865] The generative AI model generates emotion-adaptive feedback based on the results of video analysis and emotion analysis. The generative AI model integrates the results of video analysis with feedback that takes into account the player's emotional state. For example, if a player feels "disappointed" or "frustrated" after making a mistake, it creates feedback that includes technical advice and emotional support. The input is the results of video analysis and emotion analysis, and the output is emotion-adaptive feedback.

[0866] Step 8:

[0867] The server sends the generated feedback and emotion analysis results to the player's device. The server sends the feedback received from the generative AI model to the player's device and performs notification. The input is the generated feedback and emotion analysis results, and the output is the feedback and emotion analysis results sent to the player's device.

[0868] Step 9:

[0869] The user checks the feedback and emotion analysis results on the device and uses them for training. The user opens the application on the device to check the feedback and emotion analysis results, and improves their training and play based on the feedback they receive. The input is the feedback and emotion analysis results sent to the player device, and the output is the improvement results obtained by the user.

[0870] (Application example 2)

[0871] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0872] Previous online coaching and play analysis systems focused primarily on improving players' technical skills, but did not provide feedback or content recommendations based on users' emotions or viewing behavior. This limited the overall user experience. Given this background, there was a need to develop a system that could analyze users' viewing behavior and emotional state to provide more personalized feedback and content recommendations.

[0873] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading player videos from the player terminal, means for saving the uploaded player videos in cloud storage, means for passing the player videos saved in the cloud storage to a generative AI model for analysis, means for generating feedback based on the analysis results of the player videos, means for sending the generated feedback to the player terminal, means for analyzing the user's viewing behavior and emotional state, and means for recommending personalized content based on the analysis results. This makes it possible to not only improve the user's technical skills but also to provide optimal feedback and content recommendations according to the user's viewing behavior and emotional state.

[0874] "Player Device" means an electronic device used by a User to record, store, and upload videos of his / her gameplay.

[0875] "Player video" is video data showing the play and activities of a user that is filmed using a player terminal.

[0876] "Cloud storage" is an online storage system for storing and managing data via the Internet.

[0877] A "generative AI model" is an artificial intelligence system that analyzes users' video data and question data to generate personalized feedback.

[0878] "Analysis results" are the evaluation and comment data obtained after the generative AI model analyzes the video data and question data.

[0879] "Feedback" refers to information that includes evaluations and suggestions for improvement regarding a user's playing or viewing behavior.

[0880] "Viewing behavior" refers to the actions and patterns that users take when watching video content, and is the data related to that.

[0881] "Emotional state" is data that represents the user's emotional and psychological state when viewing.

[0882] "Content recommendation" is a function that presents the most appropriate videos and information based on the user's viewing behavior and emotional state.

[0883] The online coaching and play analysis system related to the present invention includes a player terminal, cloud storage, a generative AI model, an emotion engine, and a server. The functions of each component and the processing flow for implementing the present invention are described in detail below.

[0884] Player device functions

[0885] A player device is an electronic device that allows a user to record, save, and upload their gameplay video to a server. The user also receives feedback and sentiment analysis results. Examples of such devices include smartphones, tablets, and cameras.

[0886] Server Features

[0887] The server plays a central role in receiving, storing, and analyzing gameplay videos, viewing behavior data, and emotional data sent from player devices. Furthermore, the server passes the emotional analysis results from the emotion engine to the generative AI model, helping to create personalized feedback. This allows for more personalized content recommendations to be provided to users.

[0888] Cloud storage features

[0889] Cloud storage is a place to temporarily store received gameplay videos and viewing behavior data. The server saves the gameplay videos in cloud storage and passes the URL of the saved location to the generation AI model.

[0890] Generative AI model capabilities

[0891] The generative AI model analyzes gameplay videos stored in cloud storage and identifies users' strengths and areas for improvement. Furthermore, the AI ​​model generates optimal feedback for users based on their questions and the emotional analysis results provided by the emotion engine. It also includes a function to recommend optimal content based on the user's viewing behavior and emotional state.

[0892] Emotion Engine Functions

[0893] The emotion engine is a system that recognizes and analyzes a user's emotional state. It analyzes emotions based on the user's voice and facial expression data, and provides the results to a generative AI model to improve the quality of feedback and content recommendations.

[0894] Processing flow

[0895] The player's device records a video of the game and uploads it to the server. The server saves the video to cloud storage and generates a URL for the destination. The server then passes the URL to the generated AI model and requests it to analyze the video.

[0896] The generative AI model analyzes the video and identifies the user's skill strengths and areas for improvement. Based on the analysis results, the generative AI model creates feedback including specific training advice. At the same time, the emotion engine analyzes the user's emotional state and provides the analysis results to the generative AI model. The generative AI model then generates emotion-adaptive feedback by comprehensively considering the play analysis results and the user's emotional state.

[0897] Furthermore, it recommends optimal content based on the user's viewing behavior and emotional state. The server sends the generated feedback and content recommendation results to the player device, and the user can open the application and check them.

[0898] Examples and prompts

[0899] Examples:

[0900] Video content: User watching a fitness video

[0901] Sentiment analysis results: "Enjoyed" "Excited"

[0902] Generated feedback: "You seem to be really enjoying this exercise. As a next step, try these more advanced exercises."

[0903] Example prompt sentence:

[0904] You performed sentiment analysis on the fitness video the user was watching, and the result was that the user was enjoying it. Based on this analysis, you want to generate feedback to recommend the next advanced exercise to the user.

[0905] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0906] Step 1:

[0907] The player device takes video of the user's play. The camera function of the player device is used to record the user's play and activities. The input is actual play video, and the output is digital video data.

[0908] Step 2:

[0909] The gameplay video captured by the player's device is uploaded to the server. The video data is sent to the server via the Internet. The input is the gameplay video in digital format, and the output is a video file stored on the server.

[0910] Step 3:

[0911] The server saves the uploaded gameplay video to cloud storage. Using the file saving API for cloud storage, the video data is saved to the cloud. The input is the video file on the server, and the output is the URL of the cloud storage destination.

[0912] Step 4:

[0913] The server passes the URL of the video stored in cloud storage to the generative AI model. The URL of the video is passed to the API of the generative AI model and an analysis request is made. The input is the URL of the cloud storage destination, and the output is the analysis result of the generative AI model.

[0914] Step 5:

[0915] The generative AI model analyzes the gameplay video stored in cloud storage. The generative AI model's algorithm analyzes the video data and identifies the user's skills, strengths, and areas for improvement. The input is the cloud storage destination URL, and the output is the gameplay analysis results.

[0916] Step 6:

[0917] The generative AI model generates feedback based on the results of play analysis. It also takes into account the emotional analysis results provided by the emotion engine to create personalized feedback. The input is the results of play analysis and emotional analysis, and the output is feedback that includes specific training advice.

[0918] Step 7:

[0919] The server sends the feedback received from the generative AI model to the player device. The feedback data is notified to the player device via the Internet. The input is the generated feedback, and the output is the feedback displayed on the user's player device.

[0920] Step 8:

[0921] The player device receives the feedback and the user confirms it. The application is opened and the feedback content is displayed. The input is the feedback data sent from the server, and the output is the feedback screen that the user can see.

[0922] Step 9:

[0923] The emotion engine analyzes the user's viewing behavior and emotional state. It analyzes the video being watched and the user's reactions to identify the emotional state. The input is the user's viewing data and voice / facial expression data, and the output is the emotion analysis results.

[0924] Step 10:

[0925] The generative AI model recommends personalized content based on the results of sentiment analysis. It selects the most appropriate content based on the user's emotional state and viewing behavior, and creates a recommendation list. The input is the results of sentiment analysis and viewing behavior data, and the output is a content recommendation list.

[0926] Step 11:

[0927] The server sends the generated content recommendation list to the player terminal. The recommendation list is notified to the player terminal via the Internet. The input is the generated content recommendation list, and the output is the recommended content displayed on the user's player terminal.

[0928] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0929] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0930] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0931] [Fourth embodiment]

[0932] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0933] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0934] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0935] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0936] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0937] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0938] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0939] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0940] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0941] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0942] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0943] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0944] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0945] The online coaching and play analysis system of the present invention includes a player terminal, cloud storage, a generative AI model, and a server. The functions and processing flow of each component are shown below to implement the present invention.

[0946] Player device functions

[0947] A player device is a device that is primarily used by a player to record, save, and upload their play to a server. It is also the device through which the player receives feedback provided as analytical results. Specifically, a player device may be a smartphone, tablet, or camera.

[0948] Server Features

[0949] The server plays a central role in receiving, storing, and analyzing gameplay videos and question data sent from the player's device. The server provides the URL of the destination for the received video to the AI ​​model, which then analyzes the video. It also has the function of generating feedback based on the analysis results and sending it to the player's device.

[0950] Cloud storage features

[0951] Cloud storage is a place to temporarily store the received gameplay video. The server saves the video in cloud storage and passes the URL of the saved video to the generation AI model.

[0952] Generative AI model capabilities

[0953] The generative AI model is a system that analyzes gameplay videos stored in cloud storage to identify players' strengths and areas for improvement. It has the ability to generate specific feedback based on the analysis results. Furthermore, if the player inputs specific questions, it can also generate feedback based on those questions.

[0954] Specific processing of the program

[0955] The server receives the video of the player's play. For example, the player takes a video of their dribbling and uploads it through the application. The server saves the video in cloud storage and generates a URL for the destination.

[0956] The server then passes the URL to the generative AI model, asking it to analyze the video. The generative AI model analyzes the video and identifies strengths and areas for improvement in the player's dribbling skills. Based on the analysis results, the generative AI model creates feedback including specific training advice.

[0957] The server then sends the generated feedback to the player terminal, and the player terminal receives a notification, allowing the player to open the application, check the specific feedback content, and use it to improve their practice.

[0958] For example, the feedback provided may include, "Your dribbling speed is excellent, but your timing is off, making it difficult to control the ball." The advice also includes specific drills and practice menus to improve your timing. Based on this, players can then practice to improve their own play.

[0959] In this way, the system of the present invention allows players to receive efficient and accurate feedback and specific advice on how to improve their play without the need for a coach.

[0960] The processing flow will be explained below.

[0961] Step 1:

[0962] A user takes a video of a game using a smartphone or camera and saves it on the device. For example, a user might take a video of a soccer player dribbling.

[0963] Step 2:

[0964] The user launches the application, selects a saved gameplay video, and uploads it to the server. Specifically, the user taps the "Upload Video" button, selects the video file, and sends an upload request.

[0965] Step 3:

[0966] The device sends the uploaded gameplay video to the server, which receives the HTTP POST request and saves the video file in cloud storage.

[0967] Step 4:

[0968] The server generates a URL for the video stored in cloud storage and sends an analysis request to the generative AI model. Specifically, the server passes the video URL to the generative AI model via a REST API.

[0969] Step 5:

[0970] The generative AI model receives the video URL and begins analyzing the video of the game. The generative AI model analyzes the player's movements and techniques in the video to identify strengths and areas for improvement.

[0971] Step 6:

[0972] A generative AI model uses the analysis to generate feedback, including the player's technical strengths and areas for improvement, as well as specific training advice.

[0973] Step 7:

[0974] The server receives the feedback obtained from the generative AI model and sends it to the player's device. The server returns the feedback data to the device via an HTTP response.

[0975] Step 8:

[0976] The device receives the feedback from the server and displays a notification to the user, for example, a push notification that says "Your feedback has been received."

[0977] Step 9:

[0978] The user opens the application and checks the received feedback. Specifically, the user can view the feedback on the application screen and identify strengths and areas for improvement.

[0979] Step 10:

[0980] Based on the feedback provided by the user, the system implements specific advice for training and improving play, such as "doing specific drills to improve dribbling timing."

[0981] Example 1

[0982] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0983] Conventional play analysis systems have issues such as low analysis accuracy, inability to provide instant feedback, and inability to obtain analysis results that satisfy users. Furthermore, when players themselves want feedback on specific questions, they often lack the ability to obtain specific advice. To address these issues, a system that improves the accuracy of gameplay video analysis and provides fast, specific feedback is needed.

[0984] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0985] In this invention, the server includes a means for uploading gameplay videos from the player terminal, a means for saving the uploaded gameplay videos to cloud storage, and a means for generating a destination URL for the gameplay videos saved in cloud storage and passing it to the generation AI model. This improves the accuracy of gameplay video analysis and makes it possible to provide quick and specific feedback.

[0986] "Player Device" means a device used by a player to record, store, and upload their play to the server, such as a smartphone, tablet, or camera.

[0987] "Cloud storage" refers to a group of remote servers that store data via the Internet. In particular, in the present invention, it is used to temporarily store gameplay videos.

[0988] The "generative AI model" is an artificial intelligence model that analyzes gameplay videos stored in cloud storage and identifies players' strengths and areas for improvement. This model is responsible for generating specific feedback based on the results of gameplay analysis.

[0989] "Feedback" refers to specific advice and information on areas for improvement provided by the generative AI model based on the analysis of the gameplay video. The goal is to improve the player's skills.

[0990] The online coaching and play analysis system of the present invention includes a player terminal, cloud storage, a generative AI model, and a server. The components and processing flow of this system will be described in detail below.

[0991] Player device functions

[0992] A player device is a device used by a player to record, save, and upload their play to a server. Specifically, this includes smartphones, tablets, and cameras. It is also on the player device that the player receives feedback provided as analysis results. For example, a user may film a soccer dribble using a smartphone and upload the video through an application.

[0993] Server Features

[0994] The server plays a central role in receiving gameplay videos and question data sent from the player's device and storing them in cloud storage. Specifically, the server stores the received videos in cloud storage and generates a URL for the destination. This URL is then provided to the generative AI model, which requests it to analyze the video. The server also has the function of generating feedback based on the analysis results returned by the generative AI model and sending it to the player's device.

[0995] Cloud storage features

[0996] Cloud storage is a group of remote servers that temporarily store the received gameplay videos. The server saves the videos in cloud storage and passes the URL of the saved video to the generated AI model. Specifically, a common cloud storage service (e.g., Amazon S3) is used.

[0997] Generative AI model capabilities

[0998] The generative AI model is an artificial intelligence system that analyzes gameplay videos stored in cloud storage and identifies players' strengths and areas for improvement. This model has the ability to generate specific feedback based on the analysis results. Furthermore, if a player inputs a specific question, it can generate appropriate feedback based on that question.

[0999] Feedback example

[1000] An example of specific feedback might be, "Your dribbling speed is excellent, but your timing is off, making it difficult to control the ball." The advice also includes specific drills and practice exercises to improve your timing. This gives players a concrete action plan to improve their play.

[1001] Prompt Sentence Examples

[1002] The following prompts can be used as input to a generative AI model:

[1003] "Analyze this video for dribbling skills, identify player strengths and areas for improvement, and generate specific training advice."

[1004] The system allows players to receive efficient and accurate feedback quickly, giving them specific advice on how to improve their play without the need for a coach.

[1005] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1006] Step 1:

[1007] The user launches the application on the player's device and records a video of their gameplay. The recorded video is temporarily saved on the device. Then, when the user taps the "Upload" button in the application, the device sends the video file to the server. The input to this process is the recorded video of the gameplay, and the output is the transmission of the video file to the server. Specific operations include taking a video using the camera app, saving it as a video file, and uploading the video file from the application.

[1008] Step 2:

[1009] The server receives gameplay video data sent from the player's device. The received video is temporarily stored on the server. The server then uploads the video to cloud storage and generates a destination URL for the video. The input to this process is the gameplay video data sent from the device, and the output is the URL for the video stored in cloud storage. Specific operations include saving the video file on the server, uploading it to cloud storage, and generating a destination URL.

[1010] Step 3:

[1011] The server creates a request to provide the generative AI model with the destination URL. This request includes a prompt for the generative AI model (e.g., "Please analyze the dribbling skills in this video, identify the player's strengths and areas for improvement, and generate specific training advice."). The server sends this request to the generative AI model. The inputs to this process are the URL of the video stored in cloud storage and the prompt, and the output is a request sent to the generative AI model.

[1012] Step 4:

[1013] The generative AI model retrieves the video data from cloud storage using the received URL. The AI ​​model then analyzes the video data and evaluates the player's movements. This evaluation includes detailed analysis of the player's dribbling skills, ball control, speed, etc. The generative AI model then generates specific feedback based on the analysis results. The input of this process is the video data retrieved from cloud storage, and the output is specific feedback.

[1014] Step 5:

[1015] The server receives feedback sent from the generative AI model. The received feedback is sent to the player device. When the player device is ready to receive feedback, the server sends a notification. The input of this process is feedback from the generative AI model, and the output is sending feedback and a notification to the player device.

[1016] Step 6:

[1017] The player device receives the feedback sent from the server. The user opens the application and checks the received feedback. For example, the application may display a message such as "Your dribbling speed is excellent, but your timing is off, making it difficult to control the ball," and provide specific drills and practice menus to improve your timing. The input to this process is the feedback sent from the server, and the output is the feedback displayed to the user. The user can then use this information to create a practice plan and work to improve their skills.

[1018] (Application example 1)

[1019] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1020] Real-time data analysis and feedback are essential to improving the driving performance of autonomous vehicles. However, conventional systems lack real-time capabilities and have difficulty providing quick and accurate improvement measures. This has prevented improvements in safety and efficiency.

[1021] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1022] In this invention, the server includes means for uploading data from a user terminal, means for storing the uploaded data in a data storage device, means for passing the data stored in the data storage device to a generative AI model for analysis, means for generating feedback based on the data analysis results, and means for transmitting the generated feedback to the user terminal, thereby enabling the driving data of an autonomous vehicle to be analyzed in real time and providing prompt and accurate feedback based on the analysis results.

[1023] "User Device" means a device used by a User to collect, store, or transmit data, including a smartphone, tablet, camera, etc.

[1024] A "data storage device" is a storage system for temporarily or long-term storage of collected data, and includes cloud storage and on-premise storage servers.

[1025] A "generative AI model" is an algorithm that analyzes collected data and generates feedback or advice tailored to a specific purpose, and includes models that use machine learning and deep learning.

[1026] "Analysis results" are the results or evaluations that a generative AI model derives from input data, including strengths and areas for improvement regarding specific performance.

[1027] "Feedback" refers to specific instructions or advice provided to users based on the analysis results, including specific examples and improvement measures.

[1028] The driving analysis system for autonomous vehicles according to the present invention is a real-time driving analysis system that collects and analyzes data obtained from cameras and sensors on autonomous vehicles, identifies strengths and areas for improvement in driving performance, and provides specific instructions and feedback in real time based on the analysis results.

[1029] System Components and Functions

[1030] 1. User Device:

[1031] User terminals are devices used to collect, store, and transmit data. These include cameras and various sensors (such as LiDAR and radar) installed in autonomous vehicles, which collect surrounding situation data and driving information in real time.

[1032] 2. Data Storage Device:

[1033] A data storage device is a storage system for temporarily or long-term storage of collected data. Cloud storage such as Amazon S3 is an example of this, allowing data to be stored and quickly accessed.

[1034] 3. Generative AI Model:

[1035] Generative AI models are algorithms that analyze collected data and generate feedback and advice tailored to specific objectives, including machine learning and deep learning models like OpenAI GPT-4. Based on the analysis results, they identify strengths and areas for improvement regarding the driving performance of autonomous vehicles.

[1036] 4. Server:

[1037] The server uploads data from the user's device and stores it in a data storage device. Next, it requests the generated AI model to analyze the data by passing the URL of the destination. It also generates feedback based on the analysis results from the generated AI model and sends it to the user's device. Examples of such servers include AWS EC2.

[1038] Specific processing of the program

[1039] 1. Data Collection:

[1040] Cameras and various sensors (such as LiDAR and radar) installed in autonomous vehicles collect data on the surrounding environment and driving information, which is then uploaded to cloud storage in real time.

[1041] 2. Data Analysis:

[1042] The server saves the uploaded data to cloud storage (Amazon S3) and passes the URL of the saved data to an OpenAI GPT-4-based generative AI model, which analyzes the driving data and identifies strengths and areas for improvement in vehicle performance.

[1043] 3. Feedback Generation:

[1044] Based on the analysis results, the generative AI model generates feedback including specific instructions and improvement measures, such as "When turning right, it is recommended to turn on the right turn signal 50 meters before the intersection."

[1045] 4. Real-time control:

[1046] The server sends the generated feedback to the user's terminal, and the autonomous vehicle uses the feedback to improve its driving performance.

[1047] Examples and prompts

[1048] Examples:

[1049] When an autonomous vehicle makes a right turn at an intersection, it detects obstacles and uploads the information to cloud storage in real time. A generative AI model analyzes this data and provides feedback on how to improve the timing of the right turn. Based on this feedback, the vehicle completes the right turn smoothly.

[1050] Example prompt sentence:

[1051] Driving data analysis: Cameras detect obstacles 50 meters before an intersection. The system evaluates the timing of right turns at intersections and generates optimal improvement measures.

[1052] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1053] Step 1:

[1054] The autonomous vehicle's cameras and various sensors (LiDAR, radar, etc.) collect surrounding situation data and driving information in real time. The input is information about the surrounding environment and vehicle operation information. The output is the generation and storage of this data. Specifically, the camera captures video data, and the sensors measure information such as distance and angle.

[1055] Step 2:

[1056] The server stores the collected data in cloud storage (Amazon S3). The input is the environmental information and operation information generated in step 1. The output is a notification that the data has been saved to cloud storage and the URL of the destination. Specifically, the server converts the data into an appropriate format and uploads it to cloud storage.

[1057] Step 3:

[1058] The server passes the destination URL of the data stored in cloud storage to the generative AI model. The input is the destination URL obtained in step 2. The output is an analysis request to the generative AI model and a URL. Specifically, the server constructs a URL and sends the analysis request to the generative AI model.

[1059] Step 4:

[1060] The generative AI model retrieves data from cloud storage and analyzes it. The input is the destination URL. The output is the analysis results, specifically the strengths and areas for improvement in driving performance. Specifically, the generative AI model downloads the data and analyzes it using machine learning algorithms.

[1061] Step 5:

[1062] The generative AI model generates feedback based on the analysis results. The input is the analysis results obtained in step 4. The output is a feedback message that includes specific operational instructions and improvement measures. Specifically, the generative AI model evaluates the analysis results and generates appropriate feedback statements.

[1063] Step 6:

[1064] The server sends the generated feedback to the user terminal. The input is the feedback message obtained in step 5. The output is a feedback notification to the user terminal. Specifically, the server converts the feedback message into an appropriate format and sends it to the user terminal.

[1065] Step 7:

[1066] The user device receives the feedback and reflects it in the autonomous vehicle's control system. The input is the feedback notification obtained in step 6. The output is an instruction for the control system to operate. Specifically, the user device analyzes the feedback content and updates the autonomous vehicle's control commands based on it.

[1067] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1068] The online coaching and play analysis system related to the present invention includes a player terminal, cloud storage, a generative AI model, an emotion engine, and a server. The functions and processing flow of each component are described in detail below to implement the present invention.

[1069] Player device functions

[1070] A player device is a device that allows a player to record, save, and upload their play to a server. It also serves as a receiver for the feedback and emotion analysis results provided by the player as analytical results. Examples of such devices include smartphones, tablets, and cameras.

[1071] Server Features

[1072] The server plays a central role in receiving, storing, and analyzing gameplay videos, question data, and emotion data sent from player devices. Furthermore, the server passes the emotion analysis results from the emotion engine to the generative AI model, helping to create emotion-adaptive feedback. This allows for more personalized feedback to be provided to players.

[1073] Cloud storage features

[1074] Cloud storage is a place to temporarily store the received gameplay video. The server saves the gameplay video to cloud storage and passes the URL of the saved location to the generation AI model.

[1075] Generative AI model capabilities

[1076] The generative AI model analyzes gameplay videos stored in cloud storage to identify players' strengths and areas for improvement. Furthermore, the AI ​​model generates optimal feedback for players based on the player's questions and the emotional analysis results provided by the emotion engine.

[1077] Emotion Engine Functions

[1078] The emotion engine is a system that recognizes and analyzes the player's emotional state. It analyzes emotions based on the player's voice and facial expression data, and provides the results to a generative AI model to improve the quality of feedback.

[1079] Specific processing of the program

[1080] The server receives the video of the player's play from the player's device. The user takes a video of the dribble and uploads it through the application. The server saves the video to cloud storage and generates a URL for the destination.

[1081] The server then passes the URL to the generative AI model, asking it to analyze the video. The generative AI model analyzes the video and identifies strengths and areas for improvement in the player's dribbling skills. Based on the analysis results, the generative AI model creates feedback including specific training advice.

[1082] Additionally, an emotion engine analyzes the player's emotional state and provides the results to the generative AI model, determining, for example, whether the player is nervous or stressed.

[1083] The generative AI model generates emotion-adaptive feedback by comprehensively considering the player's play analysis results and emotional state. For example, if the emotion engine detects "disappointment" or "frustration" when a player makes a mistake, the generative AI model will provide feedback including "technical advice and emotional support to maintain confidence."

[1084] The server then sends the generated feedback and emotion analysis results to the player's device, which then receives a notification. The player can then open the application to view the specific feedback and emotion analysis results and use them for practice.

[1085] For example, feedback such as "Your dribbling speed is excellent, but you have difficulty maintaining your balance" is provided. Other examples of emotion-adaptive feedback include "breathing techniques to relieve tension" and "mental training to maintain positive thinking."

[1086] Such a system would enable players to receive more effective and personalized training guidance, and would also take into account their emotional state, potentially improving their overall performance.

[1087] The processing flow will be explained below.

[1088] Step 1:

[1089] A user takes a video of a game using a smartphone or camera and saves it on the device. For example, a user might take a video of a soccer player dribbling.

[1090] Step 2:

[1091] The user launches the application, selects a saved gameplay video, and uploads it to the server. Specifically, the user taps the "Upload Video" button, selects the video file, and sends an upload request.

[1092] Step 3:

[1093] The device sends the uploaded gameplay video to the server, which receives the HTTP POST request and saves the video file in cloud storage.

[1094] Step 4:

[1095] The server generates a URL for the video stored in cloud storage and sends an analysis request to the generative AI model. Specifically, the server passes the video URL to the generative AI model via a REST API.

[1096] Step 5:

[1097] The generative AI model receives the video URL and begins analyzing the video of the game. The generative AI model analyzes the player's movements and techniques in the video to identify strengths and areas for improvement.

[1098] Step 6:

[1099] The user opens the application's question input screen, enters a question about a specific play, and sends it to the server. For example, they might enter, "How can I improve my dribbling accuracy?"

[1100] Step 7:

[1101] The server sends the question to the generative AI model and asks it to also perform analysis based on the question.

[1102] Step 8:

[1103] The generative AI model generates feedback based on the video analysis and user questions, including dribbling strengths and areas for improvement, as well as specific training advice related to the question.

[1104] Step 9:

[1105] The user activates the emotion engine to collect facial and voice data while playing. For example, a camera or microphone records the user's emotion data while playing.

[1106] Step 10:

[1107] The device sends the collected emotion data to the server, which receives the emotion data and passes it to the emotion engine.

[1108] Step 11:

[1109] The emotion engine analyzes the emotion data to identify the player's emotional state, such as "tension" or "lack of concentration."

[1110] Step 12:

[1111] The emotion engine sends the emotion analysis results to the generative AI model, which receives the emotion data and uses it as feedback.

[1112] Step 13:

[1113] The generative AI model generates emotion-adaptive feedback based on emotion analysis, video analysis, and questions, including suggestions such as "breathing techniques to relax" and "mental training to improve concentration."

[1114] Step 14:

[1115] The server sends the generated feedback and the emotion analysis results to the player device, which receives the HTTP response and retrieves the feedback data.

[1116] Step 15:

[1117] The device receives the feedback and sentiment analysis results from the server and displays a notification to the user, for example, a push notification informing the user that "feedback has been received."

[1118] Step 16:

[1119] The user opens the app and sees detailed feedback and sentiment analysis, including specific advice on how to improve dribbling and relax.

[1120] Step 17:

[1121] Based on the feedback and emotional advice provided by the user, the system implements specific actions to improve training and play, such as "training to improve dribbling timing and methods to relax mentally."

[1122] In this way, the present invention provides feedback that takes into account the player's emotional state in addition to technical analysis, enabling more effective and personalized training.

[1123] Example 2

[1124] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1125] Conventional online coaching systems often do not take into account the user's emotional state when analyzing a player's playing video and generating feedback. As a result, optimal training advice is not provided to the player, limiting the effectiveness of training. Furthermore, even if a player inputs a specific question, it is difficult to obtain feedback in real time that responds to that question.

[1126] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for uploading player videos from a player terminal, means for saving the uploaded player videos in cloud storage, means for passing the player videos saved in the cloud storage to a generative AI model for analysis, means for generating feedback based on the analysis results and emotion analysis results of the player videos, and means for transmitting the generated feedback to the player terminal. This makes it possible to provide optimal feedback in real time that takes into account the player's emotional state. In addition, the player can input a question about a specific play, and feedback based on the question can be appropriately generated and provided.

[1127] A "terminal" is an electronic device used by a user, and includes a smartphone, tablet, camera, etc.

[1128] "Video" refers to video data that records a user's play.

[1129] "Cloud storage" is a data storage service accessible via the internet that provides server space for temporary storage of video files.

[1130] "Generative AI model" refers to an artificial intelligence system that analyzes videos stored in cloud storage and generates feedback.

[1131] "Feedback" refers to advice and instruction to improve a player's skills that is created by the generative AI model based on the results of video analysis.

[1132] "Emotion analysis result" refers to information about the user's emotional state obtained as a result of the emotion engine analyzing the user's voice and facial expression data.

[1133] "Emotion-adaptive feedback" refers to the process by which a generative AI model generates feedback optimized according to the user's emotional state based on the results of video analysis and emotion analysis.

[1134] "Server" means the central computer system that receives, stores, and analyzes data transmitted from Player Devices and generates and transmits feedback.

[1135] A "question" refers to an inquiry a user inputs into a generative AI model regarding a particular play.

[1136] "Player" means a person who practices a sport or game and uses the System to analyze video of their play and receive feedback.

[1137] The online coaching and play analysis system related to the present invention includes a player terminal, cloud storage, a generative AI model, an emotion engine, and a server. The functions and processing flow of each component are described in detail below.

[1138] Player device functions

[1139] A player device is a device that allows a player to record, save, and upload gameplay videos to a server. It also receives feedback and emotion analysis results from the player. Examples of such devices include smartphones, tablets, and cameras.

[1140] Server Features

[1141] The server plays a central role in receiving, storing, and analyzing gameplay videos, question data, and emotion data sent from the player's device. The server saves the received gameplay videos in cloud storage and passes the URL of the saved video to the generative AI model. In addition, the server passes the emotion analysis results from the emotion engine to the generative AI model to help create emotion-adaptive feedback. This allows for more personalized feedback to be provided to the player.

[1142] Cloud storage features

[1143] Cloud storage is a place to temporarily store the received gameplay video. The server saves the gameplay video in cloud storage (e.g., Amazon S3 or Google Cloud Storage) and generates a URL for the destination.

[1144] Generative AI model capabilities

[1145] The generative AI model is a system that analyzes gameplay videos stored in cloud storage and identifies players' strengths and areas for improvement. Furthermore, this generative AI model generates optimal feedback for players based on the player's questions and the emotional analysis results provided by the emotion engine. By comprehensively considering the analytical and emotional analysis results, it is possible to generate emotion-adaptive feedback.

[1146] Prompt Sentence Examples

[1147] For example, the prompt sentence to be input to the generative AI model is as follows:

[1148] "Download the video from this URL and perform a skill analysis of the player. Furthermore, create feedback to provide to the player based on the emotion analysis results obtained from the voice and facial expression data."

[1149] Emotion Engine Functions

[1150] The emotion engine is a system that recognizes and analyzes a player's emotional state. By analyzing emotions based on the player's voice and facial expression data and providing the results to a generative AI model, the quality of feedback can be improved. For example, it can determine whether a player is feeling nervous or stressed and reflect that information in the feedback.

[1151] Example

[1152] As a concrete example, consider a scenario in which a user films a dribbling play and uploads the video from their device to a server. The server saves the video file in cloud storage and passes the URL to the generative AI model. The generative AI model analyzes the video and finds that "the dribbling speed is excellent, but maintaining balance is difficult." The emotion engine also detects that the user is nervous. Based on this information, the generative AI model generates emotion-adaptive feedback, including technical advice such as "breathing techniques to relieve tension" and "mental training to maintain a positive mindset."

[1153] This provides players with a system that allows them to not only improve their technical skills, but also manage their emotional state and aim to improve their overall performance.

[1154] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1155] Step 1:

[1156] The user records a video of a dribbling play on the device. The user taps the "Record new video" button, which activates the device's camera. After recording is complete, the user saves the video file. The input is the video file of the play, and the output is the video file saved on the device.

[1157] Step 2:

[1158] A user uploads a video from their device to a server. When the user taps the "Upload" button in the application, the device sends the video file to the server. The input is the video file stored on the device, and the output is the video file uploaded to the server.

[1159] Step 3:

[1160] The server saves the received video in cloud storage and generates a destination URL. The server saves the video file in cloud storage (e.g., Amazon S3) and generates a unique destination URL. The input is the video file received by the server, and the output is the destination URL in cloud storage.

[1161] Step 4:

[1162] The server sends the destination URL to the generative AI model. The server sends the URL to the generative AI model and creates a prompt for analysis. The input is the destination URL in cloud storage, and the output is the prompt sent to the generative AI model.

[1163] Step 5:

[1164] The generative AI model analyzes the video and generates feedback. The generative AI model downloads the video from cloud storage and analyzes it. As a result of the analysis, it extracts the player's technical strengths and areas for improvement. It then generates specific training advice based on this information. The input is the video file downloaded from cloud storage, and the output is feedback based on the analysis results.

[1165] Step 6:

[1166] The emotion engine analyzes the player's emotional state and sends the results to the generative AI model. If voice and facial expression data are also recorded when the player uploads a video, the emotion engine analyzes this data. It identifies the tension or stress the player is feeling and sends the analysis results to the generative AI model. The input is the player's voice and facial expression data, and the output is the emotion analysis results.

[1167] Step 7:

[1168] The generative AI model generates emotion-adaptive feedback based on the results of video analysis and emotion analysis. The generative AI model integrates the results of video analysis with feedback that takes into account the player's emotional state. For example, if a player feels "disappointed" or "frustrated" after making a mistake, it creates feedback that includes technical advice and emotional support. The input is the results of video analysis and emotion analysis, and the output is emotion-adaptive feedback.

[1169] Step 8:

[1170] The server sends the generated feedback and emotion analysis results to the player's device. The server sends the feedback received from the generative AI model to the player's device and performs notification. The input is the generated feedback and emotion analysis results, and the output is the feedback and emotion analysis results sent to the player's device.

[1171] Step 9:

[1172] The user checks the feedback and emotion analysis results on the device and uses them for training. The user opens the application on the device to check the feedback and emotion analysis results, and improves their training and play based on the feedback they receive. The input is the feedback and emotion analysis results sent to the player device, and the output is the improvement results obtained by the user.

[1173] (Application example 2)

[1174] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1175] Previous online coaching and play analysis systems focused primarily on improving players' technical skills, but did not provide feedback or content recommendations based on users' emotions or viewing behavior. This limited the overall user experience. Given this background, there was a need to develop a system that could analyze users' viewing behavior and emotional state to provide more personalized feedback and content recommendations.

[1176] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading player videos from the player terminal, means for saving the uploaded player videos in cloud storage, means for passing the player videos saved in the cloud storage to a generative AI model for analysis, means for generating feedback based on the analysis results of the player videos, means for sending the generated feedback to the player terminal, means for analyzing the user's viewing behavior and emotional state, and means for recommending personalized content based on the analysis results. This makes it possible to not only improve the user's technical skills but also to provide optimal feedback and content recommendations according to the user's viewing behavior and emotional state.

[1177] "Player Device" means an electronic device used by a User to record, store, and upload videos of his / her gameplay.

[1178] "Player video" is video data showing the play and activities of a user that is filmed using a player terminal.

[1179] "Cloud storage" is an online storage system for storing and managing data via the Internet.

[1180] A "generative AI model" is an artificial intelligence system that analyzes users' video data and question data to generate personalized feedback.

[1181] "Analysis results" are the evaluation and comment data obtained after the generative AI model analyzes the video data and question data.

[1182] "Feedback" refers to information that includes evaluations and suggestions for improvement regarding a user's playing or viewing behavior.

[1183] "Viewing behavior" refers to the actions and patterns that users take when watching video content, and is the data related to that.

[1184] "Emotional state" is data that represents the user's emotional and psychological state when viewing.

[1185] "Content recommendation" is a function that presents the most appropriate videos and information based on the user's viewing behavior and emotional state.

[1186] The online coaching and play analysis system related to the present invention includes a player terminal, cloud storage, a generative AI model, an emotion engine, and a server. The functions of each component and the processing flow for implementing the present invention are described in detail below.

[1187] Player device functions

[1188] A player device is an electronic device that allows a user to record, save, and upload their gameplay video to a server. The user also receives feedback and sentiment analysis results. Examples of such devices include smartphones, tablets, and cameras.

[1189] Server Features

[1190] The server plays a central role in receiving, storing, and analyzing gameplay videos, viewing behavior data, and emotional data sent from player devices. Furthermore, the server passes the emotional analysis results from the emotion engine to the generative AI model, helping to create personalized feedback. This allows for more personalized content recommendations to be provided to users.

[1191] Cloud storage features

[1192] Cloud storage is a place to temporarily store received gameplay videos and viewing behavior data. The server saves the gameplay videos in cloud storage and passes the URL of the saved location to the generation AI model.

[1193] Generative AI model capabilities

[1194] The generative AI model analyzes gameplay videos stored in cloud storage and identifies users' strengths and areas for improvement. Furthermore, the AI ​​model generates optimal feedback for users based on their questions and the emotional analysis results provided by the emotion engine. It also includes a function to recommend optimal content based on the user's viewing behavior and emotional state.

[1195] Emotion Engine Functions

[1196] The emotion engine is a system that recognizes and analyzes a user's emotional state. It analyzes emotions based on the user's voice and facial expression data, and provides the results to a generative AI model to improve the quality of feedback and content recommendations.

[1197] Processing flow

[1198] The player's device records a video of the game and uploads it to the server. The server saves the video to cloud storage and generates a URL for the destination. The server then passes the URL to the generated AI model and requests it to analyze the video.

[1199] The generative AI model analyzes the video and identifies the user's skill strengths and areas for improvement. Based on the analysis results, the generative AI model creates feedback including specific training advice. At the same time, the emotion engine analyzes the user's emotional state and provides the analysis results to the generative AI model. The generative AI model then generates emotion-adaptive feedback by comprehensively considering the play analysis results and the user's emotional state.

[1200] Furthermore, it recommends optimal content based on the user's viewing behavior and emotional state. The server sends the generated feedback and content recommendation results to the player device, and the user can open the application and check them.

[1201] Examples and prompts

[1202] Examples:

[1203] Video content: User watching a fitness video

[1204] Sentiment analysis results: "Enjoyed" "Excited"

[1205] Generated feedback: "You seem to be really enjoying this exercise. As a next step, try these more advanced exercises."

[1206] Example prompt sentence:

[1207] You performed sentiment analysis on the fitness video the user was watching, and the result was that the user was enjoying it. Based on this analysis, you want to generate feedback to recommend the next advanced exercise to the user.

[1208] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1209] Step 1:

[1210] The player device takes video of the user's play. The camera function of the player device is used to record the user's play and activities. The input is actual play video, and the output is digital video data.

[1211] Step 2:

[1212] The gameplay video captured by the player's device is uploaded to the server. The video data is sent to the server via the Internet. The input is the gameplay video in digital format, and the output is a video file stored on the server.

[1213] Step 3:

[1214] The server saves the uploaded gameplay video to cloud storage. Using the file saving API for cloud storage, the video data is saved to the cloud. The input is the video file on the server, and the output is the URL of the cloud storage destination.

[1215] Step 4:

[1216] The server passes the URL of the video stored in cloud storage to the generative AI model. The URL of the video is passed to the API of the generative AI model and an analysis request is made. The input is the URL of the cloud storage destination, and the output is the analysis result of the generative AI model.

[1217] Step 5:

[1218] The generative AI model analyzes the gameplay video stored in cloud storage. The generative AI model's algorithm analyzes the video data and identifies the user's skills, strengths, and areas for improvement. The input is the cloud storage destination URL, and the output is the gameplay analysis results.

[1219] Step 6:

[1220] The generative AI model generates feedback based on the results of play analysis. It also takes into account the emotional analysis results provided by the emotion engine to create personalized feedback. The input is the results of play analysis and emotional analysis, and the output is feedback that includes specific training advice.

[1221] Step 7:

[1222] The server sends the feedback received from the generative AI model to the player device. The feedback data is notified to the player device via the Internet. The input is the generated feedback, and the output is the feedback displayed on the user's player device.

[1223] Step 8:

[1224] The player device receives the feedback and the user confirms it. The application is opened and the feedback content is displayed. The input is the feedback data sent from the server, and the output is the feedback screen that the user can see.

[1225] Step 9:

[1226] The emotion engine analyzes the user's viewing behavior and emotional state. It analyzes the video being watched and the user's reactions to identify the emotional state. The input is the user's viewing data and voice / facial expression data, and the output is the emotion analysis results.

[1227] Step 10:

[1228] The generative AI model recommends personalized content based on the results of sentiment analysis. It selects the most appropriate content based on the user's emotional state and viewing behavior, and creates a recommendation list. The input is the results of sentiment analysis and viewing behavior data, and the output is a content recommendation list.

[1229] Step 11:

[1230] The server sends the generated content recommendation list to the player terminal. The recommendation list is notified to the player terminal via the Internet. The input is the generated content recommendation list, and the output is the recommended content displayed on the user's player terminal.

[1231] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1232] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1233] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1234] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1235] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1236] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1237] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1238] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1239] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1240] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1241] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1242] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1243] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1244] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1245] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1246] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1247] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1248] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1249] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1250] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1251] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1252] The following is further disclosed regarding the above embodiment.

[1253] (Claim 1)

[1254] A means of uploading player videos from the player device,

[1255] A means to save uploaded player videos to cloud storage;

[1256] A means to pass player videos stored in cloud storage to the generative AI model for analysis;

[1257] A means for generating feedback based on the analysis of the player video;

[1258] means for transmitting the generated feedback to the player terminal;

[1259] A system including:

[1260] (Claim 2)

[1261] 10. The system of claim 1, further comprising: means for a player to input a question about a particular play and send the question to the generative AI model; and means for the generative AI model to generate feedback based on the question.

[1262] (Claim 3)

[1263] 10. The system of claim 1, further comprising means for the generative AI model to generate feedback including strengths and areas for improvement for the player based on the analysis of the player video and the questions.

[1264] "Example 1"

[1265] (Claim 1)

[1266] A means to upload gameplay videos from the player's device,

[1267] A means to save uploaded gameplay videos to cloud storage,

[1268] A method to generate a URL for the gameplay video saved in cloud storage and pass it to the AI ​​model.

[1269] A generative AI model analyzes video of a player's play to identify their strengths and areas for improvement.

[1270] A means for generating feedback based on the analysis results of the play video and transmitting the feedback to the player device;

[1271] a means for the player terminal to receive feedback and for the user to review;

[1272] A system including:

[1273] (Claim 2)

[1274] 10. The system of claim 1, further comprising: means for a player to input a question about a particular play and send the question to the generative AI model; and means for the generative AI model to generate feedback based on the question.

[1275] (Claim 3)

[1276] 10. The system of claim 1, further comprising means for the generative AI model to generate feedback including player strengths and areas for improvement based on analysis of play video and questions.

[1277] "Application Example 1"

[1278] (Claim 1)

[1279] A means for uploading data from a user device;

[1280] means for storing the uploaded data in a data storage device;

[1281] means for passing data stored in a data storage device to a generative AI model for analysis;

[1282] A means for generating feedback based on the analysis of the data;

[1283] means for transmitting the generated feedback to a user terminal;

[1284] A system including:

[1285] (Claim 2)

[1286] 10. The system of claim 1, further comprising: means for a user to input a question regarding a particular operation and send the question to the generative AI model; and means for the generative AI model to generate feedback based on the question.

[1287] (Claim 3)

[1288] 10. The system of claim 1, further comprising means for the generative AI model to generate feedback including strengths and areas for improvement for the automated equipment based on the analysis of the data and the questions.

[1289] "Example 2: Combining Emotion Engines"

[1290] (Claim 1)

[1291] A means for uploading player videos from a player terminal;

[1292] A means to save uploaded player videos to cloud storage;

[1293] A means to pass player videos stored in cloud storage to the generative AI model for analysis;

[1294] A means for generating feedback based on the analysis results and sentiment analysis results of the player video;

[1295] means for transmitting the generated feedback to the player terminal;

[1296] A system including:

[1297] (Claim 2)

[1298] 10. The system of claim 1, further comprising: means for a player to input a question about a particular play and send the question to the generative AI model; and means for the generative AI model to generate feedback based on the question.

[1299] (Claim 3)

[1300] 10. The system of claim 1, further comprising means for the generative AI model to generate player strengths and improvements and emotionally adaptive feedback based on analysis of player video and questions.

[1301] "Application example 2 when combining emotion engines"

[1302] (Claim 1)

[1303] A means of uploading player videos from the player device,

[1304] A means to save uploaded player videos to cloud storage;

[1305] A means to pass player videos stored in cloud storage to the generative AI model for analysis;

[1306] A means for generating feedback based on the analysis of the player video;

[1307] means for transmitting the generated feedback to the player terminal;

[1308] means for analyzing a user's viewing behavior and emotional state;

[1309] A means for recommending personalized content based on the analysis results;

[1310] A system including:

[1311] (Claim 2)

[1312] 10. The system of claim 1, further comprising: means for a player to input a question about a particular play and send the question to the generative AI model; and means for the generative AI model to generate feedback based on the question.

[1313] (Claim 3)

[1314] 10. The system of claim 1, further comprising means for the generative AI model to generate feedback including strengths and areas for improvement for the player based on the analysis of the player video and the questions. [Explanation of symbols]

[1315] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of uploading player videos from the player device, A means to save uploaded player videos to cloud storage; A means to pass player videos stored in cloud storage to the generative AI model for analysis; A means for generating feedback based on the analysis of the player video; means for transmitting the generated feedback to the player terminal; A system including:

2. 10. The system of claim 1, further comprising: means for a player to input a question about a particular play and send the question to the generative AI model; and means for the generative AI model to generate feedback based on the question.

3. 10. The system of claim 1, further comprising means for the generative AI model to generate feedback including strengths and areas for improvement for the player based on the analysis of the player's video and the questions.

Images