system

Abstract

Provide a system. The present invention includes: a means for receiving and storing video data of a game played by a user; A means for decoding the stored video data and converting it into a data format for analysis; The converted data will be used to analyze the gameplay using an AI model and compare it with the play data of professional players. A means for identifying problems in the user's play based on the analysis results and generating methods for improvement; a means for transmitting the generated feedback to a user's terminal so that the user can check the evaluation result; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, improving a game's skills requires a significant amount of time and effort, often through trial and error. However, this process is inefficient and makes it difficult for users to obtain specific feedback that will help them effectively improve their skills. Furthermore, professional coaching is expensive, making it difficult for average gamers to access. To address these issues, a system is needed that can efficiently analyze a user's play and provide specific suggestions for improvement. [Means for solving the problem]

[0005] To solve this problem, the present invention provides the following means: a system including means for receiving and saving video data of a game played by a user, means for decoding the saved video data and converting it into a data format for analysis, means for analyzing the play using an AI model using the converted data and comparing it with the play data of professional players, means for identifying issues in the user's play based on the analysis results and generating methods for improvement, and means for sending the generated feedback to the user's device so that the user can check the evaluation results. This system allows users to easily receive specific feedback on their play and efficiently improve their gaming skills.

[0006] "User" refers to an individual who utilizes the system to improve their gameplay through feedback.

[0007] A "game" is a computer program played by a user and manipulated to achieve a particular goal.

[0008] "Video data" refers to a video file that records a user's gameplay.

[0009] "Receiving" refers to the operation of the server taking in and saving video data uploaded by a user.

[0010] "Storage" refers to storing the received video data in a specific storage device.

[0011] "Decoding" refers to the process of converting stored video data into an analyzable format.

[0012] "Analysis" or "play analysis" is the process of evaluating a user's play based on video data and comparing it with the play data of professional players.

[0013] "Data format" refers to the particular format into which video data is converted in order to be analyzed.

[0014] "AI model" refers to an algorithm or program that uses artificial intelligence technology to analyze a user's play and identify problem areas.

[0015] "Professional player play data" refers to data from plays made by game experts or professionals.

[0016] "Comparison" refers to the act of comparing the user's play with the play data of professional players and evaluating them.

[0017] "Issues" refer to areas in the user's gameplay that need improvement.

[0018] "Improvement methods" refers to specific measures and advice for the identified issues.

[0019] "Feedback" refers to information provided to the user regarding analysis results and methods for improvement.

[0020] "Terminal" means an electronic device used by a user to receive feedback information.

[0021] "Sending" refers to the action of delivering the generated feedback from the server to the user's terminal. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] The present invention is a system for providing specific feedback to users in comparison with the play of professional players to help them improve their gameplay skills.

[0044] First, the user records a video of their gameplay. This can be done using the game's in-game recording feature or external software (such as OBS Studio). The user then uploads the recorded video file to the server via a dedicated web page or application. The server receives this upload request and saves the video file in a specified directory.

[0045] The server then decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the server uses a library such as moviepy to convert the video file into image data for each frame.

[0046] The server then supplies the decoded data to an AI analysis module. The AI ​​model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players was built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between them and professional players.

[0047] Based on the analysis results output by the AI, the server uses natural language to describe the issues in the user's play and how to improve them. For example, it generates feedback such as, "Your positioning is delayed by 5 minutes and 20 seconds. The enemy team has already moved to the next objective, so you need to position yourself earlier."

[0048] The generated feedback is sent by the server to the user's device. The feedback is provided in text and / or graphics format and can be viewed by the user through a dedicated application. The user can use the received feedback as a reference and practice to improve upon the next time they play.

[0049] As a concrete example, consider the case where a user records their gameplay of "League of Legends" and uploads the video to a server. The server saves the video, decodes it, and then performs AI analysis. For example, it identifies that the player's positioning is behind at 5 minutes and 20 seconds, and generates improvement advice such as "ensure proper positioning in advance for the next battle." This advice is then sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[0050] This system allows users to quickly and specifically identify problems in their playing and improve their skills efficiently.

[0051] The processing flow will be explained below.

[0052] Step 1:

[0053] A user plays a game and records the gameplay using the in-game recording function or external software such as OBS Studio.

[0054] Step 2:

[0055] The user uploads the recorded video file to the server through a dedicated web page or application by attaching the video file to a form and clicking an upload button.

[0056] Step 3:

[0057] The server receives the upload request and saves the video file in the specified directory, for example, / path / to / save / .

[0058] Step 4:

[0059] The server decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the moviepy library is used to convert the video into frame-by-frame image data.

[0060] Step 5:

[0061] The server then sends the decoded data to the AI ​​analysis module. The converted data is then fed into the AI ​​model, which compares it with the play data of professional players. The AI ​​model analyzes the user's movements, strategies, item use, positioning, etc.

[0062] Step 6:

[0063] The server receives the analysis results from the AI ​​model and identifies issues with the user's play, such as "positioning is delayed by 5 minutes and 20 seconds."

[0064] Step 7:

[0065] Based on the analysis results, the server generates feedback in natural language describing areas for improvement and the reasons for them. For example, it may generate feedback such as, "You are 5 minutes and 20 seconds behind in positioning, so you need to secure your positioning in advance."

[0066] Step 8:

[0067] The server sends the generated feedback to the user's device, where it can be viewed through a dedicated application. A notification function can also be used to notify the user of the arrival of the feedback.

[0068] Step 9:

[0069] The user checks the feedback on their device, for example by opening a dedicated smartphone app, reading the received feedback, and using it as reference the next time they play.

[0070] Step 10:

[0071] Users can improve their gameplay based on feedback and practice using specific advice to improve their skills.

[0072] Example 1

[0073] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0074] Conventional gameplay analysis systems make it difficult for users to objectively evaluate their own play and identify specific areas for improvement. Furthermore, there is a lack of a way to compare users' play data with that of professional players and provide specific feedback. This has led to the issue of users being unable to improve their skills efficiently.

[0075] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0076] In this invention, the server includes means for receiving and storing data of electronic games played by users, means for decoding the stored data and converting it into a format for analysis, means for analyzing the converted data using an artificial intelligence model and comparing it with data from experts, means for identifying problems in the user's behavior based on the analysis results and generating improvement methods, means for transmitting the generated feedback to the user's communication device so that the user can check the evaluation results, and means for uploading video data via a dedicated web page or software application, thereby enabling users to objectively evaluate their play and receive real-time, specific feedback for making specific improvements.

[0077] "Electronic game" means a game played for competitive or recreational purposes using electronic devices.

[0078] "Decoding" refers to the process of converting data from one format to another.

[0079] An "artificial intelligence model" refers to a collection of algorithms for data analysis that are trained using techniques such as machine learning and deep learning.

[0080] "Specialized web page or software application" refers to a web interface or software program designed to accomplish a specific task.

[0081] "Communication equipment" refers to electronic devices for sending and receiving data, and specifically includes smartphones and computers.

[0082] "Feedback" refers to suggestions for improvement and evaluations provided based on the results of the analysis.

[0083] "Form for analysis" refers to a form in which data has been converted into a form that can be analyzed by an artificial intelligence model.

[0084] "Storage" refers to storing data in a location so that it can be accessed at a later time.

[0085] "Comparison" refers to the process of juxtaposing two or more data sets to assess their differences and similarities.

[0086] "Improvement methods" refer to specific measures to overcome specific issues and achieve better results.

[0087] The invention is a system that provides specific feedback to users in comparison with the play of professional players to help them improve their electronic game playing skills.

[0088] First, a user records their gameplay. This can be done using the game's in-game recording feature or external software (such as OBS Studio). The user then uploads the recorded video file to a server via a dedicated web page or software application. The server receives the upload request and saves the video file in a specified directory.

[0089] The server then decodes the stored video file. Decoding is the process of converting a video file into image data and audio data for each frame. For example, the moviepy library can be used to convert a video file into image data for each frame.

[0090] The server then supplies the decoded data to an AI analysis module. The AI ​​model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players is built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between the user and professional players.

[0091] Based on the analysis results output by the AI, the server uses natural language to describe the issues in the user's play and how to improve them. For example, it generates feedback such as, "Your positioning is delayed by 5 minutes and 20 seconds. The enemy team has already moved to the next objective, so you need to position yourself earlier."

[0092] The generated feedback is sent by the server to the user's device. The feedback is provided in text and / or graphics format and can be viewed by the user through a dedicated software application. The user can use the received feedback to improve and practice the next time they play.

[0093] As a concrete example, consider the case where a user records their play in an "electronic competitive game" and uploads the video to a server. In this case, the server saves the video, decodes it using the moviepy library, and provides the analysis to an AI analysis module. For example, it may identify that the user's positioning is behind at 5 minutes and 20 seconds, and generate feedback to the user, such as "Ensure that you are positioned well in advance for the next battle," and send it to the user via a dedicated application. The user can then take this advice and put it into practice in their next play, thereby improving their skills.

[0094] An example of a prompt to be input to a generative AI model might be, "Analyze a video of an electronic competitive game and note that the user's positioning is slow at the 5 minute 20 second mark. Advise the user to secure their positioning in advance in preparation for the next battle."

[0095] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0096] Step 1:

[0097] User-recorded gameplay and uploaded video files

[0098] Specific behavior:

[0099] Users record their gameplay using the in-game recording feature or external software such as OBS Studio, and then save the recorded video file, which they then upload through a dedicated webpage or software application.

[0100] input:

[0101] Recorded video file (e.g. gameplay_video.mp4)

[0102] output:

[0103] Video files uploaded to the server

[0104] Data processing:

[0105] Nothing in particular

[0106] Step 2:

[0107] Receiving and storing video files on the server

[0108] Specific behavior:

[0109] The server receives the upload request and saves the video file uploaded by the user in the specified directory (e.g., / uploads / ).

[0110] input:

[0111] User-uploaded video files

[0112] output:

[0113] Video files stored in a specified directory on the server

[0114] Data processing:

[0115] Nothing in particular

[0116] Step 3:

[0117] Video file decoding by the server

[0118] Specific behavior:

[0119] The server decodes the stored video file and converts it into frame-by-frame image data and audio data using the moviepy library.

[0120] input:

[0121] Saved video file (e.g. / uploads / gameplay_video.mp4)

[0122] output:

[0123] Frame-by-frame image and audio data

[0124] Data processing:

[0125] Split video files into frames and convert them into image and audio data

[0126] Step 4:

[0127] The server supplies the decoded data to the AI ​​analysis module

[0128] Specific behavior:

[0129] The server supplies the decoded data (frame-by-frame image data and audio data) to the AI ​​analysis module.

[0130] input:

[0131] Frame-by-frame image and audio data

[0132] output:

[0133] Analyzable datasets (e.g., batched frame data)

[0134] Data processing:

[0135] Batch-processing the frame data and preparing it for the AI ​​analysis module

[0136] Step 5:

[0137] Data comparison and analysis using AI analysis module

[0138] Specific behavior:

[0139] The AI ​​analysis module compares the user's play data with pre-trained expert data to identify problems.

[0140] input:

[0141] Batched frame data and expert play data

[0142] output:

[0143] Analysis results (e.g., issue list, difference data)

[0144] Data processing:

[0145] Compare user play data with expert data and extract quantified differences

[0146] Step 6:

[0147] Server generates feedback of analysis results

[0148] Specific behavior:

[0149] Based on the results of the AI ​​analysis, the server describes in natural language the issues in the user's gameplay and how to improve them.

[0150] input:

[0151] Analysis results (e.g., issue list, difference data)

[0152] output:

[0153] Feedback statement (e.g., "You're 5 minutes and 20 seconds late in positioning. The enemy team has already moved to the next objective, so you need to position yourself earlier.")

[0154] Data processing:

[0155] Based on the analysis results, feedback is generated using a natural language generation algorithm.

[0156] Step 7:

[0157] Server sends feedback

[0158] Specific behavior:

[0159] The generated feedback is sent by the server to the user's terminal, and is provided in text and / or graphics format.

[0160] input:

[0161] Feedback Statement

[0162] output:

[0163] Feedback sent to the user's device

[0164] Data processing:

[0165] Convert your feedback statement into text or graphics format and send it

[0166] Step 8:

[0167] Users review feedback and practice

[0168] Specific behavior:

[0169] Users receive feedback through a dedicated software application, which displays the feedback and uses it to improve their next practice.

[0170] input:

[0171] Feedback Statement

[0172] output:

[0173] User feedback review and application

[0174] Data processing:

[0175] Nothing in particular

[0176] Through the above steps, the system analyzes the user's play data and provides specific feedback to help the user improve their skills efficiently.

[0177] (Application example 1)

[0178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0179] Conventional gameplay analysis systems have the drawback of making it difficult for users to obtain specific feedback to improve their gameplay. Furthermore, in many cases, feedback is not provided in real time, making it difficult for users to immediately identify areas for improvement and apply them to their next play. Furthermore, when viewers stream gameplay, there is no way to obtain analysis results in real time, making it difficult for them to receive live feedback and improve their play.

[0180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0181] In this invention, the server includes means for receiving and saving video data of the game played by the user, means for decoding the saved video data and converting it into a data format for analysis, means for analyzing the play using a generative AI model using the converted data and comparing it with play data of professional players, means for identifying issues in the user's play based on the analysis results and creating ways to improve it, means for sending the generated feedback to the user's device so that the user can check the evaluation results, and means for an application to run on the viewer's device and display the feedback in real time. This allows the user to receive specific feedback in real time and improve their play on the spot.

[0182] 1. "Video data of a game played by a user" refers to a series of video data recorded from the game screen while the user is playing.

[0183] 2. "Means of receiving and storing" refers to the function for sending video data to the server and storing it.

[0184] 3. "Decoding" is the process of converting stored video data into an analyzable format.

[0185] 4. "Data format for analysis" means the specific data format required for an AI model to perform analysis.

[0186] 5. "Generative AI model" refers to an artificial intelligence model that compares a user's play with the play data of professional players and generates analytical results.

[0187] 6. "Professional player play data" refers to data on play records of multiple professional players collected in advance.

[0188] 7. "Issues" are specific points that need improvement in the user's play.

[0189] 8. "Improvement methods" are specific advice and techniques for improving issues.

[0190] 9. "Feedback" refers to the evaluation results and details of how to improve provided to the user based on the analysis results.

[0191] 10. "User's device" means a communication device used by a user, such as a personal computer or smartphone.

[0192] 11. "Means for checking evaluation results" refers to a function that allows users to check the feedback content on their devices.

[0193] 12. "The application runs on the viewer's terminal" refers to software that runs on the terminal used by the viewer.

[0194] 13. "Means for displaying feedback in real time" refers to a function that allows analysis results to be displayed instantly on the viewer's device during play.

[0195] This invention is a system for improving a user's game skills. The system analyzes videos of the user's gameplay and compares them with those of professional players to provide specific feedback.

[0196] A user records a video of their gameplay. This can be done using the in-game recording feature or external software (such as OBS Studio). The user then uploads the recorded video file to a server via a dedicated web page or application. The server receives this upload request and saves the video file in a specified directory.

[0197] The server decodes the stored video files. This decoding process converts the video files into an analyzable format, such as frame-by-frame image data and audio data. For example, the moviepy library is used to convert the video files into image data for each frame.

[0198] The server then supplies the decoded data to an AI analysis module. The generative AI model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players was built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between the user and professional players.

[0199] Based on the analysis results output by the AI, the server uses natural language to describe the issues in the user's play and how to improve them. For example, it generates feedback such as, "Your positioning is delayed by 5 minutes and 20 seconds. The opposing team has already moved to the next objective, so you need to position yourself earlier." The generated feedback is sent by the server to the user's device. The feedback is provided in text and graphics format and can be viewed by the user through a dedicated application.

[0200] This system can also be used by viewers during live streaming. When viewers upload their gameplay videos while streaming, the server decodes and analyzes them in real time, instantly generating feedback that is displayed on the viewer's device. This feature allows viewers to receive live feedback and improve their gameplay in real time.

[0201] As a concrete example, consider the case where a user records gameplay of a "fighting game" and uploads the video to a server. The server saves the video, decodes it, and then analyzes it using a generative AI model. For example, it identifies that "positioning is behind at 5 minutes and 20 seconds" and generates improvement advice such as "ensure positioning in advance in preparation for the next battle." This advice is then sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[0202] Example prompt for a generative AI model:

[0203] "Analyze the video of your gameplay and output feedback on the differences between your gameplay and that of a professional player. For example, please describe what movements are lagging at what points, along with suggestions for improvement."

[0204] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0205] Step 1:

[0206] Record video of the game you are playing using the in-game recording feature or external software (such as OBS Studio).

[0207] Input: User gameplay

[0208] Output: Video file

[0209] Step 2:

[0210] Users upload recorded video files to the server via a dedicated web page or application.

[0211] Input: Video file

[0212] Output: Video file saved on the server

[0213] Step 3:

[0214] The server receives the upload request and saves the video file in a specified directory.

[0215] Input: Uploaded video file

[0216] Output: Video file saved on the server

[0217] Step 4:

[0218] The server decodes the stored video files and converts them into a data format for analysis. This decoding process uses the moviepy library to convert the video files into frame-by-frame image data and audio data.

[0219] Input: Video file stored on the server

[0220] Output: Frame-by-frame image data and audio data

[0221] Step 5:

[0222] The server then feeds the decoded data into a generative AI model and compares it with the play data of professional players, which has been compiled in advance by collecting and analyzing the play of many professional players.

[0223] Input: Frame-by-frame image data and audio data

[0224] Output: Play analysis results

[0225] Step 6:

[0226] Based on the analysis results output by the generative AI model, the server describes in natural language the issues in the user's play and how to improve them.

[0227] Input: Play analysis results

[0228] Output: Written feedback in natural language

[0229] Step 7:

[0230] The generated feedback is sent by the server to the user's terminal.

[0231] Input: Written feedback in natural language

[0232] Output: Feedback sent to the user's device

[0233] Step 8:

[0234] The user's device displays the feedback via a dedicated application, which can be in text or graphics format.

[0235] Input: Feedback sent to the user's device

[0236] Output: Feedback displayed on the terminal

[0237] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0238] The present invention combines an emotion engine with a system that provides specific feedback to users to help them improve their gameplay skills by comparing their play with that of professional players, thereby utilizing the user's emotion data to provide more personalized feedback.

[0239] First, a user records a video of their gameplay using the game's in-game recording feature or external software such as OBS Studio. Then, the user uploads the recorded video file to a server via a dedicated webpage or application. The server receives the upload request and saves the video file in a specified directory.

[0240] The server then decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the server uses a library such as moviepy to convert the video file into image data for each frame.

[0241] The server then supplies the decoded data to an AI analysis module. The AI ​​model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players was built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between them and professional players.

[0242] Furthermore, the present invention uses an emotion engine that recognizes the user's emotions. The emotion engine collects the user's biometric data (e.g., heart rate, facial expression, voice tone, etc.) and recognizes the user's emotions based on that data. For example, it can determine whether the user is tense or relaxed.

[0243] Based on the analysis results output by the AI ​​and the emotional data recognized by the emotion engine, the server generates feedback in natural language describing issues with the user's play and how to improve them. By taking emotional data into consideration, it is possible to adjust the tone and content of the feedback to be more personalized. For example, if the user is relaxed, the server generates feedback that includes many encouraging words, and if the user is nervous, the server generates calm feedback that suggests solutions.

[0244] The generated feedback is sent by the server to the user's device. The feedback is provided in text and graphics format and can be viewed by the user through a dedicated application. The user can use the received feedback as a reference and practice to improve upon the next time they play.

[0245] As a concrete example, consider the case where a user records their gameplay of "League of Legends" and uploads the video to a server. The server saves the video, decodes it, and then performs AI analysis. For example, it identifies that the user's positioning is behind at 5 minutes and 20 seconds, and generates improvement advice such as "Ensure your positioning in advance for the next battle." Furthermore, if the emotion engine detects that the user is nervous, it adds encouraging feedback such as "It's important to stay calm and understand the situation." This is then sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[0246] This system allows users to quickly and specifically identify problems in their playing and receive personalized feedback that takes emotional data into account, allowing users to efficiently improve their skills.

[0247] The processing flow will be explained below.

[0248] Step 1:

[0249] A user plays a game and records the gameplay using the in-game recording function or external software such as OBS Studio.

[0250] Step 2:

[0251] The user uploads the recorded video file to the server through a dedicated web page or application by attaching the video file to a form and clicking an upload button.

[0252] Step 3:

[0253] The server receives the upload request and saves the video file in the specified directory, for example, / path / to / save / .

[0254] Step 4:

[0255] The server decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the moviepy library is used to convert the video into frame-by-frame image data.

[0256] Step 5:

[0257] The server then sends the decoded data to the AI ​​analysis module. The converted data is then fed into the AI ​​model, which compares it with the play data of professional players. The AI ​​model analyzes the user's movements, strategies, item use, positioning, etc.

[0258] Step 6:

[0259] The server receives the analysis results from the AI ​​model and identifies issues with the user's play, such as "positioning is delayed for 5 minutes and 20 seconds."

[0260] Step 7:

[0261] The server collects the user's biometric data and provides it to the emotion engine. The biometric data includes heart rate, facial expressions, voice tone, etc. This data is collected in real time during gameplay.

[0262] Step 8:

[0263] The server generates more appropriate improvement measures based on the emotional data recognized by the emotion engine and adds it to the analysis results. For example, if the user is in a tense state, the advice would include, "It's important to stay calm and understand the situation."

[0264] Step 9:

[0265] The server generates feedback and sends it to the user's device, where it can be viewed in text or graphics format through a dedicated application.

[0266] Step 10:

[0267] The user checks the feedback on their device. Specifically, they open a dedicated smartphone app, read the received feedback, and use it as a reference the next time they play.

[0268] Step 11:

[0269] Based on the feedback, the user can reflect improvements in their next gameplay. They can practice based on specific advice and improve their skills. For example, in their next play, they can try specific measures such as "moving with an awareness of early positioning."

[0270] Example 2

[0271] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0272] As online games become more sophisticated, players need specific and effective feedback to improve their skills. However, traditional feedback methods only provide general advice and do not take into account individual players' playing styles or emotional states. This makes it difficult for players to find optimal ways to improve, resulting in delayed skill improvement.

[0273] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving and saving video data of a game played by a user, means for decoding the saved video data and converting it into a data format for analysis, means for performing play analysis using a generative model using the converted data and comparing it with play data from experts, means for identifying problems in the user's play based on the analysis results and generating improvement methods, means for collecting biometric information of the user and recognizing their emotional state, means for personalizing feedback content taking the recognized emotional state into consideration, and means for transmitting the generated feedback to the user's terminal so that the user can check the evaluation results. This allows the player to quickly identify specific areas for improvement and receive personalized feedback based on their emotional data.

[0274] "Video data" is digital data that records the screen of a game played by a user.

[0275] A "generative model" is an algorithm based on artificial intelligence that identifies specific patterns and features from data and generates new information.

[0276] "Expert play data" is data that records the play of highly experienced players, and typically indicates high technical ability and strategy.

[0277] "Biometric information" is data that indicates the user's physical condition, and includes heart rate, facial expression, voice tone, and the like.

[0278] "Emotional state" indicates the user's current emotion, and is information for determining the user's state of tension, relaxation, excitement, etc.

[0279] "Personalizing" means adapting to the characteristics and circumstances of a particular user, adjusting to individual needs and circumstances.

[0280] "Feedback" is information that includes an evaluation of the user's play and advice and instructions for improvement.

[0281] A "dedicated application" is a specific piece of software that a user uses to receive and review feedback.

[0282] "Decoding" is the process of converting encoded data into its original or parsable form.

[0283] "Analysis results" are evaluations and conclusions of data obtained through analysis using a generative model.

[0284] ---

[0285] The present invention relates to a support system for users to improve their gameplay skills, which provides personalized feedback using the user's emotional data. The system has functions for receiving, storing, and analyzing gameplay video data, generating feedback, and recognizing emotions.

[0286] First, the user records their gameplay using the in-game recording function or external software (e.g., OBS Studio), and the recorded video data is uploaded to a server via a dedicated webpage or application.

[0287] The server receives and saves the uploaded video data. The save location is set to a specific folder in the server's file system. For example, it is saved in the " / data / uploads" folder. Next, the saved video data is decoded and converted into a data format for analysis (frame-by-frame image data and audio data). This is done using the moviepy library.

[0288] The decoded data is sent to an AI analysis module on the server. The AI ​​analysis module uses a generative model to compare the user's play with the play data of professional players and identify areas for improvement. The play data of professional players is created based on play information from multiple experts that has been collected and analyzed in advance. The AI ​​analysis module analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between the user and professional players and areas for improvement.

[0289] Additionally, the system includes an emotion engine that collects biometric information (e.g., heart rate, facial expression, and vocal tone) from the user to recognize the user's emotional state. This allows the system to personalize feedback based on the emotional data. For example, if the user is nervous, the system may generate feedback suggesting calm measures, whereas if the user is relaxed, the system may generate feedback that is full of encouraging words.

[0290] The server generates feedback written in natural language based on the output of the AI ​​analysis module and the emotional data obtained from the emotion engine. The feedback is generated using a generative AI model, detailing the user's specific issues and how to improve them. The feedback is generated in text and graphics format and sent to the user's device via a dedicated application.

[0291] The user can then review the feedback they received using a dedicated app. The feedback includes specific advice for improvement and messages that take their emotions into consideration, allowing them to refer to it and practice the next time they play.

[0292] As a concrete example, consider the case where a user records a gameplay video of "a certain online game" and uploads the video data to a server. The server saves the video data, decodes it, and performs AI analysis. For example, it identifies that "positioning is behind at 5 minutes and 20 seconds" and generates specific improvement advice such as "ensure your positioning in advance for the next battle." Furthermore, if the emotion engine detects that the user is nervous, it adds encouraging feedback such as "It's important to stay calm and understand the situation." This is sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[0293] An example of an input prompt for the generative AI model is, "Generate feedback comparing the user with an expert player. If the user's positioning is behind at 5 minutes and 20 seconds, provide feedback including suggestions for improvement. Also, add content that takes into consideration the user's emotions if they are nervous."

[0294] The system allows users to quickly identify specific areas for improvement and receive personalized feedback based on emotional data, allowing them to improve their skills efficiently.

[0295] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0296] ---

[0297] Step 1: Recording user gameplay

[0298] The user starts playing a game and records the gameplay using the in-game recording function or external software such as OBS Studio. The specific input is the video and audio during gameplay, which is then output as video data (e.g., an MP4 file).

[0299] Step 2: Upload your recording

[0300] Users upload recorded video data to the server through a dedicated web page or application. The input is the video data file stored on the device, and the output is a file upload request sent to the server. Specifically, the user drags and drops the file into the web form and clicks the submit button.

[0301] Step 3: Storing video files on the server

[0302] The server receives the upload request and saves the video data in the specified directory. The input is the uploaded video data, and the output is the file path to the save destination. Specifically, the server stores the file in the " / data / uploads" folder in the server's file system and records the save path.

[0303] Step 4: The server decodes the video file

[0304] The server decodes the stored video data and converts it into a data format for analysis. The input is the stored video data file (MP4 file), and the output is image data and audio data on a frame-by-frame basis. Specifically, the server uses the moviepy library to convert the video file into image data for each frame.

[0305] Step 5: Running AI analysis on the server

[0306] The server supplies the decoded data to an AI analysis module. The input is image data and audio data for each frame, and the output is the results of gameplay analysis. The AI ​​model compares the expert's play data with the user's play and identifies issues with the user's play. Specific actions, such as position information, character movements, and skill usage timing, are analyzed using a generative model.

[0307] Step 6: Collecting and recognizing emotional data with the emotion engine

[0308] The server collects the user's biometric information and recognizes their emotional state. The input is biometric information from the user (e.g., heart rate, facial expression, and voice tone), and the output is the recognized emotional state. Specifically, the system collects data through wearable devices and webcams and analyzes emotions using algorithms.

[0309] Step 7: Server Generates Feedback

[0310] The server generates feedback written in natural language based on the analysis results output by the AI ​​and the emotional data recognized by the emotion engine. The input is the play analysis results and emotional data, and the output is feedback text. Specifically, a "prompt sentence" is input into the generative AI model, and a feedback sentence customized for each user is generated.

[0311] Step 8: Sending feedback to the user device

[0312] The server sends the generated feedback to the user's terminal. The input is the feedback text, and the output is the transmission result to the user's terminal. Specifically, the server sends the feedback to the user's terminal in text or graphics format via a dedicated application.

[0313] Step 9: Review and implement user feedback

[0314] The user checks the received feedback and practices, focusing on the areas for improvement for the next play. The input is the received feedback, and the output is what the user applies to the next play. Specifically, the user launches a dedicated application to view the feedback and implements the suggested improvements in the next play.

[0315] ---

[0316] (Application example 2)

[0317] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0318] Conventional gameplay analysis systems provide feedback on a user's gameplay, but do not provide personalized feedback that takes into account the user's emotional state. This makes it difficult to provide more effective and appropriate improvement advice based on the user's emotional state. Furthermore, there is a lack of a means to visually grasp the analysis results on a timeline, making it difficult for users to specifically understand when a problem occurred. Furthermore, there is a need for feedback to be provided in an appropriate format, such as text or graphics.

[0319] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0320] In this invention, the server includes means for receiving and saving video data of the game played by the user, means for decoding the saved video data and converting it into a data format for analysis, means for analyzing the play using an AI model using the converted data and comparing it with play data of professional players, means for identifying issues in the user's play based on the analysis results and generating methods for improvement, means for recognizing emotions based on the user's biometric data and reflecting them in the analysis results to personalize feedback, and means for transmitting the generated feedback to the user's device so that the user can check the evaluation results. This makes it possible to identify specific issues in the user's play and provide personalized feedback that takes into account the user's emotional state.

[0321] "Game video data" is a recording that includes video and audio of the game being played by the user.

[0322] "Decoding" is the process of converting saved video data into an analyzable format, such as frame-by-frame image data or audio data.

[0323] An "AI model" is an artificial intelligence algorithm that analyzes a user's play by comparing it with the play data of professional players and identifying areas of weakness.

[0324] "Professional player play data" refers to gameplay information of professional players that has been collected and analyzed in advance.

[0325] "Improvement methods" are specific advice and suggestions for improving issues in the user's play.

[0326] "User's biometric data" refers to physical and psychological information such as the user's heart rate, facial expression, and voice tone.

[0327] "Emotion recognition" is the process of determining a user's emotional state at any given time based on their biometric data.

[0328] "Feedback" refers to improvements and new suggested information provided based on the results of a user's gameplay analysis.

[0329] "Personalized feedback" refers to individually optimized improvement advice that takes into account the user's emotional state.

[0330] "Evaluation results" are information that includes the results of analysis of the user's gameplay and specific feedback based on the results.

[0331] A "dedicated application" is special software that users use to check analysis results and feedback.

[0332] This invention combines an emotion engine with a system that provides specific feedback to users comparing their gameplay with that of professional players to help them improve their gameplay skills. The system receives and stores video data of the game played by the user, decodes the data, converts it into a data format for analysis, performs comparative analysis using an AI model, and generates personalized feedback based on the evaluation results. The feedback is then sent to the user's device.

[0333] Hardware and software used

[0334] Hardware:

[0335] Smartphone

[0336] Dedicated Server

[0337] Biometric data collection devices (e.g., smartwatches)

[0338] software:

[0339] Video recording software (e.g. OBS Studio)

[0340] Video decoding library (e.g. moviepy)

[0341] Image analysis library (e.g. opencv)

[0342] Emotion Recognition Library (Custom Library: emotion_recognition)

[0343] Feedback generation library (custom library: feedback_generator)

[0344] Server communication library (custom library: server_communication)

[0345] Data processing and calculation flow

[0346] 1. A user uses a smartphone to record video of their gameplay and uploads the video data to a server. A biometric data collection device collects the user's heart rate and facial expression data, which are also uploaded to the server.

[0347] 2. The server decodes the uploaded video data using the moviepy library and converts it into frame-by-frame image data.

[0348] 3. The server uses emotion_recognition library to perform emotion recognition and analyzes the user's biometric data to recognize their emotional state.

[0349] 4. The server uses an AI analysis module to compare the playing data of professional players with the user's playing data and identify specific issues.

[0350] 5. The server generates personalized feedback using the feedback_generator library based on the identified issues and the recognized emotional data. Based on the analysis results, it displays areas for improvement linked to specific moments in the user's gameplay video.

[0351] 6. The server sends the generated feedback to the user's device using the server_communication library. The feedback is generated in the form of text and / or graphics and is displayed to the user via a dedicated application on the user's device.

[0352] Specific examples

[0353] Consider a scenario where a user records video of themselves playing a real-time strategy game, collects biometric data using a smartwatch, and uploads both sets of data to an app. The server receives the video and biometric data, performs AI analysis, and identifies issues such as "the player delayed the use of an item at 10 minutes and 30 seconds." Based on the user's recognized emotional tension, the app generates feedback such as "Use your item early in the next battle. It's important to relax and understand the situation."

[0354] Prompt Sentence Examples

[0355] I uploaded a gameplay video. Identify specific issues in the user's gameplay compared to professional players' play and generate personalized feedback based on the provided emotional data. Include encouraging words if the user is emotionally tense. Also, include specific advice pointing to a specific time in the video (e.g., 10 minutes 30 seconds).

[0356] In this way, users can upload their recorded gameplay videos and biometric data, which the server can then use AI to analyze and provide personalized feedback to the user's dedicated application.

[0357] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0358] Step 1:

[0359] The user's smartphone records gameplay video. When the user presses the record button through the app, video and audio of the gameplay are captured using video recording software (e.g., OBS Studio). The recorded video data is temporarily stored on the smartphone.

[0360] Input: Running gameplay

[0361] Output: Recorded gameplay video file

[0362] Step 2:

[0363] Users use the application to record gameplay videos and upload biometric data, including heart rate and facial expression data obtained from biometric data collection devices (e.g., smartwatches), to a server.

[0364] Input: Recorded gameplay video, biometric data

[0365] Output: Video files and biometric data uploaded to the server

[0366] Step 3:

[0367] The server receives and saves the uploaded gameplay video. The server stores the video file in a specified directory and prepares it for analysis.

[0368] Input: Uploaded video file, biometric data

[0369] Output: Video files and biometric data stored on the server

[0370] Step 4:

[0371] The server decodes the stored video files using the moviepy library. The video decoding process converts the video data into frame-by-frame image data and audio data.

[0372] Input: Video file stored on the server

[0373] Output: Image data and audio data for each frame

[0374] Step 5:

[0375] The server uses the emotion_recognition library to analyze the uploaded biometric data and recognize the user's emotions. Emotion recognition determines whether the user is tense or relaxed based on heart rate and facial expression data.

[0376] Input: Uploaded biometric data

[0377] Output: User's emotional state data

[0378] Step 6:

[0379] The server uses an AI analysis module to compare the decoded frame-by-frame image data and the user's play data with that of professional players to identify specific issues. This analysis evaluates the user's movements, strategy, item use, positioning, etc. and uses a generative AI model.

[0380] Input: Frame-by-frame image data, professional player play data

[0381] Output: Analysis results identifying problem areas

[0382] Step 7:

[0383] The server uses the feedback_generator library to generate personalized feedback based on the identified issues and emotional data. Based on the analysis results, it suggests areas for improvement by linking them to specific moments in the user's gameplay video, and provides appropriate feedback according to the user's emotional state.

[0384] Input: Analysis results identifying issues, user emotional state data

[0385] Output: Personalized feedback

[0386] Step 8:

[0387] The server sends the generated feedback to the user's terminal using the server_communication library. The feedback is generated in text and / or graphics format and is displayed to the user via a dedicated application on the user's terminal.

[0388] Input: Personalized Feedback

[0389] Output: Feedback displayed on the user's terminal

[0390] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0391] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0392] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0393] [Second embodiment]

[0394] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0395] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0396] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0397] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0398] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0399] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0400] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0401] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0402] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0403] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0404] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0405] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0406] The present invention is a system for providing specific feedback to users in comparison with the play of professional players to help them improve their gameplay skills.

[0407] First, the user records a video of their gameplay. This can be done using the game's in-game recording feature or external software (such as OBS Studio). The user then uploads the recorded video file to the server via a dedicated web page or application. The server receives this upload request and saves the video file in a specified directory.

[0408] The server then decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the server uses a library such as moviepy to convert the video file into image data for each frame.

[0409] The server then supplies the decoded data to an AI analysis module. The AI ​​model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players was built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between them and professional players.

[0410] Based on the analysis results output by the AI, the server uses natural language to describe the issues in the user's play and how to improve them. For example, it generates feedback such as, "Your positioning is delayed by 5 minutes and 20 seconds. The enemy team has already moved to the next objective, so you need to position yourself earlier."

[0411] The generated feedback is sent by the server to the user's device. The feedback is provided in text and / or graphics format and can be viewed by the user through a dedicated application. The user can use the received feedback as a reference and practice to improve upon the next time they play.

[0412] As a concrete example, consider the case where a user records their gameplay of "League of Legends" and uploads the video to a server. The server saves the video, decodes it, and then performs AI analysis. For example, it identifies that the player's positioning is behind at 5 minutes and 20 seconds, and generates improvement advice such as "ensure proper positioning in advance for the next battle." This advice is then sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[0413] This system allows users to quickly and specifically identify problems in their playing and improve their skills efficiently.

[0414] The processing flow will be explained below.

[0415] Step 1:

[0416] A user plays a game and records the gameplay using the in-game recording function or external software such as OBS Studio.

[0417] Step 2:

[0418] The user uploads the recorded video file to the server through a dedicated web page or application by attaching the video file to a form and clicking an upload button.

[0419] Step 3:

[0420] The server receives the upload request and saves the video file in the specified directory, for example, / path / to / save / .

[0421] Step 4:

[0422] The server decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the moviepy library is used to convert the video into frame-by-frame image data.

[0423] Step 5:

[0424] The server then sends the decoded data to the AI ​​analysis module. The converted data is then fed into the AI ​​model, which compares it with the play data of professional players. The AI ​​model analyzes the user's movements, strategies, item use, positioning, etc.

[0425] Step 6:

[0426] The server receives the analysis results from the AI ​​model and identifies issues with the user's play, such as "positioning is delayed by 5 minutes and 20 seconds."

[0427] Step 7:

[0428] Based on the analysis results, the server generates feedback in natural language describing areas for improvement and the reasons for them. For example, it may generate feedback such as, "You are 5 minutes and 20 seconds behind in positioning, so you need to secure your positioning in advance."

[0429] Step 8:

[0430] The server sends the generated feedback to the user's device, where it can be viewed through a dedicated application. A notification function can also be used to notify the user of the arrival of the feedback.

[0431] Step 9:

[0432] The user checks the feedback on their device, for example by opening a dedicated smartphone app, reading the received feedback, and using it as reference the next time they play.

[0433] Step 10:

[0434] Users can improve their gameplay based on feedback and practice using specific advice to improve their skills.

[0435] Example 1

[0436] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0437] Conventional gameplay analysis systems make it difficult for users to objectively evaluate their own play and identify specific areas for improvement. Furthermore, there is a lack of a way to compare users' play data with that of professional players and provide specific feedback. This has led to the issue of users being unable to improve their skills efficiently.

[0438] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0439] In this invention, the server includes means for receiving and storing data of electronic games played by users, means for decoding the stored data and converting it into a format for analysis, means for analyzing the converted data using an artificial intelligence model and comparing it with data from experts, means for identifying problems in the user's behavior based on the analysis results and generating improvement methods, means for transmitting the generated feedback to the user's communication device so that the user can check the evaluation results, and means for uploading video data via a dedicated web page or software application, thereby enabling users to objectively evaluate their play and receive real-time, specific feedback for making specific improvements.

[0440] "Electronic game" means a game played for competitive or recreational purposes using electronic devices.

[0441] "Decoding" refers to the process of converting data from one format to another.

[0442] An "artificial intelligence model" refers to a collection of algorithms for data analysis that are trained using techniques such as machine learning and deep learning.

[0443] "Specialized web page or software application" refers to a web interface or software program designed to accomplish a specific task.

[0444] "Communication equipment" refers to electronic devices for sending and receiving data, and specifically includes smartphones and computers.

[0445] "Feedback" refers to suggestions for improvement and evaluations provided based on the results of the analysis.

[0446] "Form for analysis" refers to a form in which data has been converted into a form that can be analyzed by an artificial intelligence model.

[0447] "Storage" refers to storing data in a location so that it can be accessed at a later time.

[0448] "Comparison" refers to the process of juxtaposing two or more data sets to assess their differences and similarities.

[0449] "Improvement methods" refer to specific measures to overcome specific issues and achieve better results.

[0450] The invention is a system that provides specific feedback to users in comparison with the play of professional players to help them improve their electronic game playing skills.

[0451] First, a user records their gameplay. This can be done using the game's in-game recording feature or external software (such as OBS Studio). The user then uploads the recorded video file to a server via a dedicated web page or software application. The server receives the upload request and saves the video file in a specified directory.

[0452] The server then decodes the stored video file. Decoding is the process of converting a video file into image data and audio data for each frame. For example, the moviepy library can be used to convert a video file into image data for each frame.

[0453] The server then supplies the decoded data to an AI analysis module. The AI ​​model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players is built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between the user and professional players.

[0454] Based on the analysis results output by the AI, the server uses natural language to describe the issues in the user's play and how to improve them. For example, it generates feedback such as, "Your positioning is delayed by 5 minutes and 20 seconds. The enemy team has already moved to the next objective, so you need to position yourself earlier."

[0455] The generated feedback is sent by the server to the user's device. The feedback is provided in text and / or graphics format and can be viewed by the user through a dedicated software application. The user can use the received feedback to improve and practice the next time they play.

[0456] As a concrete example, consider the case where a user records their play in an "electronic competitive game" and uploads the video to a server. In this case, the server saves the video, decodes it using the moviepy library, and provides the analysis to an AI analysis module. For example, it may identify that the user's positioning is behind at 5 minutes and 20 seconds, and generate feedback to the user, such as "Ensure that you are positioned well in advance for the next battle," and send it to the user via a dedicated application. The user can then take this advice and put it into practice in their next play, thereby improving their skills.

[0457] An example of a prompt to be input to a generative AI model might be, "Analyze a video of an electronic competitive game and note that the user's positioning is slow at the 5 minute 20 second mark. Advise the user to secure their positioning in advance in preparation for the next battle."

[0458] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0459] Step 1:

[0460] User-recorded gameplay and uploaded video files

[0461] Specific behavior:

[0462] Users record their gameplay using the in-game recording feature or external software such as OBS Studio, and then save the recorded video file, which they then upload through a dedicated webpage or software application.

[0463] input:

[0464] Recorded video file (e.g. gameplay_video.mp4)

[0465] output:

[0466] Video files uploaded to the server

[0467] Data processing:

[0468] Nothing in particular

[0469] Step 2:

[0470] Receiving and storing video files on the server

[0471] Specific behavior:

[0472] The server receives the upload request and saves the video file uploaded by the user in the specified directory (e.g., / uploads / ).

[0473] input:

[0474] User-uploaded video files

[0475] output:

[0476] Video files stored in a specified directory on the server

[0477] Data processing:

[0478] Nothing in particular

[0479] Step 3:

[0480] Video file decoding by the server

[0481] Specific behavior:

[0482] The server decodes the stored video file and converts it into frame-by-frame image data and audio data using the moviepy library.

[0483] input:

[0484] Saved video file (e.g. / uploads / gameplay_video.mp4)

[0485] output:

[0486] Frame-by-frame image and audio data

[0487] Data processing:

[0488] Split video files into frames and convert them into image and audio data

[0489] Step 4:

[0490] The server supplies the decoded data to the AI ​​analysis module

[0491] Specific behavior:

[0492] The server supplies the decoded data (frame-by-frame image data and audio data) to the AI ​​analysis module.

[0493] input:

[0494] Frame-by-frame image and audio data

[0495] output:

[0496] Analyzable datasets (e.g., batched frame data)

[0497] Data processing:

[0498] Batch-processing the frame data and preparing it for the AI ​​analysis module

[0499] Step 5:

[0500] Data comparison and analysis using AI analysis module

[0501] Specific behavior:

[0502] The AI ​​analysis module compares the user's play data with pre-trained expert data to identify problems.

[0503] input:

[0504] Batched frame data and expert play data

[0505] output:

[0506] Analysis results (e.g., issue list, difference data)

[0507] Data processing:

[0508] Compare user play data with expert data and extract quantified differences

[0509] Step 6:

[0510] Server generates feedback of analysis results

[0511] Specific behavior:

[0512] Based on the results of the AI ​​analysis, the server describes in natural language the issues in the user's gameplay and how to improve them.

[0513] input:

[0514] Analysis results (e.g., issue list, difference data)

[0515] output:

[0516] Feedback statement (e.g., "You're 5 minutes and 20 seconds late in positioning. The enemy team has already moved to the next objective, so you need to position yourself earlier.")

[0517] Data processing:

[0518] Based on the analysis results, feedback is generated using a natural language generation algorithm.

[0519] Step 7:

[0520] Server sends feedback

[0521] Specific behavior:

[0522] The generated feedback is sent by the server to the user's terminal, and is provided in text and / or graphics format.

[0523] input:

[0524] Feedback Statement

[0525] output:

[0526] Feedback sent to the user's device

[0527] Data processing:

[0528] Convert your feedback statement into text or graphics format and send it

[0529] Step 8:

[0530] Users review feedback and practice

[0531] Specific behavior:

[0532] Users receive feedback through a dedicated software application, which displays the feedback and uses it to improve their next practice.

[0533] input:

[0534] Feedback Statement

[0535] output:

[0536] User feedback review and application

[0537] Data processing:

[0538] Nothing in particular

[0539] Through the above steps, the system analyzes the user's play data and provides specific feedback to help the user improve their skills efficiently.

[0540] (Application example 1)

[0541] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0542] Conventional gameplay analysis systems have the drawback of making it difficult for users to obtain specific feedback to improve their gameplay. Furthermore, in many cases, feedback is not provided in real time, making it difficult for users to immediately identify areas for improvement and apply them to their next play. Furthermore, when viewers stream gameplay, there is no way to obtain analysis results in real time, making it difficult for them to receive live feedback and improve their play.

[0543] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0544] In this invention, the server includes means for receiving and saving video data of the game played by the user, means for decoding the saved video data and converting it into a data format for analysis, means for analyzing the play using a generative AI model using the converted data and comparing it with play data of professional players, means for identifying issues in the user's play based on the analysis results and creating ways to improve it, means for sending the generated feedback to the user's device so that the user can check the evaluation results, and means for an application to run on the viewer's device and display the feedback in real time. This allows the user to receive specific feedback in real time and improve their play on the spot.

[0545] 1. "Video data of a game played by a user" refers to a series of video data recorded from the game screen while the user is playing.

[0546] 2. "Means of receiving and storing" refers to the function for sending video data to the server and storing it.

[0547] 3. "Decoding" is the process of converting stored video data into an analyzable format.

[0548] 4. "Data format for analysis" means the specific data format required for an AI model to perform analysis.

[0549] 5. "Generative AI model" refers to an artificial intelligence model that compares a user's play with the play data of professional players and generates analytical results.

[0550] 6. "Professional player play data" refers to data on play records of multiple professional players collected in advance.

[0551] 7. "Issues" are specific points that need improvement in the user's play.

[0552] 8. "Improvement methods" are specific advice and techniques for improving issues.

[0553] 9. "Feedback" refers to the evaluation results and details of how to improve provided to the user based on the analysis results.

[0554] 10. "User's device" means a communication device used by a user, such as a personal computer or smartphone.

[0555] 11. "Means for checking evaluation results" refers to a function that allows users to check the feedback content on their devices.

[0556] 12. "The application runs on the viewer's terminal" refers to software that runs on the terminal used by the viewer.

[0557] 13. "Means for displaying feedback in real time" refers to a function that allows analysis results to be displayed instantly on the viewer's device during play.

[0558] This invention is a system for improving a user's game skills. The system analyzes videos of the user's gameplay and compares them with those of professional players to provide specific feedback.

[0559] A user records a video of their gameplay. This can be done using the in-game recording feature or external software (such as OBS Studio). The user then uploads the recorded video file to a server via a dedicated web page or application. The server receives this upload request and saves the video file in a specified directory.

[0560] The server decodes the stored video files. This decoding process converts the video files into an analyzable format, such as frame-by-frame image data and audio data. For example, the moviepy library is used to convert the video files into image data for each frame.

[0561] The server then supplies the decoded data to an AI analysis module. The generative AI model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players was built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between the user and professional players.

[0562] Based on the analysis results output by the AI, the server uses natural language to describe the issues in the user's play and how to improve them. For example, it generates feedback such as, "Your positioning is delayed by 5 minutes and 20 seconds. The opposing team has already moved to the next objective, so you need to position yourself earlier." The generated feedback is sent by the server to the user's device. The feedback is provided in text and graphics format and can be viewed by the user through a dedicated application.

[0563] This system can also be used by viewers during live streaming. When viewers upload their gameplay videos while streaming, the server decodes and analyzes them in real time, instantly generating feedback that is displayed on the viewer's device. This feature allows viewers to receive live feedback and improve their gameplay in real time.

[0564] As a concrete example, consider the case where a user records gameplay of a "fighting game" and uploads the video to a server. The server saves the video, decodes it, and then analyzes it using a generative AI model. For example, it identifies that "positioning is behind at 5 minutes and 20 seconds" and generates improvement advice such as "ensure positioning in advance in preparation for the next battle." This advice is then sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[0565] Example prompt for a generative AI model:

[0566] "Analyze the video of your gameplay and output feedback on the differences between your gameplay and that of a professional player. For example, please describe what movements are lagging at what points, along with suggestions for improvement."

[0567] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0568] Step 1:

[0569] Record video of the game you are playing using the in-game recording feature or external software (such as OBS Studio).

[0570] Input: User gameplay

[0571] Output: Video file

[0572] Step 2:

[0573] Users upload recorded video files to the server via a dedicated web page or application.

[0574] Input: Video file

[0575] Output: Video file saved on the server

[0576] Step 3:

[0577] The server receives the upload request and saves the video file in a specified directory.

[0578] Input: Uploaded video file

[0579] Output: Video file saved on the server

[0580] Step 4:

[0581] The server decodes the stored video files and converts them into a data format for analysis. This decoding process uses the moviepy library to convert the video files into frame-by-frame image data and audio data.

[0582] Input: Video file stored on the server

[0583] Output: Frame-by-frame image data and audio data

[0584] Step 5:

[0585] The server then feeds the decoded data into a generative AI model and compares it with the play data of professional players, which has been compiled in advance by collecting and analyzing the play of many professional players.

[0586] Input: Frame-by-frame image data and audio data

[0587] Output: Play analysis results

[0588] Step 6:

[0589] Based on the analysis results output by the generative AI model, the server describes in natural language the issues in the user's play and how to improve them.

[0590] Input: Play analysis results

[0591] Output: Written feedback in natural language

[0592] Step 7:

[0593] The generated feedback is sent by the server to the user's terminal.

[0594] Input: Written feedback in natural language

[0595] Output: Feedback sent to the user's device

[0596] Step 8:

[0597] The user's device displays the feedback via a dedicated application, which can be in text or graphics format.

[0598] Input: Feedback sent to the user's device

[0599] Output: Feedback displayed on the terminal

[0600] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0601] The present invention combines an emotion engine with a system that provides specific feedback to users to help them improve their gameplay skills by comparing their play with that of professional players, thereby utilizing the user's emotion data to provide more personalized feedback.

[0602] First, a user records a video of their gameplay using the game's in-game recording feature or external software such as OBS Studio. Then, the user uploads the recorded video file to a server via a dedicated webpage or application. The server receives the upload request and saves the video file in a specified directory.

[0603] The server then decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the server uses a library such as moviepy to convert the video file into image data for each frame.

[0604] The server then supplies the decoded data to an AI analysis module. The AI ​​model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players was built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between them and professional players.

[0605] Furthermore, the present invention uses an emotion engine that recognizes the user's emotions. The emotion engine collects the user's biometric data (e.g., heart rate, facial expression, voice tone, etc.) and recognizes the user's emotions based on that data. For example, it can determine whether the user is tense or relaxed.

[0606] Based on the analysis results output by the AI ​​and the emotional data recognized by the emotion engine, the server generates feedback in natural language describing issues with the user's play and how to improve them. By taking emotional data into consideration, it is possible to adjust the tone and content of the feedback to be more personalized. For example, if the user is relaxed, the server generates feedback that includes many encouraging words, and if the user is nervous, the server generates calm feedback that suggests solutions.

[0607] The generated feedback is sent by the server to the user's device. The feedback is provided in text and graphics format and can be viewed by the user through a dedicated application. The user can use the received feedback as a reference and practice to improve upon the next time they play.

[0608] As a concrete example, consider the case where a user records their gameplay of "League of Legends" and uploads the video to a server. The server saves the video, decodes it, and then performs AI analysis. For example, it identifies that the user's positioning is behind at 5 minutes and 20 seconds, and generates improvement advice such as "Ensure your positioning in advance for the next battle." Furthermore, if the emotion engine detects that the user is nervous, it adds encouraging feedback such as "It's important to stay calm and understand the situation." This is then sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[0609] This system allows users to quickly and specifically identify problems in their playing and receive personalized feedback that takes emotional data into account, allowing users to efficiently improve their skills.

[0610] The processing flow will be explained below.

[0611] Step 1:

[0612] A user plays a game and records the gameplay using the in-game recording function or external software such as OBS Studio.

[0613] Step 2:

[0614] The user uploads the recorded video file to the server through a dedicated web page or application by attaching the video file to a form and clicking an upload button.

[0615] Step 3:

[0616] The server receives the upload request and saves the video file in the specified directory, for example, / path / to / save / .

[0617] Step 4:

[0618] The server decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the moviepy library is used to convert the video into frame-by-frame image data.

[0619] Step 5:

[0620] The server then sends the decoded data to the AI ​​analysis module. The converted data is then fed into the AI ​​model, which compares it with the play data of professional players. The AI ​​model analyzes the user's movements, strategies, item use, positioning, etc.

[0621] Step 6:

[0622] The server receives the analysis results from the AI ​​model and identifies issues with the user's play, such as "positioning is delayed for 5 minutes and 20 seconds."

[0623] Step 7:

[0624] The server collects the user's biometric data and provides it to the emotion engine. The biometric data includes heart rate, facial expressions, voice tone, etc. This data is collected in real time during gameplay.

[0625] Step 8:

[0626] The server generates more appropriate improvement measures based on the emotional data recognized by the emotion engine and adds it to the analysis results. For example, if the user is in a tense state, the advice would include, "It's important to stay calm and understand the situation."

[0627] Step 9:

[0628] The server generates feedback and sends it to the user's device, where it can be viewed in text or graphics format through a dedicated application.

[0629] Step 10:

[0630] The user checks the feedback on their device. Specifically, they open a dedicated smartphone app, read the received feedback, and use it as a reference the next time they play.

[0631] Step 11:

[0632] Based on the feedback, the user can reflect improvements in their next gameplay. They can practice based on specific advice and improve their skills. For example, in their next play, they can try specific measures such as "moving with an awareness of early positioning."

[0633] Example 2

[0634] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0635] As online games become more sophisticated, players need specific and effective feedback to improve their skills. However, traditional feedback methods only provide general advice and do not take into account individual players' playing styles or emotional states. This makes it difficult for players to find optimal ways to improve, resulting in delayed skill improvement.

[0636] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving and saving video data of a game played by a user, means for decoding the saved video data and converting it into a data format for analysis, means for performing play analysis using a generative model using the converted data and comparing it with play data from experts, means for identifying problems in the user's play based on the analysis results and generating improvement methods, means for collecting biometric information of the user and recognizing their emotional state, means for personalizing feedback content taking the recognized emotional state into consideration, and means for transmitting the generated feedback to the user's terminal so that the user can check the evaluation results. This allows the player to quickly identify specific areas for improvement and receive personalized feedback based on their emotional data.

[0637] "Video data" is digital data that records the screen of a game played by a user.

[0638] A "generative model" is an algorithm based on artificial intelligence that identifies specific patterns and features from data and generates new information.

[0639] "Expert play data" is data that records the play of highly experienced players, and typically indicates high technical ability and strategy.

[0640] "Biometric information" is data that indicates the user's physical condition, and includes heart rate, facial expression, voice tone, and the like.

[0641] "Emotional state" indicates the user's current emotion, and is information for determining the user's state of tension, relaxation, excitement, etc.

[0642] "Personalizing" means adapting to the characteristics and circumstances of a particular user, adjusting to individual needs and circumstances.

[0643] "Feedback" is information that includes an evaluation of the user's play and advice and instructions for improvement.

[0644] A "dedicated application" is a specific piece of software that a user uses to receive and review feedback.

[0645] "Decoding" is the process of converting encoded data into its original or parsable form.

[0646] "Analysis results" are evaluations and conclusions of data obtained through analysis using a generative model.

[0647] ---

[0648] The present invention relates to a support system for users to improve their gameplay skills, which provides personalized feedback using the user's emotional data. The system has functions for receiving, storing, and analyzing gameplay video data, generating feedback, and recognizing emotions.

[0649] First, the user records their gameplay using the in-game recording function or external software (e.g., OBS Studio), and the recorded video data is uploaded to a server via a dedicated webpage or application.

[0650] The server receives and saves the uploaded video data. The save location is set to a specific folder in the server's file system. For example, it is saved in the " / data / uploads" folder. Next, the saved video data is decoded and converted into a data format for analysis (frame-by-frame image data and audio data). This is done using the moviepy library.

[0651] The decoded data is sent to an AI analysis module on the server. The AI ​​analysis module uses a generative model to compare the user's play with the play data of professional players and identify areas for improvement. The play data of professional players is created based on play information from multiple experts that has been collected and analyzed in advance. The AI ​​analysis module analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between the user and professional players and areas for improvement.

[0652] Additionally, the system includes an emotion engine that collects biometric information (e.g., heart rate, facial expression, and vocal tone) from the user to recognize the user's emotional state. This allows the system to personalize feedback based on the emotional data. For example, if the user is nervous, the system may generate feedback suggesting calm measures, whereas if the user is relaxed, the system may generate feedback that is full of encouraging words.

[0653] The server generates feedback written in natural language based on the output of the AI ​​analysis module and the emotional data obtained from the emotion engine. The feedback is generated using a generative AI model, detailing the user's specific issues and how to improve them. The feedback is generated in text and graphics format and sent to the user's device via a dedicated application.

[0654] The user can then review the feedback they received using a dedicated app. The feedback includes specific advice for improvement and messages that take their emotions into consideration, allowing them to refer to it and practice the next time they play.

[0655] As a concrete example, consider the case where a user records a gameplay video of "a certain online game" and uploads the video data to a server. The server saves the video data, decodes it, and performs AI analysis. For example, it identifies that "positioning is behind at 5 minutes and 20 seconds" and generates specific improvement advice such as "ensure your positioning in advance for the next battle." Furthermore, if the emotion engine detects that the user is nervous, it adds encouraging feedback such as "It's important to stay calm and understand the situation." This is sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[0656] An example of an input prompt for the generative AI model is, "Generate feedback comparing the user with an expert player. If the user's positioning is behind at 5 minutes and 20 seconds, provide feedback including suggestions for improvement. Also, add content that takes into consideration the user's emotions if they are nervous."

[0657] The system allows users to quickly identify specific areas for improvement and receive personalized feedback based on emotional data, allowing them to improve their skills efficiently.

[0658] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0659] ---

[0660] Step 1: Recording user gameplay

[0661] The user starts playing a game and records the gameplay using the in-game recording function or external software such as OBS Studio. The specific input is the video and audio during gameplay, which is then output as video data (e.g., an MP4 file).

[0662] Step 2: Upload your recording

[0663] Users upload recorded video data to the server through a dedicated web page or application. The input is the video data file stored on the device, and the output is a file upload request sent to the server. Specifically, the user drags and drops the file into the web form and clicks the submit button.

[0664] Step 3: Storing video files on the server

[0665] The server receives the upload request and saves the video data in the specified directory. The input is the uploaded video data, and the output is the file path to the save destination. Specifically, the server stores the file in the " / data / uploads" folder in the server's file system and records the save path.

[0666] Step 4: The server decodes the video file

[0667] The server decodes the stored video data and converts it into a data format for analysis. The input is the stored video data file (MP4 file), and the output is image data and audio data on a frame-by-frame basis. Specifically, the server uses the moviepy library to convert the video file into image data for each frame.

[0668] Step 5: Running AI analysis on the server

[0669] The server supplies the decoded data to an AI analysis module. The input is image data and audio data for each frame, and the output is the results of gameplay analysis. The AI ​​model compares the expert's play data with the user's play and identifies issues with the user's play. Specific actions, such as position information, character movements, and skill usage timing, are analyzed using a generative model.

[0670] Step 6: Collecting and recognizing emotional data with the emotion engine

[0671] The server collects the user's biometric information and recognizes their emotional state. The input is biometric information from the user (e.g., heart rate, facial expression, and voice tone), and the output is the recognized emotional state. Specifically, the system collects data through wearable devices and webcams and analyzes emotions using algorithms.

[0672] Step 7: Server Generates Feedback

[0673] The server generates feedback written in natural language based on the analysis results output by the AI ​​and the emotional data recognized by the emotion engine. The input is the play analysis results and emotional data, and the output is feedback text. Specifically, a "prompt sentence" is input into the generative AI model, and a feedback sentence customized for each user is generated.

[0674] Step 8: Sending feedback to the user device

[0675] The server sends the generated feedback to the user's terminal. The input is the feedback text, and the output is the transmission result to the user's terminal. Specifically, the server sends the feedback to the user's terminal in text or graphics format via a dedicated application.

[0676] Step 9: Review and implement user feedback

[0677] The user checks the received feedback and practices, focusing on the areas for improvement for the next play. The input is the received feedback, and the output is what the user applies to the next play. Specifically, the user launches a dedicated application to view the feedback and implements the suggested improvements in the next play.

[0678] ---

[0679] (Application example 2)

[0680] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0681] Conventional gameplay analysis systems provide feedback on a user's gameplay, but do not provide personalized feedback that takes into account the user's emotional state. This makes it difficult to provide more effective and appropriate improvement advice based on the user's emotional state. Furthermore, there is a lack of a means to visually grasp the analysis results on a timeline, making it difficult for users to specifically understand when a problem occurred. Furthermore, there is a need for feedback to be provided in an appropriate format, such as text or graphics.

[0682] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0683] In this invention, the server includes means for receiving and saving video data of the game played by the user, means for decoding the saved video data and converting it into a data format for analysis, means for analyzing the play using an AI model using the converted data and comparing it with play data of professional players, means for identifying issues in the user's play based on the analysis results and generating methods for improvement, means for recognizing emotions based on the user's biometric data and reflecting them in the analysis results to personalize feedback, and means for transmitting the generated feedback to the user's device so that the user can check the evaluation results. This makes it possible to identify specific issues in the user's play and provide personalized feedback that takes into account the user's emotional state.

[0684] "Game video data" is a recording that includes video and audio of the game being played by the user.

[0685] "Decoding" is the process of converting saved video data into an analyzable format, such as frame-by-frame image data or audio data.

[0686] An "AI model" is an artificial intelligence algorithm that analyzes a user's play by comparing it with the play data of professional players and identifying areas of weakness.

[0687] "Professional player play data" refers to gameplay information of professional players that has been collected and analyzed in advance.

[0688] "Improvement methods" are specific advice and suggestions for improving issues in the user's play.

[0689] "User's biometric data" refers to physical and psychological information such as the user's heart rate, facial expression, and voice tone.

[0690] "Emotion recognition" is the process of determining a user's emotional state at any given time based on their biometric data.

[0691] "Feedback" refers to improvements and new suggested information provided based on the results of a user's gameplay analysis.

[0692] "Personalized feedback" refers to individually optimized improvement advice that takes into account the user's emotional state.

[0693] "Evaluation results" are information that includes the results of analysis of the user's gameplay and specific feedback based on the results.

[0694] A "dedicated application" is special software that users use to check analysis results and feedback.

[0695] This invention combines an emotion engine with a system that provides specific feedback to users comparing their gameplay with that of professional players to help them improve their gameplay skills. The system receives and stores video data of the game played by the user, decodes the data, converts it into a data format for analysis, performs comparative analysis using an AI model, and generates personalized feedback based on the evaluation results. The feedback is then sent to the user's device.

[0696] Hardware and software used

[0697] Hardware:

[0698] Smartphone

[0699] Dedicated Server

[0700] Biometric data collection devices (e.g., smartwatches)

[0701] software:

[0702] Video recording software (e.g. OBS Studio)

[0703] Video decoding library (e.g. moviepy)

[0704] Image analysis library (e.g. opencv)

[0705] Emotion Recognition Library (Custom Library: emotion_recognition)

[0706] Feedback generation library (custom library: feedback_generator)

[0707] Server communication library (custom library: server_communication)

[0708] Data processing and calculation flow

[0709] 1. A user uses a smartphone to record video of their gameplay and uploads the video data to a server. A biometric data collection device collects the user's heart rate and facial expression data, which are also uploaded to the server.

[0710] 2. The server decodes the uploaded video data using the moviepy library and converts it into frame-by-frame image data.

[0711] 3. The server uses emotion_recognition library to perform emotion recognition and analyzes the user's biometric data to recognize their emotional state.

[0712] 4. The server uses an AI analysis module to compare the playing data of professional players with the user's playing data and identify specific issues.

[0713] 5. The server generates personalized feedback using the feedback_generator library based on the identified issues and the recognized emotional data. Based on the analysis results, it displays areas for improvement linked to specific moments in the user's gameplay video.

[0714] 6. The server sends the generated feedback to the user's device using the server_communication library. The feedback is generated in the form of text and / or graphics and is displayed to the user via a dedicated application on the user's device.

[0715] Specific examples

[0716] Consider a scenario where a user records video of themselves playing a real-time strategy game, collects biometric data using a smartwatch, and uploads both sets of data to an app. The server receives the video and biometric data, performs AI analysis, and identifies issues such as "the player delayed the use of an item at 10 minutes and 30 seconds." Based on the user's recognized emotional tension, the app generates feedback such as "Use your item early in the next battle. It's important to relax and understand the situation."

[0717] Prompt Sentence Examples

[0718] I uploaded a gameplay video. Identify specific issues in the user's gameplay compared to professional players' play and generate personalized feedback based on the provided emotional data. Include encouraging words if the user is emotionally tense. Also, include specific advice pointing to a specific time in the video (e.g., 10 minutes 30 seconds).

[0719] In this way, users can upload their recorded gameplay videos and biometric data, which the server can then use AI to analyze and provide personalized feedback to the user's dedicated application.

[0720] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0721] Step 1:

[0722] The user's smartphone records gameplay video. When the user presses the record button through the app, video and audio of the gameplay are captured using video recording software (e.g., OBS Studio). The recorded video data is temporarily stored on the smartphone.

[0723] Input: Running gameplay

[0724] Output: Recorded gameplay video file

[0725] Step 2:

[0726] Users use the application to record gameplay videos and upload biometric data, including heart rate and facial expression data obtained from biometric data collection devices (e.g., smartwatches), to a server.

[0727] Input: Recorded gameplay video, biometric data

[0728] Output: Video files and biometric data uploaded to the server

[0729] Step 3:

[0730] The server receives and saves the uploaded gameplay video. The server stores the video file in a specified directory and prepares it for analysis.

[0731] Input: Uploaded video file, biometric data

[0732] Output: Video files and biometric data stored on the server

[0733] Step 4:

[0734] The server decodes the stored video files using the moviepy library. The video decoding process converts the video data into frame-by-frame image data and audio data.

[0735] Input: Video file stored on the server

[0736] Output: Image data and audio data for each frame

[0737] Step 5:

[0738] The server uses the emotion_recognition library to analyze the uploaded biometric data and recognize the user's emotions. Emotion recognition determines whether the user is tense or relaxed based on heart rate and facial expression data.

[0739] Input: Uploaded biometric data

[0740] Output: User's emotional state data

[0741] Step 6:

[0742] The server uses an AI analysis module to compare the decoded frame-by-frame image data and the user's play data with that of professional players to identify specific issues. This analysis evaluates the user's movements, strategy, item use, positioning, etc. and uses a generative AI model.

[0743] Input: Frame-by-frame image data, professional player play data

[0744] Output: Analysis results identifying problem areas

[0745] Step 7:

[0746] The server uses the feedback_generator library to generate personalized feedback based on the identified issues and emotional data. Based on the analysis results, it suggests areas for improvement by linking them to specific moments in the user's gameplay video, and provides appropriate feedback according to the user's emotional state.

[0747] Input: Analysis results identifying issues, user emotional state data

[0748] Output: Personalized feedback

[0749] Step 8:

[0750] The server sends the generated feedback to the user's terminal using the server_communication library. The feedback is generated in text and / or graphics format and is displayed to the user via a dedicated application on the user's terminal.

[0751] Input: Personalized Feedback

[0752] Output: Feedback displayed on the user's terminal

[0753] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0754] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0755] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0756] [Third embodiment]

[0757] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0758] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0759] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0760] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0761] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0762] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0763] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0764] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0765] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0766] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0767] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0768] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0769] The present invention is a system for providing specific feedback to users in comparison with the play of professional players to help them improve their gameplay skills.

[0770] First, the user records a video of their gameplay. This can be done using the game's in-game recording feature or external software (such as OBS Studio). The user then uploads the recorded video file to the server via a dedicated web page or application. The server receives this upload request and saves the video file in a specified directory.

[0771] The server then decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the server uses a library such as moviepy to convert the video file into image data for each frame.

[0772] The server then supplies the decoded data to an AI analysis module. The AI ​​model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players was built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between them and professional players.

[0773] Based on the analysis results output by the AI, the server uses natural language to describe the issues in the user's play and how to improve them. For example, it generates feedback such as, "Your positioning is delayed by 5 minutes and 20 seconds. The enemy team has already moved to the next objective, so you need to position yourself earlier."

[0774] The generated feedback is sent by the server to the user's device. The feedback is provided in text and / or graphics format and can be viewed by the user through a dedicated application. The user can use the received feedback as a reference and practice to improve upon the next time they play.

[0775] As a concrete example, consider the case where a user records their gameplay of "League of Legends" and uploads the video to a server. The server saves the video, decodes it, and then performs AI analysis. For example, it identifies that the player's positioning is behind at 5 minutes and 20 seconds, and generates improvement advice such as "ensure proper positioning in advance for the next battle." This advice is then sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[0776] This system allows users to quickly and specifically identify problems in their playing and improve their skills efficiently.

[0777] The processing flow will be explained below.

[0778] Step 1:

[0779] A user plays a game and records the gameplay using the in-game recording function or external software such as OBS Studio.

[0780] Step 2:

[0781] The user uploads the recorded video file to the server through a dedicated web page or application by attaching the video file to a form and clicking an upload button.

[0782] Step 3:

[0783] The server receives the upload request and saves the video file in the specified directory, for example, / path / to / save / .

[0784] Step 4:

[0785] The server decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the moviepy library is used to convert the video into frame-by-frame image data.

[0786] Step 5:

[0787] The server then sends the decoded data to the AI ​​analysis module. The converted data is then fed into the AI ​​model, which compares it with the play data of professional players. The AI ​​model analyzes the user's movements, strategies, item use, positioning, etc.

[0788] Step 6:

[0789] The server receives the analysis results from the AI ​​model and identifies issues with the user's play, such as "positioning is delayed by 5 minutes and 20 seconds."

[0790] Step 7:

[0791] Based on the analysis results, the server generates feedback in natural language describing areas for improvement and the reasons for them. For example, it may generate feedback such as, "You are 5 minutes and 20 seconds behind in positioning, so you need to secure your positioning in advance."

[0792] Step 8:

[0793] The server sends the generated feedback to the user's device, where it can be viewed through a dedicated application. A notification function can also be used to notify the user of the arrival of the feedback.

[0794] Step 9:

[0795] The user checks the feedback on their device, for example by opening a dedicated smartphone app, reading the received feedback, and using it as reference the next time they play.

[0796] Step 10:

[0797] Users can improve their gameplay based on feedback and practice using specific advice to improve their skills.

[0798] Example 1

[0799] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0800] Conventional gameplay analysis systems make it difficult for users to objectively evaluate their own play and identify specific areas for improvement. Furthermore, there is a lack of a way to compare users' play data with that of professional players and provide specific feedback. This has led to the issue of users being unable to improve their skills efficiently.

[0801] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0802] In this invention, the server includes means for receiving and storing data of electronic games played by users, means for decoding the stored data and converting it into a format for analysis, means for analyzing the converted data using an artificial intelligence model and comparing it with data from experts, means for identifying problems in the user's behavior based on the analysis results and generating improvement methods, means for transmitting the generated feedback to the user's communication device so that the user can check the evaluation results, and means for uploading video data via a dedicated web page or software application, thereby enabling users to objectively evaluate their play and receive real-time, specific feedback for making specific improvements.

[0803] "Electronic game" means a game played for competitive or recreational purposes using electronic devices.

[0804] "Decoding" refers to the process of converting data from one format to another.

[0805] An "artificial intelligence model" refers to a collection of algorithms for data analysis that are trained using techniques such as machine learning and deep learning.

[0806] "Specialized web page or software application" refers to a web interface or software program designed to accomplish a specific task.

[0807] "Communication equipment" refers to electronic devices for sending and receiving data, and specifically includes smartphones and computers.

[0808] "Feedback" refers to suggestions for improvement and evaluations provided based on the results of the analysis.

[0809] "Form for analysis" refers to a form in which data has been converted into a form that can be analyzed by an artificial intelligence model.

[0810] "Storage" refers to storing data in a location so that it can be accessed at a later time.

[0811] "Comparison" refers to the process of juxtaposing two or more data sets to assess their differences and similarities.

[0812] "Improvement methods" refer to specific measures to overcome specific issues and achieve better results.

[0813] The invention is a system that provides specific feedback to users in comparison with the play of professional players to help them improve their electronic game playing skills.

[0814] First, a user records their gameplay. This can be done using the game's in-game recording feature or external software (such as OBS Studio). The user then uploads the recorded video file to a server via a dedicated web page or software application. The server receives the upload request and saves the video file in a specified directory.

[0815] The server then decodes the stored video file. Decoding is the process of converting a video file into image data and audio data for each frame. For example, the moviepy library can be used to convert a video file into image data for each frame.

[0816] The server then supplies the decoded data to an AI analysis module. The AI ​​model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players is built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between the user and professional players.

[0817] Based on the analysis results output by the AI, the server uses natural language to describe the issues in the user's play and how to improve them. For example, it generates feedback such as, "Your positioning is delayed by 5 minutes and 20 seconds. The enemy team has already moved to the next objective, so you need to position yourself earlier."

[0818] The generated feedback is sent by the server to the user's device. The feedback is provided in text and / or graphics format and can be viewed by the user through a dedicated software application. The user can use the received feedback to improve and practice the next time they play.

[0819] As a concrete example, consider the case where a user records their play in an "electronic competitive game" and uploads the video to a server. In this case, the server saves the video, decodes it using the moviepy library, and provides the analysis to an AI analysis module. For example, it may identify that the user's positioning is behind at 5 minutes and 20 seconds, and generate feedback to the user, such as "Ensure that you are positioned well in advance for the next battle," and send it to the user via a dedicated application. The user can then take this advice and put it into practice in their next play, thereby improving their skills.

[0820] An example of a prompt to be input to a generative AI model might be, "Analyze a video of an electronic competitive game and note that the user's positioning is slow at the 5 minute 20 second mark. Advise the user to secure their positioning in advance in preparation for the next battle."

[0821] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0822] Step 1:

[0823] User-recorded gameplay and uploaded video files

[0824] Specific behavior:

[0825] Users record their gameplay using the in-game recording feature or external software such as OBS Studio, and then save the recorded video file, which they then upload through a dedicated webpage or software application.

[0826] input:

[0827] Recorded video file (e.g. gameplay_video.mp4)

[0828] output:

[0829] Video files uploaded to the server

[0830] Data processing:

[0831] Nothing in particular

[0832] Step 2:

[0833] Receiving and storing video files on the server

[0834] Specific behavior:

[0835] The server receives the upload request and saves the video file uploaded by the user in the specified directory (e.g., / uploads / ).

[0836] input:

[0837] User-uploaded video files

[0838] output:

[0839] Video files stored in a specified directory on the server

[0840] Data processing:

[0841] Nothing in particular

[0842] Step 3:

[0843] Video file decoding by the server

[0844] Specific behavior:

[0845] The server decodes the stored video file and converts it into frame-by-frame image data and audio data using the moviepy library.

[0846] input:

[0847] Saved video file (e.g. / uploads / gameplay_video.mp4)

[0848] output:

[0849] Frame-by-frame image and audio data

[0850] Data processing:

[0851] Split video files into frames and convert them into image and audio data

[0852] Step 4:

[0853] The server supplies the decoded data to the AI ​​analysis module

[0854] Specific behavior:

[0855] The server supplies the decoded data (frame-by-frame image data and audio data) to the AI ​​analysis module.

[0856] input:

[0857] Frame-by-frame image and audio data

[0858] output:

[0859] Analyzable datasets (e.g., batched frame data)

[0860] Data processing:

[0861] Batch-processing the frame data and preparing it for the AI ​​analysis module

[0862] Step 5:

[0863] Data comparison and analysis using AI analysis module

[0864] Specific behavior:

[0865] The AI ​​analysis module compares the user's play data with pre-trained expert data to identify problems.

[0866] input:

[0867] Batched frame data and expert play data

[0868] output:

[0869] Analysis results (e.g., issue list, difference data)

[0870] Data processing:

[0871] Compare user play data with expert data and extract quantified differences

[0872] Step 6:

[0873] Server generates feedback of analysis results

[0874] Specific behavior:

[0875] Based on the results of the AI ​​analysis, the server describes in natural language the issues in the user's gameplay and how to improve them.

[0876] input:

[0877] Analysis results (e.g., issue list, difference data)

[0878] output:

[0879] Feedback statement (e.g., "You're 5 minutes and 20 seconds late in positioning. The enemy team has already moved to the next objective, so you need to position yourself earlier.")

[0880] Data processing:

[0881] Based on the analysis results, feedback is generated using a natural language generation algorithm.

[0882] Step 7:

[0883] Server sends feedback

[0884] Specific behavior:

[0885] The generated feedback is sent by the server to the user's terminal, and is provided in text and / or graphics format.

[0886] input:

[0887] Feedback Statement

[0888] output:

[0889] Feedback sent to the user's device

[0890] Data processing:

[0891] Convert your feedback statement into text or graphics format and send it

[0892] Step 8:

[0893] Users review feedback and practice

[0894] Specific behavior:

[0895] Users receive feedback through a dedicated software application, which displays the feedback and uses it to improve their next practice.

[0896] input:

[0897] Feedback Statement

[0898] output:

[0899] User feedback review and application

[0900] Data processing:

[0901] Nothing in particular

[0902] Through the above steps, the system analyzes the user's play data and provides specific feedback to help the user improve their skills efficiently.

[0903] (Application example 1)

[0904] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0905] Conventional gameplay analysis systems have the drawback of making it difficult for users to obtain specific feedback to improve their gameplay. Furthermore, in many cases, feedback is not provided in real time, making it difficult for users to immediately identify areas for improvement and apply them to their next play. Furthermore, when viewers stream gameplay, there is no way to obtain analysis results in real time, making it difficult for them to receive live feedback and improve their play.

[0906] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0907] In this invention, the server includes means for receiving and saving video data of the game played by the user, means for decoding the saved video data and converting it into a data format for analysis, means for analyzing the play using a generative AI model using the converted data and comparing it with play data of professional players, means for identifying issues in the user's play based on the analysis results and creating ways to improve it, means for sending the generated feedback to the user's device so that the user can check the evaluation results, and means for an application to run on the viewer's device and display the feedback in real time. This allows the user to receive specific feedback in real time and improve their play on the spot.

[0908] 1. "Video data of a game played by a user" refers to a series of video data recorded from the game screen while the user is playing.

[0909] 2. "Means of receiving and storing" refers to the function for sending video data to the server and storing it.

[0910] 3. "Decoding" is the process of converting stored video data into an analyzable format.

[0911] 4. "Data format for analysis" means the specific data format required for an AI model to perform analysis.

[0912] 5. "Generative AI model" refers to an artificial intelligence model that compares a user's play with the play data of professional players and generates analytical results.

[0913] 6. "Professional player play data" refers to data on play records of multiple professional players collected in advance.

[0914] 7. "Issues" are specific points that need improvement in the user's play.

[0915] 8. "Improvement methods" are specific advice and techniques for improving issues.

[0916] 9. "Feedback" refers to the evaluation results and details of how to improve provided to the user based on the analysis results.

[0917] 10. "User's device" means a communication device used by a user, such as a personal computer or smartphone.

[0918] 11. "Means for checking evaluation results" refers to a function that allows users to check the feedback content on their devices.

[0919] 12. "The application runs on the viewer's terminal" refers to software that runs on the terminal used by the viewer.

[0920] 13. "Means for displaying feedback in real time" refers to a function that allows analysis results to be displayed instantly on the viewer's device during play.

[0921] This invention is a system for improving a user's game skills. The system analyzes videos of the user's gameplay and compares them with those of professional players to provide specific feedback.

[0922] A user records a video of their gameplay. This can be done using the in-game recording feature or external software (such as OBS Studio). The user then uploads the recorded video file to a server via a dedicated web page or application. The server receives this upload request and saves the video file in a specified directory.

[0923] The server decodes the stored video files. This decoding process converts the video files into an analyzable format, such as frame-by-frame image data and audio data. For example, the moviepy library is used to convert the video files into image data for each frame.

[0924] The server then supplies the decoded data to an AI analysis module. The generative AI model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players was built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between the user and professional players.

[0925] Based on the analysis results output by the AI, the server uses natural language to describe the issues in the user's play and how to improve them. For example, it generates feedback such as, "Your positioning is delayed by 5 minutes and 20 seconds. The opposing team has already moved to the next objective, so you need to position yourself earlier." The generated feedback is sent by the server to the user's device. The feedback is provided in text and graphics format and can be viewed by the user through a dedicated application.

[0926] This system can also be used by viewers during live streaming. When viewers upload their gameplay videos while streaming, the server decodes and analyzes them in real time, instantly generating feedback that is displayed on the viewer's device. This feature allows viewers to receive live feedback and improve their gameplay in real time.

[0927] As a concrete example, consider the case where a user records gameplay of a "fighting game" and uploads the video to a server. The server saves the video, decodes it, and then analyzes it using a generative AI model. For example, it identifies that "positioning is behind at 5 minutes and 20 seconds" and generates improvement advice such as "ensure positioning in advance in preparation for the next battle." This advice is then sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[0928] Example prompt for a generative AI model:

[0929] "Analyze the video of your gameplay and output feedback on the differences between your gameplay and that of a professional player. For example, please describe what movements are lagging at what points, along with suggestions for improvement."

[0930] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0931] Step 1:

[0932] Record video of the game you are playing using the in-game recording feature or external software (such as OBS Studio).

[0933] Input: User gameplay

[0934] Output: Video file

[0935] Step 2:

[0936] Users upload recorded video files to the server via a dedicated web page or application.

[0937] Input: Video file

[0938] Output: Video file saved on the server

[0939] Step 3:

[0940] The server receives the upload request and saves the video file in a specified directory.

[0941] Input: Uploaded video file

[0942] Output: Video file saved on the server

[0943] Step 4:

[0944] The server decodes the stored video files and converts them into a data format for analysis. This decoding process uses the moviepy library to convert the video files into frame-by-frame image data and audio data.

[0945] Input: Video file stored on the server

[0946] Output: Frame-by-frame image data and audio data

[0947] Step 5:

[0948] The server then feeds the decoded data into a generative AI model and compares it with the play data of professional players, which has been compiled in advance by collecting and analyzing the play of many professional players.

[0949] Input: Frame-by-frame image data and audio data

[0950] Output: Play analysis results

[0951] Step 6:

[0952] Based on the analysis results output by the generative AI model, the server describes in natural language the issues in the user's play and how to improve them.

[0953] Input: Play analysis results

[0954] Output: Written feedback in natural language

[0955] Step 7:

[0956] The generated feedback is sent by the server to the user's terminal.

[0957] Input: Written feedback in natural language

[0958] Output: Feedback sent to the user's device

[0959] Step 8:

[0960] The user's device displays the feedback via a dedicated application, which can be in text or graphics format.

[0961] Input: Feedback sent to the user's device

[0962] Output: Feedback displayed on the terminal

[0963] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0964] The present invention combines an emotion engine with a system that provides specific feedback to users to help them improve their gameplay skills by comparing their play with that of professional players, thereby utilizing the user's emotion data to provide more personalized feedback.

[0965] First, a user records a video of their gameplay using the game's in-game recording feature or external software such as OBS Studio. Then, the user uploads the recorded video file to a server via a dedicated webpage or application. The server receives the upload request and saves the video file in a specified directory.

[0966] The server then decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the server uses a library such as moviepy to convert the video file into image data for each frame.

[0967] The server then supplies the decoded data to an AI analysis module. The AI ​​model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players was built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between them and professional players.

[0968] Furthermore, the present invention uses an emotion engine that recognizes the user's emotions. The emotion engine collects the user's biometric data (e.g., heart rate, facial expression, voice tone, etc.) and recognizes the user's emotions based on that data. For example, it can determine whether the user is tense or relaxed.

[0969] Based on the analysis results output by the AI ​​and the emotional data recognized by the emotion engine, the server generates feedback in natural language describing issues with the user's play and how to improve them. By taking emotional data into consideration, it is possible to adjust the tone and content of the feedback to be more personalized. For example, if the user is relaxed, the server generates feedback that includes many encouraging words, and if the user is nervous, the server generates calm feedback that suggests solutions.

[0970] The generated feedback is sent by the server to the user's device. The feedback is provided in text and graphics format and can be viewed by the user through a dedicated application. The user can use the received feedback as a reference and practice to improve upon the next time they play.

[0971] As a concrete example, consider the case where a user records their gameplay of "League of Legends" and uploads the video to a server. The server saves the video, decodes it, and then performs AI analysis. For example, it identifies that the user's positioning is behind at 5 minutes and 20 seconds, and generates improvement advice such as "Ensure your positioning in advance for the next battle." Furthermore, if the emotion engine detects that the user is nervous, it adds encouraging feedback such as "It's important to stay calm and understand the situation." This is then sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[0972] This system allows users to quickly and specifically identify problems in their playing and receive personalized feedback that takes emotional data into account, allowing users to efficiently improve their skills.

[0973] The processing flow will be explained below.

[0974] Step 1:

[0975] A user plays a game and records the gameplay using the in-game recording function or external software such as OBS Studio.

[0976] Step 2:

[0977] The user uploads the recorded video file to the server through a dedicated web page or application by attaching the video file to a form and clicking an upload button.

[0978] Step 3:

[0979] The server receives the upload request and saves the video file in the specified directory, for example, / path / to / save / .

[0980] Step 4:

[0981] The server decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the moviepy library is used to convert the video into frame-by-frame image data.

[0982] Step 5:

[0983] The server then sends the decoded data to the AI ​​analysis module. The converted data is then fed into the AI ​​model, which compares it with the play data of professional players. The AI ​​model analyzes the user's movements, strategies, item use, positioning, etc.

[0984] Step 6:

[0985] The server receives the analysis results from the AI ​​model and identifies issues with the user's play, such as "positioning is delayed for 5 minutes and 20 seconds."

[0986] Step 7:

[0987] The server collects the user's biometric data and provides it to the emotion engine. The biometric data includes heart rate, facial expressions, voice tone, etc. This data is collected in real time during gameplay.

[0988] Step 8:

[0989] The server generates more appropriate improvement measures based on the emotional data recognized by the emotion engine and adds it to the analysis results. For example, if the user is in a tense state, the advice would include, "It's important to stay calm and understand the situation."

[0990] Step 9:

[0991] The server generates feedback and sends it to the user's device, where it can be viewed in text or graphics format through a dedicated application.

[0992] Step 10:

[0993] The user checks the feedback on their device. Specifically, they open a dedicated smartphone app, read the received feedback, and use it as a reference the next time they play.

[0994] Step 11:

[0995] Based on the feedback, the user can reflect improvements in their next gameplay. They can practice based on specific advice and improve their skills. For example, in their next play, they can try specific measures such as "moving with an awareness of early positioning."

[0996] Example 2

[0997] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0998] As online games become more sophisticated, players need specific and effective feedback to improve their skills. However, traditional feedback methods only provide general advice and do not take into account individual players' playing styles or emotional states. This makes it difficult for players to find optimal ways to improve, resulting in delayed skill improvement.

[0999] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving and saving video data of a game played by a user, means for decoding the saved video data and converting it into a data format for analysis, means for performing play analysis using a generative model using the converted data and comparing it with play data from experts, means for identifying problems in the user's play based on the analysis results and generating improvement methods, means for collecting biometric information of the user and recognizing their emotional state, means for personalizing feedback content taking the recognized emotional state into consideration, and means for transmitting the generated feedback to the user's terminal so that the user can check the evaluation results. This allows the player to quickly identify specific areas for improvement and receive personalized feedback based on their emotional data.

[1000] "Video data" is digital data that records the screen of a game played by a user.

[1001] A "generative model" is an algorithm based on artificial intelligence that identifies specific patterns and features from data and generates new information.

[1002] "Expert play data" is data that records the play of highly experienced players, and typically indicates high technical ability and strategy.

[1003] "Biometric information" is data that indicates the user's physical condition, and includes heart rate, facial expression, voice tone, and the like.

[1004] "Emotional state" indicates the user's current emotion, and is information for determining the user's state of tension, relaxation, excitement, etc.

[1005] "Personalizing" means adapting to the characteristics and circumstances of a particular user, adjusting to individual needs and circumstances.

[1006] "Feedback" is information that includes an evaluation of the user's play and advice and instructions for improvement.

[1007] A "dedicated application" is a specific piece of software that a user uses to receive and review feedback.

[1008] "Decoding" is the process of converting encoded data into its original or parsable form.

[1009] "Analysis results" are evaluations and conclusions of data obtained through analysis using a generative model.

[1010] ---

[1011] The present invention relates to a support system for users to improve their gameplay skills, which provides personalized feedback using the user's emotional data. The system has functions for receiving, storing, and analyzing gameplay video data, generating feedback, and recognizing emotions.

[1012] First, the user records their gameplay using the in-game recording function or external software (e.g., OBS Studio), and the recorded video data is uploaded to a server via a dedicated webpage or application.

[1013] The server receives and saves the uploaded video data. The save location is set to a specific folder in the server's file system. For example, it is saved in the " / data / uploads" folder. Next, the saved video data is decoded and converted into a data format for analysis (frame-by-frame image data and audio data). This is done using the moviepy library.

[1014] The decoded data is sent to an AI analysis module on the server. The AI ​​analysis module uses a generative model to compare the user's play with the play data of professional players and identify areas for improvement. The play data of professional players is created based on play information from multiple experts that has been collected and analyzed in advance. The AI ​​analysis module analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between the user and professional players and areas for improvement.

[1015] Additionally, the system includes an emotion engine that collects biometric information (e.g., heart rate, facial expression, and vocal tone) from the user to recognize the user's emotional state. This allows the system to personalize feedback based on the emotional data. For example, if the user is nervous, the system may generate feedback suggesting calm measures, whereas if the user is relaxed, the system may generate feedback that is full of encouraging words.

[1016] The server generates feedback written in natural language based on the output of the AI ​​analysis module and the emotional data obtained from the emotion engine. The feedback is generated using a generative AI model, detailing the user's specific issues and how to improve them. The feedback is generated in text and graphics format and sent to the user's device via a dedicated application.

[1017] The user can then review the feedback they received using a dedicated app. The feedback includes specific advice for improvement and messages that take their emotions into consideration, allowing them to refer to it and practice the next time they play.

[1018] As a concrete example, consider the case where a user records a gameplay video of "a certain online game" and uploads the video data to a server. The server saves the video data, decodes it, and performs AI analysis. For example, it identifies that "positioning is behind at 5 minutes and 20 seconds" and generates specific improvement advice such as "ensure your positioning in advance for the next battle." Furthermore, if the emotion engine detects that the user is nervous, it adds encouraging feedback such as "It's important to stay calm and understand the situation." This is sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[1019] An example of an input prompt for the generative AI model is, "Generate feedback comparing the user with an expert player. If the user's positioning is behind at 5 minutes and 20 seconds, provide feedback including suggestions for improvement. Also, add content that takes into consideration the user's emotions if they are nervous."

[1020] The system allows users to quickly identify specific areas for improvement and receive personalized feedback based on emotional data, allowing them to improve their skills efficiently.

[1021] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1022] ---

[1023] Step 1: Recording user gameplay

[1024] The user starts playing a game and records the gameplay using the in-game recording function or external software such as OBS Studio. The specific input is the video and audio during gameplay, which is then output as video data (e.g., an MP4 file).

[1025] Step 2: Upload your recording

[1026] Users upload recorded video data to the server through a dedicated web page or application. The input is the video data file stored on the device, and the output is a file upload request sent to the server. Specifically, the user drags and drops the file into the web form and clicks the submit button.

[1027] Step 3: Storing video files on the server

[1028] The server receives the upload request and saves the video data in the specified directory. The input is the uploaded video data, and the output is the file path to the save destination. Specifically, the server stores the file in the " / data / uploads" folder in the server's file system and records the save path.

[1029] Step 4: The server decodes the video file

[1030] The server decodes the stored video data and converts it into a data format for analysis. The input is the stored video data file (MP4 file), and the output is image data and audio data on a frame-by-frame basis. Specifically, the server uses the moviepy library to convert the video file into image data for each frame.

[1031] Step 5: Running AI analysis on the server

[1032] The server supplies the decoded data to an AI analysis module. The input is image data and audio data for each frame, and the output is the results of gameplay analysis. The AI ​​model compares the expert's play data with the user's play and identifies issues with the user's play. Specific actions, such as position information, character movements, and skill usage timing, are analyzed using a generative model.

[1033] Step 6: Collecting and recognizing emotional data with the emotion engine

[1034] The server collects the user's biometric information and recognizes their emotional state. The input is biometric information from the user (e.g., heart rate, facial expression, and voice tone), and the output is the recognized emotional state. Specifically, the system collects data through wearable devices and webcams and analyzes emotions using algorithms.

[1035] Step 7: Server Generates Feedback

[1036] The server generates feedback written in natural language based on the analysis results output by the AI ​​and the emotional data recognized by the emotion engine. The input is the play analysis results and emotional data, and the output is feedback text. Specifically, a "prompt sentence" is input into the generative AI model, and a feedback sentence customized for each user is generated.

[1037] Step 8: Sending feedback to the user device

[1038] The server sends the generated feedback to the user's terminal. The input is the feedback text, and the output is the transmission result to the user's terminal. Specifically, the server sends the feedback to the user's terminal in text or graphics format via a dedicated application.

[1039] Step 9: Review and implement user feedback

[1040] The user checks the received feedback and practices, focusing on the areas for improvement for the next play. The input is the received feedback, and the output is what the user applies to the next play. Specifically, the user launches a dedicated application to view the feedback and implements the suggested improvements in the next play.

[1041] ---

[1042] (Application example 2)

[1043] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1044] Conventional gameplay analysis systems provide feedback on a user's gameplay, but do not provide personalized feedback that takes into account the user's emotional state. This makes it difficult to provide more effective and appropriate improvement advice based on the user's emotional state. Furthermore, there is a lack of a means to visually grasp the analysis results on a timeline, making it difficult for users to specifically understand when a problem occurred. Furthermore, there is a need for feedback to be provided in an appropriate format, such as text or graphics.

[1045] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1046] In this invention, the server includes means for receiving and saving video data of the game played by the user, means for decoding the saved video data and converting it into a data format for analysis, means for analyzing the play using an AI model using the converted data and comparing it with play data of professional players, means for identifying issues in the user's play based on the analysis results and generating methods for improvement, means for recognizing emotions based on the user's biometric data and reflecting them in the analysis results to personalize feedback, and means for transmitting the generated feedback to the user's device so that the user can check the evaluation results. This makes it possible to identify specific issues in the user's play and provide personalized feedback that takes into account the user's emotional state.

[1047] "Game video data" is a recording that includes video and audio of the game being played by the user.

[1048] "Decoding" is the process of converting saved video data into an analyzable format, such as frame-by-frame image data or audio data.

[1049] An "AI model" is an artificial intelligence algorithm that analyzes a user's play by comparing it with the play data of professional players and identifying areas of weakness.

[1050] "Professional player play data" refers to gameplay information of professional players that has been collected and analyzed in advance.

[1051] "Improvement methods" are specific advice and suggestions for improving issues in the user's play.

[1052] "User's biometric data" refers to physical and psychological information such as the user's heart rate, facial expression, and voice tone.

[1053] "Emotion recognition" is the process of determining a user's emotional state at any given time based on their biometric data.

[1054] "Feedback" refers to improvements and new suggested information provided based on the results of a user's gameplay analysis.

[1055] "Personalized feedback" refers to individually optimized improvement advice that takes into account the user's emotional state.

[1056] "Evaluation results" are information that includes the results of analysis of the user's gameplay and specific feedback based on the results.

[1057] A "dedicated application" is special software that users use to check analysis results and feedback.

[1058] This invention combines an emotion engine with a system that provides specific feedback to users comparing their gameplay with that of professional players to help them improve their gameplay skills. The system receives and stores video data of the game played by the user, decodes the data, converts it into a data format for analysis, performs comparative analysis using an AI model, and generates personalized feedback based on the evaluation results. The feedback is then sent to the user's device.

[1059] Hardware and software used

[1060] Hardware:

[1061] Smartphone

[1062] Dedicated Server

[1063] Biometric data collection devices (e.g., smartwatches)

[1064] software:

[1065] Video recording software (e.g. OBS Studio)

[1066] Video decoding library (e.g. moviepy)

[1067] Image analysis library (e.g. opencv)

[1068] Emotion Recognition Library (Custom Library: emotion_recognition)

[1069] Feedback generation library (custom library: feedback_generator)

[1070] Server communication library (custom library: server_communication)

[1071] Data processing and calculation flow

[1072] 1. A user uses a smartphone to record video of their gameplay and uploads the video data to a server. A biometric data collection device collects the user's heart rate and facial expression data, which are also uploaded to the server.

[1073] 2. The server decodes the uploaded video data using the moviepy library and converts it into frame-by-frame image data.

[1074] 3. The server uses emotion_recognition library to perform emotion recognition and analyzes the user's biometric data to recognize their emotional state.

[1075] 4. The server uses an AI analysis module to compare the playing data of professional players with the user's playing data and identify specific issues.

[1076] 5. The server generates personalized feedback using the feedback_generator library based on the identified issues and the recognized emotional data. Based on the analysis results, it displays areas for improvement linked to specific moments in the user's gameplay video.

[1077] 6. The server sends the generated feedback to the user's device using the server_communication library. The feedback is generated in the form of text and / or graphics and is displayed to the user via a dedicated application on the user's device.

[1078] Specific examples

[1079] Consider a scenario where a user records video of themselves playing a real-time strategy game, collects biometric data using a smartwatch, and uploads both sets of data to an app. The server receives the video and biometric data, performs AI analysis, and identifies issues such as "the player delayed the use of an item at 10 minutes and 30 seconds." Based on the user's recognized emotional tension, the app generates feedback such as "Use your item early in the next battle. It's important to relax and understand the situation."

[1080] Prompt Sentence Examples

[1081] I uploaded a gameplay video. Identify specific issues in the user's gameplay compared to professional players' play and generate personalized feedback based on the provided emotional data. Include encouraging words if the user is emotionally tense. Also, include specific advice pointing to a specific time in the video (e.g., 10 minutes 30 seconds).

[1082] In this way, users can upload their recorded gameplay videos and biometric data, which the server can then use AI to analyze and provide personalized feedback to the user's dedicated application.

[1083] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1084] Step 1:

[1085] The user's smartphone records gameplay video. When the user presses the record button through the app, video and audio of the gameplay are captured using video recording software (e.g., OBS Studio). The recorded video data is temporarily stored on the smartphone.

[1086] Input: Running gameplay

[1087] Output: Recorded gameplay video file

[1088] Step 2:

[1089] Users use the application to record gameplay videos and upload biometric data, including heart rate and facial expression data obtained from biometric data collection devices (e.g., smartwatches), to a server.

[1090] Input: Recorded gameplay video, biometric data

[1091] Output: Video files and biometric data uploaded to the server

[1092] Step 3:

[1093] The server receives and saves the uploaded gameplay video. The server stores the video file in a specified directory and prepares it for analysis.

[1094] Input: Uploaded video file, biometric data

[1095] Output: Video files and biometric data stored on the server

[1096] Step 4:

[1097] The server decodes the stored video files using the moviepy library. The video decoding process converts the video data into frame-by-frame image data and audio data.

[1098] Input: Video file stored on the server

[1099] Output: Image data and audio data for each frame

[1100] Step 5:

[1101] The server uses the emotion_recognition library to analyze the uploaded biometric data and recognize the user's emotions. Emotion recognition determines whether the user is tense or relaxed based on heart rate and facial expression data.

[1102] Input: Uploaded biometric data

[1103] Output: User's emotional state data

[1104] Step 6:

[1105] The server uses an AI analysis module to compare the decoded frame-by-frame image data and the user's play data with that of professional players to identify specific issues. This analysis evaluates the user's movements, strategy, item use, positioning, etc. and uses a generative AI model.

[1106] Input: Frame-by-frame image data, professional player play data

[1107] Output: Analysis results identifying problem areas

[1108] Step 7:

[1109] The server uses the feedback_generator library to generate personalized feedback based on the identified issues and emotional data. Based on the analysis results, it suggests areas for improvement by linking them to specific moments in the user's gameplay video, and provides appropriate feedback according to the user's emotional state.

[1110] Input: Analysis results identifying issues, user emotional state data

[1111] Output: Personalized feedback

[1112] Step 8:

[1113] The server sends the generated feedback to the user's terminal using the server_communication library. The feedback is generated in text and / or graphics format and is displayed to the user via a dedicated application on the user's terminal.

[1114] Input: Personalized Feedback

[1115] Output: Feedback displayed on the user's terminal

[1116] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1117] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1118] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1119] [Fourth embodiment]

[1120] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1121] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1122] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1123] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1124] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1125] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1126] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1127] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1128] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1129] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1130] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1131] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1132] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1133] The present invention is a system for providing specific feedback to users in comparison with the play of professional players to help them improve their gameplay skills.

[1134] First, the user records a video of their gameplay. This can be done using the game's in-game recording feature or external software (such as OBS Studio). The user then uploads the recorded video file to the server via a dedicated web page or application. The server receives this upload request and saves the video file in a specified directory.

[1135] The server then decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the server uses a library such as moviepy to convert the video file into image data for each frame.

[1136] The server then supplies the decoded data to an AI analysis module. The AI ​​model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players was built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between them and professional players.

[1137] Based on the analysis results output by the AI, the server uses natural language to describe the issues in the user's play and how to improve them. For example, it generates feedback such as, "Your positioning is delayed by 5 minutes and 20 seconds. The enemy team has already moved to the next objective, so you need to position yourself earlier."

[1138] The generated feedback is sent by the server to the user's device. The feedback is provided in text and / or graphics format and can be viewed by the user through a dedicated application. The user can use the received feedback as a reference and practice to improve upon the next time they play.

[1139] As a concrete example, consider the case where a user records their gameplay of "League of Legends" and uploads the video to a server. The server saves the video, decodes it, and then performs AI analysis. For example, it identifies that the player's positioning is behind at 5 minutes and 20 seconds, and generates improvement advice such as "ensure proper positioning in advance for the next battle." This advice is then sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[1140] This system allows users to quickly and specifically identify problems in their playing and improve their skills efficiently.

[1141] The processing flow will be explained below.

[1142] Step 1:

[1143] A user plays a game and records the gameplay using the in-game recording function or external software such as OBS Studio.

[1144] Step 2:

[1145] The user uploads the recorded video file to the server through a dedicated web page or application by attaching the video file to a form and clicking an upload button.

[1146] Step 3:

[1147] The server receives the upload request and saves the video file in the specified directory, for example, / path / to / save / .

[1148] Step 4:

[1149] The server decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the moviepy library is used to convert the video into frame-by-frame image data.

[1150] Step 5:

[1151] The server then sends the decoded data to the AI ​​analysis module. The converted data is then fed into the AI ​​model, which compares it with the play data of professional players. The AI ​​model analyzes the user's movements, strategies, item use, positioning, etc.

[1152] Step 6:

[1153] The server receives the analysis results from the AI ​​model and identifies issues with the user's play, such as "positioning is delayed by 5 minutes and 20 seconds."

[1154] Step 7:

[1155] Based on the analysis results, the server generates feedback in natural language describing areas for improvement and the reasons for them. For example, it may generate feedback such as, "You are 5 minutes and 20 seconds behind in positioning, so you need to secure your positioning in advance."

[1156] Step 8:

[1157] The server sends the generated feedback to the user's device, where it can be viewed through a dedicated application. A notification function can also be used to notify the user of the arrival of the feedback.

[1158] Step 9:

[1159] The user checks the feedback on their device, for example by opening a dedicated smartphone app, reading the received feedback, and using it as reference the next time they play.

[1160] Step 10:

[1161] Users can improve their gameplay based on feedback and practice using specific advice to improve their skills.

[1162] Example 1

[1163] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1164] Conventional gameplay analysis systems make it difficult for users to objectively evaluate their own play and identify specific areas for improvement. Furthermore, there is a lack of a way to compare users' play data with that of professional players and provide specific feedback. This has led to the issue of users being unable to improve their skills efficiently.

[1165] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1166] In this invention, the server includes means for receiving and storing data of electronic games played by users, means for decoding the stored data and converting it into a format for analysis, means for analyzing the converted data using an artificial intelligence model and comparing it with data from experts, means for identifying problems in the user's behavior based on the analysis results and generating improvement methods, means for transmitting the generated feedback to the user's communication device so that the user can check the evaluation results, and means for uploading video data via a dedicated web page or software application, thereby enabling users to objectively evaluate their play and receive real-time, specific feedback for making specific improvements.

[1167] "Electronic game" means a game played for competitive or recreational purposes using electronic devices.

[1168] "Decoding" refers to the process of converting data from one format to another.

[1169] An "artificial intelligence model" refers to a collection of algorithms for data analysis that are trained using techniques such as machine learning and deep learning.

[1170] "Specialized web page or software application" refers to a web interface or software program designed to accomplish a specific task.

[1171] "Communication equipment" refers to electronic devices for sending and receiving data, and specifically includes smartphones and computers.

[1172] "Feedback" refers to suggestions for improvement and evaluations provided based on the results of the analysis.

[1173] "Form for analysis" refers to a form in which data has been converted into a form that can be analyzed by an artificial intelligence model.

[1174] "Storage" refers to storing data in a location so that it can be accessed at a later time.

[1175] "Comparison" refers to the process of juxtaposing two or more data sets to assess their differences and similarities.

[1176] "Improvement methods" refer to specific measures to overcome specific issues and achieve better results.

[1177] The invention is a system that provides specific feedback to users in comparison with the play of professional players to help them improve their electronic game playing skills.

[1178] First, a user records their gameplay. This can be done using the game's in-game recording feature or external software (such as OBS Studio). The user then uploads the recorded video file to a server via a dedicated web page or software application. The server receives the upload request and saves the video file in a specified directory.

[1179] The server then decodes the stored video file. Decoding is the process of converting a video file into image data and audio data for each frame. For example, the moviepy library can be used to convert a video file into image data for each frame.

[1180] The server then supplies the decoded data to an AI analysis module. The AI ​​model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players is built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between the user and professional players.

[1181] Based on the analysis results output by the AI, the server uses natural language to describe the issues in the user's play and how to improve them. For example, it generates feedback such as, "Your positioning is delayed by 5 minutes and 20 seconds. The enemy team has already moved to the next objective, so you need to position yourself earlier."

[1182] The generated feedback is sent by the server to the user's device. The feedback is provided in text and / or graphics format and can be viewed by the user through a dedicated software application. The user can use the received feedback to improve and practice the next time they play.

[1183] As a concrete example, consider the case where a user records their play in an "electronic competitive game" and uploads the video to a server. In this case, the server saves the video, decodes it using the moviepy library, and provides the analysis to an AI analysis module. For example, it may identify that the user's positioning is behind at 5 minutes and 20 seconds, and generate feedback to the user, such as "Ensure that you are positioned well in advance for the next battle," and send it to the user via a dedicated application. The user can then take this advice and put it into practice in their next play, thereby improving their skills.

[1184] An example of a prompt to be input to a generative AI model might be, "Analyze a video of an electronic competitive game and note that the user's positioning is slow at the 5 minute 20 second mark. Advise the user to secure their positioning in advance in preparation for the next battle."

[1185] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1186] Step 1:

[1187] User-recorded gameplay and uploaded video files

[1188] Specific behavior:

[1189] Users record their gameplay using the in-game recording feature or external software such as OBS Studio, and then save the recorded video file, which they then upload through a dedicated webpage or software application.

[1190] input:

[1191] Recorded video file (e.g. gameplay_video.mp4)

[1192] output:

[1193] Video files uploaded to the server

[1194] Data processing:

[1195] Nothing in particular

[1196] Step 2:

[1197] Receiving and storing video files on the server

[1198] Specific behavior:

[1199] The server receives the upload request and saves the video file uploaded by the user in the specified directory (e.g., / uploads / ).

[1200] input:

[1201] User-uploaded video files

[1202] output:

[1203] Video files stored in a specified directory on the server

[1204] Data processing:

[1205] Nothing in particular

[1206] Step 3:

[1207] Video file decoding by the server

[1208] Specific behavior:

[1209] The server decodes the stored video file and converts it into frame-by-frame image data and audio data using the moviepy library.

[1210] input:

[1211] Saved video file (e.g. / uploads / gameplay_video.mp4)

[1212] output:

[1213] Frame-by-frame image and audio data

[1214] Data processing:

[1215] Split video files into frames and convert them into image and audio data

[1216] Step 4:

[1217] The server supplies the decoded data to the AI ​​analysis module

[1218] Specific behavior:

[1219] The server supplies the decoded data (frame-by-frame image data and audio data) to the AI ​​analysis module.

[1220] input:

[1221] Frame-by-frame image and audio data

[1222] output:

[1223] Analyzable datasets (e.g., batched frame data)

[1224] Data processing:

[1225] Batch-processing the frame data and preparing it for the AI ​​analysis module

[1226] Step 5:

[1227] Data comparison and analysis using AI analysis module

[1228] Specific behavior:

[1229] The AI ​​analysis module compares the user's play data with pre-trained expert data to identify problems.

[1230] input:

[1231] Batched frame data and expert play data

[1232] output:

[1233] Analysis results (e.g., issue list, difference data)

[1234] Data processing:

[1235] Compare user play data with expert data and extract quantified differences

[1236] Step 6:

[1237] Server generates feedback of analysis results

[1238] Specific behavior:

[1239] Based on the results of the AI ​​analysis, the server describes in natural language the issues in the user's gameplay and how to improve them.

[1240] input:

[1241] Analysis results (e.g., issue list, difference data)

[1242] output:

[1243] Feedback statement (e.g., "You're 5 minutes and 20 seconds late in positioning. The enemy team has already moved to the next objective, so you need to position yourself earlier.")

[1244] Data processing:

[1245] Based on the analysis results, feedback is generated using a natural language generation algorithm.

[1246] Step 7:

[1247] Server sends feedback

[1248] Specific behavior:

[1249] The generated feedback is sent by the server to the user's terminal, and is provided in text and / or graphics format.

[1250] input:

[1251] Feedback Statement

[1252] output:

[1253] Feedback sent to the user's device

[1254] Data processing:

[1255] Convert your feedback statement into text or graphics format and send it

[1256] Step 8:

[1257] Users review feedback and practice

[1258] Specific behavior:

[1259] Users receive feedback through a dedicated software application, which displays the feedback and uses it to improve their next practice.

[1260] input:

[1261] Feedback Statement

[1262] output:

[1263] User feedback review and application

[1264] Data processing:

[1265] Nothing in particular

[1266] Through the above steps, the system analyzes the user's play data and provides specific feedback to help the user improve their skills efficiently.

[1267] (Application example 1)

[1268] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1269] Conventional gameplay analysis systems have the drawback of making it difficult for users to obtain specific feedback to improve their gameplay. Furthermore, in many cases, feedback is not provided in real time, making it difficult for users to immediately identify areas for improvement and apply them to their next play. Furthermore, when viewers stream gameplay, there is no way to obtain analysis results in real time, making it difficult for them to receive live feedback and improve their play.

[1270] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1271] In this invention, the server includes means for receiving and saving video data of the game played by the user, means for decoding the saved video data and converting it into a data format for analysis, means for analyzing the play using a generative AI model using the converted data and comparing it with play data of professional players, means for identifying issues in the user's play based on the analysis results and creating ways to improve it, means for sending the generated feedback to the user's device so that the user can check the evaluation results, and means for an application to run on the viewer's device and display the feedback in real time. This allows the user to receive specific feedback in real time and improve their play on the spot.

[1272] 1. "Video data of a game played by a user" refers to a series of video data recorded from the game screen while the user is playing.

[1273] 2. "Means of receiving and storing" refers to the function for sending video data to the server and storing it.

[1274] 3. "Decoding" is the process of converting stored video data into an analyzable format.

[1275] 4. "Data format for analysis" means the specific data format required for an AI model to perform analysis.

[1276] 5. "Generative AI model" refers to an artificial intelligence model that compares a user's play with the play data of professional players and generates analytical results.

[1277] 6. "Professional player play data" refers to data on play records of multiple professional players collected in advance.

[1278] 7. "Issues" are specific points that need improvement in the user's play.

[1279] 8. "Improvement methods" are specific advice and techniques for improving issues.

[1280] 9. "Feedback" refers to the evaluation results and details of how to improve provided to the user based on the analysis results.

[1281] 10. "User's device" means a communication device used by a user, such as a personal computer or smartphone.

[1282] 11. "Means for checking evaluation results" refers to a function that allows users to check the feedback content on their devices.

[1283] 12. "The application runs on the viewer's terminal" refers to software that runs on the terminal used by the viewer.

[1284] 13. "Means for displaying feedback in real time" refers to a function that allows analysis results to be displayed instantly on the viewer's device during play.

[1285] This invention is a system for improving a user's game skills. The system analyzes videos of the user's gameplay and compares them with those of professional players to provide specific feedback.

[1286] A user records a video of their gameplay. This can be done using the in-game recording feature or external software (such as OBS Studio). The user then uploads the recorded video file to a server via a dedicated web page or application. The server receives this upload request and saves the video file in a specified directory.

[1287] The server decodes the stored video files. This decoding process converts the video files into an analyzable format, such as frame-by-frame image data and audio data. For example, the moviepy library is used to convert the video files into image data for each frame.

[1288] The server then supplies the decoded data to an AI analysis module. The generative AI model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players was built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between the user and professional players.

[1289] Based on the analysis results output by the AI, the server uses natural language to describe the issues in the user's play and how to improve them. For example, it generates feedback such as, "Your positioning is delayed by 5 minutes and 20 seconds. The opposing team has already moved to the next objective, so you need to position yourself earlier." The generated feedback is sent by the server to the user's device. The feedback is provided in text and graphics format and can be viewed by the user through a dedicated application.

[1290] This system can also be used by viewers during live streaming. When viewers upload their gameplay videos while streaming, the server decodes and analyzes them in real time, instantly generating feedback that is displayed on the viewer's device. This feature allows viewers to receive live feedback and improve their gameplay in real time.

[1291] As a concrete example, consider the case where a user records gameplay of a "fighting game" and uploads the video to a server. The server saves the video, decodes it, and then analyzes it using a generative AI model. For example, it identifies that "positioning is behind at 5 minutes and 20 seconds" and generates improvement advice such as "ensure positioning in advance in preparation for the next battle." This advice is then sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[1292] Example prompt for a generative AI model:

[1293] "Analyze the video of your gameplay and output feedback on the differences between your gameplay and that of a professional player. For example, please describe what movements are lagging at what points, along with suggestions for improvement."

[1294] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1295] Step 1:

[1296] Record video of the game you are playing using the in-game recording feature or external software (such as OBS Studio).

[1297] Input: User gameplay

[1298] Output: Video file

[1299] Step 2:

[1300] Users upload recorded video files to the server via a dedicated web page or application.

[1301] Input: Video file

[1302] Output: Video file saved on the server

[1303] Step 3:

[1304] The server receives the upload request and saves the video file in a specified directory.

[1305] Input: Uploaded video file

[1306] Output: Video file saved on the server

[1307] Step 4:

[1308] The server decodes the stored video files and converts them into a data format for analysis. This decoding process uses the moviepy library to convert the video files into frame-by-frame image data and audio data.

[1309] Input: Video file stored on the server

[1310] Output: Frame-by-frame image data and audio data

[1311] Step 5:

[1312] The server then feeds the decoded data into a generative AI model and compares it with the play data of professional players, which has been compiled in advance by collecting and analyzing the play of many professional players.

[1313] Input: Frame-by-frame image data and audio data

[1314] Output: Play analysis results

[1315] Step 6:

[1316] Based on the analysis results output by the generative AI model, the server describes in natural language the issues in the user's play and how to improve them.

[1317] Input: Play analysis results

[1318] Output: Written feedback in natural language

[1319] Step 7:

[1320] The generated feedback is sent by the server to the user's terminal.

[1321] Input: Written feedback in natural language

[1322] Output: Feedback sent to the user's device

[1323] Step 8:

[1324] The user's device displays the feedback via a dedicated application, which can be in text or graphics format.

[1325] Input: Feedback sent to the user's device

[1326] Output: Feedback displayed on the terminal

[1327] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1328] The present invention combines an emotion engine with a system that provides specific feedback to users to help them improve their gameplay skills by comparing their play with that of professional players, thereby utilizing the user's emotion data to provide more personalized feedback.

[1329] First, a user records a video of their gameplay using the game's in-game recording feature or external software such as OBS Studio. Then, the user uploads the recorded video file to a server via a dedicated webpage or application. The server receives the upload request and saves the video file in a specified directory.

[1330] The server then decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the server uses a library such as moviepy to convert the video file into image data for each frame.

[1331] The server then supplies the decoded data to an AI analysis module. The AI ​​model compares the user's play with the play data of professional players and identifies issues in the user's play. The play data of professional players was built in advance by collecting and analyzing the play of many professional players. The AI ​​analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between them and professional players.

[1332] Furthermore, the present invention uses an emotion engine that recognizes the user's emotions. The emotion engine collects the user's biometric data (e.g., heart rate, facial expression, voice tone, etc.) and recognizes the user's emotions based on that data. For example, it can determine whether the user is tense or relaxed.

[1333] Based on the analysis results output by the AI ​​and the emotional data recognized by the emotion engine, the server generates feedback in natural language describing issues with the user's play and how to improve them. By taking emotional data into consideration, it is possible to adjust the tone and content of the feedback to be more personalized. For example, if the user is relaxed, the server generates feedback that includes many encouraging words, and if the user is nervous, the server generates calm feedback that suggests solutions.

[1334] The generated feedback is sent by the server to the user's device. The feedback is provided in text and graphics format and can be viewed by the user through a dedicated application. The user can use the received feedback as a reference and practice to improve upon the next time they play.

[1335] As a concrete example, consider the case where a user records their gameplay of "League of Legends" and uploads the video to a server. The server saves the video, decodes it, and then performs AI analysis. For example, it identifies that the user's positioning is behind at 5 minutes and 20 seconds, and generates improvement advice such as "Ensure your positioning in advance for the next battle." Furthermore, if the emotion engine detects that the user is nervous, it adds encouraging feedback such as "It's important to stay calm and understand the situation." This is then sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[1336] This system allows users to quickly and specifically identify problems in their playing and receive personalized feedback that takes emotional data into account, allowing users to efficiently improve their skills.

[1337] The processing flow will be explained below.

[1338] Step 1:

[1339] A user plays a game and records the gameplay using the in-game recording function or external software such as OBS Studio.

[1340] Step 2:

[1341] The user uploads the recorded video file to the server through a dedicated web page or application by attaching the video file to a form and clicking an upload button.

[1342] Step 3:

[1343] The server receives the upload request and saves the video file in the specified directory, for example, / path / to / save / .

[1344] Step 4:

[1345] The server decodes the stored video file. Decoding is the process of converting the video file into an analyzable format, such as frame-by-frame image data and audio data. For example, the moviepy library is used to convert the video into frame-by-frame image data.

[1346] Step 5:

[1347] The server then sends the decoded data to the AI ​​analysis module. The converted data is then fed into the AI ​​model, which compares it with the play data of professional players. The AI ​​model analyzes the user's movements, strategies, item use, positioning, etc.

[1348] Step 6:

[1349] The server receives the analysis results from the AI ​​model and identifies issues with the user's play, such as "positioning is delayed for 5 minutes and 20 seconds."

[1350] Step 7:

[1351] The server collects the user's biometric data and provides it to the emotion engine. The biometric data includes heart rate, facial expressions, voice tone, etc. This data is collected in real time during gameplay.

[1352] Step 8:

[1353] The server generates more appropriate improvement measures based on the emotional data recognized by the emotion engine and adds it to the analysis results. For example, if the user is in a tense state, the advice would include, "It's important to stay calm and understand the situation."

[1354] Step 9:

[1355] The server generates feedback and sends it to the user's device, where it can be viewed in text or graphics format through a dedicated application.

[1356] Step 10:

[1357] The user checks the feedback on their device. Specifically, they open a dedicated smartphone app, read the received feedback, and use it as a reference the next time they play.

[1358] Step 11:

[1359] Based on the feedback, the user can reflect improvements in their next gameplay. They can practice based on specific advice and improve their skills. For example, in their next play, they can try specific measures such as "moving with an awareness of early positioning."

[1360] Example 2

[1361] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1362] As online games become more sophisticated, players need specific and effective feedback to improve their skills. However, traditional feedback methods only provide general advice and do not take into account individual players' playing styles or emotional states. This makes it difficult for players to find optimal ways to improve, resulting in delayed skill improvement.

[1363] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving and saving video data of a game played by a user, means for decoding the saved video data and converting it into a data format for analysis, means for performing play analysis using a generative model using the converted data and comparing it with play data from experts, means for identifying problems in the user's play based on the analysis results and generating improvement methods, means for collecting biometric information of the user and recognizing their emotional state, means for personalizing feedback content taking the recognized emotional state into consideration, and means for transmitting the generated feedback to the user's terminal so that the user can check the evaluation results. This allows the player to quickly identify specific areas for improvement and receive personalized feedback based on their emotional data.

[1364] "Video data" is digital data that records the screen of a game played by a user.

[1365] A "generative model" is an algorithm based on artificial intelligence that identifies specific patterns and features from data and generates new information.

[1366] "Expert play data" is data that records the play of highly experienced players, and typically indicates high technical ability and strategy.

[1367] "Biometric information" is data that indicates the user's physical condition, and includes heart rate, facial expression, voice tone, and the like.

[1368] "Emotional state" indicates the user's current emotion, and is information for determining the user's state of tension, relaxation, excitement, etc.

[1369] "Personalizing" means adapting to the characteristics and circumstances of a particular user, adjusting to individual needs and circumstances.

[1370] "Feedback" is information that includes an evaluation of the user's play and advice and instructions for improvement.

[1371] A "dedicated application" is a specific piece of software that a user uses to receive and review feedback.

[1372] "Decoding" is the process of converting encoded data into its original or parsable form.

[1373] "Analysis results" are evaluations and conclusions of data obtained through analysis using a generative model.

[1374] ---

[1375] The present invention relates to a support system for users to improve their gameplay skills, which provides personalized feedback using the user's emotional data. The system has functions for receiving, storing, and analyzing gameplay video data, generating feedback, and recognizing emotions.

[1376] First, the user records their gameplay using the in-game recording function or external software (e.g., OBS Studio), and the recorded video data is uploaded to a server via a dedicated webpage or application.

[1377] The server receives and saves the uploaded video data. The save location is set to a specific folder in the server's file system. For example, it is saved in the " / data / uploads" folder. Next, the saved video data is decoded and converted into a data format for analysis (frame-by-frame image data and audio data). This is done using the moviepy library.

[1378] The decoded data is sent to an AI analysis module on the server. The AI ​​analysis module uses a generative model to compare the user's play with the play data of professional players and identify areas for improvement. The play data of professional players is created based on play information from multiple experts that has been collected and analyzed in advance. The AI ​​analysis module analyzes the user's movements, strategies, item use, positioning, etc., and identifies differences between the user and professional players and areas for improvement.

[1379] Additionally, the system includes an emotion engine that collects biometric information (e.g., heart rate, facial expression, and vocal tone) from the user to recognize the user's emotional state. This allows the system to personalize feedback based on the emotional data. For example, if the user is nervous, the system may generate feedback suggesting calm measures, whereas if the user is relaxed, the system may generate feedback that is full of encouraging words.

[1380] The server generates feedback written in natural language based on the output of the AI ​​analysis module and the emotional data obtained from the emotion engine. The feedback is generated using a generative AI model, detailing the user's specific issues and how to improve them. The feedback is generated in text and graphics format and sent to the user's device via a dedicated application.

[1381] The user can then review the feedback they received using a dedicated app. The feedback includes specific advice for improvement and messages that take their emotions into consideration, allowing them to refer to it and practice the next time they play.

[1382] As a concrete example, consider the case where a user records a gameplay video of "a certain online game" and uploads the video data to a server. The server saves the video data, decodes it, and performs AI analysis. For example, it identifies that "positioning is behind at 5 minutes and 20 seconds" and generates specific improvement advice such as "ensure your positioning in advance for the next battle." Furthermore, if the emotion engine detects that the user is nervous, it adds encouraging feedback such as "It's important to stay calm and understand the situation." This is sent to the user via a dedicated application, and the user puts this advice into practice the next time they play.

[1383] An example of an input prompt for the generative AI model is, "Generate feedback comparing the user with an expert player. If the user's positioning is behind at 5 minutes and 20 seconds, provide feedback including suggestions for improvement. Also, add content that takes into consideration the user's emotions if they are nervous."

[1384] The system allows users to quickly identify specific areas for improvement and receive personalized feedback based on emotional data, allowing them to improve their skills efficiently.

[1385] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1386] ---

[1387] Step 1: Recording user gameplay

[1388] The user starts playing a game and records the gameplay using the in-game recording function or external software such as OBS Studio. The specific input is the video and audio during gameplay, which is then output as video data (e.g., an MP4 file).

[1389] Step 2: Upload your recording

[1390] Users upload recorded video data to the server through a dedicated web page or application. The input is the video data file stored on the device, and the output is a file upload request sent to the server. Specifically, the user drags and drops the file into the web form and clicks the submit button.

[1391] Step 3: Storing video files on the server

[1392] The server receives the upload request and saves the video data in the specified directory. The input is the uploaded video data, and the output is the file path to the save destination. Specifically, the server stores the file in the " / data / uploads" folder in the server's file system and records the save path.

[1393] Step 4: The server decodes the video file

[1394] The server decodes the stored video data and converts it into a data format for analysis. The input is the stored video data file (MP4 file), and the output is image data and audio data on a frame-by-frame basis. Specifically, the server uses the moviepy library to convert the video file into image data for each frame.

[1395] Step 5: Running AI analysis on the server

[1396] The server supplies the decoded data to an AI analysis module. The input is image data and audio data for each frame, and the output is the results of gameplay analysis. The AI ​​model compares the expert's play data with the user's play and identifies issues with the user's play. Specific actions, such as position information, character movements, and skill usage timing, are analyzed using a generative model.

[1397] Step 6: Collecting and recognizing emotional data with the emotion engine

[1398] The server collects the user's biometric information and recognizes their emotional state. The input is biometric information from the user (e.g., heart rate, facial expression, and voice tone), and the output is the recognized emotional state. Specifically, the system collects data through wearable devices and webcams and analyzes emotions using algorithms.

[1399] Step 7: Server Generates Feedback

[1400] The server generates feedback written in natural language based on the analysis results output by the AI ​​and the emotional data recognized by the emotion engine. The input is the play analysis results and emotional data, and the output is feedback text. Specifically, a "prompt sentence" is input into the generative AI model, and a feedback sentence customized for each user is generated.

[1401] Step 8: Sending feedback to the user device

[1402] The server sends the generated feedback to the user's terminal. The input is the feedback text, and the output is the transmission result to the user's terminal. Specifically, the server sends the feedback to the user's terminal in text or graphics format via a dedicated application.

[1403] Step 9: Review and implement user feedback

[1404] The user checks the received feedback and practices, focusing on the areas for improvement for the next play. The input is the received feedback, and the output is what the user applies to the next play. Specifically, the user launches a dedicated application to view the feedback and implements the suggested improvements in the next play.

[1405] ---

[1406] (Application example 2)

[1407] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1408] Conventional gameplay analysis systems provide feedback on a user's gameplay, but do not provide personalized feedback that takes into account the user's emotional state. This makes it difficult to provide more effective and appropriate improvement advice based on the user's emotional state. Furthermore, there is a lack of a means to visually grasp the analysis results on a timeline, making it difficult for users to specifically understand when a problem occurred. Furthermore, there is a need for feedback to be provided in an appropriate format, such as text or graphics.

[1409] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1410] In this invention, the server includes means for receiving and saving video data of the game played by the user, means for decoding the saved video data and converting it into a data format for analysis, means for analyzing the play using an AI model using the converted data and comparing it with play data of professional players, means for identifying issues in the user's play based on the analysis results and generating methods for improvement, means for recognizing emotions based on the user's biometric data and reflecting them in the analysis results to personalize feedback, and means for transmitting the generated feedback to the user's device so that the user can check the evaluation results. This makes it possible to identify specific issues in the user's play and provide personalized feedback that takes into account the user's emotional state.

[1411] "Game video data" is a recording that includes video and audio of the game being played by the user.

[1412] "Decoding" is the process of converting saved video data into an analyzable format, such as frame-by-frame image data or audio data.

[1413] An "AI model" is an artificial intelligence algorithm that analyzes a user's play by comparing it with the play data of professional players and identifying areas of weakness.

[1414] "Professional player play data" refers to gameplay information of professional players that has been collected and analyzed in advance.

[1415] "Improvement methods" are specific advice and suggestions for improving issues in the user's play.

[1416] "User's biometric data" refers to physical and psychological information such as the user's heart rate, facial expression, and voice tone.

[1417] "Emotion recognition" is the process of determining a user's emotional state at any given time based on their biometric data.

[1418] "Feedback" refers to improvements and new suggested information provided based on the results of a user's gameplay analysis.

[1419] "Personalized feedback" refers to individually optimized improvement advice that takes into account the user's emotional state.

[1420] "Evaluation results" are information that includes the results of analysis of the user's gameplay and specific feedback based on the results.

[1421] A "dedicated application" is special software that users use to check analysis results and feedback.

[1422] This invention combines an emotion engine with a system that provides specific feedback to users comparing their gameplay with that of professional players to help them improve their gameplay skills. The system receives and stores video data of the game played by the user, decodes the data, converts it into a data format for analysis, performs comparative analysis using an AI model, and generates personalized feedback based on the evaluation results. The feedback is then sent to the user's device.

[1423] Hardware and software used

[1424] Hardware:

[1425] Smartphone

[1426] Dedicated Server

[1427] Biometric data collection devices (e.g., smartwatches)

[1428] software:

[1429] Video recording software (e.g. OBS Studio)

[1430] Video decoding library (e.g. moviepy)

[1431] Image analysis library (e.g. opencv)

[1432] Emotion Recognition Library (Custom Library: emotion_recognition)

[1433] Feedback generation library (custom library: feedback_generator)

[1434] Server communication library (custom library: server_communication)

[1435] Data processing and calculation flow

[1436] 1. A user uses a smartphone to record video of their gameplay and uploads the video data to a server. A biometric data collection device collects the user's heart rate and facial expression data, which are also uploaded to the server.

[1437] 2. The server decodes the uploaded video data using the moviepy library and converts it into frame-by-frame image data.

[1438] 3. The server uses emotion_recognition library to perform emotion recognition and analyzes the user's biometric data to recognize their emotional state.

[1439] 4. The server uses an AI analysis module to compare the playing data of professional players with the user's playing data and identify specific issues.

[1440] 5. The server generates personalized feedback using the feedback_generator library based on the identified issues and the recognized emotional data. Based on the analysis results, it displays areas for improvement linked to specific moments in the user's gameplay video.

[1441] 6. The server sends the generated feedback to the user's device using the server_communication library. The feedback is generated in the form of text and / or graphics and is displayed to the user via a dedicated application on the user's device.

[1442] Specific examples

[1443] Consider a scenario where a user records video of themselves playing a real-time strategy game, collects biometric data using a smartwatch, and uploads both sets of data to an app. The server receives the video and biometric data, performs AI analysis, and identifies issues such as "the player delayed the use of an item at 10 minutes and 30 seconds." Based on the user's recognized emotional tension, the app generates feedback such as "Use your item early in the next battle. It's important to relax and understand the situation."

[1444] Prompt Sentence Examples

[1445] I uploaded a gameplay video. Identify specific issues in the user's gameplay compared to professional players' play and generate personalized feedback based on the provided emotional data. Include encouraging words if the user is emotionally tense. Also, include specific advice pointing to a specific time in the video (e.g., 10 minutes 30 seconds).

[1446] In this way, users can upload their recorded gameplay videos and biometric data, which the server can then use AI to analyze and provide personalized feedback to the user's dedicated application.

[1447] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1448] Step 1:

[1449] The user's smartphone records gameplay video. When the user presses the record button through the app, video and audio of the gameplay are captured using video recording software (e.g., OBS Studio). The recorded video data is temporarily stored on the smartphone.

[1450] Input: Running gameplay

[1451] Output: Recorded gameplay video file

[1452] Step 2:

[1453] Users use the application to record gameplay videos and upload biometric data, including heart rate and facial expression data obtained from biometric data collection devices (e.g., smartwatches), to a server.

[1454] Input: Recorded gameplay video, biometric data

[1455] Output: Video files and biometric data uploaded to the server

[1456] Step 3:

[1457] The server receives and saves the uploaded gameplay video. The server stores the video file in a specified directory and prepares it for analysis.

[1458] Input: Uploaded video file, biometric data

[1459] Output: Video files and biometric data stored on the server

[1460] Step 4:

[1461] The server decodes the stored video files using the moviepy library. The video decoding process converts the video data into frame-by-frame image data and audio data.

[1462] Input: Video file stored on the server

[1463] Output: Image data and audio data for each frame

[1464] Step 5:

[1465] The server uses the emotion_recognition library to analyze the uploaded biometric data and recognize the user's emotions. Emotion recognition determines whether the user is tense or relaxed based on heart rate and facial expression data.

[1466] Input: Uploaded biometric data

[1467] Output: User's emotional state data

[1468] Step 6:

[1469] The server uses an AI analysis module to compare the decoded frame-by-frame image data and the user's play data with that of professional players to identify specific issues. This analysis evaluates the user's movements, strategy, item use, positioning, etc. and uses a generative AI model.

[1470] Input: Frame-by-frame image data, professional player play data

[1471] Output: Analysis results identifying problem areas

[1472] Step 7:

[1473] The server uses the feedback_generator library to generate personalized feedback based on the identified issues and emotional data. Based on the analysis results, it suggests areas for improvement by linking them to specific moments in the user's gameplay video, and provides appropriate feedback according to the user's emotional state.

[1474] Input: Analysis results identifying issues, user emotional state data

[1475] Output: Personalized feedback

[1476] Step 8:

[1477] The server sends the generated feedback to the user's terminal using the server_communication library. The feedback is generated in text and / or graphics format and is displayed to the user via a dedicated application on the user's terminal.

[1478] Input: Personalized Feedback

[1479] Output: Feedback displayed on the user's terminal

[1480] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1481] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1482] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1483] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1484] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1485] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1486] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1487] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1488] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1489] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1490] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1491] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1492] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1493] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1494] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1495] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1496] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1497] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1498] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1499] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1500] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1501] The following is further disclosed regarding the above embodiment.

[1502] (Claim 1)

[1503] A means for receiving and storing video data of a game played by a user;

[1504] A means for decoding the stored video data and converting it into a data format for analysis;

[1505] The converted data will be used to analyze the gameplay using an AI model and compare it with the play data of professional players.

[1506] A means for identifying problems in the user's play based on the analysis results and generating methods for improvement;

[1507] a means for transmitting the generated feedback to a user's terminal so that the user can check the evaluation result;

[1508] A system including:

[1509] (Claim 2)

[1510] The system according to claim 1, further comprising a means for displaying areas for improvement based on the analysis results and linking them to specific timings in the user's gameplay video.

[1511] (Claim 3)

[1512] 2. The system of claim 1, further comprising means for generating said feedback in the form of text and / or graphics and displaying said feedback via a dedicated application on a user terminal.

[1513] "Example 1"

[1514] (Claim 1)

[1515] means for receiving and storing data of an electronic game played by a user;

[1516] means for decoding the stored data and converting it into a format for analysis;

[1517] A means for analyzing the converted data using an artificial intelligence model and comparing it with expert data;

[1518] A means for identifying problems in user behavior based on the analysis results and generating methods for improvement;

[1519] means for transmitting the generated feedback to a communication device of the user so that the user can check the evaluation result;

[1520] A means of uploading video data via a dedicated web page or software application;

[1521] A system including:

[1522] (Claim 2)

[1523] The system according to claim 1, further comprising means for displaying points for improvement based on the analysis results and linking them to specific timings in the user's behavioral data.

[1524] (Claim 3)

[1525] 10. The system of claim 1, further comprising means for generating said feedback in the form of text and / or graphics and displaying said feedback via an application on a user's communication device.

[1526] "Application Example 1"

[1527] (Claim 1)

[1528] A means for receiving and storing video data of a game played by a user;

[1529] A means for decoding the stored video data and converting it into a data format for analysis;

[1530] The converted data is used to analyze the play using a generative AI model and compare it with the play data of professional players.

[1531] A means for identifying problems in the user's play based on the analysis results and generating methods for improvement;

[1532] a means for transmitting the generated feedback to a user's terminal so that the user can check the evaluation result;

[1533] A means for the application to run on the viewer's device and display feedback in real time;

[1534] A system including:

[1535] (Claim 2)

[1536] The system according to claim 1, further comprising a means for displaying areas for improvement based on the analysis results and linking them to specific timings in the user's gameplay video.

[1537] (Claim 3)

[1538] 2. The system of claim 1, further comprising means for generating said feedback in the form of text and / or graphics and displaying said feedback via a dedicated application on a user terminal.

[1539] "Example 2: Combining Emotion Engines"

[1540] ---

[1541] (Claim 1)

[1542] means for receiving and storing video data of a game played by a user;

[1543] means for decoding the stored video data and converting it into a data format for analysis;

[1544] A means for performing play analysis using a generative model using the converted data and comparing it with expert play data;

[1545] A means for identifying problems in the user's play based on the analysis results and generating methods for improvement;

[1546] means for collecting biometric information of a user and recognizing an emotional state;

[1547] a means of personalizing the content of feedback taking into account the perceived emotional state;

[1548] a means for transmitting the generated feedback to a user's terminal so that the user can check the evaluation result;

[1549] A system including:

[1550] (Claim 2)

[1551] The system according to claim 1, further comprising means for displaying areas for improvement based on the analysis results and linking them to specific timings in the user's gameplay footage.

[1552] (Claim 3)

[1553] 2. The system according to claim 1, further comprising means for generating the feedback in the form of text or graphics and displaying it via a dedicated application on a user terminal.

[1554] ---

[1555] "Application example 2 when combining emotion engines"

[1556] (Claim 1)

[1557] A means for receiving and storing video data of a game played by a user;

[1558] A means for decoding the stored video data and converting it into a data format for analysis;

[1559] The converted data will be used to analyze the gameplay using an AI model and compare it with the play data of professional players.

[1560] A means for identifying problems in the user's play based on the analysis results and generating methods for improvement;

[1561] A means of recognizing emotions based on the user's biometric data and reflecting them in the analysis results to provide personalized feedback;

[1562] a means for transmitting the generated feedback to a user's terminal so that the user can check the evaluation result;

[1563] A system including:

[1564] (Claim 2)

[1565] The system according to claim 1, further comprising a means for displaying areas for improvement based on the analysis results and linking them to specific timings in the user's gameplay video.

[1566] (Claim 3)

[1567] 2. The system of claim 1, further comprising means for generating said feedback in the form of text and / or graphics and displaying said feedback via a dedicated application on a user terminal. [Explanation of symbols]

[1568] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving and storing video data of a game played by a user; A means for decoding the stored video data and converting it into a data format for analysis; The converted data will be used to analyze the gameplay using an AI model and compare it with the play data of professional players. A means for identifying problems in the user's play based on the analysis results and generating methods for improvement; a means for transmitting the generated feedback to a user's terminal so that the user can check the evaluation result; A system including:

2. The system according to claim 1, further comprising means for displaying points for improvement based on the analysis results and linking them to specific timings in the user's gameplay video.

3. 2. The system of claim 1, further comprising means for generating said feedback in the form of text and / or graphics and displaying said feedback via a dedicated application on a user terminal.

Images