System

Abstract

A system is provided.SOLUTION: A system comprising: means for uploading a video of a game played by a user; means for analyzing the uploaded video to detect an important game event; means for generating a live commentary in a natural language based on the detected game event; means for generating a live video by overlaying the generated live commentary on the video; means for providing a community function for sharing the generated live video; means for evaluating the shared live video by other users; and means for officially characterizing a good evaluation live video based on an evaluation result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today's live game streaming is extremely popular, with many users earning revenue by streaming their gameplay videos. However, creating live game videos requires advanced technical skills and a significant amount of time, presenting a significant obstacle for many users. Streaming game videos also requires the appropriate content creation capabilities, equipment, and technical knowledge. Furthermore, the content created has few opportunities for recognition and appreciation, limiting the opportunities for monetization. A system that solves these challenges and allows users to easily and effectively create, share, and monetize live game videos is needed. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system that includes a means for users to upload videos of games they play, a means for analyzing the uploaded videos and detecting important game events, a means for generating commentary comments in natural language based on the detected game events, a means for generating commentary videos by overlaying the generated commentary comments on videos, a means for providing a community function for sharing the generated commentary videos, a means for other users to rate the shared commentary videos, and a means for officially featuring commentary videos that receive high ratings based on the rating results. Furthermore, this system includes a means for storing commentary comments from professional commentators and celebrities in a database and appropriately selecting and adding these comments based on detected game events, thereby enabling the generation of higher-quality commentary videos. Furthermore, the system also includes a means for users to post the generated commentary videos on third-party distribution platforms and monetize them, thereby providing a convenient means for users to earn revenue.

[0006] "User" refers to an entity that uses the system to upload their own gameplay videos and create, share, and evaluate live gameplay videos.

[0007] A "gameplay video" is a video file that records a user actually playing a game.

[0008] "Means for uploading" refers to the functionality that allows users to transfer and save their own gameplay videos to the system.

[0009] The "means for analysis" is a function that analyzes uploaded gameplay videos and automatically detects important events.

[0010] A "significant game event" refers to a notable action or occurrence that occurs in a game as a result of a user's play, such as defeating an enemy, leveling up, or fighting a boss.

[0011] The "means for generating live commentary in natural language" is a function for generating comments based on detected game events in a format similar to the live commentary that a human would normally give.

[0012] The "overlaying means" is a function that performs a process of displaying the generated commentary overlaid on the original gameplay video.

[0013] A "play-by-play video" is a video generated by overlaying play-by-play commentary onto a gameplay video.

[0014] The "community function" is a function of the online platform that allows users to share their created live videos and interact with and rate them with other users.

[0015] The "means for rating" is a function that allows users to rate a shared live video, such as by giving a "like" or comment.

[0016] "Official Feature Option" is a feature that allows highly rated live commentary videos to be featured on the system's featured page or official platform.

[0017] "Comments from professional commentators and celebrities" are past comments made by people with specialized skills and notoriety, and can be used as material to generate high-quality commentary videos.

[0018] "Third-party distribution platform" means an internet service, such as YouTube, that enables users to post user-generated content and make it available to other viewers.

[0019] "Means to monetize" refers to the ability to post user-generated live videos to third-party distribution platforms and generate revenue through advertising or other revenue models. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The system of this invention allows users to easily convert gameplay videos into live commentary videos, share and rate them within the community, and also supports monetization of the videos. This system is mainly composed of three entities: a server, a device, and users.

[0042] Main features

[0043] Game video upload

[0044] User

[0045] Users upload videos of the games they play to the system via their terminals.

[0046] Select the video file and click the upload button to send the video to the system.

[0047] Terminal

[0048] The terminal converts the game video file selected by the user into an HTTP request and sends it to the server.

[0049] Notify the user that the video was successfully sent.

[0050] server

[0051] The received video file is saved in storage, and the video metadata (file name, upload date and time, etc.) is recorded in the database.

[0052] Video Analysis

[0053] server

[0054] The server runs software to analyze the stored video.

[0055] Using image and voice recognition technology, it detects important events in the game (e.g., defeating enemies, boss battles, level-ups, etc.) along with their timeline.

[0056] Generate a list of detected event information (event type, occurrence time).

[0057] Generate live commentary

[0058] server

[0059] Based on the detected event list, a natural language generation (NLG) engine is used to generate corresponding commentary.

[0060] The generated commentary includes phrases such as "Great! You defeated the enemy!" for "You defeated the enemy."

[0061] Select appropriate comments from a database of professional commentators and celebrities and add them as needed.

[0062] Rendering live video

[0063] server

[0064] The generated commentary is overlaid onto the gameplay video.

[0065] Use a video editing engine to overlay text onto the video based on the corresponding timestamps.

[0066] The completed live video is saved to storage and metadata is added to the database.

[0067] Share and rate videos

[0068] User

[0069] Users log in to the system to share the generated live video on a community page.

[0070] Click the share button and enter a title and description for your video.

[0071] You can watch live videos shared by other users and leave likes and comments.

[0072] Terminal

[0073] Receives share and rating requests and sends them to the server.

[0074] Display a community page where users can easily watch and rate other videos.

[0075] server

[0076] Processes incoming share and rating requests and updates the database with video rating scores and comments.

[0077] We will periodically tally rating scores and officially feature the top-rated videos.

[0078] Monetization support

[0079] User

[0080] The generated live commentary videos will be posted on third-party distribution platforms in order to generate revenue.

[0081] Get a download link from the system and upload the video to a distribution platform.

[0082] server

[0083] Provide links to facilitate posting and provide users with monetization guidelines.

[0084] Specific examples

[0085] For example, suppose User A uploads a 10-minute video of himself playing an action game. The server analyzes the video and detects scenes where enemies are defeated or boss battles are fought. The AI ​​generates commentary such as "Great attack!" or "Great opportunity!" and overlays it on the original video. The final commentary video is shared by User A to the community and receives high ratings from other users. As a result of the high ratings, the server officially features the video, and User A can then post the video on YouTube to earn revenue.

[0086] In this way, the system of the present invention makes the process of creating, sharing, rating, and monetizing gameplay videos simple and effective.

[0087] The processing flow will be explained below.

[0088] Step 1:

[0089] User

[0090] Prepare your gameplay video on your device and access the upload page.

[0091] Click the Upload button and select the video file to upload from the file selection dialog.

[0092] Step 2:

[0093] Terminal

[0094] The selected video file is converted into an HTTP request and sent to the server.

[0095] Show the upload progress to the user.

[0096] Step 3:

[0097] server

[0098] Analyze the received HTTP request and save the video file in storage.

[0099] A unique ID is assigned to the video file, and metadata (file name, upload date and time, etc.) is stored in the database.

[0100] Step 4:

[0101] server

[0102] Load the video stored in storage for analysis.

[0103] It uses image and voice recognition technology to detect important events in the game.

[0104] List the timestamp and event details for each event.

[0105] Step 5:

[0106] server

[0107] Pass the event list to a natural language generation (NLG) engine.

[0108] The NLG engine generates commentary corresponding to events. For example, for the event "Boss battle begins," it generates a comment like "This is where the real action begins, the boss battle begins!"

[0109] If necessary, add appropriate comments from a database of professional commentators and celebrities.

[0110] Step 6:

[0111] server

[0112] A video editing engine is launched to overlay the generated commentary on the original game video.

[0113] Insert comments into the video based on appropriate timestamps to generate a play-by-play video.

[0114] The completed live video is saved to storage and metadata is added to the database.

[0115] Step 7:

[0116] User

[0117] Log in to your account and share the generated commentary video on the community page.

[0118] Click the share button and enter a title and description for your video.

[0119] Step 8:

[0120] Terminal

[0121] Sends a share request to the server.

[0122] It provides an interface for accessing community pages and watching live videos shared by other users.

[0123] Step 9:

[0124] server

[0125] Process incoming share requests and add new play-by-play video entries to the community page.

[0126] When other users watch or rate a video, the rating data is saved in a database.

[0127] Step 10:

[0128] server

[0129] We periodically tally the rating scores and select the top-rated videos.

[0130] Selected videos will be featured on the official special page.

[0131] Step 11:

[0132] User

[0133] Check out the featured videos.

[0134] Get a link to download the video or post it to a third-party distribution platform.

[0135] Step 12:

[0136] User

[0137] Post videos to distribution platforms and monetize them. After posting, use revenue models such as advertising revenue and Super Chat.

[0138] The above is the flow of specific processing steps for carrying out the invention.

[0139] Example 1

[0140] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0141] Conventional gameplay video sharing systems require users to manually edit videos and add commentary, which is time-consuming and labor-intensive. Furthermore, the process of sharing, rating, and monetizing the created videos is complicated, making it difficult for users to use. The objective of this invention is to solve these problems and provide a system that allows users to easily convert gameplay videos into commentary videos and effectively share, rate, and monetize them.

[0142] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0143] In this invention, the server includes: means for users to upload videos of games they play; means for converting the video files into HTTP requests and sending them to the server; means for saving the received video files in storage and recording metadata in a database; means for analyzing the uploaded videos, detecting in-game events using image recognition technology and voice recognition technology, and generating a list of event information; means for generating commentary comments using a natural language generation engine based on the detected game events and comparing them with a database to select appropriate comments; means for overlaying the generated commentary comments on videos to generate commentary videos; means for providing a community function for sharing the generated commentary videos; means for other users to rate the shared commentary videos; means for officially featuring commentary videos with good ratings based on the rating results; and means for providing a link for posting the generated commentary videos to a third-party distribution platform for monetization. This enables users to easily analyze, edit, and share gameplay videos and effectively monetize them.

[0144] "User" means any individual or legal entity that uses the System to upload, analyze, share, and rate gameplay videos.

[0145] A "gameplay video" is a digital file that contains video and audio of a game being played by a user.

[0146] "Uploading" is the process by which a user transfers a video file from their device to the system.

[0147] An "HTTP request" is a request message for sending data to a server based on a protocol used for communication over the Internet.

[0148] A "server" is a computer system that processes and stores data received from users and provides the required services.

[0149] "Storage" refers to a storage device for saving digital data, including hard disk drives (HDDs) and solid-state drives (SSDs).

[0150] "Metadata" is data that describes the characteristics and content of a video file, and includes, for example, the file name, upload date and time, and video length.

[0151] A "database" is a system for efficiently storing, retrieving, and updating structured information.

[0152] "Image recognition technology" is a technology that analyzes the content of images and videos and identifies specific objects and movements.

[0153] "Speech recognition technology" is a technology that analyzes voice data and converts it into meaningful text or commands.

[0154] A "natural language generation engine" is an artificial intelligence technology that generates text based on input data in a form that is easy for humans to understand.

[0155] "Commentary" is text that includes commentary and impressions about specific events in gameplay videos.

[0156] "Overlay" is a technique for displaying text or images over specific locations on a video.

[0157] "Community Features" are features within the system that allow users to share and rate generated content.

[0158] A "rating" is feedback or a score that a user gives to another user's content.

[0159] A "feature" is a display of specific content that stands out more than other content.

[0160] "Monetization" is the process of utilizing generated content for financial gain.

[0161] This invention is a system that allows users to easily upload gameplay videos and then analyze, edit, share, rate, and monetize them. This system operates in cooperation with a server, devices, and users.

[0162] To upload a video of a game that a user has played, the user first logs in to the system. After logging in, the user selects the gameplay video file from their device and clicks the upload button. At this time, the device converts the video file into an HTTP request and sends it to the server. The uploaded video file is received by the server and saved in storage. The server also records the video's metadata (file name, upload date and time, etc.) in a database.

[0163] Next, the server launches software to analyze the saved video file. Specifically, it uses image recognition technology (e.g., OpenCV) and speech recognition technology (e.g., Google Speech-to-Text API) to detect important in-game events (e.g., defeating an enemy, leveling up, etc.). The server then generates a list of detected event information (event type, occurrence time). The server then uses this event list to generate commentary using a natural language generation (NLG) engine (e.g., GPT-3). The generated commentary is then compared with a database of comments from professional commentators and celebrities to select appropriate comments.

[0164] The server then uses a video editing engine (e.g., FFmpeg) to overlay the commentary onto the video. During this process, commentary generated based on timestamps is overlaid onto specific locations in the gameplay video. The completed commentary video is saved to storage, and the metadata is added to a database.

[0165] Users can share the live commentary videos they have created on the system's community page. When sharing, users enter a title and description for the video. The device then sends a sharing request to the server, allowing other users to view and rate the shared live commentary video. The server processes the received rating request and updates the video's rating score and comments in the database. Based on the rating results, live commentary videos that receive good ratings are officially featured.

[0166] Additionally, users can monetize their generated live streams by posting them to third-party distribution platforms, and the server will provide users with links and guidelines to support monetization.

[0167] As a concrete example, suppose User A uploads a 10-minute video of himself playing an action game. The server analyzes the video and detects scenes where enemies are defeated and boss battles. The AI ​​generates commentary such as "Great attack!" or "Great opportunity!" and overlays it on the original video. The final commentary video is shared by User A with the community and receives high ratings from other users. As a result of the high ratings, the server officially features the video, and User A can then post the video on distribution platforms such as YouTube to earn revenue.

[0168] An example of a prompt might be:

[0169] "I want to create a program that analyzes gameplay video, generates commentary based on important events, and then creates a commentary video. Please explain the steps below:

[0170] 1. Users upload game videos

[0171] 2. Analyze video on the server and detect important events

[0172] 3. Create live commentary using natural language generation

[0173] 4. Overlay your comments on the video and you're done!

[0174] The system operates based on this content, improving user convenience.

[0175] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0176] Step 1:

[0177] User logs in and uploads game video

[0178] To log in to the system, the user enters their username and password. After successful login, the user selects a gameplay video file from the terminal and clicks the upload button. The input is the user information and the game video file, and the output is an HTTP request. When the upload button is clicked, the terminal converts the selected video file into an outgoing packet as an HTTP POST request and sends it to the server.

[0179] Step 2:

[0180] The device converts the video file into an HTTP request and sends it to the server.

[0181] The device follows the user's instructions, converts the selected video file into an HTTP request, and sends it to the server using the POST method. The input is the video file, and the output is an HTTP request. Specifically, the device embeds the file's binary data in the body of the HTTP POST request and sends it to the specified URL.

[0182] Step 3:

[0183] The server receives the video file and stores it in storage.

[0184] The server extracts the video file from the body of the HTTP request received and saves it to storage. The input is the HTTP request, and the output is the saved video file. Specifically, the file system API is used to save the video file to a specified directory.

[0185] Step 4:

[0186] The server records the video metadata in a database

[0187] The server extracts metadata from the saved video files (file name, upload date and time, video length, etc.) and records it in a database. The input is the saved video file, and the output is a database record. Specifically, it obtains the file attribute information and inserts the record using the database API.

[0188] Step 5:

[0189] The server analyzes the video using video analysis software.

[0190] The server runs video analysis software (e.g., OpenCV) and uses image recognition technology to detect important in-game events (e.g., defeating an enemy, leveling up, etc.). The input is the saved video file, and the output is a list of event information. Specifically, it processes each frame of the video, detecting specific in-game events, and lists them along with their timestamps.

[0191] Step 6:

[0192] The server uses speech recognition technology to identify audio events and record them as metadata.

[0193] The server uses speech recognition technology (e.g., Google Speech-to-Text API) to identify important audio events from the video's audio track and record them as metadata. The input is the saved video file and the event information list, and the output is an updated event information list. Specifically, the server analyzes the audio data, converts it into text strings in each language, and records them along with the time when the audio events occurred.

[0194] Step 7:

[0195] The server generates live commentary using a natural language generation engine

[0196] The server uses a natural language generation (NLG) engine (e.g., GPT-3) to generate commentary based on the detected event information. The input is a list of event information, and the output is a list of commentary. Specifically, a prompt sentence is input to the generative AI model, which then generates appropriate commentary.

[0197] Step 8:

[0198] The server compares the generated live commentary with the database and selects the appropriate comment.

[0199] The server compares the generated live commentary comments with a database of comments from professional commentators and celebrities to select appropriate comments. The input is a list of generated live commentary comments, and the output is a list of matched live commentary comments. Specifically, it uses a string search algorithm to extract matching comments from the database.

[0200] Step 9:

[0201] The server uses a video editing engine to overlay commentary onto the video.

[0202] The server uses a video editing engine (e.g., FFmpeg) to overlay the generated commentary based on the corresponding timestamps of the video. The input is a gameplay video file and a list of commentary, and the output is the finished commentary video. Specifically, it applies a text overlay filter to specific frames of the video.

[0203] Step 10:

[0204] The server saves the completed live video to storage and adds the metadata to the database.

[0205] The server saves the completed live video in storage and adds the video's metadata (such as the creation date and time, video playback time, and comment list) to the database. The input is the completed live video and a list of live comments, and the output is an updated database record. Specifically, the server saves the video file to the storage system and inserts its metadata into the database.

[0206] Step 11:

[0207] Share user-generated play-by-play videos on your community page

[0208] A user logs in to the system and performs an operation to share the generated live video on a community page. The input is the generated live video, and the output is a sharing request. The user clicks the share button and enters the video title and description.

[0209] Step 12:

[0210] The device sends a share request to the server

[0211] The device sends the user's sharing request to the server as an HTTP POST request. The input is the sharing request, and the output is an HTTP POST request. Specifically, the sharing request is packetized along with the video metadata and sent to the server.

[0212] Step 13:

[0213] The server processes share and rating requests and updates the rating scores and comments to the database.

[0214] The server processes the received share requests, adds new video information to the database, and processes rating requests made by other users on the shared live videos, updating the rating scores and comments in the database. The input is the share request and the rating request, and the output is the updated database record.

[0215] Step 14:

[0216] The server will officially feature well-rated playthrough videos based on the evaluation results.

[0217] The server periodically aggregates the rating scores and officially features the top-rated live videos. The input is the rating scores and comments, and the output is a list of featured videos. Specifically, it calculates the ranking of the rating scores and displays the top-rated videos in a specific section.

[0218] Step 15:

[0219] Monetize user-generated live videos by posting them to a streaming platform

[0220] Users post their generated commentary videos to third-party distribution platforms (e.g., YouTube, Twitch) and monetize them. The input is the generated commentary video, and the output is the post to the distribution platform. The system provides users with links and guidelines to support monetization.

[0221] Step 16:

[0222] Provide users with links and guidelines to help your server monetize

[0223] The server displays links and guidelines for video monetization to users. The input is a user request, and the output is monetization links and guidelines. Specifically, the server displays information related to monetization on the system dashboard for easy access by users.

[0224] The above are the specific processing steps of this system.

[0225] (Application example 1)

[0226] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0227] Currently, it takes a lot of time and effort for users to easily edit, rate, and monetize their gameplay videos. Furthermore, further technological advancements are required to realize advanced features such as seamless integration with content distribution platforms and highly accurate comment generation and overlays using AI technology. The purpose of this invention is to solve these issues and provide a simple system that allows users to easily and effectively create and share gameplay videos and further monetize them.

[0228] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0229] In this invention, the server includes means for allowing users to upload videos of games they have played, means for analyzing the uploaded videos and detecting important game events, means for generating commentary comments in natural language based on the detected game events, means for generating commentary videos by overlaying the generated commentary comments on videos, means for providing a community function for sharing the generated commentary videos, means for other users to rate the shared commentary videos, means for officially featuring commentary videos that have received positive reviews based on the rating results, means for seamlessly sharing the generated commentary videos to a third-party content distribution platform, and means for accurately overlaying commentary comments based on timestamps of game events using AI technology. This allows users to easily and effectively create and share commentary videos, enabling monetization through highly accurate commentary generation and seamless distribution.

[0230] "User" means an individual or organization that uses this system to upload gameplay videos and create and share live gameplay videos.

[0231] "Game videos" refer to video files recorded by users playing games.

[0232] "Uploading" refers to the act of a user sending gameplay video from their device to a server.

[0233] "Analysis" refers to the process by which the server processes the uploaded video and detects important game events.

[0234] "Game Event" refers to an important event that occurs within the game (e.g., defeating an enemy, fighting a boss, leveling up, etc.).

[0235] "Natural language" refers to the language used by humans on a daily basis, and is used in this system to generate live commentary.

[0236] "Live commentary" refers to comments that explain and evaluate gameplay in real time or near real time.

[0237] "Overlay" refers to the process of displaying the generated commentary overlaid on the game video.

[0238] "Live commentary video" refers to a video file created by overlaying live commentary on gameplay video.

[0239] "Community Function" refers to a platform where users can share live videos they have created and other users can watch, rate, and comment on them.

[0240] "Evaluation" refers to the act of users within a community rating a posted live video by giving it a "like" or comment.

[0241] "Featured" means that a live commentary video that has received positive reviews will be officially featured and introduced to more users.

[0242] "Content distribution platform" refers to a video sharing service provided by a third party, such as YouTube or other similar services.

[0243] "Seamless" means that the operations performed by the user are smoothly linked together, allowing a series of tasks to be performed without interruption.

[0244] "AI technology" refers to artificial intelligence technology, which in this system is used for natural language generation and analysis of game events.

[0245] A "timestamp" refers to data that records the time when a particular event occurred.

[0246] "Accuracy" refers to the degree to which live commentary and analysis results generated using AI technology are accurate as expected by users.

[0247] The embodiment of the present invention will be described in detail below. This system is mainly composed of three components: a server, a terminal, and a user.

[0248] 1. Uploading game videos

[0249] User

[0250] Users use a smartphone application to select the game video they have played and click the upload button to send it to the server.

[0251] Terminal

[0252] The device converts the game video file selected by the user into an HTTP request and sends it to the server. The device notifies the user that the video has been sent successfully.

[0253] server

[0254] The server stores the received video file in storage and records the video's metadata (file name, upload date and time, etc.) in a database.

[0255] 2. Video Analysis

[0256] server

[0257] The server runs software to analyze the saved video. The software used includes Google Cloud Vision API and Google Speech-to-Text. These are used for image and voice recognition to detect important events in the game (e.g., defeating an enemy, boss battles, level-ups, etc.). The results of this detection are compiled into an event list.

[0258] 3. Generating live commentary

[0259] server

[0260] Based on the detected event list, the server uses a natural language generation engine (e.g., GPT-4) to generate corresponding commentary, such as "When an enemy is defeated: Amazing! You defeated the enemy!". It also selects appropriate comments from a database of professional commentators and celebrities and adds them as needed.

[0261] An example prompt is "Generate a commentary for the following gaming event: 'Player defeats the boss in an epic fight'".

[0262] 4. Rendering the Live Video

[0263] server

[0264] The server processes the generated commentary onto the gameplay video. FFmpeg is used as the video editing software. The commentary is accurately overlaid onto the video based on the corresponding timestamp. The completed video is saved to storage, and metadata is added to the database.

[0265] 5. Share and rate the video

[0266] User

[0267] Users log in to the system to share the created live commentary video on the community page. They click the share button and enter the video title and description. They can also watch the live commentary videos shared by other users and leave "likes" and comments.

[0268] Terminal

[0269] The device receives and sends share and rating requests to the server, and also displays a community page where users can easily watch and rate other videos.

[0270] server

[0271] The server processes the incoming share and rating requests, updates the video's rating score and comments to a database, and periodically aggregates the rating scores and officially features the top-rated play-by-play videos.

[0272] 6. Monetization Support

[0273] User

[0274] Users can monetize the generated live videos by posting them to third-party distribution platforms. Users obtain a download link from the system and upload the video to the distribution platform.

[0275] server

[0276] The server provides links to facilitate posting and provides users with monetization guidelines.

[0277] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0278] Step 1:

[0279] Game video upload

[0280] Process flow:

[0281] A user launches the application on their smartphone, selects a gameplay video file from their device, and then clicks the upload button, which generates an HTTP request to send the video to the server.

[0282] Input and Output:

[0283] Input: Gameplay video file.

[0284] Output: HTTP request to the server.

[0285] Specific behavior:

[0286] When the user selects a video file and clicks the send button, the terminal generates an HTTP request for sending the selected video file to the server, and executes the transmission of the video.

[0287] Step 2:

[0288] Receiving and storing videos

[0289] Process flow:

[0290] The server receives the HTTP request sent from the device, saves the video file in storage, and records the video file's metadata (file name, upload date and time, etc.) in a database.

[0291] Input and Output:

[0292] Input: The gameplay video file sent in the HTTP request.

[0293] Output: Video files saved on storage, along with their metadata.

[0294] Specific behavior:

[0295] When the server receives an HTTP request, it saves the video data in storage and records the metadata of the video file in a database.

[0296] Step 3:

[0297] Video Analysis

[0298] Process flow:

[0299] The server runs image and voice recognition software to analyze the stored video, using the Google Cloud Vision API and Google Speech-to-Text to detect important game events (e.g., enemy defeats, boss battles, level-ups, etc.) and generate an event list.

[0300] Input and Output:

[0301] Input: Gameplay video file saved in storage.

[0302] Output: A list of game events.

[0303] Specific behavior:

[0304] The server loads the video file, performs image recognition using the Google Cloud Vision API and speech recognition using Google Speech-to-Text, and outputs a list of detected game events with timestamps.

[0305] Step 4:

[0306] Generate live commentary

[0307] Process flow:

[0308] The server generates commentary using a generative AI model (e.g., GPT-4) based on the detected event list. The comment is a phrase such as "When an enemy is defeated: Great! You defeated the enemy!"

[0309] Input and Output:

[0310] Input: A list of detected game events.

[0311] Output: A list of live commentary.

[0312] Specific behavior:

[0313] The server uses a natural language generation engine to generate the best commentary for each event from the event list using the prompt text as input. An example of a specific prompt text is "Generate a commentary for the following gaming event: 'Player defeats the boss in an epic fight'".

[0314] Step 5:

[0315] Rendering live video

[0316] Process flow:

[0317] The server processes the generated commentary overlays onto the gameplay video, using video editing software (e.g., FFmpeg) to overlay each commentary onto the video based on its corresponding timestamp.

[0318] Input and Output:

[0319] Input: Gameplay video file, commentary list.

[0320] Output: A commentary video file with commentary overlays.

[0321] Specific behavior:

[0322] The server uses FFmpeg to overlay the commentary onto the relevant parts of the video, generating the final commentary video.

[0323] Step 6:

[0324] Share and rate videos

[0325] Process flow:

[0326] Users can upload the created live commentary videos to a community page, where other users can view and rate them. Other users can also like and comment on the live commentary videos.

[0327] Input and Output:

[0328] Input: Live video file, share request.

[0329] Output: Play-by-play videos uploaded to the community page, and rating data.

[0330] Specific behavior:

[0331] A user clicks the share button, enters the title and description of the video, and posts it to the community page. The server receives the share request and performs the necessary processing to display the video on the community page. It also manages rating requests from other users and records the ratings in a database.

[0332] Step 7:

[0333] Monetization support

[0334] Process flow:

[0335] Users can then post their live streams to third-party platforms to monetize them, and the server will provide users with links and guidelines for monetization.

[0336] Input and Output:

[0337] Input: Play-by-play video file.

[0338] Output: Links to distribution platforms, monetization guidelines.

[0339] Specific behavior:

[0340] The server provides a link to facilitate posting and displays appropriate monetization guidelines to the user, who then uses the provided link to upload their live video to the distribution platform.

[0341] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0342] The system of the present invention allows users to easily convert gameplay videos into commentary videos, share and rate them within the community, and further supports monetization of the created videos. By combining this system with an emotion engine that recognizes the user's emotions, it is possible to generate more natural and realistic commentary comments. Specific embodiments of the present invention are described below.

[0343] Main features

[0344] Game video upload

[0345] User

[0346] The user prepares the game video they played on their device and accesses the upload page.

[0347] Click the Upload button and select the video file to upload from the file selection dialog.

[0348] Terminal

[0349] The selected video file is converted into an HTTP request and sent to the server.

[0350] Show the upload progress to the user.

[0351] server

[0352] Analyze the received HTTP request and save the video file in storage.

[0353] A unique ID is assigned to the video file, and metadata (file name, upload date and time, etc.) is stored in the database.

[0354] Video Analysis

[0355] server

[0356] Load the video stored in storage for analysis.

[0357] It uses image and voice recognition technology to detect important events in the game.

[0358] List the timestamp and event details for each event.

[0359] Introducing the Emotion Engine

[0360] server

[0361] It analyzes the user's voice and facial expressions in the video and activates an emotion engine to detect emotions.

[0362] The emotion engine adds detected emotion data (e.g., joy, surprise, anger, etc.) to the analysis results.

[0363] Adjust the tone and content of commentary based on detected emotional data.

[0364] Generate live commentary

[0365] server

[0366] Pass the event list to a natural language generation (NLG) engine.

[0367] The NLG engine generates commentary corresponding to the event. For example, if the detected emotion for the event "Boss battle begins" is "surprise," it generates a comment like "This is unexpected! The boss battle has begun!"

[0368] If necessary, add appropriate comments from a database of professional commentators and celebrities.

[0369] Rendering live video

[0370] server

[0371] A video editing engine is launched to overlay the generated commentary onto the gameplay video.

[0372] Comments adjusted based on the emotion data are inserted into the video based on the corresponding timestamps to generate a live video.

[0373] The completed live video is saved to storage and metadata is added to the database.

[0374] Share and rate videos

[0375] User

[0376] Log in to your account and share the generated commentary video on the community page.

[0377] Click the share button and enter a title and description for your video.

[0378] Terminal

[0379] Sends a share request to the server.

[0380] It provides an interface for accessing community pages and watching live videos shared by other users.

[0381] server

[0382] Process incoming share requests and add new play-by-play video entries to the community page.

[0383] When other users watch or rate a video, the rating data is saved in a database.

[0384] Monetization support

[0385] User

[0386] The generated live commentary videos will be posted on third-party distribution platforms in order to generate revenue.

[0387] Get a download link from the system and upload the video to a distribution platform.

[0388] server

[0389] Provide links to facilitate posting and provide users with monetization guidelines.

[0390] Specific examples

[0391] For example, suppose User B uploads a 15-minute video of himself playing a horror game. The server analyzes the video and detects scenes where monsters appear or scenes where the user is surprised. The emotion engine detects the user's surprise, and based on that emotion data, the AI ​​generates commentary such as "I can't believe it! A monster appeared at this exact moment!" These comments are overlaid on the original video, and User B shares the completed commentary video with the community. After receiving high ratings, the video is officially featured, and User B can further post the video on YouTube, earning revenue opportunities.

[0392] In this way, the system of the present invention uses an emotion engine to generate immersive live videos that correspond to the user's emotions, allowing for a simple and effective process of sharing, rating, and monetizing.

[0393] The processing flow will be explained below.

[0394] Step 1:

[0395] User

[0396] Save your gameplay videos to your device.

[0397] Access the game commentary generation system, log in, and proceed to the video upload page.

[0398] Click the upload button and select the gameplay video file.

[0399] Once selected, confirm the upload.

[0400] Step 2:

[0401] Terminal

[0402] The selected gameplay video file is sent to the server as an HTTP request.

[0403] Show a progress bar for video uploads and notify the user once it's complete.

[0404] Step 3:

[0405] server

[0406] The received video file is saved in storage.

[0407] A unique ID is assigned to the video file, and metadata such as the file name and upload date and time are recorded in the database.

[0408] Step 4:

[0409] server

[0410] The saved video is passed to the analysis module.

[0411] Uses image recognition technology to detect important events in the game (e.g., defeating enemies, boss battles, level-ups, etc.).

[0412] Uses voice recognition technology to detect the user's voice and reactions.

[0413] Lists important events detected and audio reactions with timestamps.

[0414] Step 5:

[0415] server

[0416] The emotion engine is activated and emotions are detected by analyzing the user's voice and facial expressions in the video.

[0417] Add detected emotion data (e.g., happiness, surprise, anger, etc.) to the analysis list.

[0418] Step 6:

[0419] server

[0420] Pass the event list and sentiment data to a natural language generation (NLG) engine.

[0421] The NLG engine generates live commentary corresponding to events. For example, if the user is recognized as "surprised" by the event "Boss battle begins," it generates a comment such as "The boss appeared here! What a surprise!"

[0422] Add appropriate comments from a database of professional commentators and celebrities, if needed.

[0423] Step 7:

[0424] server

[0425] The generated commentary is passed to a video editing engine and overlaid on the original gameplay video.

[0426] Based on the comment timestamps, text and audio are inserted into the video to generate the final commentary video.

[0427] The completed live video is saved to storage and the metadata is updated in the database.

[0428] Step 8:

[0429] User

[0430] Log in to your user account and check the generated live video.

[0431] Enter the settings (title, description, etc.) for sharing on the community page.

[0432] Click the share button to post your video to the community.

[0433] Step 9:

[0434] Terminal

[0435] The input contents of the shared settings are sent to the server as an HTTP request.

[0436] It displays the community page and provides an interface where users can view other users' videos.

[0437] Step 10:

[0438] server

[0439] Receives the share request and updates the video's database entry.

[0440] Add a new playthrough video entry to the community page.

[0441] Step 11:

[0442] User

[0443] Watch other users' live videos and rate them with likes and comments.

[0444] Step 12:

[0445] Terminal

[0446] The input contents of the rating and comment are sent to the server as an HTTP request.

[0447] Step 13:

[0448] server

[0449] The received rating data and comments are stored in a database and the video's rating score is updated.

[0450] Step 14:

[0451] server

[0452] We periodically tally the rating scores and select the top-rated videos.

[0453] Selected videos will be featured on the official special page.

[0454] Step 15:

[0455] User

[0456] See which videos are featured and get links to download or post your videos to third-party distribution platforms.

[0457] Step 16:

[0458] User

[0459] They post videos on third-party distribution platforms and aim to monetize through advertising revenue, Super Chats, etc.

[0460] Example 2

[0461] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0462] Previously, the process for users to convert their gameplay videos into play-by-play videos and then share and rate them was complicated and time-consuming. It was also difficult to generate natural play-by-play comments that reflected the user's emotions, resulting in content that often lacked realism. Furthermore, support for monetizing the play-by-play videos was insufficient, making the process complicated for users.

[0463] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for users to upload videos of electronic games they have played; means for analyzing the uploaded videos and detecting important events; means for analyzing users' emotions and adding the detected emotion data to the analysis results; means for generating live commentary comments in natural language based on the detected game events and emotion data; means for generating live commentary videos by overlaying the generated live commentary comments on videos; means for providing a community function for sharing the generated live commentary videos; means for other users to rate the shared live commentary videos; means for officially featuring well-rated live commentary videos based on the rating results; means for storing live commentary comments from professional commentators and celebrities in a database and appropriately selecting and adding these comments based on the detected game events and emotion data; and means for users to post the generated live commentary videos to third-party distribution platforms and monetize them. This allows users to easily convert their own gameplay videos into natural and realistic live commentary videos, and efficiently share, rate, and monetize them.

[0464] "User" refers to any individual or entity who uploads electronic gameplay videos to the system and generates, shares, and monetizes live gameplay videos.

[0465] "Electronic game" refers to an interactive entertainment activity that operates on a computer, gaming console, mobile device, or the like.

[0466] "Video" refers to a series of digital files containing the video and audio of an electronic game played by a user.

[0467] "Upload" refers to the act of a user sending a video file from their own device to the system's server.

[0468] "Analysis" refers to the process by which the system automatically analyzes uploaded videos to detect important events and user sentiment.

[0469] "Events" refer to important actions or scenes that occur within the game, such as the start of a boss battle or the acquisition of an item.

[0470] "Emotion" refers to psychological reactions detected from a user's facial expressions and tone of voice, including joy, surprise, anger, etc.

[0471] "Natural language" refers to the language used by humans on a daily basis and is used to generate commentary in text or audio form.

[0472] "Live commentary" refers to text such as explanations and impressions generated based on events that occur during gameplay and the user's emotions.

[0473] "Overlay" refers to the process of displaying commentary at specific timestamps on the original game footage.

[0474] The "community function" refers to an online platform that allows users to share their own live videos with other users and exchange opinions and ratings.

[0475] "Rating" refers to the rating actions other users take on a shared live video, including comments, likes, and number of views.

[0476] "Feature" refers to the system officially recommending well-rated live videos so that they can be viewed by more users.

[0477] A "professional commentator" refers to an individual or group that professionally broadcasts games and provides high-quality commentary using their knowledge and experience.

[0478] "Celebrities" are publicly known or influential individuals whose live commentary is stored and used in a database.

[0479] "Database" refers to a software system that stores information in an organized manner and makes it available for retrieval and use as needed.

[0480] "Third-party distribution platforms" refers to online services, such as YouTube, that are used to share and monetize the live-action videos generated.

[0481] "Monetization" refers to the process of earning advertising revenue, donations from viewers, etc. from user-generated live videos.

[0482] As an embodiment of the invention, the program processing of this system is specifically described below. This system allows users to easily convert gameplay videos into live commentary videos, share and rate them within the community, and further supports monetization of the created videos.

[0483] composition

[0484] The system includes the following major hardware and software components:

[0485] User's device (e.g. PC or smartphone)

[0486] Server (cloud-based)

[0487] Storage systems (e.g. cloud storage)

[0488] Database (e.g. MySQL)

[0489] Image recognition technology (e.g., OpenCV)

[0490] Voice recognition technology (e.g., Google Speech-to-Text)

[0491] Sentiment analysis engine (e.g. Affectiva SDK)

[0492] Natural Language Generation (NLG) engines (e.g., GPT-3)

[0493] Video editing engine (e.g. FFmpeg)

[0494] Distribution platform (e.g. YouTube)

[0495] Operation flow

[0496] (1) The user saves the game video they played on their device and accesses the system's upload page. The user clicks the upload button, opens a file selection dialog, and selects the video file.

[0497] (2) The device converts the selected video file into an HTTP POST request and sends it to the server. The server analyzes the received HTTP POST request and saves the uploaded video file in cloud storage. At the same time, it assigns a unique ID to the video file and saves its metadata (file name, upload date and time, etc.) in a database.

[0498] (3) The server reads the video files stored in the storage for analysis and detects important events in the game using image recognition technology (e.g., OpenCV) and voice recognition technology (e.g., Google Speech-to-Text). For each event, the timestamp and event content are saved in a list format.

[0499] (4) Based on the analysis results of the detected game events, the server launches an emotion engine (e.g., Affectiva SDK) to analyze the user's facial expressions and tone of voice in the video, extracting emotional appeal points and adding the detected emotion data (e.g., joy, surprise, anger, etc.) to the analysis results.

[0500] (5) The server passes the event list and emotion data to a natural language generation (NLG) engine (e.g., GPT-3) to generate commentary corresponding to each event. For example, if surprise is detected at the start of a boss battle, the server generates a comment such as "I can't believe it! A monster appeared at this exact moment!" If necessary, appropriate comments can be selected and added from a database of comments by professional commentators or celebrities.

[0501] (6) The server passes the generated commentary and the original game video to a video editing engine (e.g., FFmpeg), which overlays the commentary on the video based on the corresponding timestamps to generate a commentary video. The completed commentary video is saved in cloud storage, and its metadata is added to a database.

[0502] (7) The user logs in to their account and shares the created live video on the community page. The user clicks the share button, enters the video title and description, and shares it.

[0503] (8) The device sends a sharing request to the server, which adds a new live video entry to the community page. Other users watch and rate the video, and the rating data is stored in a database.

[0504] (9) Users can post their live streams to third-party platforms (e.g., YouTube) and monetize them. The system provides links to facilitate posting and provides users with monetization guidelines.

[0505] Specific examples

[0506] For example, suppose User B uploads a 15-minute video of himself playing a horror game. The server analyzes the video and detects scenes in which monsters appear and scenes in which the user is surprised. The emotion engine detects the user's surprise, and based on that emotional data, the AI ​​generates commentary such as "I can't believe it! A monster appeared at this exact moment!". The generated commentary is then overlaid on the original video to complete the commentary video. User B shares the completed commentary video with the community, and if it receives high ratings from other users, it can be posted on distribution platforms such as YouTube, providing an opportunity for monetization.

[0507] Example of input prompt for generative AI model

[0508] "Please explain the system that analyzes gameplay videos uploaded by users, generates live commentary using an emotion engine and a natural language generation engine, and supports sharing and monetization."

[0509] In this way, by utilizing the emotion engine, the present invention can generate immersive live videos that correspond to the user's emotions, making the sharing and monetization process simple and effective.

[0510] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0511] Step 1: User uploads game videos

[0512] Input: Gameplay video file

[0513] Output: Notification of upload completion to the server

[0514] explanation:

[0515] Users access the upload page to upload gameplay videos stored on their device. The user clicks the upload button, opens a file selection dialog, and selects a video file. The device converts the selected video file into an HTTP POST request and sends it to the server. This process also includes displaying the video file upload progress to the user in real time. The server analyzes the received HTTP POST request and saves the uploaded video file in cloud storage. At the same time, it assigns a unique ID to the video file and saves its metadata (file name, upload date and time, etc.) in a database. After the upload is complete, the server returns a notification that the upload is complete.

[0516] Step 2: Analyze the video

[0517] Input: Video file stored in cloud storage

[0518] Output: Event list, timestamp list

[0519] explanation:

[0520] The server passes the video files stored in cloud storage to an analysis module. This analysis module uses image recognition technology (e.g., OpenCV) to analyze each frame and detect important events in the game. It also uses speech recognition technology (e.g., Google Speech-to-Text) to analyze audio data and identify important event sounds. The analysis results are saved in a list format, with the timestamp and event content for each event.

[0521] Step 3: Introducing the Emotion Engine

[0522] Input: Video file, event list, timestamp list

[0523] Output: Emotion data list

[0524] explanation:

[0525] Based on the analysis, the server launches an emotion engine (e.g., Affectiva SDK). The emotion engine analyzes the user's facial expressions and tone of voice in the video to detect emotions such as joy, surprise, and anger. The detected emotion data is added to an emotion data list along with a timestamp for each event.

[0526] Step 4: Generate commentary

[0527] Input: Event list, timestamp list, emotion data list

[0528] Output: List of live commentary

[0529] explanation:

[0530] The server passes the event list and emotion data to a natural language generation (NLG) engine (e.g., GPT-3) to generate commentary corresponding to each event. For example, if surprise is detected for the event "Boss battle begins," the server generates a comment such as "That's unexpected! The boss battle has begun!". Furthermore, it is possible to select and add appropriate comments from a database of comments from professional commentators and celebrities. This generates a list of commentary comments.

[0531] Step 5: Rendering the play-by-play video

[0532] Input: Game video file, commentary list, timestamp list

[0533] Output: Finished playthrough video file

[0534] explanation:

[0535] The server then passes the generated commentary and the original game video to a video editing engine (e.g., FFmpeg), which overlays the commentary on the video based on the corresponding timestamps to generate a commentary video. During this process, it also adjusts effects, text position, font size, etc., and finally generates a completed commentary video file that is saved in cloud storage. The server then adds the metadata to a database.

[0536] Step 6: Share your video

[0537] Input: Finished playthrough video file

[0538] Output: Shared play-by-play video entries

[0539] explanation:

[0540] The user logs in to their account and shares the created live commentary video on the community page. The user clicks the share button and enters the video title and description. The device sends a share request to the server, and the server adds a new live commentary video entry to the community page. When other users watch or rate this entry, the rating data is stored in the database.

[0541] Step 7: Monetization Support

[0542] Input: Completed play-by-play video files, shared play-by-play video entries

[0543] Output: Monetized link

[0544] explanation:

[0545] Users can post the live stream videos they create to third-party distribution platforms (e.g., YouTube) and monetize them. The system provides links to facilitate posting and provides users with monetization guidelines, allowing users to smoothly progress through the steps toward monetization.

[0546] (Application example 2)

[0547] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0548] In today's game streaming, simply sharing gameplay videos is no longer enough to maintain user participation and viewer interest. There is a need for a simple method for users to breathe life into their gameplay videos, create more immersive commentary videos, and monetize them. In particular, incorporating natural and realistic commentary that reflects the user's emotions into the videos is crucial to sustaining viewer interest.

[0549] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0550] In this invention, the server includes means for users to upload videos of games they have played, means for analyzing the uploaded videos and detecting important game events, means for detecting user emotions, means for generating commentary comments in natural language based on the detected emotions, means for generating commentary videos by overlaying the generated commentary comments on videos, means for providing a community function for sharing the generated commentary videos, means for other users to rate the shared commentary videos, and means for officially featuring commentary videos that have received good ratings based on the rating results. This enables users to easily generate immersive commentary videos that reflect their own emotions, have them shared and rated within the community, and further monetize them.

[0551] "User" refers to a person who uses this system to upload their own gameplay videos and create and share live play videos.

[0552] "Gameplay video" is video data that records a user playing a game.

[0553] "Means for uploading" refers to the function that allows users to send videos from their own devices to the server.

[0554] "Means for analyzing videos" refers to a function that analyzes the content of uploaded videos using computer vision technology, etc., and detects important events.

[0555] "Significant game events" are notable occurrences or progressions within the game, including boss battles, item acquisition, etc.

[0556] "Means for detecting emotions" refers to technology that reads emotions such as joy, surprise, and anger from the user's voice and facial expressions in gameplay videos.

[0557] "Live commentary" refers to text data of narration and explanation generated for game events.

[0558] "Natural language generation" is a technology that automatically creates natural language using a computer program.

[0559] "Overlay" is a technique that displays additional information (text or images) on top of gameplay video.

[0560] A "play-by-play video" is a new video generated by overlaying play-by-play commentary on the original gameplay video.

[0561] The "Community Function" refers to an online platform where users can share their created live videos and have other users watch and rate them.

[0562] "Means for sharing" refers to a function for making the generated live video available to other users.

[0563] "Means for rating" refers to a function that allows users to give ratings (e.g., favorites, comments, rating points) to shared live videos.

[0564] A "good review video" is a review video that has received high ratings from other users.

[0565] "Official Feature Option" is a feature that allows highly rated live videos to be featured and viewed by more users.

[0566] A "professional commentator" is a person or organization that specializes in creating live commentary.

[0567] A "celebrity" is someone who is widely known and influential to the public.

[0568] A "database" is a structured collection of information that stores data systematically and makes it easy to search and update.

[0569] A "distribution platform" is an online service that allows you to make videos available to other viewers and earn revenue.

[0570] "Means of monetization" refers to the methods and functions for generating revenue such as advertising revenue and viewing fees through the generated live video.

[0571] The system embodying the present invention allows users to easily upload gameplay videos, generate realistic commentary using an emotion engine based on the analyzed video, share and rate the commentary videos within the community, and ultimately monetize them. This system is composed of the following means:

[0572] Upload and analyze

[0573] User

[0574] Users upload videos of the games they play to the server from their smartphones or other devices. The uploaded videos are sent to the server as HTTP requests.

[0575] server

[0576] The server stores the received video in storage and assigns a unique ID to the video. Next, it analyzes the stored video and uses computer vision technology to detect important events in the game. As a result of the analysis, a list of timestamps and content for each event is generated.

[0577] Emotion detection

[0578] server

[0579] The server activates an emotion engine to detect emotions such as joy, surprise, and anger from the user's voice and facial expressions in the video. The detected emotion data is added to the analysis results.

[0580] Generate live commentary

[0581] server

[0582] The server passes the analysis results to a natural language generation (NLG) engine, which generates appropriate commentary based on the detected game events and emotional data. For example, if the server detects a user's surprise, it might generate a comment like, "That's unexpected! The boss battle has begun!"

[0583] Rendering live video

[0584] server

[0585] The server then launches a video editing engine to overlay the generated commentary onto the video. The comments, adjusted based on the emotion data, are inserted into the gameplay video based on the corresponding timestamp. The completed video is then saved back to storage, and metadata is added to the database.

[0586] Share and rate videos

[0587] User

[0588] Users log in to their accounts and share the created live commentary videos on the community page. Other users can watch the shared live commentary videos and rate them. This rating data is stored on the server.

[0589] server

[0590] The server processes the incoming share requests and adds new live video entries to the community page. Highly rated videos are officially featured.

[0591] Monetization support

[0592] User

[0593] Users can then post the live commentary videos they create on third-party distribution platforms in an effort to generate revenue.

[0594] server

[0595] The server provides links to facilitate posting and provides users with monetization guidelines.

[0596] Examples and prompts

[0597] As a concrete example, let's say a user uploads a 15-minute video of themselves playing a horror game. The server analyzes the video and detects scenes in which monsters appear and scenes in which the user is surprised. The emotion engine detects the user's surprise, and based on that emotional data, the AI ​​generates commentary comments such as "I can't believe it! A monster appeared at this exact moment!" The commentary video, which is completed by overlaying these comments on the original video, is then shared by the user within the community.

[0598] Example prompt for a generative AI model:

[0599] Prompt: This video contains a scene in which a monster suddenly appears while the user is playing a horror game. The user's facial expression shows a look of surprise. Please generate a commentary that matches this situation.

[0600] Expected output: Unbelievable! A monster appeared at this exact moment! This is so scary!

[0601] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0602] Step 1:

[0603] The user prepares a video file of the game they played and accesses the system's upload page from their smartphone or other device. They select the target video file from the file selection dialog and click the upload button. At this time, the input is the user's gameplay video file, and the output is a notification that the video file has been uploaded.

[0604] Step 2:

[0605] The device converts the selected video file into an HTTP request and sends it to the server. The server analyzes the received HTTP request and saves the video file in storage. The server assigns a unique ID to this video file and saves the metadata (file name, upload date and time, etc.) in a database. The input is the uploaded video file, and the output is the video file and its metadata saved in storage.

[0606] Step 3:

[0607] The server analyzes the video stored in storage. First, it launches a video analysis engine and uses computer vision techniques to detect important game events in the video. This includes image processing and object recognition between specific frames. The output is a list of the timestamp and content of each event.

[0608] Step 4:

[0609] The server activates an emotion engine to analyze the user's voice and facial expressions in the video to detect the user's emotions. The emotion engine uses voice analysis and facial expression recognition technology to extract the user's emotional data (e.g., joy, surprise, etc.) at a specific time in the video. The input is the user's voice and facial expression data, and the output is emotional data.

[0610] Step 5:

[0611] The server passes the detected game events and emotion data to a natural language generation (NLG) engine. The NLG engine generates commentary corresponding to each event based on this data. For example, if the emotion of surprise is detected, the commentary generated is "This is unexpected! The boss battle has begun!" The input is the game events and emotion data, and the output is the generated commentary.

[0612] Step 6:

[0613] The server launches a video editing engine to overlay the generated commentary on the gameplay video. The video editing engine inserts the comments at the appropriate positions based on the comment timestamps and generates the final commentary video. The input is the commentary and the original gameplay video, and the output is the commentary video.

[0614] Step 7:

[0615] The server saves the completed commentary video in storage and adds its metadata to the database. The generated commentary video is uploaded to the community page in a viewable state. The input is the completed commentary video, and the output is the commentary video saved in storage.

[0616] Step 8:

[0617] Users log in to their accounts and share the generated commentary videos on the community page. Other users can watch and rate the shared commentary videos. The input is the generated commentary video, and the output is the user's rating data.

[0618] Step 9:

[0619] The server stores the evaluation results in a database and officially features the highly rated live videos, allowing them to be displayed to a wider range of users. The input is the evaluation results, and the output is the featured videos.

[0620] Step 10:

[0621] Users can then post the live commentary videos they create to third-party distribution platforms for monetization. The server provides links and monetization guidelines to support posting. The input is the live commentary video and guidelines, and the output is a notification that posting has been completed to the distribution platform.

[0622] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0623] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0624] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0625] [Second embodiment]

[0626] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0627] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0628] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0629] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0630] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0631] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0632] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0633] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0634] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0635] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0636] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0637] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0638] The system of this invention allows users to easily convert gameplay videos into live commentary videos, share and rate them within the community, and also supports monetization of the videos. This system is mainly composed of three entities: a server, a device, and users.

[0639] Main features

[0640] Game video upload

[0641] User

[0642] Users upload videos of the games they play to the system via their terminals.

[0643] Select the video file and click the upload button to send the video to the system.

[0644] Terminal

[0645] The terminal converts the game video file selected by the user into an HTTP request and sends it to the server.

[0646] Notify the user that the video was successfully sent.

[0647] server

[0648] The received video file is saved in storage, and the video metadata (file name, upload date and time, etc.) is recorded in the database.

[0649] Video Analysis

[0650] server

[0651] The server runs software to analyze the stored video.

[0652] Using image and voice recognition technology, it detects important events in the game (e.g., defeating enemies, boss battles, level-ups, etc.) along with their timeline.

[0653] Generate a list of detected event information (event type, occurrence time).

[0654] Generate live commentary

[0655] server

[0656] Based on the detected event list, a natural language generation (NLG) engine is used to generate corresponding commentary.

[0657] The generated commentary includes phrases such as "Great! You defeated the enemy!" for "You defeated the enemy."

[0658] Select appropriate comments from a database of professional commentators and celebrities and add them as needed.

[0659] Rendering live video

[0660] server

[0661] The generated commentary is overlaid onto the gameplay video.

[0662] Use a video editing engine to overlay text onto the video based on the corresponding timestamps.

[0663] The completed live video is saved to storage and metadata is added to the database.

[0664] Share and rate videos

[0665] User

[0666] Users log in to the system to share the generated live video on a community page.

[0667] Click the share button and enter a title and description for your video.

[0668] You can watch live videos shared by other users and leave likes and comments.

[0669] Terminal

[0670] Receives share and rating requests and sends them to the server.

[0671] Display a community page where users can easily watch and rate other videos.

[0672] server

[0673] Processes incoming share and rating requests and updates the database with video rating scores and comments.

[0674] We will periodically tally rating scores and officially feature the top-rated videos.

[0675] Monetization support

[0676] User

[0677] The generated live commentary videos will be posted on third-party distribution platforms in order to generate revenue.

[0678] Get a download link from the system and upload the video to a distribution platform.

[0679] server

[0680] Provide links to facilitate posting and provide users with monetization guidelines.

[0681] Specific examples

[0682] For example, suppose User A uploads a 10-minute video of himself playing an action game. The server analyzes the video and detects scenes where enemies are defeated or boss battles are fought. The AI ​​generates commentary such as "Great attack!" or "Great opportunity!" and overlays it on the original video. The final commentary video is shared by User A to the community and receives high ratings from other users. As a result of the high ratings, the server officially features the video, and User A can then post the video on YouTube to earn revenue.

[0683] In this way, the system of the present invention makes the process of creating, sharing, rating, and monetizing gameplay videos simple and effective.

[0684] The processing flow will be explained below.

[0685] Step 1:

[0686] User

[0687] Prepare your gameplay video on your device and access the upload page.

[0688] Click the Upload button and select the video file to upload from the file selection dialog.

[0689] Step 2:

[0690] Terminal

[0691] The selected video file is converted into an HTTP request and sent to the server.

[0692] Show the upload progress to the user.

[0693] Step 3:

[0694] server

[0695] Analyze the received HTTP request and save the video file in storage.

[0696] A unique ID is assigned to the video file, and metadata (file name, upload date and time, etc.) is stored in the database.

[0697] Step 4:

[0698] server

[0699] Load the video stored in storage for analysis.

[0700] It uses image and voice recognition technology to detect important events in the game.

[0701] List the timestamp and event details for each event.

[0702] Step 5:

[0703] server

[0704] Pass the event list to a natural language generation (NLG) engine.

[0705] The NLG engine generates commentary corresponding to events. For example, for the event "Boss battle begins," it generates a comment like "This is where the real action begins, the boss battle begins!"

[0706] If necessary, add appropriate comments from a database of professional commentators and celebrities.

[0707] Step 6:

[0708] server

[0709] A video editing engine is launched to overlay the generated commentary on the original game video.

[0710] Insert comments into the video based on appropriate timestamps to generate a play-by-play video.

[0711] The completed live video is saved to storage and metadata is added to the database.

[0712] Step 7:

[0713] User

[0714] Log in to your account and share the generated commentary video on the community page.

[0715] Click the share button and enter a title and description for your video.

[0716] Step 8:

[0717] Terminal

[0718] Sends a share request to the server.

[0719] It provides an interface for accessing community pages and watching live videos shared by other users.

[0720] Step 9:

[0721] server

[0722] Process incoming share requests and add new play-by-play video entries to the community page.

[0723] When other users watch or rate a video, the rating data is saved in a database.

[0724] Step 10:

[0725] server

[0726] We periodically tally the rating scores and select the top-rated videos.

[0727] Selected videos will be featured on the official special page.

[0728] Step 11:

[0729] User

[0730] Check out the featured videos.

[0731] Get a link to download the video or post it to a third-party distribution platform.

[0732] Step 12:

[0733] User

[0734] Post videos to distribution platforms and monetize them. After posting, use revenue models such as advertising revenue and Super Chat.

[0735] The above is the flow of specific processing steps for carrying out the invention.

[0736] Example 1

[0737] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0738] Conventional gameplay video sharing systems require users to manually edit videos and add commentary, which is time-consuming and labor-intensive. Furthermore, the process of sharing, rating, and monetizing the created videos is complicated, making it difficult for users to use. The objective of this invention is to solve these problems and provide a system that allows users to easily convert gameplay videos into commentary videos and effectively share, rate, and monetize them.

[0739] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0740] In this invention, the server includes: means for users to upload videos of games they play; means for converting the video files into HTTP requests and sending them to the server; means for saving the received video files in storage and recording metadata in a database; means for analyzing the uploaded videos, detecting in-game events using image recognition technology and voice recognition technology, and generating a list of event information; means for generating commentary comments using a natural language generation engine based on the detected game events and comparing them with a database to select appropriate comments; means for overlaying the generated commentary comments on videos to generate commentary videos; means for providing a community function for sharing the generated commentary videos; means for other users to rate the shared commentary videos; means for officially featuring commentary videos with good ratings based on the rating results; and means for providing a link for posting the generated commentary videos to a third-party distribution platform for monetization. This enables users to easily analyze, edit, and share gameplay videos and effectively monetize them.

[0741] "User" means any individual or legal entity that uses the System to upload, analyze, share, and rate gameplay videos.

[0742] A "gameplay video" is a digital file that contains video and audio of a game being played by a user.

[0743] "Uploading" is the process by which a user transfers a video file from their device to the system.

[0744] An "HTTP request" is a request message for sending data to a server based on a protocol used for communication over the Internet.

[0745] A "server" is a computer system that processes and stores data received from users and provides the required services.

[0746] "Storage" refers to a storage device for saving digital data, including hard disk drives (HDDs) and solid-state drives (SSDs).

[0747] "Metadata" is data that describes the characteristics and content of a video file, and includes, for example, the file name, upload date and time, and video length.

[0748] A "database" is a system for efficiently storing, retrieving, and updating structured information.

[0749] "Image recognition technology" is a technology that analyzes the content of images and videos and identifies specific objects and movements.

[0750] "Speech recognition technology" is a technology that analyzes voice data and converts it into meaningful text or commands.

[0751] A "natural language generation engine" is an artificial intelligence technology that generates text based on input data in a form that is easy for humans to understand.

[0752] "Commentary" is text that includes commentary and impressions about specific events in gameplay videos.

[0753] "Overlay" is a technique for displaying text or images over specific locations on a video.

[0754] "Community Features" are features within the system that allow users to share and rate generated content.

[0755] A "rating" is feedback or a score that a user gives to another user's content.

[0756] A "feature" is a display of specific content that stands out more than other content.

[0757] "Monetization" is the process of utilizing generated content for financial gain.

[0758] This invention is a system that allows users to easily upload gameplay videos and then analyze, edit, share, rate, and monetize them. This system operates in cooperation with a server, devices, and users.

[0759] To upload a video of a game that a user has played, the user first logs in to the system. After logging in, the user selects the gameplay video file from their device and clicks the upload button. At this time, the device converts the video file into an HTTP request and sends it to the server. The uploaded video file is received by the server and saved in storage. The server also records the video's metadata (file name, upload date and time, etc.) in a database.

[0760] Next, the server launches software to analyze the saved video file. Specifically, it uses image recognition technology (e.g., OpenCV) and speech recognition technology (e.g., Google Speech-to-Text API) to detect important in-game events (e.g., defeating an enemy, leveling up, etc.). The server then generates a list of detected event information (event type, occurrence time). The server then uses this event list to generate commentary using a natural language generation (NLG) engine (e.g., GPT-3). The generated commentary is then compared with a database of comments from professional commentators and celebrities to select appropriate comments.

[0761] The server then uses a video editing engine (e.g., FFmpeg) to overlay the commentary onto the video. During this process, commentary generated based on timestamps is overlaid onto specific locations in the gameplay video. The completed commentary video is saved to storage, and the metadata is added to a database.

[0762] Users can share the live commentary videos they have created on the system's community page. When sharing, users enter a title and description for the video. The device then sends a sharing request to the server, allowing other users to view and rate the shared live commentary video. The server processes the received rating request and updates the video's rating score and comments in the database. Based on the rating results, live commentary videos that receive good ratings are officially featured.

[0763] Additionally, users can monetize their generated live streams by posting them to third-party distribution platforms, and the server will provide users with links and guidelines to support monetization.

[0764] As a concrete example, suppose User A uploads a 10-minute video of himself playing an action game. The server analyzes the video and detects scenes where enemies are defeated and boss battles. The AI ​​generates commentary such as "Great attack!" or "Great opportunity!" and overlays it on the original video. The final commentary video is shared by User A with the community and receives high ratings from other users. As a result of the high ratings, the server officially features the video, and User A can then post the video on distribution platforms such as YouTube to earn revenue.

[0765] An example of a prompt might be:

[0766] "I want to create a program that analyzes gameplay video, generates commentary based on important events, and then creates a commentary video. Please explain the steps below:

[0767] 1. Users upload game videos

[0768] 2. Analyze video on the server and detect important events

[0769] 3. Create live commentary using natural language generation

[0770] 4. Overlay your comments on the video and you're done!

[0771] The system operates based on this content, improving user convenience.

[0772] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0773] Step 1:

[0774] User logs in and uploads game video

[0775] To log in to the system, the user enters their username and password. After successful login, the user selects a gameplay video file from the terminal and clicks the upload button. The input is the user information and the game video file, and the output is an HTTP request. When the upload button is clicked, the terminal converts the selected video file into an outgoing packet as an HTTP POST request and sends it to the server.

[0776] Step 2:

[0777] The device converts the video file into an HTTP request and sends it to the server.

[0778] The device follows the user's instructions, converts the selected video file into an HTTP request, and sends it to the server using the POST method. The input is the video file, and the output is an HTTP request. Specifically, the device embeds the file's binary data in the body of the HTTP POST request and sends it to the specified URL.

[0779] Step 3:

[0780] The server receives the video file and stores it in storage.

[0781] The server extracts the video file from the body of the HTTP request received and saves it to storage. The input is the HTTP request, and the output is the saved video file. Specifically, the file system API is used to save the video file to a specified directory.

[0782] Step 4:

[0783] The server records the video metadata in a database

[0784] The server extracts metadata from the saved video files (file name, upload date and time, video length, etc.) and records it in a database. The input is the saved video file, and the output is a database record. Specifically, it obtains the file attribute information and inserts the record using the database API.

[0785] Step 5:

[0786] The server analyzes the video using video analysis software.

[0787] The server runs video analysis software (e.g., OpenCV) and uses image recognition technology to detect important in-game events (e.g., defeating an enemy, leveling up, etc.). The input is the saved video file, and the output is a list of event information. Specifically, it processes each frame of the video, detecting specific in-game events, and lists them along with their timestamps.

[0788] Step 6:

[0789] The server uses speech recognition technology to identify audio events and record them as metadata.

[0790] The server uses speech recognition technology (e.g., Google Speech-to-Text API) to identify important audio events from the video's audio track and record them as metadata. The input is the saved video file and the event information list, and the output is an updated event information list. Specifically, the server analyzes the audio data, converts it into text strings in each language, and records them along with the time when the audio events occurred.

[0791] Step 7:

[0792] The server generates live commentary using a natural language generation engine

[0793] The server uses a natural language generation (NLG) engine (e.g., GPT-3) to generate commentary based on the detected event information. The input is a list of event information, and the output is a list of commentary. Specifically, a prompt sentence is input to the generative AI model, which then generates appropriate commentary.

[0794] Step 8:

[0795] The server compares the generated live commentary with the database and selects the appropriate comment.

[0796] The server compares the generated live commentary comments with a database of comments from professional commentators and celebrities to select appropriate comments. The input is a list of generated live commentary comments, and the output is a list of matched live commentary comments. Specifically, it uses a string search algorithm to extract matching comments from the database.

[0797] Step 9:

[0798] The server uses a video editing engine to overlay commentary onto the video.

[0799] The server uses a video editing engine (e.g., FFmpeg) to overlay the generated commentary based on the corresponding timestamps of the video. The input is a gameplay video file and a list of commentary, and the output is the finished commentary video. Specifically, it applies a text overlay filter to specific frames of the video.

[0800] Step 10:

[0801] The server saves the completed live video to storage and adds the metadata to the database.

[0802] The server saves the completed live video in storage and adds the video's metadata (such as the creation date and time, video playback time, and comment list) to the database. The input is the completed live video and a list of live comments, and the output is an updated database record. Specifically, the server saves the video file to the storage system and inserts its metadata into the database.

[0803] Step 11:

[0804] Share user-generated play-by-play videos on your community page

[0805] A user logs in to the system and performs an operation to share the generated live video on a community page. The input is the generated live video, and the output is a sharing request. The user clicks the share button and enters the video title and description.

[0806] Step 12:

[0807] The device sends a share request to the server

[0808] The device sends the user's sharing request to the server as an HTTP POST request. The input is the sharing request, and the output is an HTTP POST request. Specifically, the sharing request is packetized along with the video metadata and sent to the server.

[0809] Step 13:

[0810] The server processes share and rating requests and updates the rating scores and comments to the database.

[0811] The server processes the received share requests, adds new video information to the database, and processes rating requests made by other users on the shared live videos, updating the rating scores and comments in the database. The input is the share request and the rating request, and the output is the updated database record.

[0812] Step 14:

[0813] The server will officially feature well-rated playthrough videos based on the evaluation results.

[0814] The server periodically aggregates the rating scores and officially features the top-rated live videos. The input is the rating scores and comments, and the output is a list of featured videos. Specifically, it calculates the ranking of the rating scores and displays the top-rated videos in a specific section.

[0815] Step 15:

[0816] Monetize user-generated live videos by posting them to a streaming platform

[0817] Users post their generated commentary videos to third-party distribution platforms (e.g., YouTube, Twitch) and monetize them. The input is the generated commentary video, and the output is the post to the distribution platform. The system provides users with links and guidelines to support monetization.

[0818] Step 16:

[0819] Provide users with links and guidelines to help your server monetize

[0820] The server displays links and guidelines for video monetization to users. The input is a user request, and the output is monetization links and guidelines. Specifically, the server displays information related to monetization on the system dashboard for easy access by users.

[0821] The above are the specific processing steps of this system.

[0822] (Application example 1)

[0823] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0824] Currently, it takes a lot of time and effort for users to easily edit, rate, and monetize their gameplay videos. Furthermore, further technological advancements are required to realize advanced features such as seamless integration with content distribution platforms and highly accurate comment generation and overlays using AI technology. The purpose of this invention is to solve these issues and provide a simple system that allows users to easily and effectively create and share gameplay videos and further monetize them.

[0825] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0826] In this invention, the server includes means for allowing users to upload videos of games they have played, means for analyzing the uploaded videos and detecting important game events, means for generating commentary comments in natural language based on the detected game events, means for generating commentary videos by overlaying the generated commentary comments on videos, means for providing a community function for sharing the generated commentary videos, means for other users to rate the shared commentary videos, means for officially featuring commentary videos that have received positive reviews based on the rating results, means for seamlessly sharing the generated commentary videos to a third-party content distribution platform, and means for accurately overlaying commentary comments based on timestamps of game events using AI technology. This allows users to easily and effectively create and share commentary videos, enabling monetization through highly accurate commentary generation and seamless distribution.

[0827] "User" means an individual or organization that uses this system to upload gameplay videos and create and share live gameplay videos.

[0828] "Game videos" refer to video files recorded by users playing games.

[0829] "Uploading" refers to the act of a user sending gameplay video from their device to a server.

[0830] "Analysis" refers to the process by which the server processes the uploaded video and detects important game events.

[0831] "Game Event" refers to an important event that occurs within the game (e.g., defeating an enemy, fighting a boss, leveling up, etc.).

[0832] "Natural language" refers to the language used by humans on a daily basis, and is used in this system to generate live commentary.

[0833] "Live commentary" refers to comments that explain and evaluate gameplay in real time or near real time.

[0834] "Overlay" refers to the process of displaying the generated commentary overlaid on the game video.

[0835] "Live commentary video" refers to a video file created by overlaying live commentary on gameplay video.

[0836] "Community Function" refers to a platform where users can share live videos they have created and other users can watch, rate, and comment on them.

[0837] "Evaluation" refers to the act of users within a community rating a posted live video by giving it a "like" or comment.

[0838] "Featured" means that a live commentary video that has received positive reviews will be officially featured and introduced to more users.

[0839] "Content distribution platform" refers to a video sharing service provided by a third party, such as YouTube or other similar services.

[0840] "Seamless" means that the operations performed by the user are smoothly linked together, allowing a series of tasks to be performed without interruption.

[0841] "AI technology" refers to artificial intelligence technology, which in this system is used for natural language generation and analysis of game events.

[0842] A "timestamp" refers to data that records the time when a particular event occurred.

[0843] "Accuracy" refers to the degree to which live commentary and analysis results generated using AI technology are accurate as expected by users.

[0844] The embodiment of the present invention will be described in detail below. This system is mainly composed of three components: a server, a terminal, and a user.

[0845] 1. Uploading game videos

[0846] User

[0847] Users use a smartphone application to select the game video they have played and click the upload button to send it to the server.

[0848] Terminal

[0849] The device converts the game video file selected by the user into an HTTP request and sends it to the server. The device notifies the user that the video has been sent successfully.

[0850] server

[0851] The server stores the received video file in storage and records the video's metadata (file name, upload date and time, etc.) in a database.

[0852] 2. Video Analysis

[0853] server

[0854] The server runs software to analyze the saved video. The software used includes Google Cloud Vision API and Google Speech-to-Text. These are used for image and voice recognition to detect important events in the game (e.g., defeating an enemy, boss battles, level-ups, etc.). The results of this detection are compiled into an event list.

[0855] 3. Generating live commentary

[0856] server

[0857] Based on the detected event list, the server uses a natural language generation engine (e.g., GPT-4) to generate corresponding commentary, such as "When an enemy is defeated: Amazing! You defeated the enemy!". It also selects appropriate comments from a database of professional commentators and celebrities and adds them as needed.

[0858] An example prompt is "Generate a commentary for the following gaming event: 'Player defeats the boss in an epic fight'".

[0859] 4. Rendering the Live Video

[0860] server

[0861] The server processes the generated commentary onto the gameplay video. FFmpeg is used as the video editing software. The commentary is accurately overlaid onto the video based on the corresponding timestamp. The completed video is saved to storage, and metadata is added to the database.

[0862] 5. Share and rate the video

[0863] User

[0864] Users log in to the system to share the created live commentary video on the community page. They click the share button and enter the video title and description. They can also watch the live commentary videos shared by other users and leave "likes" and comments.

[0865] Terminal

[0866] The device receives and sends share and rating requests to the server, and also displays a community page where users can easily watch and rate other videos.

[0867] server

[0868] The server processes the incoming share and rating requests, updates the video's rating score and comments to a database, and periodically aggregates the rating scores and officially features the top-rated play-by-play videos.

[0869] 6. Monetization Support

[0870] User

[0871] Users can monetize the generated live videos by posting them to third-party distribution platforms. Users obtain a download link from the system and upload the video to the distribution platform.

[0872] server

[0873] The server provides links to facilitate posting and provides users with monetization guidelines.

[0874] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0875] Step 1:

[0876] Game video upload

[0877] Process flow:

[0878] A user launches the application on their smartphone, selects a gameplay video file from their device, and then clicks the upload button, which generates an HTTP request to send the video to the server.

[0879] Input and Output:

[0880] Input: Gameplay video file.

[0881] Output: HTTP request to the server.

[0882] Specific behavior:

[0883] When the user selects a video file and clicks the send button, the terminal generates an HTTP request for sending the selected video file to the server, and executes the transmission of the video.

[0884] Step 2:

[0885] Receiving and storing videos

[0886] Process flow:

[0887] The server receives the HTTP request sent from the device, saves the video file in storage, and records the video file's metadata (file name, upload date and time, etc.) in a database.

[0888] Input and Output:

[0889] Input: The gameplay video file sent in the HTTP request.

[0890] Output: Video files saved on storage, along with their metadata.

[0891] Specific behavior:

[0892] When the server receives an HTTP request, it saves the video data in storage and records the metadata of the video file in a database.

[0893] Step 3:

[0894] Video Analysis

[0895] Process flow:

[0896] The server runs image and voice recognition software to analyze the stored video, using the Google Cloud Vision API and Google Speech-to-Text to detect important game events (e.g., enemy defeats, boss battles, level-ups, etc.) and generate an event list.

[0897] Input and Output:

[0898] Input: Gameplay video file saved in storage.

[0899] Output: A list of game events.

[0900] Specific behavior:

[0901] The server loads the video file, performs image recognition using the Google Cloud Vision API and speech recognition using Google Speech-to-Text, and outputs a list of detected game events with timestamps.

[0902] Step 4:

[0903] Generate live commentary

[0904] Process flow:

[0905] The server generates commentary using a generative AI model (e.g., GPT-4) based on the detected event list. The comment is a phrase such as "When an enemy is defeated: Great! You defeated the enemy!"

[0906] Input and Output:

[0907] Input: A list of detected game events.

[0908] Output: A list of live commentary.

[0909] Specific behavior:

[0910] The server uses a natural language generation engine to generate the best commentary for each event from the event list using the prompt text as input. An example of a specific prompt text is "Generate a commentary for the following gaming event: 'Player defeats the boss in an epic fight'".

[0911] Step 5:

[0912] Rendering live video

[0913] Process flow:

[0914] The server processes the generated commentary overlays onto the gameplay video, using video editing software (e.g., FFmpeg) to overlay each commentary onto the video based on its corresponding timestamp.

[0915] Input and Output:

[0916] Input: Gameplay video file, commentary list.

[0917] Output: A commentary video file with commentary overlays.

[0918] Specific behavior:

[0919] The server uses FFmpeg to overlay the commentary onto the relevant parts of the video, generating the final commentary video.

[0920] Step 6:

[0921] Share and rate videos

[0922] Process flow:

[0923] Users can upload the created live commentary videos to a community page, where other users can view and rate them. Other users can also like and comment on the live commentary videos.

[0924] Input and Output:

[0925] Input: Live video file, share request.

[0926] Output: Play-by-play videos uploaded to the community page, and rating data.

[0927] Specific behavior:

[0928] A user clicks the share button, enters the title and description of the video, and posts it to the community page. The server receives the share request and performs the necessary processing to display the video on the community page. It also manages rating requests from other users and records the ratings in a database.

[0929] Step 7:

[0930] Monetization support

[0931] Process flow:

[0932] Users can then post their live streams to third-party platforms to monetize them, and the server will provide users with links and guidelines for monetization.

[0933] Input and Output:

[0934] Input: Play-by-play video file.

[0935] Output: Links to distribution platforms, monetization guidelines.

[0936] Specific behavior:

[0937] The server provides a link to facilitate posting and displays appropriate monetization guidelines to the user, who then uses the provided link to upload their live video to the distribution platform.

[0938] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0939] The system of the present invention allows users to easily convert gameplay videos into commentary videos, share and rate them within the community, and further supports monetization of the created videos. By combining this system with an emotion engine that recognizes the user's emotions, it is possible to generate more natural and realistic commentary comments. Specific embodiments of the present invention are described below.

[0940] Main features

[0941] Game video upload

[0942] User

[0943] The user prepares the game video they played on their device and accesses the upload page.

[0944] Click the Upload button and select the video file to upload from the file selection dialog.

[0945] Terminal

[0946] The selected video file is converted into an HTTP request and sent to the server.

[0947] Show the upload progress to the user.

[0948] server

[0949] Analyze the received HTTP request and save the video file in storage.

[0950] A unique ID is assigned to the video file, and metadata (file name, upload date and time, etc.) is stored in the database.

[0951] Video Analysis

[0952] server

[0953] Load the video stored in storage for analysis.

[0954] It uses image and voice recognition technology to detect important events in the game.

[0955] List the timestamp and event details for each event.

[0956] Introducing the Emotion Engine

[0957] server

[0958] It analyzes the user's voice and facial expressions in the video and activates an emotion engine to detect emotions.

[0959] The emotion engine adds detected emotion data (e.g., joy, surprise, anger, etc.) to the analysis results.

[0960] Adjust the tone and content of commentary based on detected emotional data.

[0961] Generate live commentary

[0962] server

[0963] Pass the event list to a natural language generation (NLG) engine.

[0964] The NLG engine generates commentary corresponding to the event. For example, if the detected emotion for the event "Boss battle begins" is "surprise," it generates a comment like "This is unexpected! The boss battle has begun!"

[0965] If necessary, add appropriate comments from a database of professional commentators and celebrities.

[0966] Rendering live video

[0967] server

[0968] A video editing engine is launched to overlay the generated commentary onto the gameplay video.

[0969] Comments adjusted based on the emotion data are inserted into the video based on the corresponding timestamps to generate a live video.

[0970] The completed live video is saved to storage and metadata is added to the database.

[0971] Share and rate videos

[0972] User

[0973] Log in to your account and share the generated commentary video on the community page.

[0974] Click the share button and enter a title and description for your video.

[0975] Terminal

[0976] Sends a share request to the server.

[0977] It provides an interface for accessing community pages and watching live videos shared by other users.

[0978] server

[0979] Process incoming share requests and add new play-by-play video entries to the community page.

[0980] When other users watch or rate a video, the rating data is saved in a database.

[0981] Monetization support

[0982] User

[0983] The generated live commentary videos will be posted on third-party distribution platforms in order to generate revenue.

[0984] Get a download link from the system and upload the video to a distribution platform.

[0985] server

[0986] Provide links to facilitate posting and provide users with monetization guidelines.

[0987] Specific examples

[0988] For example, suppose User B uploads a 15-minute video of himself playing a horror game. The server analyzes the video and detects scenes where monsters appear or scenes where the user is surprised. The emotion engine detects the user's surprise, and based on that emotion data, the AI ​​generates commentary such as "I can't believe it! A monster appeared at this exact moment!" These comments are overlaid on the original video, and User B shares the completed commentary video with the community. After receiving high ratings, the video is officially featured, and User B can further post the video on YouTube, earning revenue opportunities.

[0989] In this way, the system of the present invention uses an emotion engine to generate immersive live videos that correspond to the user's emotions, allowing for a simple and effective process of sharing, rating, and monetizing.

[0990] The processing flow will be explained below.

[0991] Step 1:

[0992] User

[0993] Save your gameplay videos to your device.

[0994] Access the game commentary generation system, log in, and proceed to the video upload page.

[0995] Click the upload button and select the gameplay video file.

[0996] Once selected, confirm the upload.

[0997] Step 2:

[0998] Terminal

[0999] The selected gameplay video file is sent to the server as an HTTP request.

[1000] Show a progress bar for video uploads and notify the user once it's complete.

[1001] Step 3:

[1002] server

[1003] The received video file is saved in storage.

[1004] A unique ID is assigned to the video file, and metadata such as the file name and upload date and time are recorded in the database.

[1005] Step 4:

[1006] server

[1007] The saved video is passed to the analysis module.

[1008] Uses image recognition technology to detect important events in the game (e.g., defeating enemies, boss battles, level-ups, etc.).

[1009] Uses voice recognition technology to detect the user's voice and reactions.

[1010] Lists important events detected and audio reactions with timestamps.

[1011] Step 5:

[1012] server

[1013] The emotion engine is activated and emotions are detected by analyzing the user's voice and facial expressions in the video.

[1014] Add detected emotion data (e.g., happiness, surprise, anger, etc.) to the analysis list.

[1015] Step 6:

[1016] server

[1017] Pass the event list and sentiment data to a natural language generation (NLG) engine.

[1018] The NLG engine generates live commentary corresponding to events. For example, if the user is recognized as "surprised" by the event "Boss battle begins," it generates a comment such as "The boss appeared here! What a surprise!"

[1019] Add appropriate comments from a database of professional commentators and celebrities, if needed.

[1020] Step 7:

[1021] server

[1022] The generated commentary is passed to a video editing engine and overlaid on the original gameplay video.

[1023] Based on the comment timestamps, text and audio are inserted into the video to generate the final commentary video.

[1024] The completed live video is saved to storage and the metadata is updated in the database.

[1025] Step 8:

[1026] User

[1027] Log in to your user account and check the generated live video.

[1028] Enter the settings (title, description, etc.) for sharing on the community page.

[1029] Click the share button to post your video to the community.

[1030] Step 9:

[1031] Terminal

[1032] The input contents of the shared settings are sent to the server as an HTTP request.

[1033] It displays the community page and provides an interface where users can view other users' videos.

[1034] Step 10:

[1035] server

[1036] Receives the share request and updates the video's database entry.

[1037] Add a new playthrough video entry to the community page.

[1038] Step 11:

[1039] User

[1040] Watch other users' live videos and rate them with likes and comments.

[1041] Step 12:

[1042] Terminal

[1043] The input contents of the rating and comment are sent to the server as an HTTP request.

[1044] Step 13:

[1045] server

[1046] The received rating data and comments are stored in a database and the video's rating score is updated.

[1047] Step 14:

[1048] server

[1049] We periodically tally the rating scores and select the top-rated videos.

[1050] Selected videos will be featured on the official special page.

[1051] Step 15:

[1052] User

[1053] See which videos are featured and get links to download or post your videos to third-party distribution platforms.

[1054] Step 16:

[1055] User

[1056] They post videos on third-party distribution platforms and aim to monetize through advertising revenue, Super Chats, etc.

[1057] Example 2

[1058] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1059] Previously, the process for users to convert their gameplay videos into play-by-play videos and then share and rate them was complicated and time-consuming. It was also difficult to generate natural play-by-play comments that reflected the user's emotions, resulting in content that often lacked realism. Furthermore, support for monetizing the play-by-play videos was insufficient, making the process complicated for users.

[1060] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for users to upload videos of electronic games they have played; means for analyzing the uploaded videos and detecting important events; means for analyzing users' emotions and adding the detected emotion data to the analysis results; means for generating live commentary comments in natural language based on the detected game events and emotion data; means for generating live commentary videos by overlaying the generated live commentary comments on videos; means for providing a community function for sharing the generated live commentary videos; means for other users to rate the shared live commentary videos; means for officially featuring well-rated live commentary videos based on the rating results; means for storing live commentary comments from professional commentators and celebrities in a database and appropriately selecting and adding these comments based on the detected game events and emotion data; and means for users to post the generated live commentary videos to third-party distribution platforms and monetize them. This allows users to easily convert their own gameplay videos into natural and realistic live commentary videos, and efficiently share, rate, and monetize them.

[1061] "User" refers to any individual or entity who uploads electronic gameplay videos to the system and generates, shares, and monetizes live gameplay videos.

[1062] "Electronic game" refers to an interactive entertainment activity that operates on a computer, gaming console, mobile device, or the like.

[1063] "Video" refers to a series of digital files containing the video and audio of an electronic game played by a user.

[1064] "Upload" refers to the act of a user sending a video file from their own device to the system's server.

[1065] "Analysis" refers to the process by which the system automatically analyzes uploaded videos to detect important events and user sentiment.

[1066] "Events" refer to important actions or scenes that occur within the game, such as the start of a boss battle or the acquisition of an item.

[1067] "Emotion" refers to psychological reactions detected from a user's facial expressions and tone of voice, including joy, surprise, anger, etc.

[1068] "Natural language" refers to the language used by humans on a daily basis and is used to generate commentary in text or audio form.

[1069] "Live commentary" refers to text such as explanations and impressions generated based on events that occur during gameplay and the user's emotions.

[1070] "Overlay" refers to the process of displaying commentary at specific timestamps on the original game footage.

[1071] The "community function" refers to an online platform that allows users to share their own live videos with other users and exchange opinions and ratings.

[1072] "Rating" refers to the rating actions other users take on a shared live video, including comments, likes, and number of views.

[1073] "Feature" refers to the system officially recommending well-rated live videos so that they can be viewed by more users.

[1074] A "professional commentator" refers to an individual or group that professionally broadcasts games and provides high-quality commentary using their knowledge and experience.

[1075] "Celebrities" are publicly known or influential individuals whose live commentary is stored and used in a database.

[1076] "Database" refers to a software system that stores information in an organized manner and makes it available for retrieval and use as needed.

[1077] "Third-party distribution platforms" refers to online services, such as YouTube, that are used to share and monetize the live-action videos generated.

[1078] "Monetization" refers to the process of earning advertising revenue, donations from viewers, etc. from user-generated live videos.

[1079] As an embodiment of the invention, the program processing of this system is specifically described below. This system allows users to easily convert gameplay videos into live commentary videos, share and rate them within the community, and further supports monetization of the created videos.

[1080] composition

[1081] The system includes the following major hardware and software components:

[1082] User's device (e.g. PC or smartphone)

[1083] Server (cloud-based)

[1084] Storage systems (e.g. cloud storage)

[1085] Database (e.g. MySQL)

[1086] Image recognition technology (e.g., OpenCV)

[1087] Voice recognition technology (e.g., Google Speech-to-Text)

[1088] Sentiment analysis engine (e.g. Affectiva SDK)

[1089] Natural Language Generation (NLG) engines (e.g., GPT-3)

[1090] Video editing engine (e.g. FFmpeg)

[1091] Distribution platform (e.g. YouTube)

[1092] Operation flow

[1093] (1) The user saves the game video they played on their device and accesses the system's upload page. The user clicks the upload button, opens a file selection dialog, and selects the video file.

[1094] (2) The device converts the selected video file into an HTTP POST request and sends it to the server. The server analyzes the received HTTP POST request and saves the uploaded video file in cloud storage. At the same time, it assigns a unique ID to the video file and saves its metadata (file name, upload date and time, etc.) in a database.

[1095] (3) The server reads the video files stored in the storage for analysis and detects important events in the game using image recognition technology (e.g., OpenCV) and voice recognition technology (e.g., Google Speech-to-Text). For each event, the timestamp and event content are saved in a list format.

[1096] (4) Based on the analysis results of the detected game events, the server launches an emotion engine (e.g., Affectiva SDK) to analyze the user's facial expressions and tone of voice in the video, extracting emotional appeal points and adding the detected emotion data (e.g., joy, surprise, anger, etc.) to the analysis results.

[1097] (5) The server passes the event list and emotion data to a natural language generation (NLG) engine (e.g., GPT-3) to generate commentary corresponding to each event. For example, if surprise is detected at the start of a boss battle, the server generates a comment such as "I can't believe it! A monster appeared at this exact moment!" If necessary, appropriate comments can be selected and added from a database of comments by professional commentators or celebrities.

[1098] (6) The server passes the generated commentary and the original game video to a video editing engine (e.g., FFmpeg), which overlays the commentary on the video based on the corresponding timestamps to generate a commentary video. The completed commentary video is saved in cloud storage, and its metadata is added to a database.

[1099] (7) The user logs in to their account and shares the created live video on the community page. The user clicks the share button, enters the video title and description, and shares it.

[1100] (8) The device sends a sharing request to the server, which adds a new live video entry to the community page. Other users watch and rate the video, and the rating data is stored in a database.

[1101] (9) Users can post their live streams to third-party platforms (e.g., YouTube) and monetize them. The system provides links to facilitate posting and provides users with monetization guidelines.

[1102] Specific examples

[1103] For example, suppose User B uploads a 15-minute video of himself playing a horror game. The server analyzes the video and detects scenes in which monsters appear and scenes in which the user is surprised. The emotion engine detects the user's surprise, and based on that emotional data, the AI ​​generates commentary such as "I can't believe it! A monster appeared at this exact moment!". The generated commentary is then overlaid on the original video to complete the commentary video. User B shares the completed commentary video with the community, and if it receives high ratings from other users, it can be posted on distribution platforms such as YouTube, providing an opportunity for monetization.

[1104] Example of input prompt for generative AI model

[1105] "Please explain the system that analyzes gameplay videos uploaded by users, generates live commentary using an emotion engine and a natural language generation engine, and supports sharing and monetization."

[1106] In this way, by utilizing the emotion engine, the present invention can generate immersive live videos that correspond to the user's emotions, making the sharing and monetization process simple and effective.

[1107] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1108] Step 1: User uploads game videos

[1109] Input: Gameplay video file

[1110] Output: Notification of upload completion to the server

[1111] explanation:

[1112] Users access the upload page to upload gameplay videos stored on their device. The user clicks the upload button, opens a file selection dialog, and selects a video file. The device converts the selected video file into an HTTP POST request and sends it to the server. This process also includes displaying the video file upload progress to the user in real time. The server analyzes the received HTTP POST request and saves the uploaded video file in cloud storage. At the same time, it assigns a unique ID to the video file and saves its metadata (file name, upload date and time, etc.) in a database. After the upload is complete, the server returns a notification that the upload is complete.

[1113] Step 2: Analyze the video

[1114] Input: Video file stored in cloud storage

[1115] Output: Event list, timestamp list

[1116] explanation:

[1117] The server passes the video files stored in cloud storage to an analysis module. This analysis module uses image recognition technology (e.g., OpenCV) to analyze each frame and detect important events in the game. It also uses speech recognition technology (e.g., Google Speech-to-Text) to analyze audio data and identify important event sounds. The analysis results are saved in a list format, with the timestamp and event content for each event.

[1118] Step 3: Introducing the Emotion Engine

[1119] Input: Video file, event list, timestamp list

[1120] Output: Emotion data list

[1121] explanation:

[1122] Based on the analysis, the server launches an emotion engine (e.g., Affectiva SDK). The emotion engine analyzes the user's facial expressions and tone of voice in the video to detect emotions such as joy, surprise, and anger. The detected emotion data is added to an emotion data list along with a timestamp for each event.

[1123] Step 4: Generate commentary

[1124] Input: Event list, timestamp list, emotion data list

[1125] Output: List of live commentary

[1126] explanation:

[1127] The server passes the event list and emotion data to a natural language generation (NLG) engine (e.g., GPT-3) to generate commentary corresponding to each event. For example, if surprise is detected for the event "Boss battle begins," the server generates a comment such as "That's unexpected! The boss battle has begun!". Furthermore, it is possible to select and add appropriate comments from a database of comments from professional commentators and celebrities. This generates a list of commentary comments.

[1128] Step 5: Rendering the play-by-play video

[1129] Input: Game video file, commentary list, timestamp list

[1130] Output: Finished playthrough video file

[1131] explanation:

[1132] The server then passes the generated commentary and the original game video to a video editing engine (e.g., FFmpeg), which overlays the commentary on the video based on the corresponding timestamps to generate a commentary video. During this process, it also adjusts effects, text position, font size, etc., and finally generates a completed commentary video file that is saved in cloud storage. The server then adds the metadata to a database.

[1133] Step 6: Share your video

[1134] Input: Finished playthrough video file

[1135] Output: Shared play-by-play video entries

[1136] explanation:

[1137] The user logs in to their account and shares the created live commentary video on the community page. The user clicks the share button and enters the video title and description. The device sends a share request to the server, and the server adds a new live commentary video entry to the community page. When other users watch or rate this entry, the rating data is stored in the database.

[1138] Step 7: Monetization Support

[1139] Input: Completed play-by-play video files, shared play-by-play video entries

[1140] Output: Monetized link

[1141] explanation:

[1142] Users can post the live stream videos they create to third-party distribution platforms (e.g., YouTube) and monetize them. The system provides links to facilitate posting and provides users with monetization guidelines, allowing users to smoothly progress through the steps toward monetization.

[1143] (Application example 2)

[1144] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1145] In today's game streaming, simply sharing gameplay videos is no longer enough to maintain user participation and viewer interest. There is a need for a simple method for users to breathe life into their gameplay videos, create more immersive commentary videos, and monetize them. In particular, incorporating natural and realistic commentary that reflects the user's emotions into the videos is crucial to sustaining viewer interest.

[1146] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1147] In this invention, the server includes means for users to upload videos of games they have played, means for analyzing the uploaded videos and detecting important game events, means for detecting user emotions, means for generating commentary comments in natural language based on the detected emotions, means for generating commentary videos by overlaying the generated commentary comments on videos, means for providing a community function for sharing the generated commentary videos, means for other users to rate the shared commentary videos, and means for officially featuring commentary videos that have received good ratings based on the rating results. This enables users to easily generate immersive commentary videos that reflect their own emotions, have them shared and rated within the community, and further monetize them.

[1148] "User" refers to a person who uses this system to upload their own gameplay videos and create and share live play videos.

[1149] "Gameplay video" is video data that records a user playing a game.

[1150] "Means for uploading" refers to the function that allows users to send videos from their own devices to the server.

[1151] "Means for analyzing videos" refers to a function that analyzes the content of uploaded videos using computer vision technology, etc., and detects important events.

[1152] "Significant game events" are notable occurrences or progressions within the game, including boss battles, item acquisition, etc.

[1153] "Means for detecting emotions" refers to technology that reads emotions such as joy, surprise, and anger from the user's voice and facial expressions in gameplay videos.

[1154] "Live commentary" refers to text data of narration and explanation generated for game events.

[1155] "Natural language generation" is a technology that automatically creates natural language using a computer program.

[1156] "Overlay" is a technique that displays additional information (text or images) on top of gameplay video.

[1157] A "play-by-play video" is a new video generated by overlaying play-by-play commentary on the original gameplay video.

[1158] The "Community Function" refers to an online platform where users can share their created live videos and have other users watch and rate them.

[1159] "Means for sharing" refers to a function for making the generated live video available to other users.

[1160] "Means for rating" refers to a function that allows users to give ratings (e.g., favorites, comments, rating points) to shared live videos.

[1161] A "good review video" is a review video that has received high ratings from other users.

[1162] "Official Feature Option" is a feature that allows highly rated live videos to be featured and viewed by more users.

[1163] A "professional commentator" is a person or organization that specializes in creating live commentary.

[1164] A "celebrity" is someone who is widely known and influential to the public.

[1165] A "database" is a structured collection of information that stores data systematically and makes it easy to search and update.

[1166] A "distribution platform" is an online service that allows you to make videos available to other viewers and earn revenue.

[1167] "Means of monetization" refers to the methods and functions for generating revenue such as advertising revenue and viewing fees through the generated live video.

[1168] The system embodying the present invention allows users to easily upload gameplay videos, generate realistic commentary using an emotion engine based on the analyzed video, share and rate the commentary videos within the community, and ultimately monetize them. This system is composed of the following means:

[1169] Upload and analyze

[1170] User

[1171] Users upload videos of the games they play to the server from their smartphones or other devices. The uploaded videos are sent to the server as HTTP requests.

[1172] server

[1173] The server stores the received video in storage and assigns a unique ID to the video. Next, it analyzes the stored video and uses computer vision technology to detect important events in the game. As a result of the analysis, a list of timestamps and content for each event is generated.

[1174] Emotion detection

[1175] server

[1176] The server activates an emotion engine to detect emotions such as joy, surprise, and anger from the user's voice and facial expressions in the video. The detected emotion data is added to the analysis results.

[1177] Generate live commentary

[1178] server

[1179] The server passes the analysis results to a natural language generation (NLG) engine, which generates appropriate commentary based on the detected game events and emotional data. For example, if the server detects a user's surprise, it might generate a comment like, "That's unexpected! The boss battle has begun!"

[1180] Rendering live video

[1181] server

[1182] The server then launches a video editing engine to overlay the generated commentary onto the video. The comments, adjusted based on the emotion data, are inserted into the gameplay video based on the corresponding timestamp. The completed video is then saved back to storage, and metadata is added to the database.

[1183] Share and rate videos

[1184] User

[1185] Users log in to their accounts and share the created live commentary videos on the community page. Other users can watch the shared live commentary videos and rate them. This rating data is stored on the server.

[1186] server

[1187] The server processes the incoming share requests and adds new live video entries to the community page. Highly rated videos are officially featured.

[1188] Monetization support

[1189] User

[1190] Users can then post the live commentary videos they create on third-party distribution platforms in an effort to generate revenue.

[1191] server

[1192] The server provides links to facilitate posting and provides users with monetization guidelines.

[1193] Examples and prompts

[1194] As a concrete example, let's say a user uploads a 15-minute video of themselves playing a horror game. The server analyzes the video and detects scenes in which monsters appear and scenes in which the user is surprised. The emotion engine detects the user's surprise, and based on that emotional data, the AI ​​generates commentary comments such as "I can't believe it! A monster appeared at this exact moment!" The commentary video, which is completed by overlaying these comments on the original video, is then shared by the user within the community.

[1195] Example prompt for a generative AI model:

[1196] Prompt: This video contains a scene in which a monster suddenly appears while the user is playing a horror game. The user's facial expression shows a look of surprise. Please generate a commentary that matches this situation.

[1197] Expected output: Unbelievable! A monster appeared at this exact moment! This is so scary!

[1198] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1199] Step 1:

[1200] The user prepares a video file of the game they played and accesses the system's upload page from their smartphone or other device. They select the target video file from the file selection dialog and click the upload button. At this time, the input is the user's gameplay video file, and the output is a notification that the video file has been uploaded.

[1201] Step 2:

[1202] The device converts the selected video file into an HTTP request and sends it to the server. The server analyzes the received HTTP request and saves the video file in storage. The server assigns a unique ID to this video file and saves the metadata (file name, upload date and time, etc.) in a database. The input is the uploaded video file, and the output is the video file and its metadata saved in storage.

[1203] Step 3:

[1204] The server analyzes the video stored in storage. First, it launches a video analysis engine and uses computer vision techniques to detect important game events in the video. This includes image processing and object recognition between specific frames. The output is a list of the timestamp and content of each event.

[1205] Step 4:

[1206] The server activates an emotion engine to analyze the user's voice and facial expressions in the video to detect the user's emotions. The emotion engine uses voice analysis and facial expression recognition technology to extract the user's emotional data (e.g., joy, surprise, etc.) at a specific time in the video. The input is the user's voice and facial expression data, and the output is emotional data.

[1207] Step 5:

[1208] The server passes the detected game events and emotion data to a natural language generation (NLG) engine. The NLG engine generates commentary corresponding to each event based on this data. For example, if the emotion of surprise is detected, the commentary generated is "This is unexpected! The boss battle has begun!" The input is the game events and emotion data, and the output is the generated commentary.

[1209] Step 6:

[1210] The server launches a video editing engine to overlay the generated commentary on the gameplay video. The video editing engine inserts the comments at the appropriate positions based on the comment timestamps and generates the final commentary video. The input is the commentary and the original gameplay video, and the output is the commentary video.

[1211] Step 7:

[1212] The server saves the completed commentary video in storage and adds its metadata to the database. The generated commentary video is uploaded to the community page in a viewable state. The input is the completed commentary video, and the output is the commentary video saved in storage.

[1213] Step 8:

[1214] Users log in to their accounts and share the generated commentary videos on the community page. Other users can watch and rate the shared commentary videos. The input is the generated commentary video, and the output is the user's rating data.

[1215] Step 9:

[1216] The server stores the evaluation results in a database and officially features the highly rated live videos, allowing them to be displayed to a wider range of users. The input is the evaluation results, and the output is the featured videos.

[1217] Step 10:

[1218] Users can then post the live commentary videos they create to third-party distribution platforms for monetization. The server provides links and monetization guidelines to support posting. The input is the live commentary video and guidelines, and the output is a notification that posting has been completed to the distribution platform.

[1219] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1220] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1221] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1222] [Third embodiment]

[1223] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1224] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1225] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1226] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1227] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1228] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1229] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1230] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1231] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1232] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1233] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1234] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1235] The system of this invention allows users to easily convert gameplay videos into live commentary videos, share and rate them within the community, and also supports monetization of the videos. This system is mainly composed of three entities: a server, a device, and users.

[1236] Main features

[1237] Game video upload

[1238] User

[1239] Users upload videos of the games they play to the system via their terminals.

[1240] Select the video file and click the upload button to send the video to the system.

[1241] Terminal

[1242] The terminal converts the game video file selected by the user into an HTTP request and sends it to the server.

[1243] Notify the user that the video was successfully sent.

[1244] server

[1245] The received video file is saved in storage, and the video metadata (file name, upload date and time, etc.) is recorded in the database.

[1246] Video Analysis

[1247] server

[1248] The server runs software to analyze the stored video.

[1249] Using image and voice recognition technology, it detects important events in the game (e.g., defeating enemies, boss battles, level-ups, etc.) along with their timeline.

[1250] Generate a list of detected event information (event type, occurrence time).

[1251] Generate live commentary

[1252] server

[1253] Based on the detected event list, a natural language generation (NLG) engine is used to generate corresponding commentary.

[1254] The generated commentary includes phrases such as "Great! You defeated the enemy!" for "You defeated the enemy."

[1255] Select appropriate comments from a database of professional commentators and celebrities and add them as needed.

[1256] Rendering live video

[1257] server

[1258] The generated commentary is overlaid onto the gameplay video.

[1259] Use a video editing engine to overlay text onto the video based on the corresponding timestamps.

[1260] The completed live video is saved to storage and metadata is added to the database.

[1261] Share and rate videos

[1262] User

[1263] Users log in to the system to share the generated live video on a community page.

[1264] Click the share button and enter a title and description for your video.

[1265] You can watch live videos shared by other users and leave likes and comments.

[1266] Terminal

[1267] Receives share and rating requests and sends them to the server.

[1268] Display a community page where users can easily watch and rate other videos.

[1269] server

[1270] Processes incoming share and rating requests and updates the database with video rating scores and comments.

[1271] We will periodically tally rating scores and officially feature the top-rated videos.

[1272] Monetization support

[1273] User

[1274] The generated live commentary videos will be posted on third-party distribution platforms in order to generate revenue.

[1275] Get a download link from the system and upload the video to a distribution platform.

[1276] server

[1277] Provide links to facilitate posting and provide users with monetization guidelines.

[1278] Specific examples

[1279] For example, suppose User A uploads a 10-minute video of himself playing an action game. The server analyzes the video and detects scenes where enemies are defeated or boss battles are fought. The AI ​​generates commentary such as "Great attack!" or "Great opportunity!" and overlays it on the original video. The final commentary video is shared by User A to the community and receives high ratings from other users. As a result of the high ratings, the server officially features the video, and User A can then post the video on YouTube to earn revenue.

[1280] In this way, the system of the present invention makes the process of creating, sharing, rating, and monetizing gameplay videos simple and effective.

[1281] The processing flow will be explained below.

[1282] Step 1:

[1283] User

[1284] Prepare your gameplay video on your device and access the upload page.

[1285] Click the Upload button and select the video file to upload from the file selection dialog.

[1286] Step 2:

[1287] Terminal

[1288] The selected video file is converted into an HTTP request and sent to the server.

[1289] Show the upload progress to the user.

[1290] Step 3:

[1291] server

[1292] Analyze the received HTTP request and save the video file in storage.

[1293] A unique ID is assigned to the video file, and metadata (file name, upload date and time, etc.) is stored in the database.

[1294] Step 4:

[1295] server

[1296] Load the video stored in storage for analysis.

[1297] It uses image and voice recognition technology to detect important events in the game.

[1298] List the timestamp and event details for each event.

[1299] Step 5:

[1300] server

[1301] Pass the event list to a natural language generation (NLG) engine.

[1302] The NLG engine generates commentary corresponding to events. For example, for the event "Boss battle begins," it generates a comment like "This is where the real action begins, the boss battle begins!"

[1303] If necessary, add appropriate comments from a database of professional commentators and celebrities.

[1304] Step 6:

[1305] server

[1306] A video editing engine is launched to overlay the generated commentary on the original game video.

[1307] Insert comments into the video based on appropriate timestamps to generate a play-by-play video.

[1308] The completed live video is saved to storage and metadata is added to the database.

[1309] Step 7:

[1310] User

[1311] Log in to your account and share the generated commentary video on the community page.

[1312] Click the share button and enter a title and description for your video.

[1313] Step 8:

[1314] Terminal

[1315] Sends a share request to the server.

[1316] It provides an interface for accessing community pages and watching live videos shared by other users.

[1317] Step 9:

[1318] server

[1319] Process incoming share requests and add new play-by-play video entries to the community page.

[1320] When other users watch or rate a video, the rating data is saved in a database.

[1321] Step 10:

[1322] server

[1323] We periodically tally the rating scores and select the top-rated videos.

[1324] Selected videos will be featured on the official special page.

[1325] Step 11:

[1326] User

[1327] Check out the featured videos.

[1328] Get a link to download the video or post it to a third-party distribution platform.

[1329] Step 12:

[1330] User

[1331] Post videos to distribution platforms and monetize them. After posting, use revenue models such as advertising revenue and Super Chat.

[1332] The above is the flow of specific processing steps for carrying out the invention.

[1333] Example 1

[1334] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1335] Conventional gameplay video sharing systems require users to manually edit videos and add commentary, which is time-consuming and labor-intensive. Furthermore, the process of sharing, rating, and monetizing the created videos is complicated, making it difficult for users to use. The objective of this invention is to solve these problems and provide a system that allows users to easily convert gameplay videos into commentary videos and effectively share, rate, and monetize them.

[1336] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1337] In this invention, the server includes: means for users to upload videos of games they play; means for converting the video files into HTTP requests and sending them to the server; means for saving the received video files in storage and recording metadata in a database; means for analyzing the uploaded videos, detecting in-game events using image recognition technology and voice recognition technology, and generating a list of event information; means for generating commentary comments using a natural language generation engine based on the detected game events and comparing them with a database to select appropriate comments; means for overlaying the generated commentary comments on videos to generate commentary videos; means for providing a community function for sharing the generated commentary videos; means for other users to rate the shared commentary videos; means for officially featuring commentary videos with good ratings based on the rating results; and means for providing a link for posting the generated commentary videos to a third-party distribution platform for monetization. This enables users to easily analyze, edit, and share gameplay videos and effectively monetize them.

[1338] "User" means any individual or legal entity that uses the System to upload, analyze, share, and rate gameplay videos.

[1339] A "gameplay video" is a digital file that contains video and audio of a game being played by a user.

[1340] "Uploading" is the process by which a user transfers a video file from their device to the system.

[1341] An "HTTP request" is a request message for sending data to a server based on a protocol used for communication over the Internet.

[1342] A "server" is a computer system that processes and stores data received from users and provides the required services.

[1343] "Storage" refers to a storage device for saving digital data, including hard disk drives (HDDs) and solid-state drives (SSDs).

[1344] "Metadata" is data that describes the characteristics and content of a video file, and includes, for example, the file name, upload date and time, and video length.

[1345] A "database" is a system for efficiently storing, retrieving, and updating structured information.

[1346] "Image recognition technology" is a technology that analyzes the content of images and videos and identifies specific objects and movements.

[1347] "Speech recognition technology" is a technology that analyzes voice data and converts it into meaningful text or commands.

[1348] A "natural language generation engine" is an artificial intelligence technology that generates text based on input data in a form that is easy for humans to understand.

[1349] "Commentary" is text that includes commentary and impressions about specific events in gameplay videos.

[1350] "Overlay" is a technique for displaying text or images over specific locations on a video.

[1351] "Community Features" are features within the system that allow users to share and rate generated content.

[1352] A "rating" is feedback or a score that a user gives to another user's content.

[1353] A "feature" is a display of specific content that stands out more than other content.

[1354] "Monetization" is the process of utilizing generated content for financial gain.

[1355] This invention is a system that allows users to easily upload gameplay videos and then analyze, edit, share, rate, and monetize them. This system operates in cooperation with a server, devices, and users.

[1356] To upload a video of a game that a user has played, the user first logs in to the system. After logging in, the user selects the gameplay video file from their device and clicks the upload button. At this time, the device converts the video file into an HTTP request and sends it to the server. The uploaded video file is received by the server and saved in storage. The server also records the video's metadata (file name, upload date and time, etc.) in a database.

[1357] Next, the server launches software to analyze the saved video file. Specifically, it uses image recognition technology (e.g., OpenCV) and speech recognition technology (e.g., Google Speech-to-Text API) to detect important in-game events (e.g., defeating an enemy, leveling up, etc.). The server then generates a list of detected event information (event type, occurrence time). The server then uses this event list to generate commentary using a natural language generation (NLG) engine (e.g., GPT-3). The generated commentary is then compared with a database of comments from professional commentators and celebrities to select appropriate comments.

[1358] The server then uses a video editing engine (e.g., FFmpeg) to overlay the commentary onto the video. During this process, commentary generated based on timestamps is overlaid onto specific locations in the gameplay video. The completed commentary video is saved to storage, and the metadata is added to a database.

[1359] Users can share the live commentary videos they have created on the system's community page. When sharing, users enter a title and description for the video. The device then sends a sharing request to the server, allowing other users to view and rate the shared live commentary video. The server processes the received rating request and updates the video's rating score and comments in the database. Based on the rating results, live commentary videos that receive good ratings are officially featured.

[1360] Additionally, users can monetize their generated live streams by posting them to third-party distribution platforms, and the server will provide users with links and guidelines to support monetization.

[1361] As a concrete example, suppose User A uploads a 10-minute video of himself playing an action game. The server analyzes the video and detects scenes where enemies are defeated and boss battles. The AI ​​generates commentary such as "Great attack!" or "Great opportunity!" and overlays it on the original video. The final commentary video is shared by User A with the community and receives high ratings from other users. As a result of the high ratings, the server officially features the video, and User A can then post the video on distribution platforms such as YouTube to earn revenue.

[1362] An example of a prompt might be:

[1363] "I want to create a program that analyzes gameplay video, generates commentary based on important events, and then creates a commentary video. Please explain the steps below:

[1364] 1. Users upload game videos

[1365] 2. Analyze video on the server and detect important events

[1366] 3. Create live commentary using natural language generation

[1367] 4. Overlay your comments on the video and you're done!

[1368] The system operates based on this content, improving user convenience.

[1369] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1370] Step 1:

[1371] User logs in and uploads game video

[1372] To log in to the system, the user enters their username and password. After successful login, the user selects a gameplay video file from the terminal and clicks the upload button. The input is the user information and the game video file, and the output is an HTTP request. When the upload button is clicked, the terminal converts the selected video file into an outgoing packet as an HTTP POST request and sends it to the server.

[1373] Step 2:

[1374] The device converts the video file into an HTTP request and sends it to the server.

[1375] The device follows the user's instructions, converts the selected video file into an HTTP request, and sends it to the server using the POST method. The input is the video file, and the output is an HTTP request. Specifically, the device embeds the file's binary data in the body of the HTTP POST request and sends it to the specified URL.

[1376] Step 3:

[1377] The server receives the video file and stores it in storage.

[1378] The server extracts the video file from the body of the HTTP request received and saves it to storage. The input is the HTTP request, and the output is the saved video file. Specifically, the file system API is used to save the video file to a specified directory.

[1379] Step 4:

[1380] The server records the video metadata in a database

[1381] The server extracts metadata from the saved video files (file name, upload date and time, video length, etc.) and records it in a database. The input is the saved video file, and the output is a database record. Specifically, it obtains the file attribute information and inserts the record using the database API.

[1382] Step 5:

[1383] The server analyzes the video using video analysis software.

[1384] The server runs video analysis software (e.g., OpenCV) and uses image recognition technology to detect important in-game events (e.g., defeating an enemy, leveling up, etc.). The input is the saved video file, and the output is a list of event information. Specifically, it processes each frame of the video, detecting specific in-game events, and lists them along with their timestamps.

[1385] Step 6:

[1386] The server uses speech recognition technology to identify audio events and record them as metadata.

[1387] The server uses speech recognition technology (e.g., Google Speech-to-Text API) to identify important audio events from the video's audio track and record them as metadata. The input is the saved video file and the event information list, and the output is an updated event information list. Specifically, the server analyzes the audio data, converts it into text strings in each language, and records them along with the time when the audio events occurred.

[1388] Step 7:

[1389] The server generates live commentary using a natural language generation engine

[1390] The server uses a natural language generation (NLG) engine (e.g., GPT-3) to generate commentary based on the detected event information. The input is a list of event information, and the output is a list of commentary. Specifically, a prompt sentence is input to the generative AI model, which then generates appropriate commentary.

[1391] Step 8:

[1392] The server compares the generated live commentary with the database and selects the appropriate comment.

[1393] The server compares the generated live commentary comments with a database of comments from professional commentators and celebrities to select appropriate comments. The input is a list of generated live commentary comments, and the output is a list of matched live commentary comments. Specifically, it uses a string search algorithm to extract matching comments from the database.

[1394] Step 9:

[1395] The server uses a video editing engine to overlay commentary onto the video.

[1396] The server uses a video editing engine (e.g., FFmpeg) to overlay the generated commentary based on the corresponding timestamps of the video. The input is a gameplay video file and a list of commentary, and the output is the finished commentary video. Specifically, it applies a text overlay filter to specific frames of the video.

[1397] Step 10:

[1398] The server saves the completed live video to storage and adds the metadata to the database.

[1399] The server saves the completed live video in storage and adds the video's metadata (such as the creation date and time, video playback time, and comment list) to the database. The input is the completed live video and a list of live comments, and the output is an updated database record. Specifically, the server saves the video file to the storage system and inserts its metadata into the database.

[1400] Step 11:

[1401] Share user-generated play-by-play videos on your community page

[1402] A user logs in to the system and performs an operation to share the generated live video on a community page. The input is the generated live video, and the output is a sharing request. The user clicks the share button and enters the video title and description.

[1403] Step 12:

[1404] The device sends a share request to the server

[1405] The device sends the user's sharing request to the server as an HTTP POST request. The input is the sharing request, and the output is an HTTP POST request. Specifically, the sharing request is packetized along with the video metadata and sent to the server.

[1406] Step 13:

[1407] The server processes share and rating requests and updates the rating scores and comments to the database.

[1408] The server processes the received share requests, adds new video information to the database, and processes rating requests made by other users on the shared live videos, updating the rating scores and comments in the database. The input is the share request and the rating request, and the output is the updated database record.

[1409] Step 14:

[1410] The server will officially feature well-rated playthrough videos based on the evaluation results.

[1411] The server periodically aggregates the rating scores and officially features the top-rated live videos. The input is the rating scores and comments, and the output is a list of featured videos. Specifically, it calculates the ranking of the rating scores and displays the top-rated videos in a specific section.

[1412] Step 15:

[1413] Monetize user-generated live videos by posting them to a streaming platform

[1414] Users post their generated commentary videos to third-party distribution platforms (e.g., YouTube, Twitch) and monetize them. The input is the generated commentary video, and the output is the post to the distribution platform. The system provides users with links and guidelines to support monetization.

[1415] Step 16:

[1416] Provide users with links and guidelines to help your server monetize

[1417] The server displays links and guidelines for video monetization to users. The input is a user request, and the output is monetization links and guidelines. Specifically, the server displays information related to monetization on the system dashboard for easy access by users.

[1418] The above are the specific processing steps of this system.

[1419] (Application example 1)

[1420] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1421] Currently, it takes a lot of time and effort for users to easily edit, rate, and monetize their gameplay videos. Furthermore, further technological advancements are required to realize advanced features such as seamless integration with content distribution platforms and highly accurate comment generation and overlays using AI technology. The purpose of this invention is to solve these issues and provide a simple system that allows users to easily and effectively create and share gameplay videos and further monetize them.

[1422] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1423] In this invention, the server includes means for allowing users to upload videos of games they have played, means for analyzing the uploaded videos and detecting important game events, means for generating commentary comments in natural language based on the detected game events, means for generating commentary videos by overlaying the generated commentary comments on videos, means for providing a community function for sharing the generated commentary videos, means for other users to rate the shared commentary videos, means for officially featuring commentary videos that have received positive reviews based on the rating results, means for seamlessly sharing the generated commentary videos to a third-party content distribution platform, and means for accurately overlaying commentary comments based on timestamps of game events using AI technology. This allows users to easily and effectively create and share commentary videos, enabling monetization through highly accurate commentary generation and seamless distribution.

[1424] "User" means an individual or organization that uses this system to upload gameplay videos and create and share live gameplay videos.

[1425] "Game videos" refer to video files recorded by users playing games.

[1426] "Uploading" refers to the act of a user sending gameplay video from their device to a server.

[1427] "Analysis" refers to the process by which the server processes the uploaded video and detects important game events.

[1428] "Game Event" refers to an important event that occurs within the game (e.g., defeating an enemy, fighting a boss, leveling up, etc.).

[1429] "Natural language" refers to the language used by humans on a daily basis, and is used in this system to generate live commentary.

[1430] "Live commentary" refers to comments that explain and evaluate gameplay in real time or near real time.

[1431] "Overlay" refers to the process of displaying the generated commentary overlaid on the game video.

[1432] "Live commentary video" refers to a video file created by overlaying live commentary on gameplay video.

[1433] "Community Function" refers to a platform where users can share live videos they have created and other users can watch, rate, and comment on them.

[1434] "Evaluation" refers to the act of users within a community rating a posted live video by giving it a "like" or comment.

[1435] "Featured" means that a live commentary video that has received positive reviews will be officially featured and introduced to more users.

[1436] "Content distribution platform" refers to a video sharing service provided by a third party, such as YouTube or other similar services.

[1437] "Seamless" means that the operations performed by the user are smoothly linked together, allowing a series of tasks to be performed without interruption.

[1438] "AI technology" refers to artificial intelligence technology, which in this system is used for natural language generation and analysis of game events.

[1439] A "timestamp" refers to data that records the time when a particular event occurred.

[1440] "Accuracy" refers to the degree to which live commentary and analysis results generated using AI technology are accurate as expected by users.

[1441] The embodiment of the present invention will be described in detail below. This system is mainly composed of three components: a server, a terminal, and a user.

[1442] 1. Uploading game videos

[1443] User

[1444] Users use a smartphone application to select the game video they have played and click the upload button to send it to the server.

[1445] Terminal

[1446] The device converts the game video file selected by the user into an HTTP request and sends it to the server. The device notifies the user that the video has been sent successfully.

[1447] server

[1448] The server stores the received video file in storage and records the video's metadata (file name, upload date and time, etc.) in a database.

[1449] 2. Video Analysis

[1450] server

[1451] The server runs software to analyze the saved video. The software used includes Google Cloud Vision API and Google Speech-to-Text. These are used for image and voice recognition to detect important events in the game (e.g., defeating an enemy, boss battles, level-ups, etc.). The results of this detection are compiled into an event list.

[1452] 3. Generating live commentary

[1453] server

[1454] Based on the detected event list, the server uses a natural language generation engine (e.g., GPT-4) to generate corresponding commentary, such as "When an enemy is defeated: Amazing! You defeated the enemy!". It also selects appropriate comments from a database of professional commentators and celebrities and adds them as needed.

[1455] An example prompt is "Generate a commentary for the following gaming event: 'Player defeats the boss in an epic fight'".

[1456] 4. Rendering the Live Video

[1457] server

[1458] The server processes the generated commentary onto the gameplay video. FFmpeg is used as the video editing software. The commentary is accurately overlaid onto the video based on the corresponding timestamp. The completed video is saved to storage, and metadata is added to the database.

[1459] 5. Share and rate the video

[1460] User

[1461] Users log in to the system to share the created live commentary video on the community page. They click the share button and enter the video title and description. They can also watch the live commentary videos shared by other users and leave "likes" and comments.

[1462] Terminal

[1463] The device receives and sends share and rating requests to the server, and also displays a community page where users can easily watch and rate other videos.

[1464] server

[1465] The server processes the incoming share and rating requests, updates the video's rating score and comments to a database, and periodically aggregates the rating scores and officially features the top-rated play-by-play videos.

[1466] 6. Monetization Support

[1467] User

[1468] Users can monetize the generated live videos by posting them to third-party distribution platforms. Users obtain a download link from the system and upload the video to the distribution platform.

[1469] server

[1470] The server provides links to facilitate posting and provides users with monetization guidelines.

[1471] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1472] Step 1:

[1473] Game video upload

[1474] Process flow:

[1475] A user launches the application on their smartphone, selects a gameplay video file from their device, and then clicks the upload button, which generates an HTTP request to send the video to the server.

[1476] Input and Output:

[1477] Input: Gameplay video file.

[1478] Output: HTTP request to the server.

[1479] Specific behavior:

[1480] When the user selects a video file and clicks the send button, the terminal generates an HTTP request for sending the selected video file to the server, and executes the transmission of the video.

[1481] Step 2:

[1482] Receiving and storing videos

[1483] Process flow:

[1484] The server receives the HTTP request sent from the device, saves the video file in storage, and records the video file's metadata (file name, upload date and time, etc.) in a database.

[1485] Input and Output:

[1486] Input: The gameplay video file sent in the HTTP request.

[1487] Output: Video files saved on storage, along with their metadata.

[1488] Specific behavior:

[1489] When the server receives an HTTP request, it saves the video data in storage and records the metadata of the video file in a database.

[1490] Step 3:

[1491] Video Analysis

[1492] Process flow:

[1493] The server runs image and voice recognition software to analyze the stored video, using the Google Cloud Vision API and Google Speech-to-Text to detect important game events (e.g., enemy defeats, boss battles, level-ups, etc.) and generate an event list.

[1494] Input and Output:

[1495] Input: Gameplay video file saved in storage.

[1496] Output: A list of game events.

[1497] Specific behavior:

[1498] The server loads the video file, performs image recognition using the Google Cloud Vision API and speech recognition using Google Speech-to-Text, and outputs a list of detected game events with timestamps.

[1499] Step 4:

[1500] Generate live commentary

[1501] Process flow:

[1502] The server generates commentary using a generative AI model (e.g., GPT-4) based on the detected event list. The comment is a phrase such as "When an enemy is defeated: Great! You defeated the enemy!"

[1503] Input and Output:

[1504] Input: A list of detected game events.

[1505] Output: A list of live commentary.

[1506] Specific behavior:

[1507] The server uses a natural language generation engine to generate the best commentary for each event from the event list using the prompt text as input. An example of a specific prompt text is "Generate a commentary for the following gaming event: 'Player defeats the boss in an epic fight'".

[1508] Step 5:

[1509] Rendering live video

[1510] Process flow:

[1511] The server processes the generated commentary overlays onto the gameplay video, using video editing software (e.g., FFmpeg) to overlay each commentary onto the video based on its corresponding timestamp.

[1512] Input and Output:

[1513] Input: Gameplay video file, commentary list.

[1514] Output: A commentary video file with commentary overlays.

[1515] Specific behavior:

[1516] The server uses FFmpeg to overlay the commentary onto the relevant parts of the video, generating the final commentary video.

[1517] Step 6:

[1518] Share and rate videos

[1519] Process flow:

[1520] Users can upload the created live commentary videos to a community page, where other users can view and rate them. Other users can also like and comment on the live commentary videos.

[1521] Input and Output:

[1522] Input: Live video file, share request.

[1523] Output: Play-by-play videos uploaded to the community page, and rating data.

[1524] Specific behavior:

[1525] A user clicks the share button, enters the title and description of the video, and posts it to the community page. The server receives the share request and performs the necessary processing to display the video on the community page. It also manages rating requests from other users and records the ratings in a database.

[1526] Step 7:

[1527] Monetization support

[1528] Process flow:

[1529] Users can then post their live streams to third-party platforms to monetize them, and the server will provide users with links and guidelines for monetization.

[1530] Input and Output:

[1531] Input: Play-by-play video file.

[1532] Output: Links to distribution platforms, monetization guidelines.

[1533] Specific behavior:

[1534] The server provides a link to facilitate posting and displays appropriate monetization guidelines to the user, who then uses the provided link to upload their live video to the distribution platform.

[1535] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1536] The system of the present invention allows users to easily convert gameplay videos into commentary videos, share and rate them within the community, and further supports monetization of the created videos. By combining this system with an emotion engine that recognizes the user's emotions, it is possible to generate more natural and realistic commentary comments. Specific embodiments of the present invention are described below.

[1537] Main features

[1538] Game video upload

[1539] User

[1540] The user prepares the game video they played on their device and accesses the upload page.

[1541] Click the Upload button and select the video file to upload from the file selection dialog.

[1542] Terminal

[1543] The selected video file is converted into an HTTP request and sent to the server.

[1544] Show the upload progress to the user.

[1545] server

[1546] Analyze the received HTTP request and save the video file in storage.

[1547] A unique ID is assigned to the video file, and metadata (file name, upload date and time, etc.) is stored in the database.

[1548] Video Analysis

[1549] server

[1550] Load the video stored in storage for analysis.

[1551] It uses image and voice recognition technology to detect important events in the game.

[1552] List the timestamp and event details for each event.

[1553] Introducing the Emotion Engine

[1554] server

[1555] It analyzes the user's voice and facial expressions in the video and activates an emotion engine to detect emotions.

[1556] The emotion engine adds detected emotion data (e.g., joy, surprise, anger, etc.) to the analysis results.

[1557] Adjust the tone and content of commentary based on detected emotional data.

[1558] Generate live commentary

[1559] server

[1560] Pass the event list to a natural language generation (NLG) engine.

[1561] The NLG engine generates commentary corresponding to the event. For example, if the detected emotion for the event "Boss battle begins" is "surprise," it generates a comment like "This is unexpected! The boss battle has begun!"

[1562] If necessary, add appropriate comments from a database of professional commentators and celebrities.

[1563] Rendering live video

[1564] server

[1565] A video editing engine is launched to overlay the generated commentary onto the gameplay video.

[1566] Comments adjusted based on the emotion data are inserted into the video based on the corresponding timestamps to generate a live video.

[1567] The completed live video is saved to storage and metadata is added to the database.

[1568] Share and rate videos

[1569] User

[1570] Log in to your account and share the generated commentary video on the community page.

[1571] Click the share button and enter a title and description for your video.

[1572] Terminal

[1573] Sends a share request to the server.

[1574] It provides an interface for accessing community pages and watching live videos shared by other users.

[1575] server

[1576] Process incoming share requests and add new play-by-play video entries to the community page.

[1577] When other users watch or rate a video, the rating data is saved in a database.

[1578] Monetization support

[1579] User

[1580] The generated live commentary videos will be posted on third-party distribution platforms in order to generate revenue.

[1581] Get a download link from the system and upload the video to a distribution platform.

[1582] server

[1583] Provide links to facilitate posting and provide users with monetization guidelines.

[1584] Specific examples

[1585] For example, suppose User B uploads a 15-minute video of himself playing a horror game. The server analyzes the video and detects scenes where monsters appear or scenes where the user is surprised. The emotion engine detects the user's surprise, and based on that emotion data, the AI ​​generates commentary such as "I can't believe it! A monster appeared at this exact moment!" These comments are overlaid on the original video, and User B shares the completed commentary video with the community. After receiving high ratings, the video is officially featured, and User B can further post the video on YouTube, earning revenue opportunities.

[1586] In this way, the system of the present invention uses an emotion engine to generate immersive live videos that correspond to the user's emotions, allowing for a simple and effective process of sharing, rating, and monetizing.

[1587] The processing flow will be explained below.

[1588] Step 1:

[1589] User

[1590] Save your gameplay videos to your device.

[1591] Access the game commentary generation system, log in, and proceed to the video upload page.

[1592] Click the upload button and select the gameplay video file.

[1593] Once selected, confirm the upload.

[1594] Step 2:

[1595] Terminal

[1596] The selected gameplay video file is sent to the server as an HTTP request.

[1597] Show a progress bar for video uploads and notify the user once it's complete.

[1598] Step 3:

[1599] server

[1600] The received video file is saved in storage.

[1601] A unique ID is assigned to the video file, and metadata such as the file name and upload date and time are recorded in the database.

[1602] Step 4:

[1603] server

[1604] The saved video is passed to the analysis module.

[1605] Uses image recognition technology to detect important events in the game (e.g., defeating enemies, boss battles, level-ups, etc.).

[1606] Uses voice recognition technology to detect the user's voice and reactions.

[1607] Lists important events detected and audio reactions with timestamps.

[1608] Step 5:

[1609] server

[1610] The emotion engine is activated and emotions are detected by analyzing the user's voice and facial expressions in the video.

[1611] Add detected emotion data (e.g., happiness, surprise, anger, etc.) to the analysis list.

[1612] Step 6:

[1613] server

[1614] Pass the event list and sentiment data to a natural language generation (NLG) engine.

[1615] The NLG engine generates live commentary corresponding to events. For example, if the user is recognized as "surprised" by the event "Boss battle begins," it generates a comment such as "The boss appeared here! What a surprise!"

[1616] Add appropriate comments from a database of professional commentators and celebrities, if needed.

[1617] Step 7:

[1618] server

[1619] The generated commentary is passed to a video editing engine and overlaid on the original gameplay video.

[1620] Based on the comment timestamps, text and audio are inserted into the video to generate the final commentary video.

[1621] The completed live video is saved to storage and the metadata is updated in the database.

[1622] Step 8:

[1623] User

[1624] Log in to your user account and check the generated live video.

[1625] Enter the settings (title, description, etc.) for sharing on the community page.

[1626] Click the share button to post your video to the community.

[1627] Step 9:

[1628] Terminal

[1629] The input contents of the shared settings are sent to the server as an HTTP request.

[1630] It displays the community page and provides an interface where users can view other users' videos.

[1631] Step 10:

[1632] server

[1633] Receives the share request and updates the video's database entry.

[1634] Add a new playthrough video entry to the community page.

[1635] Step 11:

[1636] User

[1637] Watch other users' live videos and rate them with likes and comments.

[1638] Step 12:

[1639] Terminal

[1640] The input contents of the rating and comment are sent to the server as an HTTP request.

[1641] Step 13:

[1642] server

[1643] The received rating data and comments are stored in a database and the video's rating score is updated.

[1644] Step 14:

[1645] server

[1646] We periodically tally the rating scores and select the top-rated videos.

[1647] Selected videos will be featured on the official special page.

[1648] Step 15:

[1649] User

[1650] See which videos are featured and get links to download or post your videos to third-party distribution platforms.

[1651] Step 16:

[1652] User

[1653] They post videos on third-party distribution platforms and aim to monetize through advertising revenue, Super Chats, etc.

[1654] Example 2

[1655] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1656] Previously, the process for users to convert their gameplay videos into play-by-play videos and then share and rate them was complicated and time-consuming. It was also difficult to generate natural play-by-play comments that reflected the user's emotions, resulting in content that often lacked realism. Furthermore, support for monetizing the play-by-play videos was insufficient, making the process complicated for users.

[1657] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for users to upload videos of electronic games they have played; means for analyzing the uploaded videos and detecting important events; means for analyzing users' emotions and adding the detected emotion data to the analysis results; means for generating live commentary comments in natural language based on the detected game events and emotion data; means for generating live commentary videos by overlaying the generated live commentary comments on videos; means for providing a community function for sharing the generated live commentary videos; means for other users to rate the shared live commentary videos; means for officially featuring well-rated live commentary videos based on the rating results; means for storing live commentary comments from professional commentators and celebrities in a database and appropriately selecting and adding these comments based on the detected game events and emotion data; and means for users to post the generated live commentary videos to third-party distribution platforms and monetize them. This allows users to easily convert their own gameplay videos into natural and realistic live commentary videos, and efficiently share, rate, and monetize them.

[1658] "User" refers to any individual or entity who uploads electronic gameplay videos to the system and generates, shares, and monetizes live gameplay videos.

[1659] "Electronic game" refers to an interactive entertainment activity that operates on a computer, gaming console, mobile device, or the like.

[1660] "Video" refers to a series of digital files containing the video and audio of an electronic game played by a user.

[1661] "Upload" refers to the act of a user sending a video file from their own device to the system's server.

[1662] "Analysis" refers to the process by which the system automatically analyzes uploaded videos to detect important events and user sentiment.

[1663] "Events" refer to important actions or scenes that occur within the game, such as the start of a boss battle or the acquisition of an item.

[1664] "Emotion" refers to psychological reactions detected from a user's facial expressions and tone of voice, including joy, surprise, anger, etc.

[1665] "Natural language" refers to the language used by humans on a daily basis and is used to generate commentary in text or audio form.

[1666] "Live commentary" refers to text such as explanations and impressions generated based on events that occur during gameplay and the user's emotions.

[1667] "Overlay" refers to the process of displaying commentary at specific timestamps on the original game footage.

[1668] The "community function" refers to an online platform that allows users to share their own live videos with other users and exchange opinions and ratings.

[1669] "Rating" refers to the rating actions other users take on a shared live video, including comments, likes, and number of views.

[1670] "Feature" refers to the system officially recommending well-rated live videos so that they can be viewed by more users.

[1671] A "professional commentator" refers to an individual or group that professionally broadcasts games and provides high-quality commentary using their knowledge and experience.

[1672] "Celebrities" are publicly known or influential individuals whose live commentary is stored and used in a database.

[1673] "Database" refers to a software system that stores information in an organized manner and makes it available for retrieval and use as needed.

[1674] "Third-party distribution platforms" refers to online services, such as YouTube, that are used to share and monetize the live-action videos generated.

[1675] "Monetization" refers to the process of earning advertising revenue, donations from viewers, etc. from user-generated live videos.

[1676] As an embodiment of the invention, the program processing of this system is specifically described below. This system allows users to easily convert gameplay videos into live commentary videos, share and rate them within the community, and further supports monetization of the created videos.

[1677] composition

[1678] The system includes the following major hardware and software components:

[1679] User's device (e.g. PC or smartphone)

[1680] Server (cloud-based)

[1681] Storage systems (e.g. cloud storage)

[1682] Database (e.g. MySQL)

[1683] Image recognition technology (e.g., OpenCV)

[1684] Voice recognition technology (e.g., Google Speech-to-Text)

[1685] Sentiment analysis engine (e.g. Affectiva SDK)

[1686] Natural Language Generation (NLG) engines (e.g., GPT-3)

[1687] Video editing engine (e.g. FFmpeg)

[1688] Distribution platform (e.g. YouTube)

[1689] Operation flow

[1690] (1) The user saves the game video they played on their device and accesses the system's upload page. The user clicks the upload button, opens a file selection dialog, and selects the video file.

[1691] (2) The device converts the selected video file into an HTTP POST request and sends it to the server. The server analyzes the received HTTP POST request and saves the uploaded video file in cloud storage. At the same time, it assigns a unique ID to the video file and saves its metadata (file name, upload date and time, etc.) in a database.

[1692] (3) The server reads the video files stored in the storage for analysis and detects important events in the game using image recognition technology (e.g., OpenCV) and voice recognition technology (e.g., Google Speech-to-Text). For each event, the timestamp and event content are saved in a list format.

[1693] (4) Based on the analysis results of the detected game events, the server launches an emotion engine (e.g., Affectiva SDK) to analyze the user's facial expressions and tone of voice in the video, extracting emotional appeal points and adding the detected emotion data (e.g., joy, surprise, anger, etc.) to the analysis results.

[1694] (5) The server passes the event list and emotion data to a natural language generation (NLG) engine (e.g., GPT-3) to generate commentary corresponding to each event. For example, if surprise is detected at the start of a boss battle, the server generates a comment such as "I can't believe it! A monster appeared at this exact moment!" If necessary, appropriate comments can be selected and added from a database of comments by professional commentators or celebrities.

[1695] (6) The server passes the generated commentary and the original game video to a video editing engine (e.g., FFmpeg), which overlays the commentary on the video based on the corresponding timestamps to generate a commentary video. The completed commentary video is saved in cloud storage, and its metadata is added to a database.

[1696] (7) The user logs in to their account and shares the created live video on the community page. The user clicks the share button, enters the video title and description, and shares it.

[1697] (8) The device sends a sharing request to the server, which adds a new live video entry to the community page. Other users watch and rate the video, and the rating data is stored in a database.

[1698] (9) Users can post their live streams to third-party platforms (e.g., YouTube) and monetize them. The system provides links to facilitate posting and provides users with monetization guidelines.

[1699] Specific examples

[1700] For example, suppose User B uploads a 15-minute video of himself playing a horror game. The server analyzes the video and detects scenes in which monsters appear and scenes in which the user is surprised. The emotion engine detects the user's surprise, and based on that emotional data, the AI ​​generates commentary such as "I can't believe it! A monster appeared at this exact moment!". The generated commentary is then overlaid on the original video to complete the commentary video. User B shares the completed commentary video with the community, and if it receives high ratings from other users, it can be posted on distribution platforms such as YouTube, providing an opportunity for monetization.

[1701] Example of input prompt for generative AI model

[1702] "Please explain the system that analyzes gameplay videos uploaded by users, generates live commentary using an emotion engine and a natural language generation engine, and supports sharing and monetization."

[1703] In this way, by utilizing the emotion engine, the present invention can generate immersive live videos that correspond to the user's emotions, making the sharing and monetization process simple and effective.

[1704] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1705] Step 1: User uploads game videos

[1706] Input: Gameplay video file

[1707] Output: Notification of upload completion to the server

[1708] explanation:

[1709] Users access the upload page to upload gameplay videos stored on their device. The user clicks the upload button, opens a file selection dialog, and selects a video file. The device converts the selected video file into an HTTP POST request and sends it to the server. This process also includes displaying the video file upload progress to the user in real time. The server analyzes the received HTTP POST request and saves the uploaded video file in cloud storage. At the same time, it assigns a unique ID to the video file and saves its metadata (file name, upload date and time, etc.) in a database. After the upload is complete, the server returns a notification that the upload is complete.

[1710] Step 2: Analyze the video

[1711] Input: Video file stored in cloud storage

[1712] Output: Event list, timestamp list

[1713] explanation:

[1714] The server passes the video files stored in cloud storage to an analysis module. This analysis module uses image recognition technology (e.g., OpenCV) to analyze each frame and detect important events in the game. It also uses speech recognition technology (e.g., Google Speech-to-Text) to analyze audio data and identify important event sounds. The analysis results are saved in a list format, with the timestamp and event content for each event.

[1715] Step 3: Introducing the Emotion Engine

[1716] Input: Video file, event list, timestamp list

[1717] Output: Emotion data list

[1718] explanation:

[1719] Based on the analysis, the server launches an emotion engine (e.g., Affectiva SDK). The emotion engine analyzes the user's facial expressions and tone of voice in the video to detect emotions such as joy, surprise, and anger. The detected emotion data is added to an emotion data list along with a timestamp for each event.

[1720] Step 4: Generate commentary

[1721] Input: Event list, timestamp list, emotion data list

[1722] Output: List of live commentary

[1723] explanation:

[1724] The server passes the event list and emotion data to a natural language generation (NLG) engine (e.g., GPT-3) to generate commentary corresponding to each event. For example, if surprise is detected for the event "Boss battle begins," the server generates a comment such as "That's unexpected! The boss battle has begun!". Furthermore, it is possible to select and add appropriate comments from a database of comments from professional commentators and celebrities. This generates a list of commentary comments.

[1725] Step 5: Rendering the play-by-play video

[1726] Input: Game video file, commentary list, timestamp list

[1727] Output: Finished playthrough video file

[1728] explanation:

[1729] The server then passes the generated commentary and the original game video to a video editing engine (e.g., FFmpeg), which overlays the commentary on the video based on the corresponding timestamps to generate a commentary video. During this process, it also adjusts effects, text position, font size, etc., and finally generates a completed commentary video file that is saved in cloud storage. The server then adds the metadata to a database.

[1730] Step 6: Share your video

[1731] Input: Finished playthrough video file

[1732] Output: Shared play-by-play video entries

[1733] explanation:

[1734] The user logs in to their account and shares the created live commentary video on the community page. The user clicks the share button and enters the video title and description. The device sends a share request to the server, and the server adds a new live commentary video entry to the community page. When other users watch or rate this entry, the rating data is stored in the database.

[1735] Step 7: Monetization Support

[1736] Input: Completed play-by-play video files, shared play-by-play video entries

[1737] Output: Monetized link

[1738] explanation:

[1739] Users can post the live stream videos they create to third-party distribution platforms (e.g., YouTube) and monetize them. The system provides links to facilitate posting and provides users with monetization guidelines, allowing users to smoothly progress through the steps toward monetization.

[1740] (Application example 2)

[1741] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1742] In today's game streaming, simply sharing gameplay videos is no longer enough to maintain user participation and viewer interest. There is a need for a simple method for users to breathe life into their gameplay videos, create more immersive commentary videos, and monetize them. In particular, incorporating natural and realistic commentary that reflects the user's emotions into the videos is crucial to sustaining viewer interest.

[1743] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1744] In this invention, the server includes means for users to upload videos of games they have played, means for analyzing the uploaded videos and detecting important game events, means for detecting user emotions, means for generating commentary comments in natural language based on the detected emotions, means for generating commentary videos by overlaying the generated commentary comments on videos, means for providing a community function for sharing the generated commentary videos, means for other users to rate the shared commentary videos, and means for officially featuring commentary videos that have received good ratings based on the rating results. This enables users to easily generate immersive commentary videos that reflect their own emotions, have them shared and rated within the community, and further monetize them.

[1745] "User" refers to a person who uses this system to upload their own gameplay videos and create and share live play videos.

[1746] "Gameplay video" is video data that records a user playing a game.

[1747] "Means for uploading" refers to the function that allows users to send videos from their own devices to the server.

[1748] "Means for analyzing videos" refers to a function that analyzes the content of uploaded videos using computer vision technology, etc., and detects important events.

[1749] "Significant game events" are notable occurrences or progressions within the game, including boss battles, item acquisition, etc.

[1750] "Means for detecting emotions" refers to technology that reads emotions such as joy, surprise, and anger from the user's voice and facial expressions in gameplay videos.

[1751] "Live commentary" refers to text data of narration and explanation generated for game events.

[1752] "Natural language generation" is a technology that automatically creates natural language using a computer program.

[1753] "Overlay" is a technique that displays additional information (text or images) on top of gameplay video.

[1754] A "play-by-play video" is a new video generated by overlaying play-by-play commentary on the original gameplay video.

[1755] The "Community Function" refers to an online platform where users can share their created live videos and have other users watch and rate them.

[1756] "Means for sharing" refers to a function for making the generated live video available to other users.

[1757] "Means for rating" refers to a function that allows users to give ratings (e.g., favorites, comments, rating points) to shared live videos.

[1758] A "good review video" is a review video that has received high ratings from other users.

[1759] "Official Feature Option" is a feature that allows highly rated live videos to be featured and viewed by more users.

[1760] A "professional commentator" is a person or organization that specializes in creating live commentary.

[1761] A "celebrity" is someone who is widely known and influential to the public.

[1762] A "database" is a structured collection of information that stores data systematically and makes it easy to search and update.

[1763] A "distribution platform" is an online service that allows you to make videos available to other viewers and earn revenue.

[1764] "Means of monetization" refers to the methods and functions for generating revenue such as advertising revenue and viewing fees through the generated live video.

[1765] The system embodying the present invention allows users to easily upload gameplay videos, generate realistic commentary using an emotion engine based on the analyzed video, share and rate the commentary videos within the community, and ultimately monetize them. This system is composed of the following means:

[1766] Upload and analyze

[1767] User

[1768] Users upload videos of the games they play to the server from their smartphones or other devices. The uploaded videos are sent to the server as HTTP requests.

[1769] server

[1770] The server stores the received video in storage and assigns a unique ID to the video. Next, it analyzes the stored video and uses computer vision technology to detect important events in the game. As a result of the analysis, a list of timestamps and content for each event is generated.

[1771] Emotion detection

[1772] server

[1773] The server activates an emotion engine to detect emotions such as joy, surprise, and anger from the user's voice and facial expressions in the video. The detected emotion data is added to the analysis results.

[1774] Generate live commentary

[1775] server

[1776] The server passes the analysis results to a natural language generation (NLG) engine, which generates appropriate commentary based on the detected game events and emotional data. For example, if the server detects a user's surprise, it might generate a comment like, "That's unexpected! The boss battle has begun!"

[1777] Rendering live video

[1778] server

[1779] The server then launches a video editing engine to overlay the generated commentary onto the video. The comments, adjusted based on the emotion data, are inserted into the gameplay video based on the corresponding timestamp. The completed video is then saved back to storage, and metadata is added to the database.

[1780] Share and rate videos

[1781] User

[1782] Users log in to their accounts and share the created live commentary videos on the community page. Other users can watch the shared live commentary videos and rate them. This rating data is stored on the server.

[1783] server

[1784] The server processes the incoming share requests and adds new live video entries to the community page. Highly rated videos are officially featured.

[1785] Monetization support

[1786] User

[1787] Users can then post the live commentary videos they create on third-party distribution platforms in an effort to generate revenue.

[1788] server

[1789] The server provides links to facilitate posting and provides users with monetization guidelines.

[1790] Examples and prompts

[1791] As a concrete example, let's say a user uploads a 15-minute video of themselves playing a horror game. The server analyzes the video and detects scenes in which monsters appear and scenes in which the user is surprised. The emotion engine detects the user's surprise, and based on that emotional data, the AI ​​generates commentary comments such as "I can't believe it! A monster appeared at this exact moment!" The commentary video, which is completed by overlaying these comments on the original video, is then shared by the user within the community.

[1792] Example prompt for a generative AI model:

[1793] Prompt: This video contains a scene in which a monster suddenly appears while the user is playing a horror game. The user's facial expression shows a look of surprise. Please generate a commentary that matches this situation.

[1794] Expected output: Unbelievable! A monster appeared at this exact moment! This is so scary!

[1795] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1796] Step 1:

[1797] The user prepares a video file of the game they played and accesses the system's upload page from their smartphone or other device. They select the target video file from the file selection dialog and click the upload button. At this time, the input is the user's gameplay video file, and the output is a notification that the video file has been uploaded.

[1798] Step 2:

[1799] The device converts the selected video file into an HTTP request and sends it to the server. The server analyzes the received HTTP request and saves the video file in storage. The server assigns a unique ID to this video file and saves the metadata (file name, upload date and time, etc.) in a database. The input is the uploaded video file, and the output is the video file and its metadata saved in storage.

[1800] Step 3:

[1801] The server analyzes the video stored in storage. First, it launches a video analysis engine and uses computer vision techniques to detect important game events in the video. This includes image processing and object recognition between specific frames. The output is a list of the timestamp and content of each event.

[1802] Step 4:

[1803] The server activates an emotion engine to analyze the user's voice and facial expressions in the video to detect the user's emotions. The emotion engine uses voice analysis and facial expression recognition technology to extract the user's emotional data (e.g., joy, surprise, etc.) at a specific time in the video. The input is the user's voice and facial expression data, and the output is emotional data.

[1804] Step 5:

[1805] The server passes the detected game events and emotion data to a natural language generation (NLG) engine. The NLG engine generates commentary corresponding to each event based on this data. For example, if the emotion of surprise is detected, the commentary generated is "This is unexpected! The boss battle has begun!" The input is the game events and emotion data, and the output is the generated commentary.

[1806] Step 6:

[1807] The server launches a video editing engine to overlay the generated commentary on the gameplay video. The video editing engine inserts the comments at the appropriate positions based on the comment timestamps and generates the final commentary video. The input is the commentary and the original gameplay video, and the output is the commentary video.

[1808] Step 7:

[1809] The server saves the completed commentary video in storage and adds its metadata to the database. The generated commentary video is uploaded to the community page in a viewable state. The input is the completed commentary video, and the output is the commentary video saved in storage.

[1810] Step 8:

[1811] Users log in to their accounts and share the generated commentary videos on the community page. Other users can watch and rate the shared commentary videos. The input is the generated commentary video, and the output is the user's rating data.

[1812] Step 9:

[1813] The server stores the evaluation results in a database and officially features the highly rated live videos, allowing them to be displayed to a wider range of users. The input is the evaluation results, and the output is the featured videos.

[1814] Step 10:

[1815] Users can then post the live commentary videos they create to third-party distribution platforms for monetization. The server provides links and monetization guidelines to support posting. The input is the live commentary video and guidelines, and the output is a notification that posting has been completed to the distribution platform.

[1816] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1817] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1818] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1819] [Fourth embodiment]

[1820] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1821] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1822] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1823] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1824] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1825] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1826] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1827] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1828] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1829] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1830] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1831] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1832] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1833] The system of this invention allows users to easily convert gameplay videos into live commentary videos, share and rate them within the community, and also supports monetization of the videos. This system is mainly composed of three entities: a server, a device, and users.

[1834] Main features

[1835] Game video upload

[1836] User

[1837] Users upload videos of the games they play to the system via their terminals.

[1838] Select the video file and click the upload button to send the video to the system.

[1839] Terminal

[1840] The terminal converts the game video file selected by the user into an HTTP request and sends it to the server.

[1841] Notify the user that the video was successfully sent.

[1842] server

[1843] The received video file is saved in storage, and the video metadata (file name, upload date and time, etc.) is recorded in the database.

[1844] Video Analysis

[1845] server

[1846] The server runs software to analyze the stored video.

[1847] Using image and voice recognition technology, it detects important events in the game (e.g., defeating enemies, boss battles, level-ups, etc.) along with their timeline.

[1848] Generate a list of detected event information (event type, occurrence time).

[1849] Generate live commentary

[1850] server

[1851] Based on the detected event list, a natural language generation (NLG) engine is used to generate corresponding commentary.

[1852] The generated commentary includes phrases such as "Great! You defeated the enemy!" for "You defeated the enemy."

[1853] Select appropriate comments from a database of professional commentators and celebrities and add them as needed.

[1854] Rendering live video

[1855] server

[1856] The generated commentary is overlaid onto the gameplay video.

[1857] Use a video editing engine to overlay text onto the video based on the corresponding timestamps.

[1858] The completed live video is saved to storage and metadata is added to the database.

[1859] Share and rate videos

[1860] User

[1861] Users log in to the system to share the generated live video on a community page.

[1862] Click the share button and enter a title and description for your video.

[1863] You can watch live videos shared by other users and leave likes and comments.

[1864] Terminal

[1865] Receives share and rating requests and sends them to the server.

[1866] Display a community page where users can easily watch and rate other videos.

[1867] server

[1868] Processes incoming share and rating requests and updates the database with video rating scores and comments.

[1869] We will periodically tally rating scores and officially feature the top-rated videos.

[1870] Monetization support

[1871] User

[1872] The generated live commentary videos will be posted on third-party distribution platforms in order to generate revenue.

[1873] Get a download link from the system and upload the video to a distribution platform.

[1874] server

[1875] Provide links to facilitate posting and provide users with monetization guidelines.

[1876] Specific examples

[1877] For example, suppose User A uploads a 10-minute video of himself playing an action game. The server analyzes the video and detects scenes where enemies are defeated or boss battles are fought. The AI ​​generates commentary such as "Great attack!" or "Great opportunity!" and overlays it on the original video. The final commentary video is shared by User A to the community and receives high ratings from other users. As a result of the high ratings, the server officially features the video, and User A can then post the video on YouTube to earn revenue.

[1878] In this way, the system of the present invention makes the process of creating, sharing, rating, and monetizing gameplay videos simple and effective.

[1879] The processing flow will be explained below.

[1880] Step 1:

[1881] User

[1882] Prepare your gameplay video on your device and access the upload page.

[1883] Click the Upload button and select the video file to upload from the file selection dialog.

[1884] Step 2:

[1885] Terminal

[1886] The selected video file is converted into an HTTP request and sent to the server.

[1887] Show the upload progress to the user.

[1888] Step 3:

[1889] server

[1890] Analyze the received HTTP request and save the video file in storage.

[1891] A unique ID is assigned to the video file, and metadata (file name, upload date and time, etc.) is stored in the database.

[1892] Step 4:

[1893] server

[1894] Load the video stored in storage for analysis.

[1895] It uses image and voice recognition technology to detect important events in the game.

[1896] List the timestamp and event details for each event.

[1897] Step 5:

[1898] server

[1899] Pass the event list to a natural language generation (NLG) engine.

[1900] The NLG engine generates commentary corresponding to events. For example, for the event "Boss battle begins," it generates a comment like "This is where the real action begins, the boss battle begins!"

[1901] If necessary, add appropriate comments from a database of professional commentators and celebrities.

[1902] Step 6:

[1903] server

[1904] A video editing engine is launched to overlay the generated commentary on the original game video.

[1905] Insert comments into the video based on appropriate timestamps to generate a play-by-play video.

[1906] The completed live video is saved to storage and metadata is added to the database.

[1907] Step 7:

[1908] User

[1909] Log in to your account and share the generated commentary video on the community page.

[1910] Click the share button and enter a title and description for your video.

[1911] Step 8:

[1912] Terminal

[1913] Sends a share request to the server.

[1914] It provides an interface for accessing community pages and watching live videos shared by other users.

[1915] Step 9:

[1916] server

[1917] Process incoming share requests and add new play-by-play video entries to the community page.

[1918] When other users watch or rate a video, the rating data is saved in a database.

[1919] Step 10:

[1920] server

[1921] We periodically tally the rating scores and select the top-rated videos.

[1922] Selected videos will be featured on the official special page.

[1923] Step 11:

[1924] User

[1925] Check out the featured videos.

[1926] Get a link to download the video or post it to a third-party distribution platform.

[1927] Step 12:

[1928] User

[1929] Post videos to distribution platforms and monetize them. After posting, use revenue models such as advertising revenue and Super Chat.

[1930] The above is the flow of specific processing steps for carrying out the invention.

[1931] Example 1

[1932] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1933] Conventional gameplay video sharing systems require users to manually edit videos and add commentary, which is time-consuming and labor-intensive. Furthermore, the process of sharing, rating, and monetizing the created videos is complicated, making it difficult for users to use. The objective of this invention is to solve these problems and provide a system that allows users to easily convert gameplay videos into commentary videos and effectively share, rate, and monetize them.

[1934] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1935] In this invention, the server includes: means for users to upload videos of games they play; means for converting the video files into HTTP requests and sending them to the server; means for saving the received video files in storage and recording metadata in a database; means for analyzing the uploaded videos, detecting in-game events using image recognition technology and voice recognition technology, and generating a list of event information; means for generating commentary comments using a natural language generation engine based on the detected game events and comparing them with a database to select appropriate comments; means for overlaying the generated commentary comments on videos to generate commentary videos; means for providing a community function for sharing the generated commentary videos; means for other users to rate the shared commentary videos; means for officially featuring commentary videos with good ratings based on the rating results; and means for providing a link for posting the generated commentary videos to a third-party distribution platform for monetization. This enables users to easily analyze, edit, and share gameplay videos and effectively monetize them.

[1936] "User" means any individual or legal entity that uses the System to upload, analyze, share, and rate gameplay videos.

[1937] A "gameplay video" is a digital file that contains video and audio of a game being played by a user.

[1938] "Uploading" is the process by which a user transfers a video file from their device to the system.

[1939] An "HTTP request" is a request message for sending data to a server based on a protocol used for communication over the Internet.

[1940] A "server" is a computer system that processes and stores data received from users and provides the required services.

[1941] "Storage" refers to a storage device for saving digital data, including hard disk drives (HDDs) and solid-state drives (SSDs).

[1942] "Metadata" is data that describes the characteristics and content of a video file, and includes, for example, the file name, upload date and time, and video length.

[1943] A "database" is a system for efficiently storing, retrieving, and updating structured information.

[1944] "Image recognition technology" is a technology that analyzes the content of images and videos and identifies specific objects and movements.

[1945] "Speech recognition technology" is a technology that analyzes voice data and converts it into meaningful text or commands.

[1946] A "natural language generation engine" is an artificial intelligence technology that generates text based on input data in a form that is easy for humans to understand.

[1947] "Commentary" is text that includes commentary and impressions about specific events in gameplay videos.

[1948] "Overlay" is a technique for displaying text or images over specific locations on a video.

[1949] "Community Features" are features within the system that allow users to share and rate generated content.

[1950] A "rating" is feedback or a score that a user gives to another user's content.

[1951] A "feature" is a display of specific content that stands out more than other content.

[1952] "Monetization" is the process of utilizing generated content for financial gain.

[1953] This invention is a system that allows users to easily upload gameplay videos and then analyze, edit, share, rate, and monetize them. This system operates in cooperation with a server, devices, and users.

[1954] To upload a video of a game that a user has played, the user first logs in to the system. After logging in, the user selects the gameplay video file from their device and clicks the upload button. At this time, the device converts the video file into an HTTP request and sends it to the server. The uploaded video file is received by the server and saved in storage. The server also records the video's metadata (file name, upload date and time, etc.) in a database.

[1955] Next, the server launches software to analyze the saved video file. Specifically, it uses image recognition technology (e.g., OpenCV) and speech recognition technology (e.g., Google Speech-to-Text API) to detect important in-game events (e.g., defeating an enemy, leveling up, etc.). The server then generates a list of detected event information (event type, occurrence time). The server then uses this event list to generate commentary using a natural language generation (NLG) engine (e.g., GPT-3). The generated commentary is then compared with a database of comments from professional commentators and celebrities to select appropriate comments.

[1956] The server then uses a video editing engine (e.g., FFmpeg) to overlay the commentary onto the video. During this process, commentary generated based on timestamps is overlaid onto specific locations in the gameplay video. The completed commentary video is saved to storage, and the metadata is added to a database.

[1957] Users can share the live commentary videos they have created on the system's community page. When sharing, users enter a title and description for the video. The device then sends a sharing request to the server, allowing other users to view and rate the shared live commentary video. The server processes the received rating request and updates the video's rating score and comments in the database. Based on the rating results, live commentary videos that receive good ratings are officially featured.

[1958] Additionally, users can monetize their generated live streams by posting them to third-party distribution platforms, and the server will provide users with links and guidelines to support monetization.

[1959] As a concrete example, suppose User A uploads a 10-minute video of himself playing an action game. The server analyzes the video and detects scenes where enemies are defeated and boss battles. The AI ​​generates commentary such as "Great attack!" or "Great opportunity!" and overlays it on the original video. The final commentary video is shared by User A with the community and receives high ratings from other users. As a result of the high ratings, the server officially features the video, and User A can then post the video on distribution platforms such as YouTube to earn revenue.

[1960] An example of a prompt might be:

[1961] "I want to create a program that analyzes gameplay video, generates commentary based on important events, and then creates a commentary video. Please explain the steps below:

[1962] 1. Users upload game videos

[1963] 2. Analyze video on the server and detect important events

[1964] 3. Create live commentary using natural language generation

[1965] 4. Overlay your comments on the video and you're done!

[1966] The system operates based on this content, improving user convenience.

[1967] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1968] Step 1:

[1969] User logs in and uploads game video

[1970] To log in to the system, the user enters their username and password. After successful login, the user selects a gameplay video file from the terminal and clicks the upload button. The input is the user information and the game video file, and the output is an HTTP request. When the upload button is clicked, the terminal converts the selected video file into an outgoing packet as an HTTP POST request and sends it to the server.

[1971] Step 2:

[1972] The device converts the video file into an HTTP request and sends it to the server.

[1973] The device follows the user's instructions, converts the selected video file into an HTTP request, and sends it to the server using the POST method. The input is the video file, and the output is an HTTP request. Specifically, the device embeds the file's binary data in the body of the HTTP POST request and sends it to the specified URL.

[1974] Step 3:

[1975] The server receives the video file and stores it in storage.

[1976] The server extracts the video file from the body of the HTTP request received and saves it to storage. The input is the HTTP request, and the output is the saved video file. Specifically, the file system API is used to save the video file to a specified directory.

[1977] Step 4:

[1978] The server records the video metadata in a database

[1979] The server extracts metadata from the saved video files (file name, upload date and time, video length, etc.) and records it in a database. The input is the saved video file, and the output is a database record. Specifically, it obtains the file attribute information and inserts the record using the database API.

[1980] Step 5:

[1981] The server analyzes the video using video analysis software.

[1982] The server runs video analysis software (e.g., OpenCV) and uses image recognition technology to detect important in-game events (e.g., defeating an enemy, leveling up, etc.). The input is the saved video file, and the output is a list of event information. Specifically, it processes each frame of the video, detecting specific in-game events, and lists them along with their timestamps.

[1983] Step 6:

[1984] The server uses speech recognition technology to identify audio events and record them as metadata.

[1985] The server uses speech recognition technology (e.g., Google Speech-to-Text API) to identify important audio events from the video's audio track and record them as metadata. The input is the saved video file and the event information list, and the output is an updated event information list. Specifically, the server analyzes the audio data, converts it into text strings in each language, and records them along with the time when the audio events occurred.

[1986] Step 7:

[1987] The server generates live commentary using a natural language generation engine

[1988] The server uses a natural language generation (NLG) engine (e.g., GPT-3) to generate commentary based on the detected event information. The input is a list of event information, and the output is a list of commentary. Specifically, a prompt sentence is input to the generative AI model, which then generates appropriate commentary.

[1989] Step 8:

[1990] The server compares the generated live commentary with the database and selects the appropriate comment.

[1991] The server compares the generated live commentary comments with a database of comments from professional commentators and celebrities to select appropriate comments. The input is a list of generated live commentary comments, and the output is a list of matched live commentary comments. Specifically, it uses a string search algorithm to extract matching comments from the database.

[1992] Step 9:

[1993] The server uses a video editing engine to overlay commentary onto the video.

[1994] The server uses a video editing engine (e.g., FFmpeg) to overlay the generated commentary based on the corresponding timestamps of the video. The input is a gameplay video file and a list of commentary, and the output is the finished commentary video. Specifically, it applies a text overlay filter to specific frames of the video.

[1995] Step 10:

[1996] The server saves the completed live video to storage and adds the metadata to the database.

[1997] The server saves the completed live video in storage and adds the video's metadata (such as the creation date and time, video playback time, and comment list) to the database. The input is the completed live video and a list of live comments, and the output is an updated database record. Specifically, the server saves the video file to the storage system and inserts its metadata into the database.

[1998] Step 11:

[1999] Share user-generated play-by-play videos on your community page

[2000] A user logs in to the system and performs an operation to share the generated live video on a community page. The input is the generated live video, and the output is a sharing request. The user clicks the share button and enters the video title and description.

[2001] Step 12:

[2002] The device sends a share request to the server

[2003] The device sends the user's sharing request to the server as an HTTP POST request. The input is the sharing request, and the output is an HTTP POST request. Specifically, the sharing request is packetized along with the video metadata and sent to the server.

[2004] Step 13:

[2005] The server processes share and rating requests and updates the rating scores and comments to the database.

[2006] The server processes the received share requests, adds new video information to the database, and processes rating requests made by other users on the shared live videos, updating the rating scores and comments in the database. The input is the share request and the rating request, and the output is the updated database record.

[2007] Step 14:

[2008] The server will officially feature well-rated playthrough videos based on the evaluation results.

[2009] The server periodically aggregates the rating scores and officially features the top-rated live videos. The input is the rating scores and comments, and the output is a list of featured videos. Specifically, it calculates the ranking of the rating scores and displays the top-rated videos in a specific section.

[2010] Step 15:

[2011] Monetize user-generated live videos by posting them to a streaming platform

[2012] Users post their generated commentary videos to third-party distribution platforms (e.g., YouTube, Twitch) and monetize them. The input is the generated commentary video, and the output is the post to the distribution platform. The system provides users with links and guidelines to support monetization.

[2013] Step 16:

[2014] Provide users with links and guidelines to help your server monetize

[2015] The server displays links and guidelines for video monetization to users. The input is a user request, and the output is monetization links and guidelines. Specifically, the server displays information related to monetization on the system dashboard for easy access by users.

[2016] The above are the specific processing steps of this system.

[2017] (Application example 1)

[2018] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2019] Currently, it takes a lot of time and effort for users to easily edit, rate, and monetize their gameplay videos. Furthermore, further technological advancements are required to realize advanced features such as seamless integration with content distribution platforms and highly accurate comment generation and overlays using AI technology. The purpose of this invention is to solve these issues and provide a simple system that allows users to easily and effectively create and share gameplay videos and further monetize them.

[2020] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2021] In this invention, the server includes means for allowing users to upload videos of games they have played, means for analyzing the uploaded videos and detecting important game events, means for generating commentary comments in natural language based on the detected game events, means for generating commentary videos by overlaying the generated commentary comments on videos, means for providing a community function for sharing the generated commentary videos, means for other users to rate the shared commentary videos, means for officially featuring commentary videos that have received positive reviews based on the rating results, means for seamlessly sharing the generated commentary videos to a third-party content distribution platform, and means for accurately overlaying commentary comments based on timestamps of game events using AI technology. This allows users to easily and effectively create and share commentary videos, enabling monetization through highly accurate commentary generation and seamless distribution.

[2022] "User" means an individual or organization that uses this system to upload gameplay videos and create and share live gameplay videos.

[2023] "Game videos" refer to video files recorded by users playing games.

[2024] "Uploading" refers to the act of a user sending gameplay video from their device to a server.

[2025] "Analysis" refers to the process by which the server processes the uploaded video and detects important game events.

[2026] "Game Event" refers to an important event that occurs within the game (e.g., defeating an enemy, fighting a boss, leveling up, etc.).

[2027] "Natural language" refers to the language used by humans on a daily basis, and is used in this system to generate live commentary.

[2028] "Live commentary" refers to comments that explain and evaluate gameplay in real time or near real time.

[2029] "Overlay" refers to the process of displaying the generated commentary overlaid on the game video.

[2030] "Live commentary video" refers to a video file created by overlaying live commentary on gameplay video.

[2031] "Community Function" refers to a platform where users can share live videos they have created and other users can watch, rate, and comment on them.

[2032] "Evaluation" refers to the act of users within a community rating a posted live video by giving it a "like" or comment.

[2033] "Featured" means that a live commentary video that has received positive reviews will be officially featured and introduced to more users.

[2034] "Content distribution platform" refers to a video sharing service provided by a third party, such as YouTube or other similar services.

[2035] "Seamless" means that the operations performed by the user are smoothly linked together, allowing a series of tasks to be performed without interruption.

[2036] "AI technology" refers to artificial intelligence technology, which in this system is used for natural language generation and analysis of game events.

[2037] A "timestamp" refers to data that records the time when a particular event occurred.

[2038] "Accuracy" refers to the degree to which live commentary and analysis results generated using AI technology are accurate as expected by users.

[2039] The embodiment of the present invention will be described in detail below. This system is mainly composed of three components: a server, a terminal, and a user.

[2040] 1. Uploading game videos

[2041] User

[2042] Users use a smartphone application to select the game video they have played and click the upload button to send it to the server.

[2043] Terminal

[2044] The device converts the game video file selected by the user into an HTTP request and sends it to the server. The device notifies the user that the video has been sent successfully.

[2045] server

[2046] The server stores the received video file in storage and records the video's metadata (file name, upload date and time, etc.) in a database.

[2047] 2. Video Analysis

[2048] server

[2049] The server runs software to analyze the saved video. The software used includes Google Cloud Vision API and Google Speech-to-Text. These are used for image and voice recognition to detect important events in the game (e.g., defeating an enemy, boss battles, level-ups, etc.). The results of this detection are compiled into an event list.

[2050] 3. Generating live commentary

[2051] server

[2052] Based on the detected event list, the server uses a natural language generation engine (e.g., GPT-4) to generate corresponding commentary, such as "When an enemy is defeated: Amazing! You defeated the enemy!". It also selects appropriate comments from a database of professional commentators and celebrities and adds them as needed.

[2053] An example prompt is "Generate a commentary for the following gaming event: 'Player defeats the boss in an epic fight'".

[2054] 4. Rendering the Live Video

[2055] server

[2056] The server processes the generated commentary onto the gameplay video. FFmpeg is used as the video editing software. The commentary is accurately overlaid onto the video based on the corresponding timestamp. The completed video is saved to storage, and metadata is added to the database.

[2057] 5. Share and rate the video

[2058] User

[2059] Users log in to the system to share the created live commentary video on the community page. They click the share button and enter the video title and description. They can also watch the live commentary videos shared by other users and leave "likes" and comments.

[2060] Terminal

[2061] The device receives and sends share and rating requests to the server, and also displays a community page where users can easily watch and rate other videos.

[2062] server

[2063] The server processes the incoming share and rating requests, updates the video's rating score and comments to a database, and periodically aggregates the rating scores and officially features the top-rated play-by-play videos.

[2064] 6. Monetization Support

[2065] User

[2066] Users can monetize the generated live videos by posting them to third-party distribution platforms. Users obtain a download link from the system and upload the video to the distribution platform.

[2067] server

[2068] The server provides links to facilitate posting and provides users with monetization guidelines.

[2069] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2070] Step 1:

[2071] Game video upload

[2072] Process flow:

[2073] A user launches the application on their smartphone, selects a gameplay video file from their device, and then clicks the upload button, which generates an HTTP request to send the video to the server.

[2074] Input and Output:

[2075] Input: Gameplay video file.

[2076] Output: HTTP request to the server.

[2077] Specific behavior:

[2078] When the user selects a video file and clicks the send button, the terminal generates an HTTP request for sending the selected video file to the server, and executes the transmission of the video.

[2079] Step 2:

[2080] Receiving and storing videos

[2081] Process flow:

[2082] The server receives the HTTP request sent from the device, saves the video file in storage, and records the video file's metadata (file name, upload date and time, etc.) in a database.

[2083] Input and Output:

[2084] Input: The gameplay video file sent in the HTTP request.

[2085] Output: Video files saved on storage, along with their metadata.

[2086] Specific behavior:

[2087] When the server receives an HTTP request, it saves the video data in storage and records the metadata of the video file in a database.

[2088] Step 3:

[2089] Video Analysis

[2090] Process flow:

[2091] The server runs image and voice recognition software to analyze the stored video, using the Google Cloud Vision API and Google Speech-to-Text to detect important game events (e.g., enemy defeats, boss battles, level-ups, etc.) and generate an event list.

[2092] Input and Output:

[2093] Input: Gameplay video file saved in storage.

[2094] Output: A list of game events.

[2095] Specific behavior:

[2096] The server loads the video file, performs image recognition using the Google Cloud Vision API and speech recognition using Google Speech-to-Text, and outputs a list of detected game events with timestamps.

[2097] Step 4:

[2098] Generate live commentary

[2099] Process flow:

[2100] The server generates commentary using a generative AI model (e.g., GPT-4) based on the detected event list. The comment is a phrase such as "When an enemy is defeated: Great! You defeated the enemy!"

[2101] Input and Output:

[2102] Input: A list of detected game events.

[2103] Output: A list of live commentary.

[2104] Specific behavior:

[2105] The server uses a natural language generation engine to generate the best commentary for each event from the event list using the prompt text as input. An example of a specific prompt text is "Generate a commentary for the following gaming event: 'Player defeats the boss in an epic fight'".

[2106] Step 5:

[2107] Rendering live video

[2108] Process flow:

[2109] The server processes the generated commentary overlays onto the gameplay video, using video editing software (e.g., FFmpeg) to overlay each commentary onto the video based on its corresponding timestamp.

[2110] Input and Output:

[2111] Input: Gameplay video file, commentary list.

[2112] Output: A commentary video file with commentary overlays.

[2113] Specific behavior:

[2114] The server uses FFmpeg to overlay the commentary onto the relevant parts of the video, generating the final commentary video.

[2115] Step 6:

[2116] Share and rate videos

[2117] Process flow:

[2118] Users can upload the created live commentary videos to a community page, where other users can view and rate them. Other users can also like and comment on the live commentary videos.

[2119] Input and Output:

[2120] Input: Live video file, share request.

[2121] Output: Play-by-play videos uploaded to the community page, and rating data.

[2122] Specific behavior:

[2123] A user clicks the share button, enters the title and description of the video, and posts it to the community page. The server receives the share request and performs the necessary processing to display the video on the community page. It also manages rating requests from other users and records the ratings in a database.

[2124] Step 7:

[2125] Monetization support

[2126] Process flow:

[2127] Users can then post their live streams to third-party platforms to monetize them, and the server will provide users with links and guidelines for monetization.

[2128] Input and Output:

[2129] Input: Play-by-play video file.

[2130] Output: Links to distribution platforms, monetization guidelines.

[2131] Specific behavior:

[2132] The server provides a link to facilitate posting and displays appropriate monetization guidelines to the user, who then uses the provided link to upload their live video to the distribution platform.

[2133] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2134] The system of the present invention allows users to easily convert gameplay videos into commentary videos, share and rate them within the community, and further supports monetization of the created videos. By combining this system with an emotion engine that recognizes the user's emotions, it is possible to generate more natural and realistic commentary comments. Specific embodiments of the present invention are described below.

[2135] Main features

[2136] Game video upload

[2137] User

[2138] The user prepares the game video they played on their device and accesses the upload page.

[2139] Click the Upload button and select the video file to upload from the file selection dialog.

[2140] Terminal

[2141] The selected video file is converted into an HTTP request and sent to the server.

[2142] Show the upload progress to the user.

[2143] server

[2144] Analyze the received HTTP request and save the video file in storage.

[2145] A unique ID is assigned to the video file, and metadata (file name, upload date and time, etc.) is stored in the database.

[2146] Video Analysis

[2147] server

[2148] Load the video stored in storage for analysis.

[2149] It uses image and voice recognition technology to detect important events in the game.

[2150] List the timestamp and event details for each event.

[2151] Introducing the Emotion Engine

[2152] server

[2153] It analyzes the user's voice and facial expressions in the video and activates an emotion engine to detect emotions.

[2154] The emotion engine adds detected emotion data (e.g., joy, surprise, anger, etc.) to the analysis results.

[2155] Adjust the tone and content of commentary based on detected emotional data.

[2156] Generate live commentary

[2157] server

[2158] Pass the event list to a natural language generation (NLG) engine.

[2159] The NLG engine generates commentary corresponding to the event. For example, if the detected emotion for the event "Boss battle begins" is "surprise," it generates a comment like "This is unexpected! The boss battle has begun!"

[2160] If necessary, add appropriate comments from a database of professional commentators and celebrities.

[2161] Rendering live video

[2162] server

[2163] A video editing engine is launched to overlay the generated commentary onto the gameplay video.

[2164] Comments adjusted based on the emotion data are inserted into the video based on the corresponding timestamps to generate a live video.

[2165] The completed live video is saved to storage and metadata is added to the database.

[2166] Share and rate videos

[2167] User

[2168] Log in to your account and share the generated commentary video on the community page.

[2169] Click the share button and enter a title and description for your video.

[2170] Terminal

[2171] Sends a share request to the server.

[2172] It provides an interface for accessing community pages and watching live videos shared by other users.

[2173] server

[2174] Process incoming share requests and add new play-by-play video entries to the community page.

[2175] When other users watch or rate a video, the rating data is saved in a database.

[2176] Monetization support

[2177] User

[2178] The generated live commentary videos will be posted on third-party distribution platforms in order to generate revenue.

[2179] Get a download link from the system and upload the video to a distribution platform.

[2180] server

[2181] Provide links to facilitate posting and provide users with monetization guidelines.

[2182] Specific examples

[2183] For example, suppose User B uploads a 15-minute video of himself playing a horror game. The server analyzes the video and detects scenes where monsters appear or scenes where the user is surprised. The emotion engine detects the user's surprise, and based on that emotion data, the AI ​​generates commentary such as "I can't believe it! A monster appeared at this exact moment!" These comments are overlaid on the original video, and User B shares the completed commentary video with the community. After receiving high ratings, the video is officially featured, and User B can further post the video on YouTube, earning revenue opportunities.

[2184] In this way, the system of the present invention uses an emotion engine to generate immersive live videos that correspond to the user's emotions, allowing for a simple and effective process of sharing, rating, and monetizing.

[2185] The processing flow will be explained below.

[2186] Step 1:

[2187] User

[2188] Save your gameplay videos to your device.

[2189] Access the game commentary generation system, log in, and proceed to the video upload page.

[2190] Click the upload button and select the gameplay video file.

[2191] Once selected, confirm the upload.

[2192] Step 2:

[2193] Terminal

[2194] The selected gameplay video file is sent to the server as an HTTP request.

[2195] Show a progress bar for video uploads and notify the user once it's complete.

[2196] Step 3:

[2197] server

[2198] The received video file is saved in storage.

[2199] A unique ID is assigned to the video file, and metadata such as the file name and upload date and time are recorded in the database.

[2200] Step 4:

[2201] server

[2202] The saved video is passed to the analysis module.

[2203] Uses image recognition technology to detect important events in the game (e.g., defeating enemies, boss battles, level-ups, etc.).

[2204] Uses voice recognition technology to detect the user's voice and reactions.

[2205] Lists important events detected and audio reactions with timestamps.

[2206] Step 5:

[2207] server

[2208] The emotion engine is activated and emotions are detected by analyzing the user's voice and facial expressions in the video.

[2209] Add detected emotion data (e.g., happiness, surprise, anger, etc.) to the analysis list.

[2210] Step 6:

[2211] server

[2212] Pass the event list and sentiment data to a natural language generation (NLG) engine.

[2213] The NLG engine generates live commentary corresponding to events. For example, if the user is recognized as "surprised" by the event "Boss battle begins," it generates a comment such as "The boss appeared here! What a surprise!"

[2214] Add appropriate comments from a database of professional commentators and celebrities, if needed.

[2215] Step 7:

[2216] server

[2217] The generated commentary is passed to a video editing engine and overlaid on the original gameplay video.

[2218] Based on the comment timestamps, text and audio are inserted into the video to generate the final commentary video.

[2219] The completed live video is saved to storage and the metadata is updated in the database.

[2220] Step 8:

[2221] User

[2222] Log in to your user account and check the generated live video.

[2223] Enter the settings (title, description, etc.) for sharing on the community page.

[2224] Click the share button to post your video to the community.

[2225] Step 9:

[2226] Terminal

[2227] The input contents of the shared settings are sent to the server as an HTTP request.

[2228] It displays the community page and provides an interface where users can view other users' videos.

[2229] Step 10:

[2230] server

[2231] Receives the share request and updates the video's database entry.

[2232] Add a new playthrough video entry to the community page.

[2233] Step 11:

[2234] User

[2235] Watch other users' live videos and rate them with likes and comments.

[2236] Step 12:

[2237] Terminal

[2238] The input contents of the rating and comment are sent to the server as an HTTP request.

[2239] Step 13:

[2240] server

[2241] The received rating data and comments are stored in a database and the video's rating score is updated.

[2242] Step 14:

[2243] server

[2244] We periodically tally the rating scores and select the top-rated videos.

[2245] Selected videos will be featured on the official special page.

[2246] Step 15:

[2247] User

[2248] See which videos are featured and get links to download or post your videos to third-party distribution platforms.

[2249] Step 16:

[2250] User

[2251] They post videos on third-party distribution platforms and aim to monetize through advertising revenue, Super Chats, etc.

[2252] Example 2

[2253] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2254] Previously, the process for users to convert their gameplay videos into play-by-play videos and then share and rate them was complicated and time-consuming. It was also difficult to generate natural play-by-play comments that reflected the user's emotions, resulting in content that often lacked realism. Furthermore, support for monetizing the play-by-play videos was insufficient, making the process complicated for users.

[2255] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for users to upload videos of electronic games they have played; means for analyzing the uploaded videos and detecting important events; means for analyzing users' emotions and adding the detected emotion data to the analysis results; means for generating live commentary comments in natural language based on the detected game events and emotion data; means for generating live commentary videos by overlaying the generated live commentary comments on videos; means for providing a community function for sharing the generated live commentary videos; means for other users to rate the shared live commentary videos; means for officially featuring well-rated live commentary videos based on the rating results; means for storing live commentary comments from professional commentators and celebrities in a database and appropriately selecting and adding these comments based on the detected game events and emotion data; and means for users to post the generated live commentary videos to third-party distribution platforms and monetize them. This allows users to easily convert their own gameplay videos into natural and realistic live commentary videos, and efficiently share, rate, and monetize them.

[2256] "User" refers to any individual or entity who uploads electronic gameplay videos to the system and generates, shares, and monetizes live gameplay videos.

[2257] "Electronic game" refers to an interactive entertainment activity that operates on a computer, gaming console, mobile device, or the like.

[2258] "Video" refers to a series of digital files containing the video and audio of an electronic game played by a user.

[2259] "Upload" refers to the act of a user sending a video file from their own device to the system's server.

[2260] "Analysis" refers to the process by which the system automatically analyzes uploaded videos to detect important events and user sentiment.

[2261] "Events" refer to important actions or scenes that occur within the game, such as the start of a boss battle or the acquisition of an item.

[2262] "Emotion" refers to psychological reactions detected from a user's facial expressions and tone of voice, including joy, surprise, anger, etc.

[2263] "Natural language" refers to the language used by humans on a daily basis and is used to generate commentary in text or audio form.

[2264] "Live commentary" refers to text such as explanations and impressions generated based on events that occur during gameplay and the user's emotions.

[2265] "Overlay" refers to the process of displaying commentary at specific timestamps on the original game footage.

[2266] The "community function" refers to an online platform that allows users to share their own live videos with other users and exchange opinions and ratings.

[2267] "Rating" refers to the rating actions other users take on a shared live video, including comments, likes, and number of views.

[2268] "Feature" refers to the system officially recommending well-rated live videos so that they can be viewed by more users.

[2269] A "professional commentator" refers to an individual or group that professionally broadcasts games and provides high-quality commentary using their knowledge and experience.

[2270] "Celebrities" are publicly known or influential individuals whose live commentary is stored and used in a database.

[2271] "Database" refers to a software system that stores information in an organized manner and makes it available for retrieval and use as needed.

[2272] "Third-party distribution platforms" refers to online services, such as YouTube, that are used to share and monetize the live-action videos generated.

[2273] "Monetization" refers to the process of earning advertising revenue, donations from viewers, etc. from user-generated live videos.

[2274] As an embodiment of the invention, the program processing of this system is specifically described below. This system allows users to easily convert gameplay videos into live commentary videos, share and rate them within the community, and further supports monetization of the created videos.

[2275] composition

[2276] The system includes the following major hardware and software components:

[2277] User's device (e.g. PC or smartphone)

[2278] Server (cloud-based)

[2279] Storage systems (e.g. cloud storage)

[2280] Database (e.g. MySQL)

[2281] Image recognition technology (e.g., OpenCV)

[2282] Voice recognition technology (e.g., Google Speech-to-Text)

[2283] Sentiment analysis engine (e.g. Affectiva SDK)

[2284] Natural Language Generation (NLG) engines (e.g., GPT-3)

[2285] Video editing engine (e.g. FFmpeg)

[2286] Distribution platform (e.g. YouTube)

[2287] Operation flow

[2288] (1) The user saves the game video they played on their device and accesses the system's upload page. The user clicks the upload button, opens a file selection dialog, and selects the video file.

[2289] (2) The device converts the selected video file into an HTTP POST request and sends it to the server. The server analyzes the received HTTP POST request and saves the uploaded video file in cloud storage. At the same time, it assigns a unique ID to the video file and saves its metadata (file name, upload date and time, etc.) in a database.

[2290] (3) The server reads the video files stored in the storage for analysis and detects important events in the game using image recognition technology (e.g., OpenCV) and voice recognition technology (e.g., Google Speech-to-Text). For each event, the timestamp and event content are saved in a list format.

[2291] (4) Based on the analysis results of the detected game events, the server launches an emotion engine (e.g., Affectiva SDK) to analyze the user's facial expressions and tone of voice in the video, extracting emotional appeal points and adding the detected emotion data (e.g., joy, surprise, anger, etc.) to the analysis results.

[2292] (5) The server passes the event list and emotion data to a natural language generation (NLG) engine (e.g., GPT-3) to generate commentary corresponding to each event. For example, if surprise is detected at the start of a boss battle, the server generates a comment such as "I can't believe it! A monster appeared at this exact moment!" If necessary, appropriate comments can be selected and added from a database of comments by professional commentators or celebrities.

[2293] (6) The server passes the generated commentary and the original game video to a video editing engine (e.g., FFmpeg), which overlays the commentary on the video based on the corresponding timestamps to generate a commentary video. The completed commentary video is saved in cloud storage, and its metadata is added to a database.

[2294] (7) The user logs in to their account and shares the created live video on the community page. The user clicks the share button, enters the video title and description, and shares it.

[2295] (8) The device sends a sharing request to the server, which adds a new live video entry to the community page. Other users watch and rate the video, and the rating data is stored in a database.

[2296] (9) Users can post their live streams to third-party platforms (e.g., YouTube) and monetize them. The system provides links to facilitate posting and provides users with monetization guidelines.

[2297] Specific examples

[2298] For example, suppose User B uploads a 15-minute video of himself playing a horror game. The server analyzes the video and detects scenes in which monsters appear and scenes in which the user is surprised. The emotion engine detects the user's surprise, and based on that emotional data, the AI ​​generates commentary such as "I can't believe it! A monster appeared at this exact moment!". The generated commentary is then overlaid on the original video to complete the commentary video. User B shares the completed commentary video with the community, and if it receives high ratings from other users, it can be posted on distribution platforms such as YouTube, providing an opportunity for monetization.

[2299] Example of input prompt for generative AI model

[2300] "Please explain the system that analyzes gameplay videos uploaded by users, generates live commentary using an emotion engine and a natural language generation engine, and supports sharing and monetization."

[2301] In this way, by utilizing the emotion engine, the present invention can generate immersive live...

Claims

1. A means for users to upload videos of games they play; A means of analyzing uploaded videos to detect important game events; means for generating commentary in natural language based on the detected game events; A means for generating a commentary video by overlaying the generated commentary comments on a video; A means for providing a community function for sharing the generated live commentary video; A means for other users to rate the shared live video; A means to officially feature well-rated play-by-play videos based on the evaluation results, A system including:

2. 2. The system of claim 1, further comprising means for storing live commentary from professional commentators and celebrities in a database and for selecting and adding such commentary as appropriate based on detected game events.

3. The system of claim 1 , further comprising means for enabling users to post the generated live video to a third-party distribution platform for monetization.

Images