System

Abstract

A system is provided.SOLUTION: A system including means for selecting an event that a user wants to view, means for transmitting a request for a live stream to a server, means for collecting posts on social media related to the event designated by the server in real time, means for analyzing the collected posts and specifying a peak time of excitement or emotion, means for notifying the user of an optimal cheering timing on the basis of the specified peak time, and means for providing a notification of the live stream and the cheering timing to a terminal of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional online sports viewing has had issues such as difficulty in communicating with other fans in real time, resulting in a lack of a sense of unity. Furthermore, despite a strong desire for detailed commentary and background information during the game in real time, there are limited ways to easily obtain this information. Furthermore, there is a lack of ways to view customized highlights tailored to individual needs. For these reasons, there is a demand for an environment that allows viewers to enjoy the game more deeply and with greater satisfaction. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means.

[0006] The system provides a means for users to select the event they want to watch and send a live stream request to the server. The server collects social media posts related to the specified event in real time and analyzes the collected posts to identify peak times of excitement and emotion. Based on the identified peak times, the system notifies users of the optimal timing to cheer. The system also includes a system that provides live streams and notifications of the timing to cheer to users' devices.

[0007] Furthermore, the system includes a server that receives the live-stream video data, a commentary assistant that analyzes the progress of the match, a means for transmitting commentary text generated by the commentary assistant to a user's terminal in real time, and a means for displaying the commentary text on the user's terminal.

[0008] After the game ends, the server saves all game video data and runs a highlight generation engine to automatically extract important scenes. A customized highlight video is generated based on the user's settings and viewing history, and the generated highlight video is sent to the user's device. The system also includes a means for the user's device to provide the sent highlight video in a playable format.

[0009] A "user" is a person who uses the system to watch a sporting event.

[0010] A "terminal" is a device used by a user, such as a personal computer, smartphone, or tablet.

[0011] "Live streaming" is a technology that broadcasts video of sporting events in real time over the Internet.

[0012] A "server" is a computer system that responds to user requests, distributes live streams, and performs data analysis.

[0013] "Social media" refers to communication platforms that use networks such as Twitter, Instagram, and Facebook.

[0014] A "post" is content such as text, images, and videos that a user uploads to social media.

[0015] "Collection" is the act or process of gathering specific information.

[0016] "Analysis" is the process of using statistics and algorithms to find specific information and patterns in collected data.

[0017] "Peak times of excitement and emotion" are specific times when social media posts are most popular and emotions are at their highest.

[0018] A "notification" is a message or alert that the system uses to inform the user of specific information.

[0019] A "commentary assistant" is software or a system that uses AI to analyze the progress of a match and the content of the play in real time and generate commentary text.

[0020] "Explanatory text" is text that explains the situation and background information of the match.

[0021] The "highlight generation engine" is an AI system that automatically extracts and edits important scenes from game footage.

[0022] "Key moments" are moments in a game that are particularly noteworthy, such as goals, fouls, or good plays.

[0023] "Customized highlights" are highlight videos that are edited based on the user's settings and past viewing history and are tailored to the individual preferences of the user.

[0024] "Viewing history" is a record of the matches and related content that a user has viewed to date. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0026] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0027] First, the terms used in the following description will be explained.

[0028] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0029] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0030] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0031] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0033] [First embodiment]

[0034] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0035] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0036] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0038] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0041] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0043] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0044] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0045] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0046] The present invention is a system that integrates live streaming with social media, real-time AI commentary, automatic highlight generation, and more to enhance users' sporting event viewing experience.

[0047] A way for users to select the event they want to watch and submit a request for a live stream

[0048] Users access the system using their terminals and select the sporting event they want to watch. The terminals then send the selection information to the server, requesting a live stream. As a result, users can watch their desired sporting event in real time.

[0049] A means by which the server collects social media posts related to a specified event in real time.

[0050] The server collects social media posts in real time using specific hashtags and keywords related to the designated event, and updates the collected posts as the event progresses and provides them to users.

[0051] A method for identifying peak times of excitement and emotion by analyzing collected posts

[0052] The server analyzes the collected social media posts and uses natural language processing technology to identify peak times of excitement and emotion, allowing it to understand in real time at which points during a game users are most excited.

[0053] A means of notifying users of the optimal timing for cheering based on identified peak times

[0054] The server then notifies the user of the optimal time to cheer based on the identified peak times. The notification is sent via the device and displayed on the screen, allowing the user to cheer while feeling a sense of unity with other fans.

[0055] A means for the server to receive the live stream video data and run a commentary assistant to analyze the progress of the match

[0056] The server receives live-stream video data in real time and runs an AI commentary assistant, which analyzes the progress of the game, understands player movements, goals scored, tactics, etc., and generates commentary text.

[0057] A means for transmitting the explanatory text generated by the explanatory assistant to the user's terminal in real time and displaying the explanatory text on the user's terminal.

[0058] The server transmits the generated commentary text to the user's device in real time, and the user's device displays the received commentary text overlaid on the live stream video, providing real-time commentary information to the user while watching the game.

[0059] After the match, the server saves all the match video data and runs a highlight generation engine to automatically extract important scenes.

[0060] After the game ends, the server stores all game video data, then runs a highlight generation engine to automatically extract important scenes (scores, fouls, good plays, etc.).

[0061] A means to generate customized highlight videos based on user preferences and viewing history

[0062] Users set their preferences and interests through their devices. The server references the user's settings and viewing history to generate a customized highlight video. The generated highlight video is then edited to suit the user's individual preferences.

[0063] A means for transmitting the generated highlight video to a user's terminal, and for the user's terminal to provide the transmitted highlight video in a reproducible form.

[0064] The server transmits the generated customized highlight video to the user's device, which then displays the received highlight video in a playable format, allowing the user to easily review and watch it.

[0065] Specific examples

[0066] Example 1: Real-time support

[0067] A user named "Yamada" uses a device to watch a basketball game. The server receives a request from Yamada's device and broadcasts the live stream while simultaneously collecting related social media posts. The server analyzes the posts, identifies moments of high excitement, and notifies Yamada of those moments. This allows Yamada to enjoy cheering at the most exciting moments.

[0068] Example 2: AI real-time commentary

[0069] A user named "Sasaki" is watching a soccer match. The server receives the live-stream video data, and the AI ​​commentary assistant analyzes the progress of the match and generates commentary text in real time. The commentary text is sent to Sasaki's device and displayed on the screen. This allows Sasaki to understand the flow of the match and the players' movements in detail.

[0070] Example 3: Automatic highlight generation

[0071] After a baseball game, user "Suzuki-san" wants to watch highlights of games he has previously watched. The server saves all video data at the end of the game and runs an AI highlight generation engine to extract important scenes. A customized highlight video is generated based on Suzuki-san's preferences and viewing history and sent to his device. Suzuki-san can easily enjoy highlights including his favorite scenes.

[0072] The above is an embodiment of the present invention. The present invention allows users to enjoy watching sports while feeling a sense of unity with other fans, and also allows users to gain a deeper understanding of sports through detailed commentary and customized highlights.

[0073] The processing flow will be explained below.

[0074] Processing steps for live streaming linked to social media in real time

[0075] Step 1:

[0076] A user uses a terminal to select a sporting event that they wish to watch.

[0077] Step 2:

[0078] The device sends a request for a live stream to the server.

[0079] Step 3:

[0080] The server receives a request for a live stream and retrieves the corresponding stream.

[0081] Step 4:

[0082] The server distributes the acquired live stream to the terminal, allowing the user to view it.

[0083] Step 5:

[0084] The server collects social media posts in real time using keywords and hashtags related to the specified event.

[0085] Step 6:

[0086] The server filters the collected posts and selects the most relevant posts.

[0087] Step 7:

[0088] The server analyzes the collected posts and identifies peak times of excitement and emotion.

[0089] Step 8:

[0090] The server generates data for notifying the user of the optimal cheering timing based on the identified peak time.

[0091] Step 9:

[0092] The server transmits the generated notification data to the terminal.

[0093] Step 10:

[0094] The notification received by the terminal is displayed on the screen to notify the user.

[0095] AI real-time commentary processing steps

[0096] Step 1:

[0097] The server receives the live stream video data in real time.

[0098] Step 2:

[0099] The server runs an AI commentary assistant and analyzes the video data.

[0100] Step 3:

[0101] The AI ​​commentary assistant recognizes the progress of the game, player movements, scoring scenes, etc.

[0102] Step 4:

[0103] The AI ​​commentary assistant generates explanatory text based on the analysis results.

[0104] Step 5:

[0105] The server transmits the generated explanatory text to the terminal in real time.

[0106] Step 6:

[0107] The explanatory text received by the terminal is displayed on the screen together with the live stream video and provided to the user.

[0108] Processing steps for automatic highlight generation

[0109] Step 1:

[0110] After the match ends, the server stores all the match video data.

[0111] Step 2:

[0112] The server inputs the saved video data into the AI ​​highlight generation engine.

[0113] Step 3:

[0114] The AI ​​highlight generation engine automatically extracts important scenes from the game (scores, fouls, good plays, etc.).

[0115] Step 4:

[0116] The user uses the device to set their preferences and interests.

[0117] Step 5:

[0118] The server refers to the user's setting information and past viewing history and determines a policy for generating a customized highlight video.

[0119] Step 6:

[0120] The server uses an AI highlight generation engine to generate a customized highlight video.

[0121] Step 7:

[0122] The server transmits the generated customized highlight video to the user's terminal.

[0123] Step 8:

[0124] The highlight video received by the terminal is provided to the user in a reproducible form.

[0125] Example 1

[0126] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0127] Conventional sports viewing systems have struggled to provide real-time commentary, improve the viewing experience, and share the excitement and emotion of users. Furthermore, it has been difficult to customize the post-match highlights based on individual user interests and viewing history. This has limited the enjoyment of sports viewing and prevented the user experience from being maximized.

[0128] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0129] In this invention, the server includes means for a user to select an event they wish to watch, means for sending a live stream request to the server, means for the server to collect social media posts related to the designated event in real time, means for analyzing the collected posts to identify peak times of excitement and emotion, means for notifying the user of the optimal cheering timing based on the identified peak times, means for providing the live stream and a notification of the cheering timing to the user's terminal, means for analyzing the social media posts using a generative AI model, means for notifying the user based on the peak times identified by the generative AI model, and a commentary assistant for the server to receive the live stream video data and analyze the progress of the match. The system includes a means for operating a game server, a means for transmitting commentary text generated by a commentary assistant to a user's device in real time, a means for displaying the commentary text on the user's device, a means for using a generative AI model for generating commentary text in real time, a means for a server to save all game video data after the game ends, a means for the server to operate a highlight generation engine and automatically extract important scenes, a means for generating a customized highlight video based on a user's settings and viewing history, a means for transmitting the generated highlight video to the user's device, a means for customizing and generating the highlight video using a generative AI model that uses the user's viewing history, and a means for the user's device to provide the transmitted highlight video in a playable form. This enables users to watch a game with in-depth commentary while sharing the excitement and emotion with other fans in real time, and to enjoy a highlight video that suits their interests after the game ends.

[0130] The "System" is an integrated mechanism that provides users with the optimal viewing experience by broadcasting the sporting events they want to watch in real time and collecting and analyzing related social media information.

[0131] "User" means an individual who uses the System to watch sporting events and receive services such as live streams, commentary, and highlight videos.

[0132] A "server" is a computer device that is the core of the system, and is responsible for processing requests from users, distributing live streams, analyzing data, generating highlights, and so on.

[0133] "Terminal" means a device used by a user to access the system and watch a sporting event, and includes a smartphone, tablet, PC, etc.

[0134] A "live stream" is video data of sporting events and performances broadcast in real time over the Internet.

[0135] "Social media" refers to a platform on the Internet where users can share and interact with each other, and includes Twitter, Facebook, Instagram, etc.

[0136] A "generative AI model" is a machine learning algorithm that is trained using artificial intelligence techniques to perform a specific task (e.g., text generation, sentiment analysis, etc.).

[0137] The "Commentary Assistant" is an artificial intelligence system that runs on a server, analyzes live-stream video data, and generates commentary text in real time.

[0138] The "highlight generation engine" is a software component that analyzes video data from matches, automatically extracts and edits important scenes, and generates highlight videos.

[0139] "Viewing history" is a record of sporting events that a user has viewed in the past and their contents.

[0140] The present invention provides a system that integrates live streaming, social media, real-time commentary using generative AI models, and automatic highlight generation to enhance users' sporting event viewing experience.

[0141] First, a user accesses the system using a terminal and selects the sporting event they want to watch. At this time, the terminal sends the user's selection information to the server and requests a live stream. The server receives the request and provides live footage of the specified sporting event.

[0142] The server then collects social media posts in real time using specific hashtags or keywords related to the specified event. These posts are retrieved using social media APIs (e.g., Twitter API, Facebook API). The collected posts are then analyzed using a generative AI model to identify peak times of excitement and emotion.

[0143] The server notifies users of the best time to cheer based on the identified peak times. The notification is sent via a pop-up message or notification bar on the user's device, allowing users to send cheering messages and share their excitement at the best possible time.

[0144] The server also receives live-stream video data in real time and runs a commentary assistant to analyze the progress of the match. The commentary assistant analyzes the player movements, goals scored, tactics, and other aspects of the video, and generates commentary text in real time. The generated commentary text is sent to the user's device and overlaid on the live video.

[0145] After the game ends, the server saves all game video data and runs a highlight generation engine. The highlight generation engine automatically extracts and edits important scenes, such as goals and great plays, to generate a customized highlight video. The customization is based on the user's settings and viewing history, and the generated highlight video is sent to the user's device.

[0146] As a concrete example, consider a user named "Yamada" using a device to watch a basketball game. The server receives a request from Yamada's device and delivers the live stream while simultaneously collecting and analyzing related social media posts. The server identifies the moments when excitement levels rise and notifies Yamada, allowing him to enjoy cheering without missing the most exciting moments.

[0147] In addition, when a user named "Sasaki" is watching a soccer match, the server receives the live-stream video data and uses the generative AI model to generate and transmit commentary text in real time, allowing Sasaki to watch the match while gaining a detailed understanding of the progress of the match.

[0148] Furthermore, if user "Suzuki-san" wants to watch highlights after a baseball game, the server saves all video data at the end of the game and activates the AI ​​highlight generation engine. A customized highlight video is generated and sent based on Suzuki-san's preferences and viewing history. Suzuki-san can easily watch highlight videos containing important scenes.

[0149] Example prompt sentence:

[0150] "How can Yamada know the optimal timing to cheer while watching a basketball game in real time?"

[0151] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0152] Step 1:

[0153] A user accesses the system using a terminal and selects the sporting event they wish to watch.

[0154] Input: User event selection information

[0155] Specific operation: The user selects the menu or list on the device screen, selects the event they want to watch, and confirms the selection by pressing the Enter key or touching the screen.

[0156] Output: Event selection information is recorded on the terminal and sent to the server.

[0157] Step 2:

[0158] The terminal transmits the user's selection information to the server.

[0159] Input: User event selection information

[0160] What it does: Your device sends the selected information to a server over the internet, using a secure protocol such as HTTPS to protect the data.

[0161] Output: The selection information is received by the server.

[0162] Step 3:

[0163] The server processes the live stream request based on the received selection information and provides the live video to the user.

[0164] Input: Selection information

[0165] Specific operation: The server accesses the streaming service, acquires live video of the specified event, and streams the acquired video data to the user's device.

[0166] Output: The live stream video is displayed on the user's device.

[0167] Step 4:

[0168] The server collects social media posts related to a specified event in real time.

[0169] Input: Event hashtags or keywords

[0170] How it works: The server uses social media APIs to collect posts related to the specified hashtags and keywords. The collection process is done in real time.

[0171] Output: Collected social media post data

[0172] Step 5:

[0173] The server analyzes the collected posts using a generative AI model to identify peak times of excitement and emotion.

[0174] Input: Collected submission data

[0175] How it works: Using a generative AI model, post data is analyzed and scored for excitement and emotional impact. Peak times are identified based on the scoring results.

[0176] Output: Identified peak times of excitement and emotion

[0177] Step 6:

[0178] The server notifies the user of the optimal cheering timing based on the identified peak time.

[0179] Input: Identified peak times

[0180] Specific operation: The server sends a notification message to the user's device, causing a pop-up message or notification bar to appear on the user's screen.

[0181] Output: Notification of the cheering timing displayed on the device screen

[0182] Step 7:

[0183] The server receives the live stream video data in real time, runs the commentary assistant, and analyzes the progress of the match.

[0184] Input: Live stream video data

[0185] Specific operation: The server analyzes the video data using an AI commentary assistant (e.g., image recognition model and natural language processing model).

[0186] Output: Real-time generated explanatory text

[0187] Step 8:

[0188] The server transmits the generated commentary text to the user's terminal in real time, and the commentary text is displayed on the user's terminal.

[0189] Input: Generated description text

[0190] Specific operation: The server sends explanatory text to the user's device, which then overlays the received explanatory text on the live video.

[0191] Output: Descriptive text that is displayed on the device screen

[0192] Step 9:

[0193] After the match ends, the server stores all match video data and runs a highlight generation engine to automatically extract important scenes.

[0194] Input: Match video data

[0195] How it works: The server stores the video data in a database, then uses a highlight generation engine to automatically extract and edit important scenes (scoring scenes, great plays, etc.).

[0196] Output: Clips of extracted key scenes

[0197] Step 10:

[0198] The server generates a customized highlight video based on the user's settings and viewing history and transmits it to the user's terminal.

[0199] Input: User settings and viewing history, extracted clips of important scenes

[0200] Specific operation: The server references the user's settings and viewing history, combines clips of key scenes, and generates a customized highlight video. The generated video is then sent to the user's device.

[0201] Output: A customized highlight video sent to the user's device

[0202] Step 11:

[0203] To provide a highlight video received by a user terminal in a reproducible form.

[0204] Input: Custom highlight video

[0205] Specific operation: The device uses an application (e.g., a video player) to play the received highlight video and displays it in a playable format for the user.

[0206] Output: A highlight video that can be viewed by the user

[0207] (Application example 1)

[0208] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0209] Conventional sports viewing systems lack real-time commentary, highlight generation, and notification functions that allow viewers to cheer on the most exciting moments. This makes it difficult for users to gain a deeper understanding of the content and enjoy it without missing any exciting moments. Furthermore, they do not provide customized highlight videos tailored to individual users' preferences. Therefore, a system that integrates these functions is needed to improve the user experience.

[0210] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0211] In this invention, the server includes means for a user to select content they wish to view, means for sending a streaming request to the server, means for the server to collect online posts related to the specified content in real time, means for analyzing the collected posts to identify peak times of excitement and emotion, means for notifying the user of the optimal timing for cheering based on the identified peak times, and means for providing streaming and notification of the timing for cheering to the user's information terminal. This allows the user to know the timing for cheering in real time, watch the game with a sense of unity with other viewers, and enjoy customized highlights that suit their interests.

[0212] "Means for a user to select content they wish to view" refers to the interface or process used to select the particular content a user wishes to view.

[0213] The "means for sending a streaming request to the server" refers to a mechanism for requesting the server to stream content based on the content selected by the user.

[0214] "Means for the server to collect Internet posts related to specified content in real time" refers to the function of the server to obtain posts and comments related to specific content on the Internet in real time.

[0215] "Means of analyzing collected posts to identify peak times of excitement and emotion" refers to the process of analyzing collected internet posts to identify the times when users are most excited or moved.

[0216] "Means for notifying users of the optimal timing to cheer based on identified peak times" refers to a system that notifies users of the optimal timing to cheer based on peak times of excitement and emotion identified through analysis.

[0217] "Means for providing streaming and cheering timing notifications to a user's information terminal" refers to a function for delivering streaming data and cheering timing information to a device used by a user.

[0218] "Commentary assistant for analyzing content progress" refers to AI or software that analyzes the progress of content being streamed in real time and generates commentary based on that content.

[0219] "Means for transmitting explanatory text generated by the explanatory assistant in real time to the user's information terminal" refers to a process for instantly transmitting explanatory text generated by the explanatory assistant to the user's device.

[0220] "Means for displaying explanatory text on the user's information terminal" refers to a mechanism for displaying explanatory text on the screen of the user's device.

[0221] "Means for the server to save all video data after the end of the match" refers to the function of storing all streamed video data on the server after the end of the match.

[0222] "Means for the server to operate the highlight generation engine and automatically extract important scenes" refers to a system that uses the highlight generation engine in the server to automatically select important scenes from video data.

[0223] The "means for generating a customized highlight video based on user settings and viewing history" refers to a function for generating an individually customized highlight video based on preferences set by the user and past viewing history.

[0224] "Means for transmitting the generated highlight video to the user's information terminal" refers to a process for transmitting the customized highlight video to the user's device.

[0225] The "means for providing the transmitted highlight video in a format that can be played back by the user's information terminal" refers to a system that provides the transmitted highlight video in a format that can be played back by the user's device.

[0226] This invention is a system for improving a user's sports viewing experience, utilizing a server, a user's information terminal, and AI technology. This system is implemented in the following steps.

[0227] First, a user selects the content they want to watch using an information terminal such as a smartphone or tablet. The selection information is sent from the terminal to the server, and the server receives a streaming request based on that information. The server then begins streaming the specified content.

[0228] The server then collects real-time internet posts related to the specified content, including searches using specific hashtags or keywords, and analyzes these posts using natural language processing techniques to identify peak times of excitement and emotion.

[0229] Once the data collection and analysis is complete, the server notifies users of the optimal time to cheer based on the identified peak times. The notification is sent to the user's information terminal and displayed on the screen in real time.

[0230] The server also runs an AI commentary assistant to analyze the streaming video. The AI ​​commentary assistant analyzes the progress of the game in real time and generates commentary text about players' movements, scoring scenes, etc. The generated commentary text is sent in real time to the user's information terminal and displayed on the screen.

[0231] After the game ends, the server saves all video data and runs a highlight generation engine. The highlight generation engine automatically extracts important scenes and generates a customized highlight video based on the user's settings and viewing history. This highlight video is sent to the user's information terminal, allowing the user to watch the highlights at any time.

[0232] The hardware and software used include the following: Hardware includes information terminals such as smartphones and tablets, and servers. Software includes TensorFlow (AI model), Flask / Django (server-side framework), OpenCV (video data analysis), Requests (HTTP request processing), and NLTK (natural language processing) used on the server side.

[0233] As a concrete example, consider the case where a user named "Suzuki" wants to watch a basketball game. Suzuki uses his smartphone to select the game he wants to watch and sends a request to the server. The server receives the request and provides live streaming, while also collecting and analyzing related posts from social media to identify peak times of excitement. The server notifies Suzuki of these peak times and also provides commentary text generated by an AI commentary assistant in real time. After the game ends, the server extracts key scenes and generates and sends customized highlights tailored to Suzuki's preferences.

[0234] An example of a prompt sentence might be:

[0235] "While watching a basketball game in real time, generate a code to identify and notify you of peak moments of excitement from social media."

[0236] In this way, the present invention can significantly improve the sports viewing experience for users.

[0237] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0238] Step 1:

[0239] The user selects the content they wish to view using an information terminal. The input is the user's viewing preference, and the output is the selected content information. The interface used by the user is intuitive, and selection can be easily made using lists and search functions.

[0240] Step 2:

[0241] A streaming request is sent from the information terminal to the server. The input is the selected content information, and the output is a confirmation of the request from the server. This information sent as an HTTP request triggers the server to start streaming.

[0242] Step 3:

[0243] The server starts streaming the specified content. The input is the requested content information, and the output is live streaming data. The server encodes the video data in real time and sends it to the user's information terminal.

[0244] Step 4:

[0245] The server collects internet posts related to the specified content in real time. The input is a specific hashtag or keyword, and the output is the post data. The server uses APIs to retrieve related posts from social media and stores them in a database.

[0246] Step 5:

[0247] The server analyzes the collected posts to identify peak times of excitement and emotion. The input is the collected post data, and the identified peak times are extracted as the output. Natural language processing technology (e.g., NLTK) is used to analyze the text data and evaluate emotions and excitement levels.

[0248] Step 6:

[0249] The server notifies the user of the optimal cheering timing based on the identified peak times. The input is the identified peak time information, and the output is generated notification data. The server generates real-time notifications and sends them to the user's information terminal.

[0250] Step 7:

[0251] The server runs an AI commentary assistant to analyze streaming video. The input is real-time video data, and the output is explanatory text. The TensorFlow model is used to analyze the video data and generate explanatory text according to the progress.

[0252] Step 8:

[0253] The server generates explanatory text and sends it to the user's information terminal in real time. The input is the generated explanatory text, and the output is a confirmation of transmission. The explanatory text is distributed along with the video data.

[0254] Step 9:

[0255] The commentary text is displayed on the user's information terminal. The input is the received commentary text, and the output is the displayed text. It is displayed overlaid on the screen of the user's information terminal to aid viewing.

[0256] Step 10:

[0257] After the game ends, the server saves all video data. The input is all video data from the game, and the output is a saved data file. The server saves the data in cloud storage such as S3.

[0258] Step 11:

[0259] The server runs a highlight generation engine to automatically extract important scenes. The input is stored video data, and the output is extracted highlight scenes. OpenCV is used to perform video analysis and extract scenes based on specific events or actions.

[0260] Step 12:

[0261] The server generates a customized highlight video based on the user's settings and viewing history. The input is the user's settings and viewing history, and the output is a customized highlight video. An AI model is used to analyze the user's preferences and combine the optimal scenes.

[0262] Step 13:

[0263] The server sends the generated highlight video to the user's information terminal. The input is the customized highlight video, and the output is a confirmation of transmission. The video data is provided to the user in a format that is easy to view.

[0264] Step 14:

[0265] The user's information terminal provides the transmitted highlight video in a playable format. The input is the received highlight video, and the output is a played video. The user can enjoy the highlight video containing their favorite scenes.

[0266] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0267] This invention is a system that integrates live streaming with social media, AI-powered real-time commentary, automatic highlight generation, and an emotion engine that recognizes user emotions to enhance users' sporting event viewing experience.

[0268] A way for users to select the event they want to watch and submit a request for a live stream

[0269] Users access the system using their terminals and select the sporting event they want to watch. The terminals then send the selection information to the server, requesting a live stream. As a result, users can watch their desired sporting event in real time.

[0270] A means by which the server collects social media posts related to a specified event in real time.

[0271] The server collects social media posts in real time using specific hashtags and keywords related to the designated event, and updates the collected posts as the event progresses and provides them to users.

[0272] A method for identifying peak times of excitement and emotion by analyzing collected posts

[0273] The server analyzes the collected social media posts and uses natural language processing technology to identify peak times of excitement and emotion, allowing it to understand in real time at which points during a game users are most excited.

[0274] A means of notifying users of the optimal timing for cheering based on identified peak times

[0275] The server then notifies the user of the optimal time to cheer based on the identified peak times. The notification is sent via the device and displayed on the screen, allowing the user to cheer while feeling a sense of unity with other fans.

[0276] A means for the server to receive the live stream video data and run a commentary assistant to analyze the progress of the match

[0277] The server receives live-stream video data in real time and runs an AI commentary assistant, which analyzes the progress of the game, understands player movements, goals scored, tactics, etc., and generates commentary text.

[0278] A means for transmitting the explanatory text generated by the explanatory assistant to the user's terminal in real time and displaying the explanatory text on the user's terminal.

[0279] The server transmits the generated commentary text to the user's device in real time, and the user's device displays the received commentary text overlaid on the live stream video, providing real-time commentary information to the user while watching the game.

[0280] After the match, the server saves all the match video data and runs a highlight generation engine to automatically extract important scenes.

[0281] After the game ends, the server stores all game video data, then runs a highlight generation engine to automatically extract important scenes (scores, fouls, good plays, etc.).

[0282] A means to generate customized highlight videos based on user preferences and viewing history

[0283] Users set their preferences and interests through their devices. The server references the user's settings and viewing history to generate a customized highlight video. The generated highlight video is then edited to suit the user's individual preferences.

[0284] A means for transmitting the generated highlight video to a user's terminal, and for the user's terminal to provide the transmitted highlight video in a reproducible form.

[0285] The server transmits the generated customized highlight video to the user's device, which then displays the received highlight video in a playable format, allowing the user to easily review and watch it.

[0286] Processing by emotion engine that recognizes user emotions

[0287] The user's device is equipped with a camera and microphone, which are used to collect the user's emotional data in real time. The emotion engine reads the user's facial expressions using the device's camera and analyzes the tone and strength of the voice using the microphone. The emotion engine then sends the collected emotional data to the server.

[0288] Server analyzes emotional data and adjusts cheering timing

[0289] The server analyzes the emotion data sent from the emotion engine and evaluates the user's level of excitement and emotion in real time. Based on this evaluation, the server further optimizes the timing of cheering notifications and supports the user's cheering.

[0290] Specific examples

[0291] Example 1: Real-time cheering and emotion recognition

[0292] A user named "Yamada" uses a device to watch a basketball game. The server receives a request from Yamada's device and broadcasts the live stream while simultaneously collecting related social media posts. The server analyzes the posts, identifies moments of high excitement, and notifies Yamada of those moments. Furthermore, the emotion engine analyzes Yamada's facial expressions and voice to provide more accurate cheering timing.

[0293] Example 2: AI real-time commentary and emotion recognition

[0294] A user named "Sasaki" is watching a soccer match. The server receives the live-stream video data, and the AI ​​commentary assistant analyzes the progress of the match and generates commentary text in real time. The commentary text is sent to Sasaki's device and displayed on the screen. Furthermore, the emotion engine analyzes Sasaki's emotions and provides additional commentary based on the moments of excitement.

[0295] Example 3: Automatic highlight generation and emotion recognition

[0296] After a baseball game, user "Suzuki-san" wants to watch highlights of games he has previously watched. The server saves all video data at the end of the game and runs an AI highlight generation engine to extract important scenes. A customized highlight video is generated based on Suzuki-san's preferences and viewing history and sent to his device. The emotion engine analyzes Suzuki-san's emotions while watching the highlights and further emphasizes scenes that are exciting.

[0297] The above is an embodiment of the present invention. The present invention allows users to enjoy watching sports while feeling a sense of unity with other fans, and also allows for a deeper understanding of sports through detailed commentary and customized highlights. Furthermore, by analyzing users' real-time emotions, a more personalized cheering and viewing experience can be realized.

[0298] The processing flow will be explained below.

[0299] Processing steps for live streaming linked to social media in real time

[0300] Step 1:

[0301] A user uses a terminal to select a sporting event that they wish to watch.

[0302] Step 2:

[0303] The device sends a request for a live stream to the server.

[0304] Step 3:

[0305] The server receives a request for a live stream and retrieves the corresponding stream.

[0306] Step 4:

[0307] The server distributes the acquired live stream to the terminal, allowing the user to view it.

[0308] Step 5:

[0309] The server collects social media posts in real time using keywords and hashtags related to the specified event.

[0310] Step 6:

[0311] The server filters the collected posts and selects the most relevant posts.

[0312] Step 7:

[0313] The server analyzes the collected posts and identifies peak times of excitement and emotion.

[0314] Step 8:

[0315] The server generates data for notifying the user of the optimal cheering timing based on the identified peak time.

[0316] Step 9:

[0317] The server transmits the generated notification data to the terminal.

[0318] Step 10:

[0319] The notification received by the terminal is displayed on the screen to notify the user.

[0320] AI real-time commentary processing steps

[0321] Step 1:

[0322] The server receives the live stream video data in real time.

[0323] Step 2:

[0324] The server runs an AI commentary assistant and analyzes the video data.

[0325] Step 3:

[0326] The AI ​​commentary assistant recognizes the progress of the game, player movements, scoring scenes, etc.

[0327] Step 4:

[0328] The AI ​​commentary assistant generates explanatory text based on the analysis results.

[0329] Step 5:

[0330] The server transmits the generated explanatory text to the terminal in real time.

[0331] Step 6:

[0332] The explanatory text received by the terminal is displayed on the screen together with the live stream video and provided to the user.

[0333] Processing steps for automatic highlight generation

[0334] Step 1:

[0335] After the match ends, the server stores all the match video data.

[0336] Step 2:

[0337] The server inputs the saved video data into the AI ​​highlight generation engine.

[0338] Step 3:

[0339] The AI ​​highlight generation engine automatically extracts important scenes from the game (scores, fouls, good plays, etc.).

[0340] Step 4:

[0341] The user uses the device to set their preferences and interests.

[0342] Step 5:

[0343] The server refers to the user's setting information and past viewing history and determines a policy for generating a customized highlight video.

[0344] Step 6:

[0345] The server uses an AI highlight generation engine to generate a customized highlight video.

[0346] Step 7:

[0347] The server transmits the generated customized highlight video to the user's terminal.

[0348] Step 8:

[0349] The highlight video received by the terminal is provided to the user in a reproducible form.

[0350] Processing steps combining emotion engines

[0351] Step 1:

[0352] When a user uses a device to watch an event, the device's camera and microphone are activated.

[0353] Step 2:

[0354] The device uses a camera to recognize the user's facial expressions and a microphone to analyze the tone and strength of the voice.

[0355] Step 3:

[0356] The emotion data acquired by the device is sent to the server in real time.

[0357] Step 4:

[0358] The server receives the emotion data and analyzes it.

[0359] Step 5:

[0360] The server evaluates the user's excitement level and changes in emotions.

[0361] Step 6:

[0362] The server adjusts the timing of cheering notifications based on the results of emotion data analysis.

[0363] Step 7:

[0364] The server transmits the adjusted cheering timing to the terminal.

[0365] Step 8:

[0366] The device displays a notification of the adjusted cheering timing to the user.

[0367] Specific examples

[0368] Example 1: Real-time cheering and emotion recognition

[0369] Step 1:

[0370] A user "Yamada" uses the terminal to watch a basketball game.

[0371] Step 2:

[0372] The device sends a request for a live stream to the server, and the server delivers the stream.

[0373] Step 3:

[0374] The server collects and analyzes social media posts to identify peak times of excitement.

[0375] Step 4:

[0376] The device's camera and microphone capture Yamada's emotions in real time.

[0377] Step 5:

[0378] The emotion data is sent to a server and analyzed.

[0379] Step 6:

[0380] Based on the analysis results, the server will notify Yamada of the optimal timing to cheer him on.

[0381] Step 7:

[0382] The device will display the optimal timing for cheering to Yamada, encouraging him to cheer in real time.

[0383] Example 2: AI real-time commentary and emotion recognition

[0384] Step 1:

[0385] User "Sasaki-san" starts watching a soccer game.

[0386] Step 2:

[0387] The server receives the live stream footage and the AI ​​commentary assistant begins analyzing it.

[0388] Step 3:

[0389] The explanatory text generated by the commentary assistant is sent to Sasaki's device in real time.

[0390] Step 4:

[0391] The device's camera and microphone capture Sasaki's emotional data.

[0392] Step 5:

[0393] The emotion data is sent to a server and analyzed.

[0394] Step 6:

[0395] The server provides additional commentary and highlights based on the results of Sasaki's sentiment analysis.

[0396] Step 7:

[0397] The device displays additional commentary information to Sasaki, helping him better understand the match.

[0398] Example 3: Automatic highlight generation and emotion recognition

[0399] Step 1:

[0400] After the baseball game ends, user "Suzuki" wants to watch the highlights.

[0401] Step 2:

[0402] The server stores all video data from the match and runs the AI ​​highlight generation engine.

[0403] Step 3:

[0404] After extracting important scenes, Suzuki's device sends his preferences and viewing history to the server.

[0405] Step 4:

[0406] The server generates a customized highlight video and transmits it to the device.

[0407] Step 5:

[0408] The device's camera and microphone capture Suzuki's emotional data.

[0409] Step 6:

[0410] The emotion data is sent to a server and analyzed.

[0411] Step 7:

[0412] The server provides emotional feedback according to the scene of interest, further emphasizing the highlights.

[0413] Step 8:

[0414] The device will provide Suzuki with the adjusted highlight video in a playable format.

[0415] Example 2

[0416] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0417] Conventional sports viewing systems often only provide simple live streaming, resulting in a one-way viewing experience for users. They also lack real-time information and commentary, and lack advanced features, particularly those utilizing social media and AI technology, preventing services from responding to users' emotions and excitement. Furthermore, highlight videos generated after the game are often generated manually, making it difficult to customize them based on individual user preferences and viewing history.

[0418] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0419] In this invention, the server includes: a means for a user to select an event they wish to watch; a means for sending a live stream request to the server; a means for the server to collect social media posts related to the specified event in real time; a means for analyzing the collected posts to identify peak times of excitement and emotion; a means for notifying the user of the optimal cheering timing based on the identified peak times; a means for providing the live stream and a notification of the cheering timing to the user's device; a means for recognizing the user's emotions in real time; and a means for analyzing the user's emotion data and adjusting the cheering timing. This allows users to enjoy a real-time, interactive viewing experience, and enables the provision of advanced information and commentary utilizing social media and AI technologies. Furthermore, highlights generated after the game are automated, allowing for customization based on the user's preferences and interests.

[0420] The "means for a user to select an event that the user wishes to watch" refers to an interface and functionality that allows a user to use a terminal to search for and select a sporting event that the user wishes to watch.

[0421] "Means for sending a request for a live stream to a server" refers to the communications capabilities and protocols for sending a request for a live stream to a server based on a user-selected event.

[0422] "Means by which the server collects social media posts related to a specified event in real time" refers to algorithms and data collection systems for collecting relevant posts from social media using specific hashtags or keywords.

[0423] "Means for analyzing collected posts to identify peak times of excitement or emotion" refers to technology and processing for analyzing social media posts using natural language processing technology and identifying peak times of excitement or emotion among users.

[0424] "Means for notifying users of the optimal timing for cheering based on identified peak times" refers to a communication function and notification system for notifying users of cheering messages at the optimal timing based on peak times of excitement and emotion.

[0425] "Means for providing a live stream and cheering timing notifications to a user's terminal" refers to a distribution system and application function for providing a live stream video and cheering timing notifications to a user's terminal.

[0426] "Means for recognizing a user's emotions in real time" refers to the technology and functions for collecting a user's facial expressions and voice in real time using the device's camera and microphone, and recognizing their emotions.

[0427] "Means for analyzing user emotional data and adjusting the timing of cheering" refers to an algorithm and processing system for analyzing collected user emotional data and optimizing the timing of cheering based on the analysis results.

[0428] "Means for receiving live-stream video data and operating a commentary assistant to analyze the progress of the match" refers to the function of receiving live-stream video data in real time and operating an AI commentary assistant that analyzes the progress of the match based on that data.

[0429] "Means for transmitting the explanatory text generated by the explanation assistant to the user's device in real time" refers to the communication function and data transmission system for transmitting the explanatory text generated by the AI ​​explanation assistant to the user's device in real time.

[0430] "Means for the server to store all game video data after the game ends" refers to a data storage and management system for storing all game video data on the server after the game ends.

[0431] "Means for operating a highlight generation engine and automatically extracting important scenes" refers to a technology and processing system for operating a highlight generation engine and automatically extracting important scenes from a match.

[0432] The "means for generating a customized highlight video based on user settings and viewing history" refers to algorithms and processing functions for generating a customized highlight video based on user preferences and viewing history.

[0433] The "means for transmitting the generated highlight video to the user's terminal" refers to a communication function and a data transmission system for transmitting the generated customized highlight video to the user's terminal.

[0434] The present invention provides a system that allows users to watch sporting events in real time and enhance their viewing experience. The system includes three main components: a user, a terminal, and a server. Each component plays a specific role and works together to enhance the user experience.

[0435] System Configuration

[0436] User terminal

[0437] Users use devices such as smartphones, tablets, and PCs (hereafter referred to as "terminals"). Terminals include an internet connection, camera, microphone, and display. Terminals access the system through a browser or dedicated application.

[0438] server

[0439] The server is located on the cloud infrastructure and is responsible for collecting, analyzing, distributing and storing data. The server has the following main functions:

[0440] Live Streaming

[0441] Social media data collection and analysis

[0442] AI commentary assistant

[0443] Highlight Generation Engine

[0444] Emotion Recognition Engine

[0445] Main processing

[0446] Request and deliver a live stream

[0447] The user selects the event they want to watch and sends a live stream request to the server via their device. The server receives the request and delivers the live stream of the event. The live stream data is sent to the user's device in real time and displayed on the screen.

[0448] Social media data collection and analysis

[0449] The server collects social media posts in real time using hashtags and keywords related to a specified event, then analyzes the posts using natural language processing (NLP) techniques to identify peak times of user excitement and emotion.

[0450] Cheering timing notification

[0451] Based on the analysis results, the server calculates the optimal cheering timing according to the identified peak times and notifies the user's device, allowing users to feel a sense of unity with other fans and cheer more enthusiastically.

[0452] AI commentary assistant

[0453] The server receives live-stream video data in real time and runs an AI commentary assistant. The AI ​​commentary assistant analyzes the progress of the game, players' movements, scoring scenes, etc., and generates commentary text. This text is sent to the device in real time and displayed overlaid on the video.

[0454] Highlight Generation

[0455] After the game ends, the server stores all game video data and runs a highlight generation engine, which generates a customized highlight video based on the user's settings and viewing history, and sends it to the device for playback.

[0456] Emotion recognition and cheer timing adjustment

[0457] The user's device is equipped with a camera and microphone, which are used to collect the user's emotional data in real time. The emotion engine analyzes facial expressions and tone of voice and sends the results to the server. The server uses this data to calculate the optimal timing for cheering, further improving the user's cheering experience.

[0458] Specific examples

[0459] Example 1: Real-time cheering and emotion recognition

[0460] User "A" uses a device to watch a basketball game. The server receives a request from A's device and delivers the live stream while simultaneously collecting related social media posts. The server analyzes the posts, identifies moments of high excitement, and notifies A of those moments. Furthermore, the emotion engine analyzes A's facial expressions and voice to provide more accurate cheering timing.

[0461] Example 2: AI real-time commentary and emotion recognition

[0462] User "B" is watching a soccer match. The server receives the live-stream video data, and the AI ​​commentary assistant analyzes the progress of the match and generates commentary text. The text is sent to B's device and displayed on the screen. Furthermore, the emotion engine analyzes B's emotions and provides additional commentary at times when B is excited.

[0463] Example 3: Automatic highlight generation and emotion recognition

[0464] After a baseball game, user "C" wants to watch highlights of games he previously watched. The server saves all video data at the end of the game and runs an AI highlight generation engine to extract important scenes. A customized highlight video based on C's preferences and viewing history is sent to C's device. The emotion engine analyzes C's emotions while watching the highlights and further emphasizes scenes that are exciting.

[0465] Prompt Sentence Examples

[0466] Imagine you are watching a basketball game. Describe a system in which an AI commentary assistant generates commentary in real time and notifies the user of the optimal timing to cheer based on social media data. It also analyzes user sentiment in real time and uses that information to optimize cheering and commentary.

[0467] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0468] Processing Step Description

[0469] Step 1:

[0470] Users select the event they want to watch

[0471] Input: The user accesses the device's interface, searches for and selects the sporting event they wish to watch.

[0472] Data processing / calculation: The terminal formats the user's selected information and prepares it as request data for the server.

[0473] Output: The terminal sends live stream request data to the server.

[0474] Specific operation: The user selects "basketball game" on the interface, and the device sends that information to the server as a "live stream request."

[0475] Step 2:

[0476] The server receives a request for a live stream and starts broadcasting it.

[0477] Input: The server receives live stream request data from the terminal.

[0478] Data processing / calculation: The server analyzes the request content and obtains the distribution URL and data of the corresponding live stream.

[0479] Output: The server sends the live stream data to the user's device.

[0480] Specific operation: The server receives the request, generates a live streaming URL, and starts sending the stream data to the device.

[0481] Step 3:

[0482] The server collects and analyzes relevant posts from social media.

[0483] Input: The server receives as input hashtags or keywords related to the specified event (e.g., "basketball game" or "score").

[0484] Data processing / calculation: Based on these hashtags and keywords, the server collects data from social media in real time and analyzes it using natural language processing (NLP) technology.

[0485] Output: The analyzed social media data is accumulated and peak time information on excitement and emotion is generated.

[0486] Specific operation: The server analyzes collected social media posts and assigns them a rating score based on excitement and emotion.

[0487] Step 4:

[0488] The server identifies and notifies you of peak times

[0489] Input: Social media data analyzed by NLP techniques.

[0490] Data processing / calculation: The server uses an algorithm to identify peak times based on the analysis results and calculate the optimal timing for cheering.

[0491] Output: The identified cheering timing notification data is generated and sent to the user's terminal.

[0492] Specific operation: The server notifies the device of the timing to cheer based on a score such as "excitement level 80%."

[0493] Step 5:

[0494] The server receives the live stream video and runs the AI ​​commentary assistant.

[0495] Input: Live stream video data.

[0496] Data processing / calculation: The AI ​​commentary assistant analyzes the video data and generates commentary text about the progress of the match and specific scenes.

[0497] Output: The generated explanatory text data is sent to the user's terminal.

[0498] Specific operation: The AI ​​generates explanatory text such as "Player A scored a goal!" and sends it to the device.

[0499] Step 6:

[0500] Recognize user emotions in real time and collect data

[0501] Input: User's facial expression data and voice data obtained from the device's camera and microphone.

[0502] Data processing / calculation: The emotion recognition engine analyzes facial expressions and voice data to identify emotional states (e.g., "excited" or "relaxed").

[0503] Output: The analyzed emotion data is sent to the server.

[0504] Specific operation: The camera recognizes the user's facial expressions such as "smile" and "surprise," and the microphone analyzes the "tone" and "pitch" of the voice.

[0505] Step 7:

[0506] The server analyzes the emotional data and adjusts the timing of cheering.

[0507] Input: User emotion data sent from the device.

[0508] Data processing / calculation: The server uses an algorithm to dynamically adjust the timing of cheering based on emotional data.

[0509] Output: The re-adjusted cheer timing notification data is sent to the user's device.

[0510] Specific operation: Based on the "excitement level 90%", the server further optimizes the cheering message and notifies the user.

[0511] Step 8:

[0512] The server stores the match video data, runs the highlight generation engine, and sends it to the user.

[0513] Input: All game video data after the game has finished.

[0514] Data processing / calculation: The highlight generation engine automatically extracts important scenes and generates a customized highlight video based on the user's settings and viewing history.

[0515] Output: The generated customized highlight video is sent to the user's device and displayed in a playable format.

[0516] Specific operation: The server extracts important parts such as "goal scenes" and generates and sends a customized highlight video for users who have set "I want to watch only goal scenes."

[0517] The above is a detailed flow of the system's processing steps, which allows users to enjoy a fulfilling real-time sports viewing experience.

[0518] (Application example 2)

[0519] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0520] In modern sports viewing, users want to enjoy the event in real time while feeling a sense of unity with other fans, and they want detailed commentary. However, existing systems have separate functions for live streaming, commentary, highlight generation, and emotion recognition, making it difficult to provide an integrated experience. Furthermore, they are unable to adequately optimize the timing of cheering based on user emotions. This results in a problem where users are unable to get the best possible viewing experience.

[0521] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for selecting an event the user wants to watch; means for sending a live stream request to the server; means for the server to collect social media posts related to the specified event in real time; means for analyzing the collected posts to identify peak times of excitement and emotion; means for notifying the user of the optimal cheering timing based on the identified peak times; means for providing the live stream and a cheering timing notification to the user's device; means for providing an emotion recognition engine for analyzing user emotion data collected from the user's device; and means for further optimizing the cheering timing notification based on the analysis results of the emotion recognition engine. This allows the user to watch the game while feeling a sense of unity with other fans, and detailed commentary and customized highlights are also provided, allowing for a deeper enjoyment of sports. Furthermore, a cheering experience that is individually optimized through real-time emotion analysis is provided.

[0522] The "means for the user to select an event that the user wishes to watch" is an interface that allows the user to select a sport event that the user wishes to watch using his / her own terminal and transmit the selection information to the system.

[0523] The "means for sending a request for a live stream to a server" is a function for requesting a live stream of a specified sporting event from a server based on the user's selection information.

[0524] The "means for the server to collect social media posts related to a designated event in real time" refers to a function that allows the server to collect social media posts using specific hashtags or keywords related to a designated sporting event.

[0525] "Means of analyzing collected posts to identify peak times of excitement and emotion" is a function that uses natural language processing technology to analyze collected social media posts and identify peak times of excitement and emotion.

[0526] The "means for notifying the user of the optimal cheering timing based on the identified peak time" is a function for notifying the user of the cheering timing based on the identified peak time.

[0527] The "means for providing a live stream and a notification of the cheering timing to a user's terminal" is a function for providing a live stream generated by the server and a notification of the cheering timing to a user's terminal.

[0528] An "emotion recognition engine that analyzes user emotion data collected from the user's device" is an engine that uses the camera and microphone installed on the user's device to collect and analyze user emotion data in real time.

[0529] The "means for further optimizing the notification of the timing to cheer based on the analysis results of the emotion recognition engine" is a function for optimizing the notification of the timing to cheer to the user with even greater accuracy based on the analysis results obtained from the emotion recognition engine.

[0530] "Means for the server to receive live-stream video data and operate an AI commentary assistant to analyze the progress of the event" refers to a function in which the server receives live-stream video data and uses AI to analyze the progress of the event.

[0531] "Means for sending explanatory text generated by the explanation assistant to the user's device in real time" is a function for sending text generated by the AI ​​explanation assistant to the user's device in real time.

[0532] "Means for the server to save all game video data after the game ends" is a function that enables the server to save all game video data after the game ends.

[0533] The "means for the server to operate a highlight generation engine and automatically extract important scenes" is a function for operating a highlight generation engine to extract important scenes from the saved game video data.

[0534] The "means for generating a customized highlight video based on the user's settings and viewing history" is a function for generating a customized highlight video based on the user's setting information and viewing history.

[0535] The "means for transmitting the generated highlight video to the user's terminal" is a function for transmitting the generated customized highlight video to the user's terminal.

[0536] The "means for providing the transmitted highlight video in a playable form to the user's terminal" is a function for providing the received highlight video in a playable form to the user's terminal.

[0537] The present invention is a system that integrates live streams, social media, AI commentary, highlight generation, and emotion recognition to enhance users' sporting event viewing experience.

[0538] The system program works as follows: First, a user selects a sporting event they want to watch using their device and sends the selection information to the server, which then starts a live stream related to the selected event and simultaneously collects social media posts using specific hashtags and keywords in real time.

[0539] The collected posts are analyzed by the server to identify peak times of excitement and emotion. Based on the identified peak times, the server then sends notifications to the user's device with the optimal timing for cheering. Furthermore, the emotion data collected from the user's device is analyzed by an emotion recognition engine, and the notification of the optimal timing for cheering is optimized based on the results.

[0540] The server receives the live-stream video data, and the AI ​​commentary assistant analyzes the progress of the match. The commentary text generated as a result of this analysis is sent to the user's device in real time, allowing the user to enjoy the commentary along with the video.

[0541] After the game ends, the server saves all game video data and runs a highlight generation engine to automatically extract important scenes. The generated highlight video is customized based on the user's settings and viewing history and sent to the user's device. The user's device then presents the received highlight video in a playable format.

[0542] The specific hardware and software required to realize this system include: Flask (a Python web framework) is used for the server; OpenCV and Dlib are used for sentiment analysis; HuggingFace Transformers is used for natural language processing; and the Twitter API is used for social media integration.

[0543] As a concrete example, consider the following scenario where a user is watching a basketball game. After the user selects a game and the live stream begins, the system collects related social media posts and identifies peak excitement times from tags like "TeamXWins." The emotion recognition engine analyzes the user's emotions and based on the results, notifies the user with "Team X is leading!", ensuring the user gets the best possible viewing experience.

[0544] An example of a prompt is as follows:

[0545] sports_event: "basketball"

[0546] social_media_hashtag: "TeamXWins"

[0547] emotion: "excited"

[0548] notify_action: "Team X is in the lead!"

[0549] As described above, the present invention is a system that highly personalizes a user's sports viewing experience and provides optimized cheering and commentary information in real time.

[0550] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0551] Step 1:

[0552] A user selects a sporting event they wish to watch using their device. The input is the event information selected by the user, and the output is the selection information sent to the server. Specifically, when the user operates the application interface to select an event and presses the "Watch" button, the selection information is sent to the server as a request.

[0553] Step 2:

[0554] The server processes requests for live streams of specified sporting events based on the selections received from the user. The input is the selections sent by the user, and the output is the URL and access information for the live stream. Specifically, the server sends a request to the live stream provider and prepares to receive the stream data.

[0555] Step 3:

[0556] The server collects social media posts related to a specified event in real time. The input is specific hashtags or keywords for the event, and the output is data on related social media posts. Specifically, it uses social media APIs (e.g., Twitter API) to collect related posts in real time.

[0557] Step 4:

[0558] The server analyzes collected social media posts and identifies peak times of excitement and emotion. The input is the collected post data, and the output is the identified peak time information. Specifically, it uses natural language processing technology to identify times when positive emotional expressions and reactions are increasing.

[0559] Step 5:

[0560] The server sends a notification to the user of the optimal time to cheer based on the identified peak times. The input is peak time information, and the output is a notification of the best time to cheer. Specifically, the server sends a notification to the user's device and displays a message such as "Now is your chance to cheer!"

[0561] Step 6:

[0562] The user's device receives the live stream and the notification of the cheering timing and provides it to the user. The input is the live stream URL and the cheering notification sent from the server, and the output is the stream video and notification displayed on the user's screen. Specifically, the application plays the stream and pops up a notification.

[0563] Step 7:

[0564] The user's device uses the device's camera and microphone to collect the user's emotional data in real time. The input is the user's facial expression data and voice data, and the output is emotional data sent to the emotion recognition engine. Specifically, the device processes the video captured by the camera and the voice recorded by the microphone.

[0565] Step 8:

[0566] The emotion recognition engine analyzes the user's emotional data and sends the results to the server. The input is the emotional data sent from the device, and the output is the emotional information as the analysis result. Specifically, it uses OpenCV and Dlib to analyze the user's facial expressions and voice.

[0567] Step 9:

[0568] The server further optimizes the timing of cheering notifications based on the analysis results obtained from the emotion recognition engine. The input is the analyzed emotion data, and the output is an optimized cheering notification. Specifically, the server adjusts the timing and content of notifications according to the emotion data.

[0569] Step 10:

[0570] The server receives the live-stream video data, runs the AI ​​commentary assistant, and analyzes the progress of the event. The input is the video data, and the output is commentary text. Specifically, the AI ​​commentary assistant analyzes players' movements and scoring scenes in real time and generates commentary text.

[0571] Step 11:

[0572] The explanatory text generated by the explanatory assistant is sent to the user's device in real time. The input is the generated explanatory text, and the output is the explanatory information displayed on the user's device. Specifically, the server sends the text data in real time, and the device displays it overlaid on the screen.

[0573] Step 12:

[0574] After the game ends, the server saves all game video data and runs a highlight generation engine. The input is game video data, and the output is a highlight clip containing important scenes. Specifically, the AI ​​engine automatically extracts important scenes such as goals and fouls and generates highlights.

[0575] Step 13:

[0576] A customized highlight video is generated based on the user's settings and viewing history and sent to the user's device. The input is the user's settings information and viewing history, and the output is a customized highlight video. Specifically, a video edited based on the individual user's preferences and past viewing history is created.

[0577] Step 14:

[0578] The user's device provides the transmitted highlight video in a playable format. The input is the highlight video data transmitted from the server, and the output is a playable highlight video. Specifically, the user's device plays the received video, allowing the user to easily review it.

[0579] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0580] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0581] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0582] [Second embodiment]

[0583] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0584] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0585] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0586] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0587] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0588] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0589] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0590] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0591] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0592] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0593] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0594] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0595] The present invention is a system that integrates live streaming with social media, real-time AI commentary, automatic highlight generation, and more to enhance users' sporting event viewing experience.

[0596] A way for users to select the event they want to watch and submit a request for a live stream

[0597] Users access the system using their terminals and select the sporting event they want to watch. The terminals then send the selection information to the server, requesting a live stream. As a result, users can watch their desired sporting event in real time.

[0598] A means by which the server collects social media posts related to a specified event in real time.

[0599] The server collects social media posts in real time using specific hashtags and keywords related to the designated event, and updates the collected posts as the event progresses and provides them to users.

[0600] A method for identifying peak times of excitement and emotion by analyzing collected posts

[0601] The server analyzes the collected social media posts and uses natural language processing technology to identify peak times of excitement and emotion, allowing it to understand in real time at which points during a game users are most excited.

[0602] A means of notifying users of the optimal timing for cheering based on identified peak times

[0603] The server then notifies the user of the optimal time to cheer based on the identified peak times. The notification is sent via the device and displayed on the screen, allowing the user to cheer while feeling a sense of unity with other fans.

[0604] A means for the server to receive the live stream video data and run a commentary assistant to analyze the progress of the match

[0605] The server receives live-stream video data in real time and runs an AI commentary assistant, which analyzes the progress of the game, understands player movements, goals scored, tactics, etc., and generates commentary text.

[0606] A means for transmitting the explanatory text generated by the explanatory assistant to the user's terminal in real time and displaying the explanatory text on the user's terminal.

[0607] The server transmits the generated commentary text to the user's device in real time, and the user's device displays the received commentary text overlaid on the live stream video, providing real-time commentary information to the user while watching the game.

[0608] After the match, the server saves all the match video data and runs a highlight generation engine to automatically extract important scenes.

[0609] After the game ends, the server stores all game video data, then runs a highlight generation engine to automatically extract important scenes (scores, fouls, good plays, etc.).

[0610] A means to generate customized highlight videos based on user preferences and viewing history

[0611] Users set their preferences and interests through their devices. The server references the user's settings and viewing history to generate a customized highlight video. The generated highlight video is then edited to suit the user's individual preferences.

[0612] A means for transmitting the generated highlight video to a user's terminal, and for the user's terminal to provide the transmitted highlight video in a reproducible form.

[0613] The server transmits the generated customized highlight video to the user's device, which then displays the received highlight video in a playable format, allowing the user to easily review and watch it.

[0614] Specific examples

[0615] Example 1: Real-time support

[0616] A user named "Yamada" uses a device to watch a basketball game. The server receives a request from Yamada's device and broadcasts the live stream while simultaneously collecting related social media posts. The server analyzes the posts, identifies moments of high excitement, and notifies Yamada of those moments. This allows Yamada to enjoy cheering at the most exciting moments.

[0617] Example 2: AI real-time commentary

[0618] A user named "Sasaki" is watching a soccer match. The server receives the live-stream video data, and the AI ​​commentary assistant analyzes the progress of the match and generates commentary text in real time. The commentary text is sent to Sasaki's device and displayed on the screen. This allows Sasaki to understand the flow of the match and the players' movements in detail.

[0619] Example 3: Automatic highlight generation

[0620] After a baseball game, user "Suzuki-san" wants to watch highlights of games he has previously watched. The server saves all video data at the end of the game and runs an AI highlight generation engine to extract important scenes. A customized highlight video is generated based on Suzuki-san's preferences and viewing history and sent to his device. Suzuki-san can easily enjoy highlights including his favorite scenes.

[0621] The above is an embodiment of the present invention. The present invention allows users to enjoy watching sports while feeling a sense of unity with other fans, and also allows users to gain a deeper understanding of sports through detailed commentary and customized highlights.

[0622] The processing flow will be explained below.

[0623] Processing steps for live streaming linked to social media in real time

[0624] Step 1:

[0625] A user uses a terminal to select a sporting event that they wish to watch.

[0626] Step 2:

[0627] The device sends a request for a live stream to the server.

[0628] Step 3:

[0629] The server receives a request for a live stream and retrieves the corresponding stream.

[0630] Step 4:

[0631] The server distributes the acquired live stream to the terminal, allowing the user to view it.

[0632] Step 5:

[0633] The server collects social media posts in real time using keywords and hashtags related to the specified event.

[0634] Step 6:

[0635] The server filters the collected posts and selects the most relevant posts.

[0636] Step 7:

[0637] The server analyzes the collected posts and identifies peak times of excitement and emotion.

[0638] Step 8:

[0639] The server generates data for notifying the user of the optimal cheering timing based on the identified peak time.

[0640] Step 9:

[0641] The server transmits the generated notification data to the terminal.

[0642] Step 10:

[0643] The notification received by the terminal is displayed on the screen to notify the user.

[0644] AI real-time commentary processing steps

[0645] Step 1:

[0646] The server receives the live stream video data in real time.

[0647] Step 2:

[0648] The server runs an AI commentary assistant and analyzes the video data.

[0649] Step 3:

[0650] The AI ​​commentary assistant recognizes the progress of the game, player movements, scoring scenes, etc.

[0651] Step 4:

[0652] The AI ​​commentary assistant generates explanatory text based on the analysis results.

[0653] Step 5:

[0654] The server transmits the generated explanatory text to the terminal in real time.

[0655] Step 6:

[0656] The explanatory text received by the terminal is displayed on the screen together with the live stream video and provided to the user.

[0657] Processing steps for automatic highlight generation

[0658] Step 1:

[0659] After the match ends, the server stores all the match video data.

[0660] Step 2:

[0661] The server inputs the saved video data into the AI ​​highlight generation engine.

[0662] Step 3:

[0663] The AI ​​highlight generation engine automatically extracts important scenes from the game (scores, fouls, good plays, etc.).

[0664] Step 4:

[0665] The user uses the device to set their preferences and interests.

[0666] Step 5:

[0667] The server refers to the user's setting information and past viewing history and determines a policy for generating a customized highlight video.

[0668] Step 6:

[0669] The server uses an AI highlight generation engine to generate a customized highlight video.

[0670] Step 7:

[0671] The server transmits the generated customized highlight video to the user's terminal.

[0672] Step 8:

[0673] The highlight video received by the terminal is provided to the user in a reproducible form.

[0674] Example 1

[0675] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0676] Conventional sports viewing systems have struggled to provide real-time commentary, improve the viewing experience, and share the excitement and emotion of users. Furthermore, it has been difficult to customize the post-match highlights based on individual user interests and viewing history. This has limited the enjoyment of sports viewing and prevented the user experience from being maximized.

[0677] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0678] In this invention, the server includes means for a user to select an event they wish to watch, means for sending a live stream request to the server, means for the server to collect social media posts related to the designated event in real time, means for analyzing the collected posts to identify peak times of excitement and emotion, means for notifying the user of the optimal cheering timing based on the identified peak times, means for providing the live stream and a notification of the cheering timing to the user's terminal, means for analyzing the social media posts using a generative AI model, means for notifying the user based on the peak times identified by the generative AI model, and a commentary assistant for the server to receive the live stream video data and analyze the progress of the match. The system includes a means for operating a game server, a means for transmitting commentary text generated by a commentary assistant to a user's device in real time, a means for displaying the commentary text on the user's device, a means for using a generative AI model for generating commentary text in real time, a means for a server to save all game video data after the game ends, a means for the server to operate a highlight generation engine and automatically extract important scenes, a means for generating a customized highlight video based on a user's settings and viewing history, a means for transmitting the generated highlight video to the user's device, a means for customizing and generating the highlight video using a generative AI model that uses the user's viewing history, and a means for the user's device to provide the transmitted highlight video in a playable form. This enables users to watch a game with in-depth commentary while sharing the excitement and emotion with other fans in real time, and to enjoy a highlight video that suits their interests after the game ends.

[0679] The "System" is an integrated mechanism that provides users with the optimal viewing experience by broadcasting the sporting events they want to watch in real time and collecting and analyzing related social media information.

[0680] "User" means an individual who uses the System to watch sporting events and receive services such as live streams, commentary, and highlight videos.

[0681] A "server" is a computer device that is the core of the system, and is responsible for processing requests from users, distributing live streams, analyzing data, generating highlights, and so on.

[0682] "Terminal" means a device used by a user to access the system and watch a sporting event, and includes a smartphone, tablet, PC, etc.

[0683] A "live stream" is video data of sporting events and performances broadcast in real time over the Internet.

[0684] "Social media" refers to a platform on the Internet where users can share and interact with each other, and includes Twitter, Facebook, Instagram, etc.

[0685] A "generative AI model" is a machine learning algorithm that is trained using artificial intelligence techniques to perform a specific task (e.g., text generation, sentiment analysis, etc.).

[0686] The "Commentary Assistant" is an artificial intelligence system that runs on a server, analyzes live-stream video data, and generates commentary text in real time.

[0687] The "highlight generation engine" is a software component that analyzes video data from matches, automatically extracts and edits important scenes, and generates highlight videos.

[0688] "Viewing history" is a record of sporting events that a user has viewed in the past and their contents.

[0689] The present invention provides a system that integrates live streaming, social media, real-time commentary using generative AI models, and automatic highlight generation to enhance users' sporting event viewing experience.

[0690] First, a user accesses the system using a terminal and selects the sporting event they want to watch. At this time, the terminal sends the user's selection information to the server and requests a live stream. The server receives the request and provides live footage of the specified sporting event.

[0691] The server then collects social media posts in real time using specific hashtags or keywords related to the specified event. These posts are retrieved using social media APIs (e.g., Twitter API, Facebook API). The collected posts are then analyzed using a generative AI model to identify peak times of excitement and emotion.

[0692] The server notifies users of the best time to cheer based on the identified peak times. The notification is sent via a pop-up message or notification bar on the user's device, allowing users to send cheering messages and share their excitement at the best possible time.

[0693] The server also receives live-stream video data in real time and runs a commentary assistant to analyze the progress of the match. The commentary assistant analyzes the player movements, goals scored, tactics, and other aspects of the video, and generates commentary text in real time. The generated commentary text is sent to the user's device and overlaid on the live video.

[0694] After the game ends, the server saves all game video data and runs a highlight generation engine. The highlight generation engine automatically extracts and edits important scenes, such as goals and great plays, to generate a customized highlight video. The customization is based on the user's settings and viewing history, and the generated highlight video is sent to the user's device.

[0695] As a concrete example, consider a user named "Yamada" using a device to watch a basketball game. The server receives a request from Yamada's device and delivers the live stream while simultaneously collecting and analyzing related social media posts. The server identifies the moments when excitement levels rise and notifies Yamada, allowing him to enjoy cheering without missing the most exciting moments.

[0696] In addition, when a user named "Sasaki" is watching a soccer match, the server receives the live-stream video data and uses the generative AI model to generate and transmit commentary text in real time, allowing Sasaki to watch the match while gaining a detailed understanding of the progress of the match.

[0697] Furthermore, if user "Suzuki-san" wants to watch highlights after a baseball game, the server saves all video data at the end of the game and activates the AI ​​highlight generation engine. A customized highlight video is generated and sent based on Suzuki-san's preferences and viewing history. Suzuki-san can easily watch highlight videos containing important scenes.

[0698] Example prompt sentence:

[0699] "How can Yamada know the optimal timing to cheer while watching a basketball game in real time?"

[0700] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0701] Step 1:

[0702] A user accesses the system using a terminal and selects the sporting event they wish to watch.

[0703] Input: User event selection information

[0704] Specific operation: The user selects the menu or list on the device screen, selects the event they want to watch, and confirms the selection by pressing the Enter key or touching the screen.

[0705] Output: Event selection information is recorded on the terminal and sent to the server.

[0706] Step 2:

[0707] The terminal transmits the user's selection information to the server.

[0708] Input: User event selection information

[0709] What it does: Your device sends the selected information to a server over the internet, using a secure protocol such as HTTPS to protect the data.

[0710] Output: The selection information is received by the server.

[0711] Step 3:

[0712] The server processes the live stream request based on the received selection information and provides the live video to the user.

[0713] Input: Selection information

[0714] Specific operation: The server accesses the streaming service, acquires live video of the specified event, and streams the acquired video data to the user's device.

[0715] Output: The live stream video is displayed on the user's device.

[0716] Step 4:

[0717] The server collects social media posts related to a specified event in real time.

[0718] Input: Event hashtags or keywords

[0719] How it works: The server uses social media APIs to collect posts related to the specified hashtags and keywords. The collection process is done in real time.

[0720] Output: Collected social media post data

[0721] Step 5:

[0722] The server analyzes the collected posts using a generative AI model to identify peak times of excitement and emotion.

[0723] Input: Collected submission data

[0724] How it works: Using a generative AI model, post data is analyzed and scored for excitement and emotional impact. Peak times are identified based on the scoring results.

[0725] Output: Identified peak times of excitement and emotion

[0726] Step 6:

[0727] The server notifies the user of the optimal cheering timing based on the identified peak time.

[0728] Input: Identified peak times

[0729] Specific operation: The server sends a notification message to the user's device, causing a pop-up message or notification bar to appear on the user's screen.

[0730] Output: Notification of the cheering timing displayed on the device screen

[0731] Step 7:

[0732] The server receives the live stream video data in real time, runs the commentary assistant, and analyzes the progress of the match.

[0733] Input: Live stream video data

[0734] Specific operation: The server analyzes the video data using an AI commentary assistant (e.g., image recognition model and natural language processing model).

[0735] Output: Real-time generated explanatory text

[0736] Step 8:

[0737] The server transmits the generated commentary text to the user's terminal in real time, and the commentary text is displayed on the user's terminal.

[0738] Input: Generated description text

[0739] Specific operation: The server sends explanatory text to the user's device, which then overlays the received explanatory text on the live video.

[0740] Output: Descriptive text that is displayed on the device screen

[0741] Step 9:

[0742] After the match ends, the server stores all match video data and runs a highlight generation engine to automatically extract important scenes.

[0743] Input: Match video data

[0744] How it works: The server stores the video data in a database, then uses a highlight generation engine to automatically extract and edit important scenes (scoring scenes, great plays, etc.).

[0745] Output: Clips of extracted key scenes

[0746] Step 10:

[0747] The server generates a customized highlight video based on the user's settings and viewing history and transmits it to the user's terminal.

[0748] Input: User settings and viewing history, extracted clips of important scenes

[0749] Specific operation: The server references the user's settings and viewing history, combines clips of key scenes, and generates a customized highlight video. The generated video is then sent to the user's device.

[0750] Output: A customized highlight video sent to the user's device

[0751] Step 11:

[0752] To provide a highlight video received by a user terminal in a reproducible form.

[0753] Input: Custom highlight video

[0754] Specific operation: The device uses an application (e.g., a video player) to play the received highlight video and displays it in a playable format for the user.

[0755] Output: A highlight video that can be viewed by the user

[0756] (Application example 1)

[0757] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0758] Conventional sports viewing systems lack real-time commentary, highlight generation, and notification functions that allow viewers to cheer on the most exciting moments. This makes it difficult for users to gain a deeper understanding of the content and enjoy it without missing any exciting moments. Furthermore, they do not provide customized highlight videos tailored to individual users' preferences. Therefore, a system that integrates these functions is needed to improve the user experience.

[0759] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0760] In this invention, the server includes means for a user to select content they wish to view, means for sending a streaming request to the server, means for the server to collect online posts related to the specified content in real time, means for analyzing the collected posts to identify peak times of excitement and emotion, means for notifying the user of the optimal timing for cheering based on the identified peak times, and means for providing streaming and notification of the timing for cheering to the user's information terminal. This allows the user to know the timing for cheering in real time, watch the game with a sense of unity with other viewers, and enjoy customized highlights that suit their interests.

[0761] "Means for a user to select content they wish to view" refers to the interface or process used to select the particular content a user wishes to view.

[0762] The "means for sending a streaming request to the server" refers to a mechanism for requesting the server to stream content based on the content selected by the user.

[0763] "Means for the server to collect Internet posts related to specified content in real time" refers to the function of the server to obtain posts and comments related to specific content on the Internet in real time.

[0764] "Means of analyzing collected posts to identify peak times of excitement and emotion" refers to the process of analyzing collected internet posts to identify the times when users are most excited or moved.

[0765] "Means for notifying users of the optimal timing to cheer based on identified peak times" refers to a system that notifies users of the optimal timing to cheer based on peak times of excitement and emotion identified through analysis.

[0766] "Means for providing streaming and cheering timing notifications to a user's information terminal" refers to a function for delivering streaming data and cheering timing information to a device used by a user.

[0767] "Commentary assistant for analyzing content progress" refers to AI or software that analyzes the progress of content being streamed in real time and generates commentary based on that content.

[0768] "Means for transmitting explanatory text generated by the explanatory assistant in real time to the user's information terminal" refers to a process for instantly transmitting explanatory text generated by the explanatory assistant to the user's device.

[0769] "Means for displaying explanatory text on the user's information terminal" refers to a mechanism for displaying explanatory text on the screen of the user's device.

[0770] "Means for the server to save all video data after the end of the match" refers to the function of storing all streamed video data on the server after the end of the match.

[0771] "Means for the server to operate the highlight generation engine and automatically extract important scenes" refers to a system that uses the highlight generation engine in the server to automatically select important scenes from video data.

[0772] The "means for generating a customized highlight video based on user settings and viewing history" refers to a function for generating an individually customized highlight video based on preferences set by the user and past viewing history.

[0773] "Means for transmitting the generated highlight video to the user's information terminal" refers to a process for transmitting the customized highlight video to the user's device.

[0774] The "means for providing the transmitted highlight video in a format that can be played back by the user's information terminal" refers to a system that provides the transmitted highlight video in a format that can be played back by the user's device.

[0775] This invention is a system for improving a user's sports viewing experience, utilizing a server, a user's information terminal, and AI technology. This system is implemented in the following steps.

[0776] First, a user selects the content they want to watch using an information terminal such as a smartphone or tablet. The selection information is sent from the terminal to the server, and the server receives a streaming request based on that information. The server then begins streaming the specified content.

[0777] The server then collects real-time internet posts related to the specified content, including searches using specific hashtags or keywords, and analyzes these posts using natural language processing techniques to identify peak times of excitement and emotion.

[0778] Once the data collection and analysis is complete, the server notifies users of the optimal time to cheer based on the identified peak times. The notification is sent to the user's information terminal and displayed on the screen in real time.

[0779] The server also runs an AI commentary assistant to analyze the streaming video. The AI ​​commentary assistant analyzes the progress of the game in real time and generates commentary text about players' movements, scoring scenes, etc. The generated commentary text is sent in real time to the user's information terminal and displayed on the screen.

[0780] After the game ends, the server saves all video data and runs a highlight generation engine. The highlight generation engine automatically extracts important scenes and generates a customized highlight video based on the user's settings and viewing history. This highlight video is sent to the user's information terminal, allowing the user to watch the highlights at any time.

[0781] The hardware and software used include the following: Hardware includes information terminals such as smartphones and tablets, and servers. Software includes TensorFlow (AI model), Flask / Django (server-side framework), OpenCV (video data analysis), Requests (HTTP request processing), and NLTK (natural language processing) used on the server side.

[0782] As a concrete example, consider the case where a user named "Suzuki" wants to watch a basketball game. Suzuki uses his smartphone to select the game he wants to watch and sends a request to the server. The server receives the request and provides live streaming, while also collecting and analyzing related posts from social media to identify peak times of excitement. The server notifies Suzuki of these peak times and also provides commentary text generated by an AI commentary assistant in real time. After the game ends, the server extracts key scenes and generates and sends customized highlights tailored to Suzuki's preferences.

[0783] An example of a prompt sentence might be:

[0784] "While watching a basketball game in real time, generate a code to identify and notify you of peak moments of excitement from social media."

[0785] In this way, the present invention can significantly improve the sports viewing experience for users.

[0786] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0787] Step 1:

[0788] The user selects the content they wish to view using an information terminal. The input is the user's viewing preference, and the output is the selected content information. The interface used by the user is intuitive, and selection can be easily made using lists and search functions.

[0789] Step 2:

[0790] A streaming request is sent from the information terminal to the server. The input is the selected content information, and the output is a confirmation of the request from the server. This information sent as an HTTP request triggers the server to start streaming.

[0791] Step 3:

[0792] The server starts streaming the specified content. The input is the requested content information, and the output is live streaming data. The server encodes the video data in real time and sends it to the user's information terminal.

[0793] Step 4:

[0794] The server collects internet posts related to the specified content in real time. The input is a specific hashtag or keyword, and the output is the post data. The server uses APIs to retrieve related posts from social media and stores them in a database.

[0795] Step 5:

[0796] The server analyzes the collected posts to identify peak times of excitement and emotion. The input is the collected post data, and the identified peak times are extracted as the output. Natural language processing technology (e.g., NLTK) is used to analyze the text data and evaluate emotions and excitement levels.

[0797] Step 6:

[0798] The server notifies the user of the optimal cheering timing based on the identified peak times. The input is the identified peak time information, and the output is generated notification data. The server generates real-time notifications and sends them to the user's information terminal.

[0799] Step 7:

[0800] The server runs an AI commentary assistant to analyze streaming video. The input is real-time video data, and the output is explanatory text. The TensorFlow model is used to analyze the video data and generate explanatory text according to the progress.

[0801] Step 8:

[0802] The server generates explanatory text and sends it to the user's information terminal in real time. The input is the generated explanatory text, and the output is a confirmation of transmission. The explanatory text is distributed along with the video data.

[0803] Step 9:

[0804] The commentary text is displayed on the user's information terminal. The input is the received commentary text, and the output is the displayed text. It is displayed overlaid on the screen of the user's information terminal to aid viewing.

[0805] Step 10:

[0806] After the game ends, the server saves all video data. The input is all video data from the game, and the output is a saved data file. The server saves the data in cloud storage such as S3.

[0807] Step 11:

[0808] The server runs a highlight generation engine to automatically extract important scenes. The input is stored video data, and the output is extracted highlight scenes. OpenCV is used to perform video analysis and extract scenes based on specific events or actions.

[0809] Step 12:

[0810] The server generates a customized highlight video based on the user's settings and viewing history. The input is the user's settings and viewing history, and the output is a customized highlight video. An AI model is used to analyze the user's preferences and combine the optimal scenes.

[0811] Step 13:

[0812] The server sends the generated highlight video to the user's information terminal. The input is the customized highlight video, and the output is a confirmation of transmission. The video data is provided to the user in a format that is easy to view.

[0813] Step 14:

[0814] The user's information terminal provides the transmitted highlight video in a playable format. The input is the received highlight video, and the output is a played video. The user can enjoy the highlight video containing their favorite scenes.

[0815] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0816] This invention is a system that integrates live streaming with social media, AI-powered real-time commentary, automatic highlight generation, and an emotion engine that recognizes user emotions to enhance users' sporting event viewing experience.

[0817] A way for users to select the event they want to watch and submit a request for a live stream

[0818] Users access the system using their terminals and select the sporting event they want to watch. The terminals then send the selection information to the server, requesting a live stream. As a result, users can watch their desired sporting event in real time.

[0819] A means by which the server collects social media posts related to a specified event in real time.

[0820] The server collects social media posts in real time using specific hashtags and keywords related to the designated event, and updates the collected posts as the event progresses and provides them to users.

[0821] A method for identifying peak times of excitement and emotion by analyzing collected posts

[0822] The server analyzes the collected social media posts and uses natural language processing technology to identify peak times of excitement and emotion, allowing it to understand in real time at which points during a game users are most excited.

[0823] A means of notifying users of the optimal timing for cheering based on identified peak times

[0824] The server then notifies the user of the optimal time to cheer based on the identified peak times. The notification is sent via the device and displayed on the screen, allowing the user to cheer while feeling a sense of unity with other fans.

[0825] A means for the server to receive the live stream video data and run a commentary assistant to analyze the progress of the match

[0826] The server receives live-stream video data in real time and runs an AI commentary assistant, which analyzes the progress of the game, understands player movements, goals scored, tactics, etc., and generates commentary text.

[0827] A means for transmitting the explanatory text generated by the explanatory assistant to the user's terminal in real time and displaying the explanatory text on the user's terminal.

[0828] The server transmits the generated commentary text to the user's device in real time, and the user's device displays the received commentary text overlaid on the live stream video, providing real-time commentary information to the user while watching the game.

[0829] After the match, the server saves all the match video data and runs a highlight generation engine to automatically extract important scenes.

[0830] After the game ends, the server stores all game video data, then runs a highlight generation engine to automatically extract important scenes (scores, fouls, good plays, etc.).

[0831] A means to generate customized highlight videos based on user preferences and viewing history

[0832] Users set their preferences and interests through their devices. The server references the user's settings and viewing history to generate a customized highlight video. The generated highlight video is then edited to suit the user's individual preferences.

[0833] A means for transmitting the generated highlight video to a user's terminal, and for the user's terminal to provide the transmitted highlight video in a reproducible form.

[0834] The server transmits the generated customized highlight video to the user's device, which then displays the received highlight video in a playable format, allowing the user to easily review and watch it.

[0835] Processing by emotion engine that recognizes user emotions

[0836] The user's device is equipped with a camera and microphone, which are used to collect the user's emotional data in real time. The emotion engine reads the user's facial expressions using the device's camera and analyzes the tone and strength of the voice using the microphone. The emotion engine then sends the collected emotional data to the server.

[0837] Server analyzes emotional data and adjusts cheering timing

[0838] The server analyzes the emotion data sent from the emotion engine and evaluates the user's level of excitement and emotion in real time. Based on this evaluation, the server further optimizes the timing of cheering notifications and supports the user's cheering.

[0839] Specific examples

[0840] Example 1: Real-time cheering and emotion recognition

[0841] A user named "Yamada" uses a device to watch a basketball game. The server receives a request from Yamada's device and broadcasts the live stream while simultaneously collecting related social media posts. The server analyzes the posts, identifies moments of high excitement, and notifies Yamada of those moments. Furthermore, the emotion engine analyzes Yamada's facial expressions and voice to provide more accurate cheering timing.

[0842] Example 2: AI real-time commentary and emotion recognition

[0843] A user named "Sasaki" is watching a soccer match. The server receives the live-stream video data, and the AI ​​commentary assistant analyzes the progress of the match and generates commentary text in real time. The commentary text is sent to Sasaki's device and displayed on the screen. Furthermore, the emotion engine analyzes Sasaki's emotions and provides additional commentary based on the moments of excitement.

[0844] Example 3: Automatic highlight generation and emotion recognition

[0845] After a baseball game, user "Suzuki-san" wants to watch highlights of games he has previously watched. The server saves all video data at the end of the game and runs an AI highlight generation engine to extract important scenes. A customized highlight video is generated based on Suzuki-san's preferences and viewing history and sent to his device. The emotion engine analyzes Suzuki-san's emotions while watching the highlights and further emphasizes scenes that are exciting.

[0846] The above is an embodiment of the present invention. The present invention allows users to enjoy watching sports while feeling a sense of unity with other fans, and also allows for a deeper understanding of sports through detailed commentary and customized highlights. Furthermore, by analyzing users' real-time emotions, a more personalized cheering and viewing experience can be realized.

[0847] The processing flow will be explained below.

[0848] Processing steps for live streaming linked to social media in real time

[0849] Step 1:

[0850] A user uses a terminal to select a sporting event that they wish to watch.

[0851] Step 2:

[0852] The device sends a request for a live stream to the server.

[0853] Step 3:

[0854] The server receives a request for a live stream and retrieves the corresponding stream.

[0855] Step 4:

[0856] The server distributes the acquired live stream to the terminal, allowing the user to view it.

[0857] Step 5:

[0858] The server collects social media posts in real time using keywords and hashtags related to the specified event.

[0859] Step 6:

[0860] The server filters the collected posts and selects the most relevant posts.

[0861] Step 7:

[0862] The server analyzes the collected posts and identifies peak times of excitement and emotion.

[0863] Step 8:

[0864] The server generates data for notifying the user of the optimal cheering timing based on the identified peak time.

[0865] Step 9:

[0866] The server transmits the generated notification data to the terminal.

[0867] Step 10:

[0868] The notification received by the terminal is displayed on the screen to notify the user.

[0869] AI real-time commentary processing steps

[0870] Step 1:

[0871] The server receives the live stream video data in real time.

[0872] Step 2:

[0873] The server runs an AI commentary assistant and analyzes the video data.

[0874] Step 3:

[0875] The AI ​​commentary assistant recognizes the progress of the game, player movements, scoring scenes, etc.

[0876] Step 4:

[0877] The AI ​​commentary assistant generates explanatory text based on the analysis results.

[0878] Step 5:

[0879] The server transmits the generated explanatory text to the terminal in real time.

[0880] Step 6:

[0881] The explanatory text received by the terminal is displayed on the screen together with the live stream video and provided to the user.

[0882] Processing steps for automatic highlight generation

[0883] Step 1:

[0884] After the match ends, the server stores all the match video data.

[0885] Step 2:

[0886] The server inputs the saved video data into the AI ​​highlight generation engine.

[0887] Step 3:

[0888] The AI ​​highlight generation engine automatically extracts important scenes from the game (scores, fouls, good plays, etc.).

[0889] Step 4:

[0890] The user uses the device to set their preferences and interests.

[0891] Step 5:

[0892] The server refers to the user's setting information and past viewing history and determines a policy for generating a customized highlight video.

[0893] Step 6:

[0894] The server uses an AI highlight generation engine to generate a customized highlight video.

[0895] Step 7:

[0896] The server transmits the generated customized highlight video to the user's terminal.

[0897] Step 8:

[0898] The highlight video received by the terminal is provided to the user in a reproducible form.

[0899] Processing steps combining emotion engines

[0900] Step 1:

[0901] When a user uses a device to watch an event, the device's camera and microphone are activated.

[0902] Step 2:

[0903] The device uses a camera to recognize the user's facial expressions and a microphone to analyze the tone and strength of the voice.

[0904] Step 3:

[0905] The emotion data acquired by the device is sent to the server in real time.

[0906] Step 4:

[0907] The server receives the emotion data and analyzes it.

[0908] Step 5:

[0909] The server evaluates the user's excitement level and changes in emotions.

[0910] Step 6:

[0911] The server adjusts the timing of cheering notifications based on the results of emotion data analysis.

[0912] Step 7:

[0913] The server transmits the adjusted cheering timing to the terminal.

[0914] Step 8:

[0915] The device displays a notification of the adjusted cheering timing to the user.

[0916] Specific examples

[0917] Example 1: Real-time cheering and emotion recognition

[0918] Step 1:

[0919] A user "Yamada" uses the terminal to watch a basketball game.

[0920] Step 2:

[0921] The device sends a request for a live stream to the server, and the server delivers the stream.

[0922] Step 3:

[0923] The server collects and analyzes social media posts to identify peak times of excitement.

[0924] Step 4:

[0925] The device's camera and microphone capture Yamada's emotions in real time.

[0926] Step 5:

[0927] The emotion data is sent to a server and analyzed.

[0928] Step 6:

[0929] Based on the analysis results, the server will notify Yamada of the optimal timing to cheer him on.

[0930] Step 7:

[0931] The device will display the optimal timing for cheering to Yamada, encouraging him to cheer in real time.

[0932] Example 2: AI real-time commentary and emotion recognition

[0933] Step 1:

[0934] User "Sasaki-san" starts watching a soccer game.

[0935] Step 2:

[0936] The server receives the live stream footage and the AI ​​commentary assistant begins analyzing it.

[0937] Step 3:

[0938] The explanatory text generated by the commentary assistant is sent to Sasaki's device in real time.

[0939] Step 4:

[0940] The device's camera and microphone capture Sasaki's emotional data.

[0941] Step 5:

[0942] The emotion data is sent to a server and analyzed.

[0943] Step 6:

[0944] The server provides additional commentary and highlights based on the results of Sasaki's sentiment analysis.

[0945] Step 7:

[0946] The device displays additional commentary information to Sasaki, helping him better understand the match.

[0947] Example 3: Automatic highlight generation and emotion recognition

[0948] Step 1:

[0949] After the baseball game ends, user "Suzuki" wants to watch the highlights.

[0950] Step 2:

[0951] The server stores all video data from the match and runs the AI ​​highlight generation engine.

[0952] Step 3:

[0953] After extracting important scenes, Suzuki's device sends his preferences and viewing history to the server.

[0954] Step 4:

[0955] The server generates a customized highlight video and transmits it to the device.

[0956] Step 5:

[0957] The device's camera and microphone capture Suzuki's emotional data.

[0958] Step 6:

[0959] The emotion data is sent to a server and analyzed.

[0960] Step 7:

[0961] The server provides emotional feedback according to the scene of interest, further emphasizing the highlights.

[0962] Step 8:

[0963] The device will provide Suzuki with the adjusted highlight video in a playable format.

[0964] Example 2

[0965] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0966] Conventional sports viewing systems often only provide simple live streaming, resulting in a one-way viewing experience for users. They also lack real-time information and commentary, and lack advanced features, particularly those utilizing social media and AI technology, preventing services from responding to users' emotions and excitement. Furthermore, highlight videos generated after the game are often generated manually, making it difficult to customize them based on individual user preferences and viewing history.

[0967] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0968] In this invention, the server includes: a means for a user to select an event they wish to watch; a means for sending a live stream request to the server; a means for the server to collect social media posts related to the specified event in real time; a means for analyzing the collected posts to identify peak times of excitement and emotion; a means for notifying the user of the optimal cheering timing based on the identified peak times; a means for providing the live stream and a notification of the cheering timing to the user's device; a means for recognizing the user's emotions in real time; and a means for analyzing the user's emotion data and adjusting the cheering timing. This allows users to enjoy a real-time, interactive viewing experience, and enables the provision of advanced information and commentary utilizing social media and AI technologies. Furthermore, highlights generated after the game are automated, allowing for customization based on the user's preferences and interests.

[0969] The "means for a user to select an event that the user wishes to watch" refers to an interface and functionality that allows a user to use a terminal to search for and select a sporting event that the user wishes to watch.

[0970] "Means for sending a request for a live stream to a server" refers to the communications capabilities and protocols for sending a request for a live stream to a server based on a user-selected event.

[0971] "Means by which the server collects social media posts related to a specified event in real time" refers to algorithms and data collection systems for collecting relevant posts from social media using specific hashtags or keywords.

[0972] "Means for analyzing collected posts to identify peak times of excitement or emotion" refers to technology and processing for analyzing social media posts using natural language processing technology and identifying peak times of excitement or emotion among users.

[0973] "Means for notifying users of the optimal timing for cheering based on identified peak times" refers to a communication function and notification system for notifying users of cheering messages at the optimal timing based on peak times of excitement and emotion.

[0974] "Means for providing a live stream and cheering timing notifications to a user's terminal" refers to a distribution system and application function for providing a live stream video and cheering timing notifications to a user's terminal.

[0975] "Means for recognizing a user's emotions in real time" refers to the technology and functions for collecting a user's facial expressions and voice in real time using the device's camera and microphone, and recognizing their emotions.

[0976] "Means for analyzing user emotional data and adjusting the timing of cheering" refers to an algorithm and processing system for analyzing collected user emotional data and optimizing the timing of cheering based on the analysis results.

[0977] "Means for receiving live-stream video data and operating a commentary assistant to analyze the progress of the match" refers to the function of receiving live-stream video data in real time and operating an AI commentary assistant that analyzes the progress of the match based on that data.

[0978] "Means for transmitting the explanatory text generated by the explanation assistant to the user's device in real time" refers to the communication function and data transmission system for transmitting the explanatory text generated by the AI ​​explanation assistant to the user's device in real time.

[0979] "Means for the server to store all game video data after the game ends" refers to a data storage and management system for storing all game video data on the server after the game ends.

[0980] "Means for operating a highlight generation engine and automatically extracting important scenes" refers to a technology and processing system for operating a highlight generation engine and automatically extracting important scenes from a match.

[0981] The "means for generating a customized highlight video based on user settings and viewing history" refers to algorithms and processing functions for generating a customized highlight video based on user preferences and viewing history.

[0982] The "means for transmitting the generated highlight video to the user's terminal" refers to a communication function and a data transmission system for transmitting the generated customized highlight video to the user's terminal.

[0983] The present invention provides a system that allows users to watch sporting events in real time and enhance their viewing experience. The system includes three main components: a user, a terminal, and a server. Each component plays a specific role and works together to enhance the user experience.

[0984] System Configuration

[0985] User terminal

[0986] Users use devices such as smartphones, tablets, and PCs (hereafter referred to as "terminals"). Terminals include an internet connection, camera, microphone, and display. Terminals access the system through a browser or dedicated application.

[0987] server

[0988] The server is located on the cloud infrastructure and is responsible for collecting, analyzing, distributing and storing data. The server has the following main functions:

[0989] Live Streaming

[0990] Social media data collection and analysis

[0991] AI commentary assistant

[0992] Highlight Generation Engine

[0993] Emotion Recognition Engine

[0994] Main processing

[0995] Request and deliver a live stream

[0996] The user selects the event they want to watch and sends a live stream request to the server via their device. The server receives the request and delivers the live stream of the event. The live stream data is sent to the user's device in real time and displayed on the screen.

[0997] Social media data collection and analysis

[0998] The server collects social media posts in real time using hashtags and keywords related to a specified event, then analyzes the posts using natural language processing (NLP) techniques to identify peak times of user excitement and emotion.

[0999] Cheering timing notification

[1000] Based on the analysis results, the server calculates the optimal cheering timing according to the identified peak times and notifies the user's device, allowing users to feel a sense of unity with other fans and cheer more enthusiastically.

[1001] AI commentary assistant

[1002] The server receives live-stream video data in real time and runs an AI commentary assistant. The AI ​​commentary assistant analyzes the progress of the game, players' movements, scoring scenes, etc., and generates commentary text. This text is sent to the device in real time and displayed overlaid on the video.

[1003] Highlight Generation

[1004] After the game ends, the server stores all game video data and runs a highlight generation engine, which generates a customized highlight video based on the user's settings and viewing history, and sends it to the device for playback.

[1005] Emotion recognition and cheer timing adjustment

[1006] The user's device is equipped with a camera and microphone, which are used to collect the user's emotional data in real time. The emotion engine analyzes facial expressions and tone of voice and sends the results to the server. The server uses this data to calculate the optimal timing for cheering, further improving the user's cheering experience.

[1007] Specific examples

[1008] Example 1: Real-time cheering and emotion recognition

[1009] User "A" uses a device to watch a basketball game. The server receives a request from A's device and delivers the live stream while simultaneously collecting related social media posts. The server analyzes the posts, identifies moments of high excitement, and notifies A of those moments. Furthermore, the emotion engine analyzes A's facial expressions and voice to provide more accurate cheering timing.

[1010] Example 2: AI real-time commentary and emotion recognition

[1011] User "B" is watching a soccer match. The server receives the live-stream video data, and the AI ​​commentary assistant analyzes the progress of the match and generates commentary text. The text is sent to B's device and displayed on the screen. Furthermore, the emotion engine analyzes B's emotions and provides additional commentary at times when B is excited.

[1012] Example 3: Automatic highlight generation and emotion recognition

[1013] After a baseball game, user "C" wants to watch highlights of games he previously watched. The server saves all video data at the end of the game and runs an AI highlight generation engine to extract important scenes. A customized highlight video based on C's preferences and viewing history is sent to C's device. The emotion engine analyzes C's emotions while watching the highlights and further emphasizes scenes that are exciting.

[1014] Prompt Sentence Examples

[1015] Imagine you are watching a basketball game. Describe a system in which an AI commentary assistant generates commentary in real time and notifies the user of the optimal timing to cheer based on social media data. It also analyzes user sentiment in real time and uses that information to optimize cheering and commentary.

[1016] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1017] Processing Step Description

[1018] Step 1:

[1019] Users select the event they want to watch

[1020] Input: The user accesses the device's interface, searches for and selects the sporting event they wish to watch.

[1021] Data processing / calculation: The terminal formats the user's selected information and prepares it as request data for the server.

[1022] Output: The terminal sends live stream request data to the server.

[1023] Specific operation: The user selects "basketball game" on the interface, and the device sends that information to the server as a "live stream request."

[1024] Step 2:

[1025] The server receives a request for a live stream and starts broadcasting it.

[1026] Input: The server receives live stream request data from the terminal.

[1027] Data processing / calculation: The server analyzes the request content and obtains the distribution URL and data of the corresponding live stream.

[1028] Output: The server sends the live stream data to the user's device.

[1029] Specific operation: The server receives the request, generates a live streaming URL, and starts sending the stream data to the device.

[1030] Step 3:

[1031] The server collects and analyzes relevant posts from social media.

[1032] Input: The server receives as input hashtags or keywords related to the specified event (e.g., "basketball game" or "score").

[1033] Data processing / calculation: Based on these hashtags and keywords, the server collects data from social media in real time and analyzes it using natural language processing (NLP) technology.

[1034] Output: The analyzed social media data is accumulated and peak time information on excitement and emotion is generated.

[1035] Specific operation: The server analyzes collected social media posts and assigns them a rating score based on excitement and emotion.

[1036] Step 4:

[1037] The server identifies and notifies you of peak times

[1038] Input: Social media data analyzed by NLP techniques.

[1039] Data processing / calculation: The server uses an algorithm to identify peak times based on the analysis results and calculate the optimal timing for cheering.

[1040] Output: The identified cheering timing notification data is generated and sent to the user's terminal.

[1041] Specific operation: The server notifies the device of the timing to cheer based on a score such as "excitement level 80%."

[1042] Step 5:

[1043] The server receives the live stream video and runs the AI ​​commentary assistant.

[1044] Input: Live stream video data.

[1045] Data processing / calculation: The AI ​​commentary assistant analyzes the video data and generates commentary text about the progress of the match and specific scenes.

[1046] Output: The generated explanatory text data is sent to the user's terminal.

[1047] Specific operation: The AI ​​generates explanatory text such as "Player A scored a goal!" and sends it to the device.

[1048] Step 6:

[1049] Recognize user emotions in real time and collect data

[1050] Input: User's facial expression data and voice data obtained from the device's camera and microphone.

[1051] Data processing / calculation: The emotion recognition engine analyzes facial expressions and voice data to identify emotional states (e.g., "excited" or "relaxed").

[1052] Output: The analyzed emotion data is sent to the server.

[1053] Specific operation: The camera recognizes the user's facial expressions such as "smile" and "surprise," and the microphone analyzes the "tone" and "pitch" of the voice.

[1054] Step 7:

[1055] The server analyzes the emotional data and adjusts the timing of cheering.

[1056] Input: User emotion data sent from the device.

[1057] Data processing / calculation: The server uses an algorithm to dynamically adjust the timing of cheering based on emotional data.

[1058] Output: The re-adjusted cheer timing notification data is sent to the user's device.

[1059] Specific operation: Based on the "excitement level 90%", the server further optimizes the cheering message and notifies the user.

[1060] Step 8:

[1061] The server stores the match video data, runs the highlight generation engine, and sends it to the user.

[1062] Input: All game video data after the game has finished.

[1063] Data processing / calculation: The highlight generation engine automatically extracts important scenes and generates a customized highlight video based on the user's settings and viewing history.

[1064] Output: The generated customized highlight video is sent to the user's device and displayed in a playable format.

[1065] Specific operation: The server extracts important parts such as "goal scenes" and generates and sends a customized highlight video for users who have set "I want to watch only goal scenes."

[1066] The above is a detailed flow of the system's processing steps, which allows users to enjoy a fulfilling real-time sports viewing experience.

[1067] (Application example 2)

[1068] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1069] In modern sports viewing, users want to enjoy the event in real time while feeling a sense of unity with other fans, and they want detailed commentary. However, existing systems have separate functions for live streaming, commentary, highlight generation, and emotion recognition, making it difficult to provide an integrated experience. Furthermore, they are unable to adequately optimize the timing of cheering based on user emotions. This results in a problem where users are unable to get the best possible viewing experience.

[1070] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for selecting an event the user wants to watch; means for sending a live stream request to the server; means for the server to collect social media posts related to the specified event in real time; means for analyzing the collected posts to identify peak times of excitement and emotion; means for notifying the user of the optimal cheering timing based on the identified peak times; means for providing the live stream and a cheering timing notification to the user's device; means for providing an emotion recognition engine for analyzing user emotion data collected from the user's device; and means for further optimizing the cheering timing notification based on the analysis results of the emotion recognition engine. This allows the user to watch the game while feeling a sense of unity with other fans, and detailed commentary and customized highlights are also provided, allowing for a deeper enjoyment of sports. Furthermore, a cheering experience that is individually optimized through real-time emotion analysis is provided.

[1071] The "means for the user to select an event that the user wishes to watch" is an interface that allows the user to select a sport event that the user wishes to watch using his / her own terminal and transmit the selection information to the system.

[1072] The "means for sending a request for a live stream to a server" is a function for requesting a live stream of a specified sporting event from a server based on the user's selection information.

[1073] The "means for the server to collect social media posts related to a designated event in real time" refers to a function that allows the server to collect social media posts using specific hashtags or keywords related to a designated sporting event.

[1074] "Means of analyzing collected posts to identify peak times of excitement and emotion" is a function that uses natural language processing technology to analyze collected social media posts and identify peak times of excitement and emotion.

[1075] The "means for notifying the user of the optimal cheering timing based on the identified peak time" is a function for notifying the user of the cheering timing based on the identified peak time.

[1076] The "means for providing a live stream and a notification of the cheering timing to a user's terminal" is a function for providing a live stream generated by the server and a notification of the cheering timing to a user's terminal.

[1077] An "emotion recognition engine that analyzes user emotion data collected from the user's device" is an engine that uses the camera and microphone installed on the user's device to collect and analyze user emotion data in real time.

[1078] The "means for further optimizing the notification of the timing to cheer based on the analysis results of the emotion recognition engine" is a function for optimizing the notification of the timing to cheer to the user with even greater accuracy based on the analysis results obtained from the emotion recognition engine.

[1079] "Means for the server to receive live-stream video data and operate an AI commentary assistant to analyze the progress of the event" refers to a function in which the server receives live-stream video data and uses AI to analyze the progress of the event.

[1080] "Means for sending explanatory text generated by the explanation assistant to the user's device in real time" is a function for sending text generated by the AI ​​explanation assistant to the user's device in real time.

[1081] "Means for the server to save all game video data after the game ends" is a function that enables the server to save all game video data after the game ends.

[1082] The "means for the server to operate a highlight generation engine and automatically extract important scenes" is a function for operating a highlight generation engine to extract important scenes from the saved game video data.

[1083] The "means for generating a customized highlight video based on the user's settings and viewing history" is a function for generating a customized highlight video based on the user's setting information and viewing history.

[1084] The "means for transmitting the generated highlight video to the user's terminal" is a function for transmitting the generated customized highlight video to the user's terminal.

[1085] The "means for providing the transmitted highlight video in a playable form to the user's terminal" is a function for providing the received highlight video in a playable form to the user's terminal.

[1086] The present invention is a system that integrates live streams, social media, AI commentary, highlight generation, and emotion recognition to enhance users' sporting event viewing experience.

[1087] The system program works as follows: First, a user selects a sporting event they want to watch using their device and sends the selection information to the server, which then starts a live stream related to the selected event and simultaneously collects social media posts using specific hashtags and keywords in real time.

[1088] The collected posts are analyzed by the server to identify peak times of excitement and emotion. Based on the identified peak times, the server then sends notifications to the user's device with the optimal timing for cheering. Furthermore, the emotion data collected from the user's device is analyzed by an emotion recognition engine, and the notification of the optimal timing for cheering is optimized based on the results.

[1089] The server receives the live-stream video data, and the AI ​​commentary assistant analyzes the progress of the match. The commentary text generated as a result of this analysis is sent to the user's device in real time, allowing the user to enjoy the commentary along with the video.

[1090] After the game ends, the server saves all game video data and runs a highlight generation engine to automatically extract important scenes. The generated highlight video is customized based on the user's settings and viewing history and sent to the user's device. The user's device then presents the received highlight video in a playable format.

[1091] The specific hardware and software required to realize this system include: Flask (a Python web framework) is used for the server; OpenCV and Dlib are used for sentiment analysis; HuggingFace Transformers is used for natural language processing; and the Twitter API is used for social media integration.

[1092] As a concrete example, consider the following scenario where a user is watching a basketball game. After the user selects a game and the live stream begins, the system collects related social media posts and identifies peak excitement times from tags like "TeamXWins." The emotion recognition engine analyzes the user's emotions and based on the results, notifies the user with "Team X is leading!", ensuring the user gets the best possible viewing experience.

[1093] An example of a prompt is as follows:

[1094] sports_event: "basketball"

[1095] social_media_hashtag: "TeamXWins"

[1096] emotion: "excited"

[1097] notify_action: "Team X is in the lead!"

[1098] As described above, the present invention is a system that highly personalizes a user's sports viewing experience and provides optimized cheering and commentary information in real time.

[1099] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1100] Step 1:

[1101] A user selects a sporting event they wish to watch using their device. The input is the event information selected by the user, and the output is the selection information sent to the server. Specifically, when the user operates the application interface to select an event and presses the "Watch" button, the selection information is sent to the server as a request.

[1102] Step 2:

[1103] The server processes requests for live streams of specified sporting events based on the selections received from the user. The input is the selections sent by the user, and the output is the URL and access information for the live stream. Specifically, the server sends a request to the live stream provider and prepares to receive the stream data.

[1104] Step 3:

[1105] The server collects social media posts related to a specified event in real time. The input is specific hashtags or keywords for the event, and the output is data on related social media posts. Specifically, it uses social media APIs (e.g., Twitter API) to collect related posts in real time.

[1106] Step 4:

[1107] The server analyzes collected social media posts and identifies peak times of excitement and emotion. The input is the collected post data, and the output is the identified peak time information. Specifically, it uses natural language processing technology to identify times when positive emotional expressions and reactions are increasing.

[1108] Step 5:

[1109] The server sends a notification to the user of the optimal time to cheer based on the identified peak times. The input is peak time information, and the output is a notification of the best time to cheer. Specifically, the server sends a notification to the user's device and displays a message such as "Now is your chance to cheer!"

[1110] Step 6:

[1111] The user's device receives the live stream and the notification of the cheering timing and provides it to the user. The input is the live stream URL and the cheering notification sent from the server, and the output is the stream video and notification displayed on the user's screen. Specifically, the application plays the stream and pops up a notification.

[1112] Step 7:

[1113] The user's device uses the device's camera and microphone to collect the user's emotional data in real time. The input is the user's facial expression data and voice data, and the output is emotional data sent to the emotion recognition engine. Specifically, the device processes the video captured by the camera and the voice recorded by the microphone.

[1114] Step 8:

[1115] The emotion recognition engine analyzes the user's emotional data and sends the results to the server. The input is the emotional data sent from the device, and the output is the emotional information as the analysis result. Specifically, it uses OpenCV and Dlib to analyze the user's facial expressions and voice.

[1116] Step 9:

[1117] The server further optimizes the timing of cheering notifications based on the analysis results obtained from the emotion recognition engine. The input is the analyzed emotion data, and the output is an optimized cheering notification. Specifically, the server adjusts the timing and content of notifications according to the emotion data.

[1118] Step 10:

[1119] The server receives the live-stream video data, runs the AI ​​commentary assistant, and analyzes the progress of the event. The input is the video data, and the output is commentary text. Specifically, the AI ​​commentary assistant analyzes players' movements and scoring scenes in real time and generates commentary text.

[1120] Step 11:

[1121] The explanatory text generated by the explanatory assistant is sent to the user's device in real time. The input is the generated explanatory text, and the output is the explanatory information displayed on the user's device. Specifically, the server sends the text data in real time, and the device displays it overlaid on the screen.

[1122] Step 12:

[1123] After the game ends, the server saves all game video data and runs a highlight generation engine. The input is game video data, and the output is a highlight clip containing important scenes. Specifically, the AI ​​engine automatically extracts important scenes such as goals and fouls and generates highlights.

[1124] Step 13:

[1125] A customized highlight video is generated based on the user's settings and viewing history and sent to the user's device. The input is the user's settings information and viewing history, and the output is a customized highlight video. Specifically, a video edited based on the individual user's preferences and past viewing history is created.

[1126] Step 14:

[1127] The user's device provides the transmitted highlight video in a playable format. The input is the highlight video data transmitted from the server, and the output is a playable highlight video. Specifically, the user's device plays the received video, allowing the user to easily review it.

[1128] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1129] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1130] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1131] [Third embodiment]

[1132] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1133] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1134] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1135] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1136] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1137] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1138] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1139] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1140] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1141] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1142] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1143] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1144] The present invention is a system that integrates live streaming with social media, real-time AI commentary, automatic highlight generation, and more to enhance users' sporting event viewing experience.

[1145] A way for users to select the event they want to watch and submit a request for a live stream

[1146] Users access the system using their terminals and select the sporting event they want to watch. The terminals then send the selection information to the server, requesting a live stream. As a result, users can watch their desired sporting event in real time.

[1147] A means by which the server collects social media posts related to a specified event in real time.

[1148] The server collects social media posts in real time using specific hashtags and keywords related to the designated event, and updates the collected posts as the event progresses and provides them to users.

[1149] A method for identifying peak times of excitement and emotion by analyzing collected posts

[1150] The server analyzes the collected social media posts and uses natural language processing technology to identify peak times of excitement and emotion, allowing it to understand in real time at which points during a game users are most excited.

[1151] A means of notifying users of the optimal timing for cheering based on identified peak times

[1152] The server then notifies the user of the optimal time to cheer based on the identified peak times. The notification is sent via the device and displayed on the screen, allowing the user to cheer while feeling a sense of unity with other fans.

[1153] A means for the server to receive the live stream video data and run a commentary assistant to analyze the progress of the match

[1154] The server receives live-stream video data in real time and runs an AI commentary assistant, which analyzes the progress of the game, understands player movements, goals scored, tactics, etc., and generates commentary text.

[1155] A means for transmitting the explanatory text generated by the explanatory assistant to the user's terminal in real time and displaying the explanatory text on the user's terminal.

[1156] The server transmits the generated commentary text to the user's device in real time, and the user's device displays the received commentary text overlaid on the live stream video, providing real-time commentary information to the user while watching the game.

[1157] After the match, the server saves all the match video data and runs a highlight generation engine to automatically extract important scenes.

[1158] After the game ends, the server stores all game video data, then runs a highlight generation engine to automatically extract important scenes (scores, fouls, good plays, etc.).

[1159] A means to generate customized highlight videos based on user preferences and viewing history

[1160] Users set their preferences and interests through their devices. The server references the user's settings and viewing history to generate a customized highlight video. The generated highlight video is then edited to suit the user's individual preferences.

[1161] A means for transmitting the generated highlight video to a user's terminal, and for the user's terminal to provide the transmitted highlight video in a reproducible form.

[1162] The server transmits the generated customized highlight video to the user's device, which then displays the received highlight video in a playable format, allowing the user to easily review and watch it.

[1163] Specific examples

[1164] Example 1: Real-time support

[1165] A user named "Yamada" uses a device to watch a basketball game. The server receives a request from Yamada's device and broadcasts the live stream while simultaneously collecting related social media posts. The server analyzes the posts, identifies moments of high excitement, and notifies Yamada of those moments. This allows Yamada to enjoy cheering at the most exciting moments.

[1166] Example 2: AI real-time commentary

[1167] A user named "Sasaki" is watching a soccer match. The server receives the live-stream video data, and the AI ​​commentary assistant analyzes the progress of the match and generates commentary text in real time. The commentary text is sent to Sasaki's device and displayed on the screen. This allows Sasaki to understand the flow of the match and the players' movements in detail.

[1168] Example 3: Automatic highlight generation

[1169] After a baseball game, user "Suzuki-san" wants to watch highlights of games he has previously watched. The server saves all video data at the end of the game and runs an AI highlight generation engine to extract important scenes. A customized highlight video is generated based on Suzuki-san's preferences and viewing history and sent to his device. Suzuki-san can easily enjoy highlights including his favorite scenes.

[1170] The above is an embodiment of the present invention. The present invention allows users to enjoy watching sports while feeling a sense of unity with other fans, and also allows users to gain a deeper understanding of sports through detailed commentary and customized highlights.

[1171] The processing flow will be explained below.

[1172] Processing steps for live streaming linked to social media in real time

[1173] Step 1:

[1174] A user uses a terminal to select a sporting event that they wish to watch.

[1175] Step 2:

[1176] The device sends a request for a live stream to the server.

[1177] Step 3:

[1178] The server receives a request for a live stream and retrieves the corresponding stream.

[1179] Step 4:

[1180] The server distributes the acquired live stream to the terminal, allowing the user to view it.

[1181] Step 5:

[1182] The server collects social media posts in real time using keywords and hashtags related to the specified event.

[1183] Step 6:

[1184] The server filters the collected posts and selects the most relevant posts.

[1185] Step 7:

[1186] The server analyzes the collected posts and identifies peak times of excitement and emotion.

[1187] Step 8:

[1188] The server generates data for notifying the user of the optimal cheering timing based on the identified peak time.

[1189] Step 9:

[1190] The server transmits the generated notification data to the terminal.

[1191] Step 10:

[1192] The notification received by the terminal is displayed on the screen to notify the user.

[1193] AI real-time commentary processing steps

[1194] Step 1:

[1195] The server receives the live stream video data in real time.

[1196] Step 2:

[1197] The server runs an AI commentary assistant and analyzes the video data.

[1198] Step 3:

[1199] The AI ​​commentary assistant recognizes the progress of the game, player movements, scoring scenes, etc.

[1200] Step 4:

[1201] The AI ​​commentary assistant generates explanatory text based on the analysis results.

[1202] Step 5:

[1203] The server transmits the generated explanatory text to the terminal in real time.

[1204] Step 6:

[1205] The explanatory text received by the terminal is displayed on the screen together with the live stream video and provided to the user.

[1206] Processing steps for automatic highlight generation

[1207] Step 1:

[1208] After the match ends, the server stores all the match video data.

[1209] Step 2:

[1210] The server inputs the saved video data into the AI ​​highlight generation engine.

[1211] Step 3:

[1212] The AI ​​highlight generation engine automatically extracts important scenes from the game (scores, fouls, good plays, etc.).

[1213] Step 4:

[1214] The user uses the device to set their preferences and interests.

[1215] Step 5:

[1216] The server refers to the user's setting information and past viewing history and determines a policy for generating a customized highlight video.

[1217] Step 6:

[1218] The server uses an AI highlight generation engine to generate a customized highlight video.

[1219] Step 7:

[1220] The server transmits the generated customized highlight video to the user's terminal.

[1221] Step 8:

[1222] The highlight video received by the terminal is provided to the user in a reproducible form.

[1223] Example 1

[1224] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1225] Conventional sports viewing systems have struggled to provide real-time commentary, improve the viewing experience, and share the excitement and emotion of users. Furthermore, it has been difficult to customize the post-match highlights based on individual user interests and viewing history. This has limited the enjoyment of sports viewing and prevented the user experience from being maximized.

[1226] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1227] In this invention, the server includes means for a user to select an event they wish to watch, means for sending a live stream request to the server, means for the server to collect social media posts related to the designated event in real time, means for analyzing the collected posts to identify peak times of excitement and emotion, means for notifying the user of the optimal cheering timing based on the identified peak times, means for providing the live stream and a notification of the cheering timing to the user's terminal, means for analyzing the social media posts using a generative AI model, means for notifying the user based on the peak times identified by the generative AI model, and a commentary assistant for the server to receive the live stream video data and analyze the progress of the match. The system includes a means for operating a game server, a means for transmitting commentary text generated by a commentary assistant to a user's device in real time, a means for displaying the commentary text on the user's device, a means for using a generative AI model for generating commentary text in real time, a means for a server to save all game video data after the game ends, a means for the server to operate a highlight generation engine and automatically extract important scenes, a means for generating a customized highlight video based on a user's settings and viewing history, a means for transmitting the generated highlight video to the user's device, a means for customizing and generating the highlight video using a generative AI model that uses the user's viewing history, and a means for the user's device to provide the transmitted highlight video in a playable form. This enables users to watch a game with in-depth commentary while sharing the excitement and emotion with other fans in real time, and to enjoy a highlight video that suits their interests after the game ends.

[1228] The "System" is an integrated mechanism that provides users with the optimal viewing experience by broadcasting the sporting events they want to watch in real time and collecting and analyzing related social media information.

[1229] "User" means an individual who uses the System to watch sporting events and receive services such as live streams, commentary, and highlight videos.

[1230] A "server" is a computer device that is the core of the system, and is responsible for processing requests from users, distributing live streams, analyzing data, generating highlights, and so on.

[1231] "Terminal" means a device used by a user to access the system and watch a sporting event, and includes a smartphone, tablet, PC, etc.

[1232] A "live stream" is video data of sporting events and performances broadcast in real time over the Internet.

[1233] "Social media" refers to a platform on the Internet where users can share and interact with each other, and includes Twitter, Facebook, Instagram, etc.

[1234] A "generative AI model" is a machine learning algorithm that is trained using artificial intelligence techniques to perform a specific task (e.g., text generation, sentiment analysis, etc.).

[1235] The "Commentary Assistant" is an artificial intelligence system that runs on a server, analyzes live-stream video data, and generates commentary text in real time.

[1236] The "highlight generation engine" is a software component that analyzes video data from matches, automatically extracts and edits important scenes, and generates highlight videos.

[1237] "Viewing history" is a record of sporting events that a user has viewed in the past and their contents.

[1238] The present invention provides a system that integrates live streaming, social media, real-time commentary using generative AI models, and automatic highlight generation to enhance users' sporting event viewing experience.

[1239] First, a user accesses the system using a terminal and selects the sporting event they want to watch. At this time, the terminal sends the user's selection information to the server and requests a live stream. The server receives the request and provides live footage of the specified sporting event.

[1240] The server then collects social media posts in real time using specific hashtags or keywords related to the specified event. These posts are retrieved using social media APIs (e.g., Twitter API, Facebook API). The collected posts are then analyzed using a generative AI model to identify peak times of excitement and emotion.

[1241] The server notifies users of the best time to cheer based on the identified peak times. The notification is sent via a pop-up message or notification bar on the user's device, allowing users to send cheering messages and share their excitement at the best possible time.

[1242] The server also receives live-stream video data in real time and runs a commentary assistant to analyze the progress of the match. The commentary assistant analyzes the player movements, goals scored, tactics, and other aspects of the video, and generates commentary text in real time. The generated commentary text is sent to the user's device and overlaid on the live video.

[1243] After the game ends, the server saves all game video data and runs a highlight generation engine. The highlight generation engine automatically extracts and edits important scenes, such as goals and great plays, to generate a customized highlight video. The customization is based on the user's settings and viewing history, and the generated highlight video is sent to the user's device.

[1244] As a concrete example, consider a user named "Yamada" using a device to watch a basketball game. The server receives a request from Yamada's device and delivers the live stream while simultaneously collecting and analyzing related social media posts. The server identifies the moments when excitement levels rise and notifies Yamada, allowing him to enjoy cheering without missing the most exciting moments.

[1245] In addition, when a user named "Sasaki" is watching a soccer match, the server receives the live-stream video data and uses the generative AI model to generate and transmit commentary text in real time, allowing Sasaki to watch the match while gaining a detailed understanding of the progress of the match.

[1246] Furthermore, if user "Suzuki-san" wants to watch highlights after a baseball game, the server saves all video data at the end of the game and activates the AI ​​highlight generation engine. A customized highlight video is generated and sent based on Suzuki-san's preferences and viewing history. Suzuki-san can easily watch highlight videos containing important scenes.

[1247] Example prompt sentence:

[1248] "How can Yamada know the optimal timing to cheer while watching a basketball game in real time?"

[1249] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1250] Step 1:

[1251] A user accesses the system using a terminal and selects the sporting event they wish to watch.

[1252] Input: User event selection information

[1253] Specific operation: The user selects the menu or list on the device screen, selects the event they want to watch, and confirms the selection by pressing the Enter key or touching the screen.

[1254] Output: Event selection information is recorded on the terminal and sent to the server.

[1255] Step 2:

[1256] The terminal transmits the user's selection information to the server.

[1257] Input: User event selection information

[1258] What it does: Your device sends the selected information to a server over the internet, using a secure protocol such as HTTPS to protect the data.

[1259] Output: The selection information is received by the server.

[1260] Step 3:

[1261] The server processes the live stream request based on the received selection information and provides the live video to the user.

[1262] Input: Selection information

[1263] Specific operation: The server accesses the streaming service, acquires live video of the specified event, and streams the acquired video data to the user's device.

[1264] Output: The live stream video is displayed on the user's device.

[1265] Step 4:

[1266] The server collects social media posts related to a specified event in real time.

[1267] Input: Event hashtags or keywords

[1268] How it works: The server uses social media APIs to collect posts related to the specified hashtags and keywords. The collection process is done in real time.

[1269] Output: Collected social media post data

[1270] Step 5:

[1271] The server analyzes the collected posts using a generative AI model to identify peak times of excitement and emotion.

[1272] Input: Collected submission data

[1273] How it works: Using a generative AI model, post data is analyzed and scored for excitement and emotional impact. Peak times are identified based on the scoring results.

[1274] Output: Identified peak times of excitement and emotion

[1275] Step 6:

[1276] The server notifies the user of the optimal cheering timing based on the identified peak time.

[1277] Input: Identified peak times

[1278] Specific operation: The server sends a notification message to the user's device, causing a pop-up message or notification bar to appear on the user's screen.

[1279] Output: Notification of the cheering timing displayed on the device screen

[1280] Step 7:

[1281] The server receives the live stream video data in real time, runs the commentary assistant, and analyzes the progress of the match.

[1282] Input: Live stream video data

[1283] Specific operation: The server analyzes the video data using an AI commentary assistant (e.g., image recognition model and natural language processing model).

[1284] Output: Real-time generated explanatory text

[1285] Step 8:

[1286] The server transmits the generated commentary text to the user's terminal in real time, and the commentary text is displayed on the user's terminal.

[1287] Input: Generated description text

[1288] Specific operation: The server sends explanatory text to the user's device, which then overlays the received explanatory text on the live video.

[1289] Output: Descriptive text that is displayed on the device screen

[1290] Step 9:

[1291] After the match ends, the server stores all match video data and runs a highlight generation engine to automatically extract important scenes.

[1292] Input: Match video data

[1293] How it works: The server stores the video data in a database, then uses a highlight generation engine to automatically extract and edit important scenes (scoring scenes, great plays, etc.).

[1294] Output: Clips of extracted key scenes

[1295] Step 10:

[1296] The server generates a customized highlight video based on the user's settings and viewing history and transmits it to the user's terminal.

[1297] Input: User settings and viewing history, extracted clips of important scenes

[1298] Specific operation: The server references the user's settings and viewing history, combines clips of key scenes, and generates a customized highlight video. The generated video is then sent to the user's device.

[1299] Output: A customized highlight video sent to the user's device

[1300] Step 11:

[1301] To provide a highlight video received by a user terminal in a reproducible form.

[1302] Input: Custom highlight video

[1303] Specific operation: The device uses an application (e.g., a video player) to play the received highlight video and displays it in a playable format for the user.

[1304] Output: A highlight video that can be viewed by the user

[1305] (Application example 1)

[1306] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1307] Conventional sports viewing systems lack real-time commentary, highlight generation, and notification functions that allow viewers to cheer on the most exciting moments. This makes it difficult for users to gain a deeper understanding of the content and enjoy it without missing any exciting moments. Furthermore, they do not provide customized highlight videos tailored to individual users' preferences. Therefore, a system that integrates these functions is needed to improve the user experience.

[1308] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1309] In this invention, the server includes means for a user to select content they wish to view, means for sending a streaming request to the server, means for the server to collect online posts related to the specified content in real time, means for analyzing the collected posts to identify peak times of excitement and emotion, means for notifying the user of the optimal timing for cheering based on the identified peak times, and means for providing streaming and notification of the timing for cheering to the user's information terminal. This allows the user to know the timing for cheering in real time, watch the game with a sense of unity with other viewers, and enjoy customized highlights that suit their interests.

[1310] "Means for a user to select content they wish to view" refers to the interface or process used to select the particular content a user wishes to view.

[1311] The "means for sending a streaming request to the server" refers to a mechanism for requesting the server to stream content based on the content selected by the user.

[1312] "Means for the server to collect Internet posts related to specified content in real time" refers to the function of the server to obtain posts and comments related to specific content on the Internet in real time.

[1313] "Means of analyzing collected posts to identify peak times of excitement and emotion" refers to the process of analyzing collected internet posts to identify the times when users are most excited or moved.

[1314] "Means for notifying users of the optimal timing to cheer based on identified peak times" refers to a system that notifies users of the optimal timing to cheer based on peak times of excitement and emotion identified through analysis.

[1315] "Means for providing streaming and cheering timing notifications to a user's information terminal" refers to a function for delivering streaming data and cheering timing information to a device used by a user.

[1316] "Commentary assistant for analyzing content progress" refers to AI or software that analyzes the progress of content being streamed in real time and generates commentary based on that content.

[1317] "Means for transmitting explanatory text generated by the explanatory assistant in real time to the user's information terminal" refers to a process for instantly transmitting explanatory text generated by the explanatory assistant to the user's device.

[1318] "Means for displaying explanatory text on the user's information terminal" refers to a mechanism for displaying explanatory text on the screen of the user's device.

[1319] "Means for the server to save all video data after the end of the match" refers to the function of storing all streamed video data on the server after the end of the match.

[1320] "Means for the server to operate the highlight generation engine and automatically extract important scenes" refers to a system that uses the highlight generation engine in the server to automatically select important scenes from video data.

[1321] The "means for generating a customized highlight video based on user settings and viewing history" refers to a function for generating an individually customized highlight video based on preferences set by the user and past viewing history.

[1322] "Means for transmitting the generated highlight video to the user's information terminal" refers to a process for transmitting the customized highlight video to the user's device.

[1323] The "means for providing the transmitted highlight video in a format that can be played back by the user's information terminal" refers to a system that provides the transmitted highlight video in a format that can be played back by the user's device.

[1324] This invention is a system for improving a user's sports viewing experience, utilizing a server, a user's information terminal, and AI technology. This system is implemented in the following steps.

[1325] First, a user selects the content they want to watch using an information terminal such as a smartphone or tablet. The selection information is sent from the terminal to the server, and the server receives a streaming request based on that information. The server then begins streaming the specified content.

[1326] The server then collects real-time internet posts related to the specified content, including searches using specific hashtags or keywords, and analyzes these posts using natural language processing techniques to identify peak times of excitement and emotion.

[1327] Once the data collection and analysis is complete, the server notifies users of the optimal time to cheer based on the identified peak times. The notification is sent to the user's information terminal and displayed on the screen in real time.

[1328] The server also runs an AI commentary assistant to analyze the streaming video. The AI ​​commentary assistant analyzes the progress of the game in real time and generates commentary text about players' movements, scoring scenes, etc. The generated commentary text is sent in real time to the user's information terminal and displayed on the screen.

[1329] After the game ends, the server saves all video data and runs a highlight generation engine. The highlight generation engine automatically extracts important scenes and generates a customized highlight video based on the user's settings and viewing history. This highlight video is sent to the user's information terminal, allowing the user to watch the highlights at any time.

[1330] The hardware and software used include the following: Hardware includes information terminals such as smartphones and tablets, and servers. Software includes TensorFlow (AI model), Flask / Django (server-side framework), OpenCV (video data analysis), Requests (HTTP request processing), and NLTK (natural language processing) used on the server side.

[1331] As a concrete example, consider the case where a user named "Suzuki" wants to watch a basketball game. Suzuki uses his smartphone to select the game he wants to watch and sends a request to the server. The server receives the request and provides live streaming, while also collecting and analyzing related posts from social media to identify peak times of excitement. The server notifies Suzuki of these peak times and also provides commentary text generated by an AI commentary assistant in real time. After the game ends, the server extracts key scenes and generates and sends customized highlights tailored to Suzuki's preferences.

[1332] An example of a prompt sentence might be:

[1333] "While watching a basketball game in real time, generate a code to identify and notify you of peak moments of excitement from social media."

[1334] In this way, the present invention can significantly improve the sports viewing experience for users.

[1335] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1336] Step 1:

[1337] The user selects the content they wish to view using an information terminal. The input is the user's viewing preference, and the output is the selected content information. The interface used by the user is intuitive, and selection can be easily made using lists and search functions.

[1338] Step 2:

[1339] A streaming request is sent from the information terminal to the server. The input is the selected content information, and the output is a confirmation of the request from the server. This information sent as an HTTP request triggers the server to start streaming.

[1340] Step 3:

[1341] The server starts streaming the specified content. The input is the requested content information, and the output is live streaming data. The server encodes the video data in real time and sends it to the user's information terminal.

[1342] Step 4:

[1343] The server collects internet posts related to the specified content in real time. The input is a specific hashtag or keyword, and the output is the post data. The server uses APIs to retrieve related posts from social media and stores them in a database.

[1344] Step 5:

[1345] The server analyzes the collected posts to identify peak times of excitement and emotion. The input is the collected post data, and the identified peak times are extracted as the output. Natural language processing technology (e.g., NLTK) is used to analyze the text data and evaluate emotions and excitement levels.

[1346] Step 6:

[1347] The server notifies the user of the optimal cheering timing based on the identified peak times. The input is the identified peak time information, and the output is generated notification data. The server generates real-time notifications and sends them to the user's information terminal.

[1348] Step 7:

[1349] The server runs an AI commentary assistant to analyze streaming video. The input is real-time video data, and the output is explanatory text. The TensorFlow model is used to analyze the video data and generate explanatory text according to the progress.

[1350] Step 8:

[1351] The server generates explanatory text and sends it to the user's information terminal in real time. The input is the generated explanatory text, and the output is a confirmation of transmission. The explanatory text is distributed along with the video data.

[1352] Step 9:

[1353] The commentary text is displayed on the user's information terminal. The input is the received commentary text, and the output is the displayed text. It is displayed overlaid on the screen of the user's information terminal to aid viewing.

[1354] Step 10:

[1355] After the game ends, the server saves all video data. The input is all video data from the game, and the output is a saved data file. The server saves the data in cloud storage such as S3.

[1356] Step 11:

[1357] The server runs a highlight generation engine to automatically extract important scenes. The input is stored video data, and the output is extracted highlight scenes. OpenCV is used to perform video analysis and extract scenes based on specific events or actions.

[1358] Step 12:

[1359] The server generates a customized highlight video based on the user's settings and viewing history. The input is the user's settings and viewing history, and the output is a customized highlight video. An AI model is used to analyze the user's preferences and combine the optimal scenes.

[1360] Step 13:

[1361] The server sends the generated highlight video to the user's information terminal. The input is the customized highlight video, and the output is a confirmation of transmission. The video data is provided to the user in a format that is easy to view.

[1362] Step 14:

[1363] The user's information terminal provides the transmitted highlight video in a playable format. The input is the received highlight video, and the output is a played video. The user can enjoy the highlight video containing their favorite scenes.

[1364] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1365] This invention is a system that integrates live streaming with social media, AI-powered real-time commentary, automatic highlight generation, and an emotion engine that recognizes user emotions to enhance users' sporting event viewing experience.

[1366] A way for users to select the event they want to watch and submit a request for a live stream

[1367] Users access the system using their terminals and select the sporting event they want to watch. The terminals then send the selection information to the server, requesting a live stream. As a result, users can watch their desired sporting event in real time.

[1368] A means by which the server collects social media posts related to a specified event in real time.

[1369] The server collects social media posts in real time using specific hashtags and keywords related to the designated event, and updates the collected posts as the event progresses and provides them to users.

[1370] A method for identifying peak times of excitement and emotion by analyzing collected posts

[1371] The server analyzes the collected social media posts and uses natural language processing technology to identify peak times of excitement and emotion, allowing it to understand in real time at which points during a game users are most excited.

[1372] A means of notifying users of the optimal timing for cheering based on identified peak times

[1373] The server then notifies the user of the optimal time to cheer based on the identified peak times. The notification is sent via the device and displayed on the screen, allowing the user to cheer while feeling a sense of unity with other fans.

[1374] A means for the server to receive the live stream video data and run a commentary assistant to analyze the progress of the match

[1375] The server receives live-stream video data in real time and runs an AI commentary assistant, which analyzes the progress of the game, understands player movements, goals scored, tactics, etc., and generates commentary text.

[1376] A means for transmitting the explanatory text generated by the explanatory assistant to the user's terminal in real time and displaying the explanatory text on the user's terminal.

[1377] The server transmits the generated commentary text to the user's device in real time, and the user's device displays the received commentary text overlaid on the live stream video, providing real-time commentary information to the user while watching the game.

[1378] After the match, the server saves all the match video data and runs a highlight generation engine to automatically extract important scenes.

[1379] After the game ends, the server stores all game video data, then runs a highlight generation engine to automatically extract important scenes (scores, fouls, good plays, etc.).

[1380] A means to generate customized highlight videos based on user preferences and viewing history

[1381] Users set their preferences and interests through their devices. The server references the user's settings and viewing history to generate a customized highlight video. The generated highlight video is then edited to suit the user's individual preferences.

[1382] A means for transmitting the generated highlight video to a user's terminal, and for the user's terminal to provide the transmitted highlight video in a reproducible form.

[1383] The server transmits the generated customized highlight video to the user's device, which then displays the received highlight video in a playable format, allowing the user to easily review and watch it.

[1384] Processing by emotion engine that recognizes user emotions

[1385] The user's device is equipped with a camera and microphone, which are used to collect the user's emotional data in real time. The emotion engine reads the user's facial expressions using the device's camera and analyzes the tone and strength of the voice using the microphone. The emotion engine then sends the collected emotional data to the server.

[1386] Server analyzes emotional data and adjusts cheering timing

[1387] The server analyzes the emotion data sent from the emotion engine and evaluates the user's level of excitement and emotion in real time. Based on this evaluation, the server further optimizes the timing of cheering notifications and supports the user's cheering.

[1388] Specific examples

[1389] Example 1: Real-time cheering and emotion recognition

[1390] A user named "Yamada" uses a device to watch a basketball game. The server receives a request from Yamada's device and broadcasts the live stream while simultaneously collecting related social media posts. The server analyzes the posts, identifies moments of high excitement, and notifies Yamada of those moments. Furthermore, the emotion engine analyzes Yamada's facial expressions and voice to provide more accurate cheering timing.

[1391] Example 2: AI real-time commentary and emotion recognition

[1392] A user named "Sasaki" is watching a soccer match. The server receives the live-stream video data, and the AI ​​commentary assistant analyzes the progress of the match and generates commentary text in real time. The commentary text is sent to Sasaki's device and displayed on the screen. Furthermore, the emotion engine analyzes Sasaki's emotions and provides additional commentary based on the moments of excitement.

[1393] Example 3: Automatic highlight generation and emotion recognition

[1394] After a baseball game, user "Suzuki-san" wants to watch highlights of games he has previously watched. The server saves all video data at the end of the game and runs an AI highlight generation engine to extract important scenes. A customized highlight video is generated based on Suzuki-san's preferences and viewing history and sent to his device. The emotion engine analyzes Suzuki-san's emotions while watching the highlights and further emphasizes scenes that are exciting.

[1395] The above is an embodiment of the present invention. The present invention allows users to enjoy watching sports while feeling a sense of unity with other fans, and also allows for a deeper understanding of sports through detailed commentary and customized highlights. Furthermore, by analyzing users' real-time emotions, a more personalized cheering and viewing experience can be realized.

[1396] The processing flow will be explained below.

[1397] Processing steps for live streaming linked to social media in real time

[1398] Step 1:

[1399] A user uses a terminal to select a sporting event that they wish to watch.

[1400] Step 2:

[1401] The device sends a request for a live stream to the server.

[1402] Step 3:

[1403] The server receives a request for a live stream and retrieves the corresponding stream.

[1404] Step 4:

[1405] The server distributes the acquired live stream to the terminal, allowing the user to view it.

[1406] Step 5:

[1407] The server collects social media posts in real time using keywords and hashtags related to the specified event.

[1408] Step 6:

[1409] The server filters the collected posts and selects the most relevant posts.

[1410] Step 7:

[1411] The server analyzes the collected posts and identifies peak times of excitement and emotion.

[1412] Step 8:

[1413] The server generates data for notifying the user of the optimal cheering timing based on the identified peak time.

[1414] Step 9:

[1415] The server transmits the generated notification data to the terminal.

[1416] Step 10:

[1417] The notification received by the terminal is displayed on the screen to notify the user.

[1418] AI real-time commentary processing steps

[1419] Step 1:

[1420] The server receives the live stream video data in real time.

[1421] Step 2:

[1422] The server runs an AI commentary assistant and analyzes the video data.

[1423] Step 3:

[1424] The AI ​​commentary assistant recognizes the progress of the game, player movements, scoring scenes, etc.

[1425] Step 4:

[1426] The AI ​​commentary assistant generates explanatory text based on the analysis results.

[1427] Step 5:

[1428] The server transmits the generated explanatory text to the terminal in real time.

[1429] Step 6:

[1430] The explanatory text received by the terminal is displayed on the screen together with the live stream video and provided to the user.

[1431] Processing steps for automatic highlight generation

[1432] Step 1:

[1433] After the match ends, the server stores all the match video data.

[1434] Step 2:

[1435] The server inputs the saved video data into the AI ​​highlight generation engine.

[1436] Step 3:

[1437] The AI ​​highlight generation engine automatically extracts important scenes from the game (scores, fouls, good plays, etc.).

[1438] Step 4:

[1439] The user uses the device to set their preferences and interests.

[1440] Step 5:

[1441] The server refers to the user's setting information and past viewing history and determines a policy for generating a customized highlight video.

[1442] Step 6:

[1443] The server uses an AI highlight generation engine to generate a customized highlight video.

[1444] Step 7:

[1445] The server transmits the generated customized highlight video to the user's terminal.

[1446] Step 8:

[1447] The highlight video received by the terminal is provided to the user in a reproducible form.

[1448] Processing steps combining emotion engines

[1449] Step 1:

[1450] When a user uses a device to watch an event, the device's camera and microphone are activated.

[1451] Step 2:

[1452] The device uses a camera to recognize the user's facial expressions and a microphone to analyze the tone and strength of the voice.

[1453] Step 3:

[1454] The emotion data acquired by the device is sent to the server in real time.

[1455] Step 4:

[1456] The server receives the emotion data and analyzes it.

[1457] Step 5:

[1458] The server evaluates the user's excitement level and changes in emotions.

[1459] Step 6:

[1460] The server adjusts the timing of cheering notifications based on the results of emotion data analysis.

[1461] Step 7:

[1462] The server transmits the adjusted cheering timing to the terminal.

[1463] Step 8:

[1464] The device displays a notification of the adjusted cheering timing to the user.

[1465] Specific examples

[1466] Example 1: Real-time cheering and emotion recognition

[1467] Step 1:

[1468] A user "Yamada" uses the terminal to watch a basketball game.

[1469] Step 2:

[1470] The device sends a request for a live stream to the server, and the server delivers the stream.

[1471] Step 3:

[1472] The server collects and analyzes social media posts to identify peak times of excitement.

[1473] Step 4:

[1474] The device's camera and microphone capture Yamada's emotions in real time.

[1475] Step 5:

[1476] The emotion data is sent to a server and analyzed.

[1477] Step 6:

[1478] Based on the analysis results, the server will notify Yamada of the optimal timing to cheer him on.

[1479] Step 7:

[1480] The device will display the optimal timing for cheering to Yamada, encouraging him to cheer in real time.

[1481] Example 2: AI real-time commentary and emotion recognition

[1482] Step 1:

[1483] User "Sasaki-san" starts watching a soccer game.

[1484] Step 2:

[1485] The server receives the live stream footage and the AI ​​commentary assistant begins analyzing it.

[1486] Step 3:

[1487] The explanatory text generated by the commentary assistant is sent to Sasaki's device in real time.

[1488] Step 4:

[1489] The device's camera and microphone capture Sasaki's emotional data.

[1490] Step 5:

[1491] The emotion data is sent to a server and analyzed.

[1492] Step 6:

[1493] The server provides additional commentary and highlights based on the results of Sasaki's sentiment analysis.

[1494] Step 7:

[1495] The device displays additional commentary information to Sasaki, helping him better understand the match.

[1496] Example 3: Automatic highlight generation and emotion recognition

[1497] Step 1:

[1498] After the baseball game ends, user "Suzuki" wants to watch the highlights.

[1499] Step 2:

[1500] The server stores all video data from the match and runs the AI ​​highlight generation engine.

[1501] Step 3:

[1502] After extracting important scenes, Suzuki's device sends his preferences and viewing history to the server.

[1503] Step 4:

[1504] The server generates a customized highlight video and transmits it to the device.

[1505] Step 5:

[1506] The device's camera and microphone capture Suzuki's emotional data.

[1507] Step 6:

[1508] The emotion data is sent to a server and analyzed.

[1509] Step 7:

[1510] The server provides emotional feedback according to the scene of interest, further emphasizing the highlights.

[1511] Step 8:

[1512] The device will provide Suzuki with the adjusted highlight video in a playable format.

[1513] Example 2

[1514] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1515] Conventional sports viewing systems often only provide simple live streaming, resulting in a one-way viewing experience for users. They also lack real-time information and commentary, and lack advanced features, particularly those utilizing social media and AI technology, preventing services from responding to users' emotions and excitement. Furthermore, highlight videos generated after the game are often generated manually, making it difficult to customize them based on individual user preferences and viewing history.

[1516] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1517] In this invention, the server includes: a means for a user to select an event they wish to watch; a means for sending a live stream request to the server; a means for the server to collect social media posts related to the specified event in real time; a means for analyzing the collected posts to identify peak times of excitement and emotion; a means for notifying the user of the optimal cheering timing based on the identified peak times; a means for providing the live stream and a notification of the cheering timing to the user's device; a means for recognizing the user's emotions in real time; and a means for analyzing the user's emotion data and adjusting the cheering timing. This allows users to enjoy a real-time, interactive viewing experience, and enables the provision of advanced information and commentary utilizing social media and AI technologies. Furthermore, highlights generated after the game are automated, allowing for customization based on the user's preferences and interests.

[1518] The "means for a user to select an event that the user wishes to watch" refers to an interface and functionality that allows a user to use a terminal to search for and select a sporting event that the user wishes to watch.

[1519] "Means for sending a request for a live stream to a server" refers to the communications capabilities and protocols for sending a request for a live stream to a server based on a user-selected event.

[1520] "Means by which the server collects social media posts related to a specified event in real time" refers to algorithms and data collection systems for collecting relevant posts from social media using specific hashtags or keywords.

[1521] "Means for analyzing collected posts to identify peak times of excitement or emotion" refers to technology and processing for analyzing social media posts using natural language processing technology and identifying peak times of excitement or emotion among users.

[1522] "Means for notifying users of the optimal timing for cheering based on identified peak times" refers to a communication function and notification system for notifying users of cheering messages at the optimal timing based on peak times of excitement and emotion.

[1523] "Means for providing a live stream and cheering timing notifications to a user's terminal" refers to a distribution system and application function for providing a live stream video and cheering timing notifications to a user's terminal.

[1524] "Means for recognizing a user's emotions in real time" refers to the technology and functions for collecting a user's facial expressions and voice in real time using the device's camera and microphone, and recognizing their emotions.

[1525] "Means for analyzing user emotional data and adjusting the timing of cheering" refers to an algorithm and processing system for analyzing collected user emotional data and optimizing the timing of cheering based on the analysis results.

[1526] "Means for receiving live-stream video data and operating a commentary assistant to analyze the progress of the match" refers to the function of receiving live-stream video data in real time and operating an AI commentary assistant that analyzes the progress of the match based on that data.

[1527] "Means for transmitting the explanatory text generated by the explanation assistant to the user's device in real time" refers to the communication function and data transmission system for transmitting the explanatory text generated by the AI ​​explanation assistant to the user's device in real time.

[1528] "Means for the server to store all game video data after the game ends" refers to a data storage and management system for storing all game video data on the server after the game ends.

[1529] "Means for operating a highlight generation engine and automatically extracting important scenes" refers to a technology and processing system for operating a highlight generation engine and automatically extracting important scenes from a match.

[1530] The "means for generating a customized highlight video based on user settings and viewing history" refers to algorithms and processing functions for generating a customized highlight video based on user preferences and viewing history.

[1531] The "means for transmitting the generated highlight video to the user's terminal" refers to a communication function and a data transmission system for transmitting the generated customized highlight video to the user's terminal.

[1532] The present invention provides a system that allows users to watch sporting events in real time and enhance their viewing experience. The system includes three main components: a user, a terminal, and a server. Each component plays a specific role and works together to enhance the user experience.

[1533] System Configuration

[1534] User terminal

[1535] Users use devices such as smartphones, tablets, and PCs (hereafter referred to as "terminals"). Terminals include an internet connection, camera, microphone, and display. Terminals access the system through a browser or dedicated application.

[1536] server

[1537] The server is located on the cloud infrastructure and is responsible for collecting, analyzing, distributing and storing data. The server has the following main functions:

[1538] Live Streaming

[1539] Social media data collection and analysis

[1540] AI commentary assistant

[1541] Highlight Generation Engine

[1542] Emotion Recognition Engine

[1543] Main processing

[1544] Request and deliver a live stream

[1545] The user selects the event they want to watch and sends a live stream request to the server via their device. The server receives the request and delivers the live stream of the event. The live stream data is sent to the user's device in real time and displayed on the screen.

[1546] Social media data collection and analysis

[1547] The server collects social media posts in real time using hashtags and keywords related to a specified event, then analyzes the posts using natural language processing (NLP) techniques to identify peak times of user excitement and emotion.

[1548] Cheering timing notification

[1549] Based on the analysis results, the server calculates the optimal cheering timing according to the identified peak times and notifies the user's device, allowing users to feel a sense of unity with other fans and cheer more enthusiastically.

[1550] AI commentary assistant

[1551] The server receives live-stream video data in real time and runs an AI commentary assistant. The AI ​​commentary assistant analyzes the progress of the game, players' movements, scoring scenes, etc., and generates commentary text. This text is sent to the device in real time and displayed overlaid on the video.

[1552] Highlight Generation

[1553] After the game ends, the server stores all game video data and runs a highlight generation engine, which generates a customized highlight video based on the user's settings and viewing history, and sends it to the device for playback.

[1554] Emotion recognition and cheer timing adjustment

[1555] The user's device is equipped with a camera and microphone, which are used to collect the user's emotional data in real time. The emotion engine analyzes facial expressions and tone of voice and sends the results to the server. The server uses this data to calculate the optimal timing for cheering, further improving the user's cheering experience.

[1556] Specific examples

[1557] Example 1: Real-time cheering and emotion recognition

[1558] User "A" uses a device to watch a basketball game. The server receives a request from A's device and delivers the live stream while simultaneously collecting related social media posts. The server analyzes the posts, identifies moments of high excitement, and notifies A of those moments. Furthermore, the emotion engine analyzes A's facial expressions and voice to provide more accurate cheering timing.

[1559] Example 2: AI real-time commentary and emotion recognition

[1560] User "B" is watching a soccer match. The server receives the live-stream video data, and the AI ​​commentary assistant analyzes the progress of the match and generates commentary text. The text is sent to B's device and displayed on the screen. Furthermore, the emotion engine analyzes B's emotions and provides additional commentary at times when B is excited.

[1561] Example 3: Automatic highlight generation and emotion recognition

[1562] After a baseball game, user "C" wants to watch highlights of games he previously watched. The server saves all video data at the end of the game and runs an AI highlight generation engine to extract important scenes. A customized highlight video based on C's preferences and viewing history is sent to C's device. The emotion engine analyzes C's emotions while watching the highlights and further emphasizes scenes that are exciting.

[1563] Prompt Sentence Examples

[1564] Imagine you are watching a basketball game. Describe a system in which an AI commentary assistant generates commentary in real time and notifies the user of the optimal timing to cheer based on social media data. It also analyzes user sentiment in real time and uses that information to optimize cheering and commentary.

[1565] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1566] Processing Step Description

[1567] Step 1:

[1568] Users select the event they want to watch

[1569] Input: The user accesses the device's interface, searches for and selects the sporting event they wish to watch.

[1570] Data processing / calculation: The terminal formats the user's selected information and prepares it as request data for the server.

[1571] Output: The terminal sends live stream request data to the server.

[1572] Specific operation: The user selects "basketball game" on the interface, and the device sends that information to the server as a "live stream request."

[1573] Step 2:

[1574] The server receives a request for a live stream and starts broadcasting it.

[1575] Input: The server receives live stream request data from the terminal.

[1576] Data processing / calculation: The server analyzes the request content and obtains the distribution URL and data of the corresponding live stream.

[1577] Output: The server sends the live stream data to the user's device.

[1578] Specific operation: The server receives the request, generates a live streaming URL, and starts sending the stream data to the device.

[1579] Step 3:

[1580] The server collects and analyzes relevant posts from social media.

[1581] Input: The server receives as input hashtags or keywords related to the specified event (e.g., "basketball game" or "score").

[1582] Data processing / calculation: Based on these hashtags and keywords, the server collects data from social media in real time and analyzes it using natural language processing (NLP) technology.

[1583] Output: The analyzed social media data is accumulated and peak time information on excitement and emotion is generated.

[1584] Specific operation: The server analyzes collected social media posts and assigns them a rating score based on excitement and emotion.

[1585] Step 4:

[1586] The server identifies and notifies you of peak times

[1587] Input: Social media data analyzed by NLP techniques.

[1588] Data processing / calculation: The server uses an algorithm to identify peak times based on the analysis results and calculate the optimal timing for cheering.

[1589] Output: The identified cheering timing notification data is generated and sent to the user's terminal.

[1590] Specific operation: The server notifies the device of the timing to cheer based on a score such as "excitement level 80%."

[1591] Step 5:

[1592] The server receives the live stream video and runs the AI ​​commentary assistant.

[1593] Input: Live stream video data.

[1594] Data processing / calculation: The AI ​​commentary assistant analyzes the video data and generates commentary text about the progress of the match and specific scenes.

[1595] Output: The generated explanatory text data is sent to the user's terminal.

[1596] Specific operation: The AI ​​generates explanatory text such as "Player A scored a goal!" and sends it to the device.

[1597] Step 6:

[1598] Recognize user emotions in real time and collect data

[1599] Input: User's facial expression data and voice data obtained from the device's camera and microphone.

[1600] Data processing / calculation: The emotion recognition engine analyzes facial expressions and voice data to identify emotional states (e.g., "excited" or "relaxed").

[1601] Output: The analyzed emotion data is sent to the server.

[1602] Specific operation: The camera recognizes the user's facial expressions such as "smile" and "surprise," and the microphone analyzes the "tone" and "pitch" of the voice.

[1603] Step 7:

[1604] The server analyzes the emotional data and adjusts the timing of cheering.

[1605] Input: User emotion data sent from the device.

[1606] Data processing / calculation: The server uses an algorithm to dynamically adjust the timing of cheering based on emotional data.

[1607] Output: The re-adjusted cheer timing notification data is sent to the user's device.

[1608] Specific operation: Based on the "excitement level 90%", the server further optimizes the cheering message and notifies the user.

[1609] Step 8:

[1610] The server stores the match video data, runs the highlight generation engine, and sends it to the user.

[1611] Input: All game video data after the game has finished.

[1612] Data processing / calculation: The highlight generation engine automatically extracts important scenes and generates a customized highlight video based on the user's settings and viewing history.

[1613] Output: The generated customized highlight video is sent to the user's device and displayed in a playable format.

[1614] Specific operation: The server extracts important parts such as "goal scenes" and generates and sends a customized highlight video for users who have set "I want to watch only goal scenes."

[1615] The above is a detailed flow of the system's processing steps, which allows users to enjoy a fulfilling real-time sports viewing experience.

[1616] (Application example 2)

[1617] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1618] In modern sports viewing, users want to enjoy the event in real time while feeling a sense of unity with other fans, and they want detailed commentary. However, existing systems have separate functions for live streaming, commentary, highlight generation, and emotion recognition, making it difficult to provide an integrated experience. Furthermore, they are unable to adequately optimize the timing of cheering based on user emotions. This results in a problem where users are unable to get the best possible viewing experience.

[1619] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for selecting an event the user wants to watch; means for sending a live stream request to the server; means for the server to collect social media posts related to the specified event in real time; means for analyzing the collected posts to identify peak times of excitement and emotion; means for notifying the user of the optimal cheering timing based on the identified peak times; means for providing the live stream and a cheering timing notification to the user's device; means for providing an emotion recognition engine for analyzing user emotion data collected from the user's device; and means for further optimizing the cheering timing notification based on the analysis results of the emotion recognition engine. This allows the user to watch the game while feeling a sense of unity with other fans, and detailed commentary and customized highlights are also provided, allowing for a deeper enjoyment of sports. Furthermore, a cheering experience that is individually optimized through real-time emotion analysis is provided.

[1620] The "means for the user to select an event that the user wishes to watch" is an interface that allows the user to select a sport event that the user wishes to watch using his / her own terminal and transmit the selection information to the system.

[1621] The "means for sending a request for a live stream to a server" is a function for requesting a live stream of a specified sporting event from a server based on the user's selection information.

[1622] The "means for the server to collect social media posts related to a designated event in real time" refers to a function that allows the server to collect social media posts using specific hashtags or keywords related to a designated sporting event.

[1623] "Means of analyzing collected posts to identify peak times of excitement and emotion" is a function that uses natural language processing technology to analyze collected social media posts and identify peak times of excitement and emotion.

[1624] The "means for notifying the user of the optimal cheering timing based on the identified peak time" is a function for notifying the user of the cheering timing based on the identified peak time.

[1625] The "means for providing a live stream and a notification of the cheering timing to a user's terminal" is a function for providing a live stream generated by the server and a notification of the cheering timing to a user's terminal.

[1626] An "emotion recognition engine that analyzes user emotion data collected from the user's device" is an engine that uses the camera and microphone installed on the user's device to collect and analyze user emotion data in real time.

[1627] The "means for further optimizing the notification of the timing to cheer based on the analysis results of the emotion recognition engine" is a function for optimizing the notification of the timing to cheer to the user with even greater accuracy based on the analysis results obtained from the emotion recognition engine.

[1628] "Means for the server to receive live-stream video data and operate an AI commentary assistant to analyze the progress of the event" refers to a function in which the server receives live-stream video data and uses AI to analyze the progress of the event.

[1629] "Means for sending explanatory text generated by the explanation assistant to the user's device in real time" is a function for sending text generated by the AI ​​explanation assistant to the user's device in real time.

[1630] "Means for the server to save all game video data after the game ends" is a function that enables the server to save all game video data after the game ends.

[1631] The "means for the server to operate a highlight generation engine and automatically extract important scenes" is a function for operating a highlight generation engine to extract important scenes from the saved game video data.

[1632] The "means for generating a customized highlight video based on the user's settings and viewing history" is a function for generating a customized highlight video based on the user's setting information and viewing history.

[1633] The "means for transmitting the generated highlight video to the user's terminal" is a function for transmitting the generated customized highlight video to the user's terminal.

[1634] The "means for providing the transmitted highlight video in a playable form to the user's terminal" is a function for providing the received highlight video in a playable form to the user's terminal.

[1635] The present invention is a system that integrates live streams, social media, AI commentary, highlight generation, and emotion recognition to enhance users' sporting event viewing experience.

[1636] The system program works as follows: First, a user selects a sporting event they want to watch using their device and sends the selection information to the server, which then starts a live stream related to the selected event and simultaneously collects social media posts using specific hashtags and keywords in real time.

[1637] The collected posts are analyzed by the server to identify peak times of excitement and emotion. Based on the identified peak times, the server then sends notifications to the user's device with the optimal timing for cheering. Furthermore, the emotion data collected from the user's device is analyzed by an emotion recognition engine, and the notification of the optimal timing for cheering is optimized based on the results.

[1638] The server receives the live-stream video data, and the AI ​​commentary assistant analyzes the progress of the match. The commentary text generated as a result of this analysis is sent to the user's device in real time, allowing the user to enjoy the commentary along with the video.

[1639] After the game ends, the server saves all game video data and runs a highlight generation engine to automatically extract important scenes. The generated highlight video is customized based on the user's settings and viewing history and sent to the user's device. The user's device then presents the received highlight video in a playable format.

[1640] The specific hardware and software required to realize this system include: Flask (a Python web framework) is used for the server; OpenCV and Dlib are used for sentiment analysis; HuggingFace Transformers is used for natural language processing; and the Twitter API is used for social media integration.

[1641] As a concrete example, consider the following scenario where a user is watching a basketball game. After the user selects a game and the live stream begins, the system collects related social media posts and identifies peak excitement times from tags like "TeamXWins." The emotion recognition engine analyzes the user's emotions and based on the results, notifies the user with "Team X is leading!", ensuring the user gets the best possible viewing experience.

[1642] An example of a prompt is as follows:

[1643] sports_event: "basketball"

[1644] social_media_hashtag: "TeamXWins"

[1645] emotion: "excited"

[1646] notify_action: "Team X is in the lead!"

[1647] As described above, the present invention is a system that highly personalizes a user's sports viewing experience and provides optimized cheering and commentary information in real time.

[1648] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1649] Step 1:

[1650] A user selects a sporting event they wish to watch using their device. The input is the event information selected by the user, and the output is the selection information sent to the server. Specifically, when the user operates the application interface to select an event and presses the "Watch" button, the selection information is sent to the server as a request.

[1651] Step 2:

[1652] The server processes requests for live streams of specified sporting events based on the selections received from the user. The input is the selections sent by the user, and the output is the URL and access information for the live stream. Specifically, the server sends a request to the live stream provider and prepares to receive the stream data.

[1653] Step 3:

[1654] The server collects social media posts related to a specified event in real time. The input is specific hashtags or keywords for the event, and the output is data on related social media posts. Specifically, it uses social media APIs (e.g., Twitter API) to collect related posts in real time.

[1655] Step 4:

[1656] The server analyzes collected social media posts and identifies peak times of excitement and emotion. The input is the collected post data, and the output is the identified peak time information. Specifically, it uses natural language processing technology to identify times when positive emotional expressions and reactions are increasing.

[1657] Step 5:

[1658] The server sends a notification to the user of the optimal time to cheer based on the identified peak times. The input is peak time information, and the output is a notification of the best time to cheer. Specifically, the server sends a notification to the user's device and displays a message such as "Now is your chance to cheer!"

[1659] Step 6:

[1660] The user's device receives the live stream and the notification of the cheering timing and provides it to the user. The input is the live stream URL and the cheering notification sent from the server, and the output is the stream video and notification displayed on the user's screen. Specifically, the application plays the stream and pops up a notification.

[1661] Step 7:

[1662] The user's device uses the device's camera and microphone to collect the user's emotional data in real time. The input is the user's facial expression data and voice data, and the output is emotional data sent to the emotion recognition engine. Specifically, the device processes the video captured by the camera and the voice recorded by the microphone.

[1663] Step 8:

[1664] The emotion recognition engine analyzes the user's emotional data and sends the results to the server. The input is the emotional data sent from the device, and the output is the emotional information as the analysis result. Specifically, it uses OpenCV and Dlib to analyze the user's facial expressions and voice.

[1665] Step 9:

[1666] The server further optimizes the timing of cheering notifications based on the analysis results obtained from the emotion recognition engine. The input is the analyzed emotion data, and the output is an optimized cheering notification. Specifically, the server adjusts the timing and content of notifications according to the emotion data.

[1667] Step 10:

[1668] The server receives the live-stream video data, runs the AI ​​commentary assistant, and analyzes the progress of the event. The input is the video data, and the output is commentary text. Specifically, the AI ​​commentary assistant analyzes players' movements and scoring scenes in real time and generates commentary text.

[1669] Step 11:

[1670] The explanatory text generated by the explanatory assistant is sent to the user's device in real time. The input is the generated explanatory text, and the output is the explanatory information displayed on the user's device. Specifically, the server sends the text data in real time, and the device displays it overlaid on the screen.

[1671] Step 12:

[1672] After the game ends, the server saves all game video data and runs a highlight generation engine. The input is game video data, and the output is a highlight clip containing important scenes. Specifically, the AI ​​engine automatically extracts important scenes such as goals and fouls and generates highlights.

[1673] Step 13:

[1674] A customized highlight video is generated based on the user's settings and viewing history and sent to the user's device. The input is the user's settings information and viewing history, and the output is a customized highlight video. Specifically, a video edited based on the individual user's preferences and past viewing history is created.

[1675] Step 14:

[1676] The user's device provides the transmitted highlight video in a playable format. The input is the highlight video data transmitted from the server, and the output is a playable highlight video. Specifically, the user's device plays the received video, allowing the user to easily review it.

[1677] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1678] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1679] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1680] [Fourth embodiment]

[1681] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1682] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1683] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1684] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1685] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1686] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1687] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1688] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1689] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1690] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1691] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1692] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1693] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1694] The present invention is a system that integrates live streaming with social media, real-time AI commentary, automatic highlight generation, and more to enhance users' sporting event viewing experience.

[1695] A way for users to select the event they want to watch and submit a request for a live stream

[1696] Users access the system using their terminals and select the sporting event they want to watch. The terminals then send the selection information to the server, requesting a live stream. As a result, users can watch their desired sporting event in real time.

[1697] A means by which the server collects social media posts related to a specified event in real time.

[1698] The server collects social media posts in real time using specific hashtags and keywords related to the designated event, and updates the collected posts as the event progresses and provides them to users.

[1699] A method for identifying peak times of excitement and emotion by analyzing collected posts

[1700] The server analyzes the collected social media posts and uses natural language processing technology to identify peak times of excitement and emotion, allowing it to understand in real time at which points during a game users are most excited.

[1701] A means of notifying users of the optimal timing for cheering based on identified peak times

[1702] The server then notifies the user of the optimal time to cheer based on the identified peak times. The notification is sent via the device and displayed on the screen, allowing the user to cheer while feeling a sense of unity with other fans.

[1703] A means for the server to receive the live stream video data and run a commentary assistant to analyze the progress of the match

[1704] The server receives live-stream video data in real time and runs an AI commentary assistant, which analyzes the progress of the game, understands player movements, goals scored, tactics, etc., and generates commentary text.

[1705] A means for transmitting the explanatory text generated by the explanatory assistant to the user's terminal in real time and displaying the explanatory text on the user's terminal.

[1706] The server transmits the generated commentary text to the user's device in real time, and the user's device displays the received commentary text overlaid on the live stream video, providing real-time commentary information to the user while watching the game.

[1707] After the match, the server saves all the match video data and runs a highlight generation engine to automatically extract important scenes.

[1708] After the game ends, the server stores all game video data, then runs a highlight generation engine to automatically extract important scenes (scores, fouls, good plays, etc.).

[1709] A means to generate customized highlight videos based on user preferences and viewing history

[1710] Users set their preferences and interests through their devices. The server references the user's settings and viewing history to generate a customized highlight video. The generated highlight video is then edited to suit the user's individual preferences.

[1711] A means for transmitting the generated highlight video to a user's terminal, and for the user's terminal to provide the transmitted highlight video in a reproducible form.

[1712] The server transmits the generated customized highlight video to the user's device, which then displays the received highlight video in a playable format, allowing the user to easily review and watch it.

[1713] Specific examples

[1714] Example 1: Real-time support

[1715] A user named "Yamada" uses a device to watch a basketball game. The server receives a request from Yamada's device and broadcasts the live stream while simultaneously collecting related social media posts. The server analyzes the posts, identifies moments of high excitement, and notifies Yamada of those moments. This allows Yamada to enjoy cheering at the most exciting moments.

[1716] Example 2: AI real-time commentary

[1717] A user named "Sasaki" is watching a soccer match. The server receives the live-stream video data, and the AI ​​commentary assistant analyzes the progress of the match and generates commentary text in real time. The commentary text is sent to Sasaki's device and displayed on the screen. This allows Sasaki to understand the flow of the match and the players' movements in detail.

[1718] Example 3: Automatic highlight generation

[1719] After a baseball game, user "Suzuki-san" wants to watch highlights of games he has previously watched. The server saves all video data at the end of the game and runs an AI highlight generation engine to extract important scenes. A customized highlight video is generated based on Suzuki-san's preferences and viewing history and sent to his device. Suzuki-san can easily enjoy highlights including his favorite scenes.

[1720] The above is an embodiment of the present invention. The present invention allows users to enjoy watching sports while feeling a sense of unity with other fans, and also allows users to gain a deeper understanding of sports through detailed commentary and customized highlights.

[1721] The processing flow will be explained below.

[1722] Processing steps for live streaming linked to social media in real time

[1723] Step 1:

[1724] A user uses a terminal to select a sporting event that they wish to watch.

[1725] Step 2:

[1726] The device sends a request for a live stream to the server.

[1727] Step 3:

[1728] The server receives a request for a live stream and retrieves the corresponding stream.

[1729] Step 4:

[1730] The server distributes the acquired live stream to the terminal, allowing the user to view it.

[1731] Step 5:

[1732] The server collects social media posts in real time using keywords and hashtags related to the specified event.

[1733] Step 6:

[1734] The server filters the collected posts and selects the most relevant posts.

[1735] Step 7:

[1736] The server analyzes the collected posts and identifies peak times of excitement and emotion.

[1737] Step 8:

[1738] The server generates data for notifying the user of the optimal cheering timing based on the identified peak time.

[1739] Step 9:

[1740] The server transmits the generated notification data to the terminal.

[1741] Step 10:

[1742] The notification received by the terminal is displayed on the screen to notify the user.

[1743] AI real-time commentary processing steps

[1744] Step 1:

[1745] The server receives the live stream video data in real time.

[1746] Step 2:

[1747] The server runs an AI commentary assistant and analyzes the video data.

[1748] Step 3:

[1749] The AI ​​commentary assistant recognizes the progress of the game, player movements, scoring scenes, etc.

[1750] Step 4:

[1751] The AI ​​commentary assistant generates explanatory text based on the analysis results.

[1752] Step 5:

[1753] The server transmits the generated explanatory text to the terminal in real time.

[1754] Step 6:

[1755] The explanatory text received by the terminal is displayed on the screen together with the live stream video and provided to the user.

[1756] Processing steps for automatic highlight generation

[1757] Step 1:

[1758] After the match ends, the server stores all the match video data.

[1759] Step 2:

[1760] The server inputs the saved video data into the AI ​​highlight generation engine.

[1761] Step 3:

[1762] The AI ​​highlight generation engine automatically extracts important scenes from the game (scores, fouls, good plays, etc.).

[1763] Step 4:

[1764] The user uses the device to set their preferences and interests.

[1765] Step 5:

[1766] The server refers to the user's setting information and past viewing history and determines a policy for generating a customized highlight video.

[1767] Step 6:

[1768] The server uses an AI highlight generation engine to generate a customized highlight video.

[1769] Step 7:

[1770] The server transmits the generated customized highlight video to the user's terminal.

[1771] Step 8:

[1772] The highlight video received by the terminal is provided to the user in a reproducible form.

[1773] Example 1

[1774] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1775] Conventional sports viewing systems have struggled to provide real-time commentary, improve the viewing experience, and share the excitement and emotion of users. Furthermore, it has been difficult to customize the post-match highlights based on individual user interests and viewing history. This has limited the enjoyment of sports viewing and prevented the user experience from being maximized.

[1776] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1777] In this invention, the server includes means for a user to select an event they wish to watch, means for sending a live stream request to the server, means for the server to collect social media posts related to the designated event in real time, means for analyzing the collected posts to identify peak times of excitement and emotion, means for notifying the user of the optimal cheering timing based on the identified peak times, means for providing the live stream and a notification of the cheering timing to the user's terminal, means for analyzing the social media posts using a generative AI model, means for notifying the user based on the peak times identified by the generative AI model, and a commentary assistant for the server to receive the live stream video data and analyze the progress of the match. The system includes a means for operating a game server, a means for transmitting commentary text generated by a commentary assistant to a user's device in real time, a means for displaying the commentary text on the user's device, a means for using a generative AI model for generating commentary text in real time, a means for a server to save all game video data after the game ends, a means for the server to operate a highlight generation engine and automatically extract important scenes, a means for generating a customized highlight video based on a user's settings and viewing history, a means for transmitting the generated highlight video to the user's device, a means for customizing and generating the highlight video using a generative AI model that uses the user's viewing history, and a means for the user's device to provide the transmitted highlight video in a playable form. This enables users to watch a game with in-depth commentary while sharing the excitement and emotion with other fans in real time, and to enjoy a highlight video that suits their interests after the game ends.

[1778] The "System" is an integrated mechanism that provides users with the optimal viewing experience by broadcasting the sporting events they want to watch in real time and collecting and analyzing related social media information.

[1779] "User" means an individual who uses the System to watch sporting events and receive services such as live streams, commentary, and highlight videos.

[1780] A "server" is a computer device that is the core of the system, and is responsible for processing requests from users, distributing live streams, analyzing data, generating highlights, and so on.

[1781] "Terminal" means a device used by a user to access the system and watch a sporting event, and includes a smartphone, tablet, PC, etc.

[1782] A "live stream" is video data of sporting events and performances broadcast in real time over the Internet.

[1783] "Social media" refers to a platform on the Internet where users can share and interact with each other, and includes Twitter, Facebook, Instagram, etc.

[1784] A "generative AI model" is a machine learning algorithm that is trained using artificial intelligence techniques to perform a specific task (e.g., text generation, sentiment analysis, etc.).

[1785] The "Commentary Assistant" is an artificial intelligence system that runs on a server, analyzes live-stream video data, and generates commentary text in real time.

[1786] The "highlight generation engine" is a software component that analyzes video data from matches, automatically extracts and edits important scenes, and generates highlight videos.

[1787] "Viewing history" is a record of sporting events that a user has viewed in the past and their contents.

[1788] The present invention provides a system that integrates live streaming, social media, real-time commentary using generative AI models, and automatic highlight generation to enhance users' sporting event viewing experience.

[1789] First, a user accesses the system using a terminal and selects the sporting event they want to watch. At this time, the terminal sends the user's selection information to the server and requests a live stream. The server receives the request and provides live footage of the specified sporting event.

[1790] The server then collects social media posts in real time using specific hashtags or keywords related to the specified event. These posts are retrieved using social media APIs (e.g., Twitter API, Facebook API). The collected posts are then analyzed using a generative AI model to identify peak times of excitement and emotion.

[1791] The server notifies users of the best time to cheer based on the identified peak times. The notification is sent via a pop-up message or notification bar on the user's device, allowing users to send cheering messages and share their excitement at the best possible time.

[1792] The server also receives live-stream video data in real time and runs a commentary assistant to analyze the progress of the match. The commentary assistant analyzes the player movements, goals scored, tactics, and other aspects of the video, and generates commentary text in real time. The generated commentary text is sent to the user's device and overlaid on the live video.

[1793] After the game ends, the server saves all game video data and runs a highlight generation engine. The highlight generation engine automatically extracts and edits important scenes, such as goals and great plays, to generate a customized highlight video. The customization is based on the user's settings and viewing history, and the generated highlight video is sent to the user's device.

[1794] As a concrete example, consider a user named "Yamada" using a device to watch a basketball game. The server receives a request from Yamada's device and delivers the live stream while simultaneously collecting and analyzing related social media posts. The server identifies the moments when excitement levels rise and notifies Yamada, allowing him to enjoy cheering without missing the most exciting moments.

[1795] In addition, when a user named "Sasaki" is watching a soccer match, the server receives the live-stream video data and uses the generative AI model to generate and transmit commentary text in real time, allowing Sasaki to watch the match while gaining a detailed understanding of the progress of the match.

[1796] Furthermore, if user "Suzuki-san" wants to watch highlights after a baseball game, the server saves all video data at the end of the game and activates the AI ​​highlight generation engine. A customized highlight video is generated and sent based on Suzuki-san's preferences and viewing history. Suzuki-san can easily watch highlight videos containing important scenes.

[1797] Example prompt sentence:

[1798] "How can Yamada know the optimal timing to cheer while watching a basketball game in real time?"

[1799] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1800] Step 1:

[1801] A user accesses the system using a terminal and selects the sporting event they wish to watch.

[1802] Input: User event selection information

[1803] Specific operation: The user selects the menu or list on the device screen, selects the event they want to watch, and confirms the selection by pressing the Enter key or touching the screen.

[1804] Output: Event selection information is recorded on the terminal and sent to the server.

[1805] Step 2:

[1806] The terminal transmits the user's selection information to the server.

[1807] Input: User event selection information

[1808] What it does: Your device sends the selected information to a server over the internet, using a secure protocol such as HTTPS to protect the data.

[1809] Output: The selection information is received by the server.

[1810] Step 3:

[1811] The server processes the live stream request based on the received selection information and provides the live video to the user.

[1812] Input: Selection information

[1813] Specific operation: The server accesses the streaming service, acquires live video of the specified event, and streams the acquired video data to the user's device.

[1814] Output: The live stream video is displayed on the user's device.

[1815] Step 4:

[1816] The server collects social media posts related to a specified event in real time.

[1817] Input: Event hashtags or keywords

[1818] How it works: The server uses social media APIs to collect posts related to the specified hashtags and keywords. The collection process is done in real time.

[1819] Output: Collected social media post data

[1820] Step 5:

[1821] The server analyzes the collected posts using a generative AI model to identify peak times of excitement and emotion.

[1822] Input: Collected submission data

[1823] How it works: Using a generative AI model, post data is analyzed and scored for excitement and emotional impact. Peak times are identified based on the scoring results.

[1824] Output: Identified peak times of excitement and emotion

[1825] Step 6:

[1826] The server notifies the user of the optimal cheering timing based on the identified peak time.

[1827] Input: Identified peak times

[1828] Specific operation: The server sends a notification message to the user's device, causing a pop-up message or notification bar to appear on the user's screen.

[1829] Output: Notification of the cheering timing displayed on the device screen

[1830] Step 7:

[1831] The server receives the live stream video data in real time, runs the commentary assistant, and analyzes the progress of the match.

[1832] Input: Live stream video data

[1833] Specific operation: The server analyzes the video data using an AI commentary assistant (e.g., image recognition model and natural language processing model).

[1834] Output: Real-time generated explanatory text

[1835] Step 8:

[1836] The server transmits the generated commentary text to the user's terminal in real time, and the commentary text is displayed on the user's terminal.

[1837] Input: Generated description text

[1838] Specific operation: The server sends explanatory text to the user's device, which then overlays the received explanatory text on the live video.

[1839] Output: Descriptive text that is displayed on the device screen

[1840] Step 9:

[1841] After the match ends, the server stores all match video data and runs a highlight generation engine to automatically extract important scenes.

[1842] Input: Match video data

[1843] How it works: The server stores the video data in a database, then uses a highlight generation engine to automatically extract and edit important scenes (scoring scenes, great plays, etc.).

[1844] Output: Clips of extracted key scenes

[1845] Step 10:

[1846] The server generates a customized highlight video based on the user's settings and viewing history and transmits it to the user's terminal.

[1847] Input: User settings and viewing history, extracted clips of important scenes

[1848] Specific operation: The server references the user's settings and viewing history, combines clips of key scenes, and generates a customized highlight video. The generated video is then sent to the user's device.

[1849] Output: A customized highlight video sent to the user's device

[1850] Step 11:

[1851] To provide a highlight video received by a user terminal in a reproducible form.

[1852] Input: Custom highlight video

[1853] Specific operation: The device uses an application (e.g., a video player) to play the received highlight video and displays it in a playable format for the user.

[1854] Output: A highlight video that can be viewed by the user

[1855] (Application example 1)

[1856] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1857] Conventional sports viewing systems lack real-time commentary, highlight generation, and notification functions that allow viewers to cheer on the most exciting moments. This makes it difficult for users to gain a deeper understanding of the content and enjoy it without missing any exciting moments. Furthermore, they do not provide customized highlight videos tailored to individual users' preferences. Therefore, a system that integrates these functions is needed to improve the user experience.

[1858] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1859] In this invention, the server includes means for a user to select content they wish to view, means for sending a streaming request to the server, means for the server to collect online posts related to the specified content in real time, means for analyzing the collected posts to identify peak times of excitement and emotion, means for notifying the user of the optimal timing for cheering based on the identified peak times, and means for providing streaming and notification of the timing for cheering to the user's information terminal. This allows the user to know the timing for cheering in real time, watch the game with a sense of unity with other viewers, and enjoy customized highlights that suit their interests.

[1860] "Means for a user to select content they wish to view" refers to the interface or process used to select the particular content a user wishes to view.

[1861] The "means for sending a streaming request to the server" refers to a mechanism for requesting the server to stream content based on the content selected by the user.

[1862] "Means for the server to collect Internet posts related to specified content in real time" refers to the function of the server to obtain posts and comments related to specific content on the Internet in real time.

[1863] "Means of analyzing collected posts to identify peak times of excitement and emotion" refers to the process of analyzing collected internet posts to identify the times when users are most excited or moved.

[1864] "Means for notifying users of the optimal timing to cheer based on identified peak times" refers to a system that notifies users of the optimal timing to cheer based on peak times of excitement and emotion identified through analysis.

[1865] "Means for providing streaming and cheering timing notifications to a user's information terminal" refers to a function for delivering streaming data and cheering timing information to a device used by a user.

[1866] "Commentary assistant for analyzing content progress" refers to AI or software that analyzes the progress of content being streamed in real time and generates commentary based on that content.

[1867] "Means for transmitting explanatory text generated by the explanatory assistant in real time to the user's information terminal" refers to a process for instantly transmitting explanatory text generated by the explanatory assistant to the user's device.

[1868] "Means for displaying explanatory text on the user's information terminal" refers to a mechanism for displaying explanatory text on the screen of the user's device.

[1869] "Means for the server to save all video data after the end of the match" refers to the function of storing all streamed video data on the server after the end of the match.

[1870] "Means for the server to operate the highlight generation engine and automatically extract important scenes" refers to a system that uses the highlight generation engine in the server to automatically select important scenes from video data.

[1871] The "means for generating a customized highlight video based on user settings and viewing history" refers to a function for generating an individually customized highlight video based on preferences set by the user and past viewing history.

[1872] "Means for transmitting the generated highlight video to the user's information terminal" refers to a process for transmitting the customized highlight video to the user's device.

[1873] The "means for providing the transmitted highlight video in a format that can be played back by the user's information terminal" refers to a system that provides the transmitted highlight video in a format that can be played back by the user's device.

[1874] This invention is a system for improving a user's sports viewing experience, utilizing a server, a user's information terminal, and AI technology. This system is implemented in the following steps.

[1875] First, a user selects the content they want to watch using an information terminal such as a smartphone or tablet. The selection information is sent from the terminal to the server, and the server receives a streaming request based on that information. The server then begins streaming the specified content.

[1876] The server then collects real-time internet posts related to the specified content, including searches using specific hashtags or keywords, and analyzes these posts using natural language processing techniques to identify peak times of excitement and emotion.

[1877] Once the data collection and analysis is complete, the server notifies users of the optimal time to cheer based on the identified peak times. The notification is sent to the user's information terminal and displayed on the screen in real time.

[1878] The server also runs an AI commentary assistant to analyze the streaming video. The AI ​​commentary assistant analyzes the progress of the game in real time and generates commentary text about players' movements, scoring scenes, etc. The generated commentary text is sent in real time to the user's information terminal and displayed on the screen.

[1879] After the game ends, the server saves all video data and runs a highlight generation engine. The highlight generation engine automatically extracts important scenes and generates a customized highlight video based on the user's settings and viewing history. This highlight video is sent to the user's information terminal, allowing the user to watch the highlights at any time.

[1880] The hardware and software used include the following: Hardware includes information terminals such as smartphones and tablets, and servers. Software includes TensorFlow (AI model), Flask / Django (server-side framework), OpenCV (video data analysis), Requests (HTTP request processing), and NLTK (natural language processing) used on the server side.

[1881] As a concrete example, consider the case where a user named "Suzuki" wants to watch a basketball game. Suzuki uses his smartphone to select the game he wants to watch and sends a request to the server. The server receives the request and provides live streaming, while also collecting and analyzing related posts from social media to identify peak times of excitement. The server notifies Suzuki of these peak times and also provides commentary text generated by an AI commentary assistant in real time. After the game ends, the server extracts key scenes and generates and sends customized highlights tailored to Suzuki's preferences.

[1882] An example of a prompt sentence might be:

[1883] "While watching a basketball game in real time, generate a code to identify and notify you of peak moments of excitement from social media."

[1884] In this way, the present invention can significantly improve the sports viewing experience for users.

[1885] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1886] Step 1:

[1887] The user selects the content they wish to view using an information terminal. The input is the user's viewing preference, and the output is the selected content information. The interface used by the user is intuitive, and selection can be easily made using lists and search functions.

[1888] Step 2:

[1889] A streaming request is sent from the information terminal to the server. The input is the selected content information, and the output is a confirmation of the request from the server. This information sent as an HTTP request triggers the server to start streaming.

[1890] Step 3:

[1891] The server starts streaming the specified content. The input is the requested content information, and the output is live streaming data. The server encodes the video data in real time and sends it to the user's information terminal.

[1892] Step 4:

[1893] The server collects internet posts related to the specified content in real time. The input is a specific hashtag or keyword, and the output is the post data. The server uses APIs to retrieve related posts from social media and stores them in a database.

[1894] Step 5:

[1895] The server analyzes the collected posts to identify peak times of excitement and emotion. The input is the collected post data, and the identified peak times are extracted as the output. Natural language processing technology (e.g., NLTK) is used to analyze the text data and evaluate emotions and excitement levels.

[1896] Step 6:

[1897] The server notifies the user of the optimal cheering timing based on the identified peak times. The input is the identified peak time information, and the output is generated notification data. The server generates real-time notifications and sends them to the user's information terminal.

[1898] Step 7:

[1899] The server runs an AI commentary assistant to analyze streaming video. The input is real-time video data, and the output is explanatory text. The TensorFlow model is used to analyze the video data and generate explanatory text according to the progress.

[1900] Step 8:

[1901] The server generates explanatory text and sends it to the user's information terminal in real time. The input is the generated explanatory text, and the output is a confirmation of transmission. The explanatory text is distributed along with the video data.

[1902] Step 9:

[1903] The commentary text is displayed on the user's information terminal. The input is the received commentary text, and the output is the displayed text. It is displayed overlaid on the screen of the user's information terminal to aid viewing.

[1904] Step 10:

[1905] After the game ends, the server saves all video data. The input is all video data from the game, and the output is a saved data file. The server saves the data in cloud storage such as S3.

[1906] Step 11:

[1907] The server runs a highlight generation engine to automatically extract important scenes. The input is stored video data, and the output is extracted highlight scenes. OpenCV is used to perform video analysis and extract scenes based on specific events or actions.

[1908] Step 12:

[1909] The server generates a customized highlight video based on the user's settings and viewing history. The input is the user's settings and viewing history, and the output is a customized highlight video. An AI model is used to analyze the user's preferences and combine the optimal scenes.

[1910] Step 13:

[1911] The server sends the generated highlight video to the user's information terminal. The input is the customized highlight video, and the output is a confirmation of transmission. The video data is provided to the user in a format that is easy to view.

[1912] Step 14:

[1913] The user's information terminal provides the transmitted highlight video in a playable format. The input is the received highlight video, and the output is a played video. The user can enjoy the highlight video containing their favorite scenes.

[1914] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1915] This invention is a system that integrates live streaming with social media, AI-powered real-time commentary, automatic highlight generation, and an emotion engine that recognizes user emotions to enhance users' sporting event viewing experience.

[1916] A way for users to select the event they want to watch and submit a request for a live stream

[1917] Users access the system using their terminals and select the sporting event they want to watch. The terminals then send the selection information to the server, requesting a live stream. As a result, users can watch their desired sporting event in real time.

[1918] A means by which the server collects social media posts related to a specified event in real time.

[1919] The server collects social media posts in real time using specific hashtags and keywords related to the designated event, and updates the collected posts as the event progresses and provides them to users.

[1920] A method for identifying peak times of excitement and emotion by analyzing collected posts

[1921] The server analyzes the collected social media posts and uses natural language processing technology to identify peak times of excitement and emotion, allowing it to understand in real time at which points during a game users are most excited.

[1922] A means of notifying users of the optimal timing for cheering based on identified peak times

[1923] The server then notifies the user of the optimal time to cheer based on the identified peak times. The notification is sent via the device and displayed on the screen, allowing the user to cheer while feeling ...

Claims

1. a means for a user to select an event they wish to view; means for sending a request for a live stream to a server; a means for the server to collect social media posts related to a designated event in real time; A method for analyzing collected posts to identify peak times of excitement and emotion, A means for notifying a user of an optimal cheering timing based on the identified peak time; A means for providing a notification of the live stream and cheering timing to a user's terminal; A system including:

2. a server receiving the live stream video data and running a commentary assistant to analyze the progress of the match; a means for transmitting the commentary text generated by the commentary assistant to the user's terminal in real time; means for displaying explanatory text on a user's terminal; The system of claim 1 , comprising:

3. After the match ends, the server will store all the match video data. A means for the server to run a highlight generation engine and automatically extract important scenes; means for generating a customized highlight video based on a user's settings and viewing history; means for transmitting the generated highlight video to a user terminal; A means for providing the transmitted highlight video in a playable form to a user's terminal; The system of claim 1 , comprising:

Images