INTERACTIVE FAN ENGAGEMENT SYSTEM AND METHOD

Abstract

The invention relates to an integrated system and method that creates narrative structures by analyzing multimodal data streams, generates personalized media content through generative artificial intelligence models, and presents this content to users / fans through augmented reality (AR) and gamification elements.

Description

1 TARIFF INTERACTIVE FAN ENGAGEMENT SYSTEM AND METHOD Technical Area The invention leverages artificial intelligence (AI), machine learning (ML), computer vision, generative artificial intelligence, and It relates to digital media content production technologies. 5 The invention specifically creates narrative structures by analyzing multimodal data streams. Generating personalized media content through generative artificial intelligence models and content through augmented reality (AR) and gamification elements It relates to an integrated system and method that offers something to users / fans. State of the Art 10 Today, user experience is crucial in the fields of digital media and sports broadcasting. The need for technologies aimed at personalization is steadily increasing, especially. AI-powered data analysis systems analyze user behavior, visual content, By processing audio data and real-time event streams into meaningful content... This makes transformation possible. However, generative artificial intelligence 15 Thanks to its models, text, image, audio and video-based content is automatically processed. It can be produced and customized according to different user profiles. Additionally, Augmented reality (AR) technologies and gamification mechanisms enable users to digitally by enabling them to interact with content in a more engaging way, especially in sports and entertainment. and is widely used in the live event industry. 20 In current technology, traditional media production processes, especially animation and high-technology, are problematic. When it comes to quality video production, you need massive infrastructure and expert personnel. and requires high costs. Visualizing a narrative; screenplay writing, Storyboard creation, voice acting, character modeling, and frame-by-frame processing. It consists of manual and time-consuming stages such as (rendering) (For example; 25 (https: / / www.youtube.com / watch?v=NmSZO4oxClQ). This situation affects content production. It is monopolized by companies with very large budgets, leaving solo content creators or sports organizations with limited budgets building vast media empires It prevents. 2 In the field of sports broadcasting, the current situation is generally based on a linear flow. Match summaries and highlights are often generated manually or with limited automation. With its tools, it is prepared in a way that will be the same for all viewers. However, modern fan profile, based on their interests, favorite players, or statistics they follow They demand content customized according to their needs. Most existing systems require a data depth of 5. (player movements, ball possession, weather conditions, etc.) and narrative context (press (meeting statements, fan sentiments, historical data) insufficient to combine It remains. As a result of the research conducted on this subject, the "Auto Curation" system with registration number US10595101B2 was identified. An application titled "and Personalization of Sports Highlights" was found. The system; 10 It calculates "excitement scores" using visual and auditory cues and provides summary clips. It proposes a system that creates [this]. However, these systems only trim the data. It merely clips, without synthesizing new visual content or narrative. Similar Although facial recognition technologies are used in stadium experiences, this data... How to process it without leading to privacy breaches and how to create special value for the fan 15 There are gaps regarding what it will create. As a result of the research conducted on this subject, it was also determined that the document numbered US8929721B2 is “Methods for Identification of Highlight Game Events and Automated Generation of Videos for An application titled "Same" was found. The system includes games related to sports competitions. By analyzing data, statistical events, and specific threshold values, key moments can be identified. 20 It proposes a structure that monitors and generates automated video summaries of these moments. However, The system in question is based solely on selecting and combining existing images. They are working on creating new media content using generative artificial intelligence models. It does not enable its synthesis. In addition, fan-generated content (FGC) production and marketing, 25 While this content holds great potential for brands, the curation, moderation, and Providing professional-quality content still largely depends on human intervention. Web3 While the rise of platforms and fan tokens has created a financial interaction space, The integration of this situation with a rich and interactive media experience is not yet fully complete. It did not happen in the truest sense. 30 In sports broadcasting, match highlights are generally the same for all viewers. is being prepared. The modern fan profile, on the other hand, is based on their favorite players or the players they follow. According to statistics, there is a demand for customized content. In the current state of the art... 3 Although there are systems that create summary clips using visual and auditory cues, The systems only clip the data, not creating new visual content or... It does not synthesize personalized narratives. Furthermore, it lacks facial expressions in stadium experiences. Even when identification technologies are used, how can this data be stored without leading to privacy breaches? There are technical gaps regarding how it will be processed. 5 In conclusion, due to the negative aspects described above and the current solutions being the subject of discussion... Due to its shortcomings, an improvement is needed in the relevant technical field. It has been made. Purpose of the Invention The invention was created by drawing inspiration from existing situations and addressing the aforementioned drawbacks. 10 It aims to solve the problem. The main purpose of the invention is to create narrative structures by analyzing multimodal data streams. Generating personalized media content through generative artificial intelligence models and content through augmented reality (AR) and gamification elements The goal is to create an integrated system and method that offers something to users / fans. 15 Another purpose of the invention is to capture live sports footage with high-resolution camera data. By analyzing these statistics, we can create personalized videos using this data at minimum cost. a system and method that transforms content into interactive fan experiences It is to create. Another aim of the invention is to create large language models (LLM), vision models and quadal video 20 using an integrated pipeline of models to process raw data automatically transforms high-value narratives, emotionally coherent video clips, and The goal is to provide a structure that transforms into interactive digital experiences. Another objective of the invention is the direct storage of biometric data on the server side. instead of being divided into distributed parts (shards) or processed on the device 25 by protecting user privacy and securing the production of fan-specific content The goal is to provide a security architecture. To achieve the objectives described above, the invention enables multimodal data flows. through generative artificial intelligence models that create narrative structures by analyzing Creating personalized media content and combining this content with augmented reality 30 4 It is a system that presents itself to users / fans through gamification elements. Accordingly system;  Images from high-resolution cameras placed inside the stadium flows, statistical data obtained from external data providers and public Data collection unit that collects the relevant texts, 5  semantically annotating the aforementioned raw data and based on a large language model Automated tagging unit that generates metadata,  Different data streams are captured with millisecond-level timestamps. normalization unit that synchronizes by matching,  10 the visual motion intensity, acoustic energy levels, and in those images Calculating excitement score using semantic significance data calculation module, The clip starts and ends depending on the calculated excitement score. Narrative 15 creates a narrative flow by dynamically optimizing its points. arc unit analysis unit,  natural language vocalization of data from the aforementioned narrative arc unit. and the large language model module that converts them into dialogue texts, 20 Identify the relevant supporter from those high-resolution wide-angle images. By performing clarity analysis, it automatically crops the image. Maintaining visual consistency and providing personalized solutions by using quaddal video models. Video synthesis unit that creates summary videos synthesis and production unit, 25  A distribution unit that delivers personalized content to fans' mobile devices,  Biometric data and user preferences are stored locally on the device. A device-mounted privacy vault that provides data security by processing data.  ensuring that each fan is presented with content tailored only to their interests. hyper-personalized broadcast unit, 30  those fans were increased in certain physical points within the stadium access to reality-based content and obtain digital collectible items AR interaction unit that enables it, those fans making predictions about events during the match gamification module that allows earning points including interaction module It includes. The invention also creates narrative structures by analyzing multimodal data streams. Generating personalized media content through generative artificial intelligence models and content through augmented reality and gamification elements It also includes the method of presenting it to users / fans. Accordingly, the method is:  Images from high-resolution cameras placed inside the stadium 10 flows, statistical data obtained from external data providers and public Collection of the texts through a data collection unit,  The aforementioned raw data is semantically analyzed through an automated labeling unit. marking as such and generating metadata based on a large language model,  Different data streams are normalized in milliseconds through a normalization unit. Synchronization by matching timestamps with high precision,  visual motion intensity, acoustic energy levels, and semantics in the images Using important data, the calculation module located within the analysis unit Calculating the excitement score through,  Depending on the calculated excitement score, the start and end points of the clip will be 20 dynamically through the narrative arc unit located within the analysis unit Creating a narrative flow by optimizing it,  Data from the aforementioned narrative arc unit, a synthesis and production unit Natural language speech through the large language model module included within it. and their transformation into dialogue texts, 25  Identifying the relevant fan from high-resolution wide-angle images automatic framing based on clarity analysis, and Personalized summaries that maintain visual consistency using quadal video models. videos, video synthesis unit located within the synthesis and production unit creation through, 30  Personalized content created is distributed to fans through the distribution unit. transmitted to their mobile devices, 6  Biometric data and user preferences are managed through an on-device privacy vault. data security is ensured by processing data locally on the relevant device. ensuring,  Each fan receives content tailored only to their interests, in a hyper-personalized way. presented via the broadcasting unit, 5  via an AR interaction unit located within an interaction module augmented reality for fans at specific physical points within the stadium. access to based content and obtaining digital collectible items ensuring,  through a gamification module located within an interaction module 10 Fans score points by making predictions about events during the match. ensuring that he / she wins It includes the steps involved in the process. The structural and characteristic features and all the advantages of the invention are given in the figures below and 15 This becomes clearer thanks to the detailed explanation written with references to these figures. This will be understood as such, and therefore the evaluation will also take these forms and detailed explanations into account. This should be done taking that into consideration. Figures that will help understand the invention. Figure 1 shows a schematic representation of the system that is the subject of the invention. 20 Explanation of Part References 100. Data collection unit 101. High-resolution camera 102. Statistical data 103. Public domain texts 25 110. Automatic labeling unit 120. Normalization unit 200. Analysis unit 7 210. Computing module 220. Narrative arc unit 300. Synthesis and production unit 310. Large language model module 320. Video synthesis unit 5 400 Distribution units 410. On-device privacy safe 420. Hyper-personalized broadcast unit 500. Interaction module 510. AR interaction unit 10 520. Gamification module Detailed Description of the Invention This detailed explanation describes the preferred system and method for the invention. Their structures are explained solely to facilitate a better understanding of the subject. The invention creates generative artificial intelligence (GAI) systems that generate narrative structures by analyzing multimodal data streams. producing personalized media content through intelligence models and distributing this content presenting to users / fans through augmented reality and gamification elements It is a system. Figure 1 shows a schematic representation of the system that is the subject of the invention. According to the system, high-resolution cameras are placed inside the stadium (101) incoming image streams, statistical data obtained from external data providers 20 (102) and data collection unit (100) collecting public domain texts (103), the aforementioned raw semantically tags data and generates metadata based on large language models. Automatic labeling unit (110) processes different data streams with millisecond accuracy. Normalization unit (120) that synchronizes by matching through the stamps, visual motion intensity, acoustic energy levels and semantic significance in images 25 Calculation module (210) which calculates excitement score using data, calculated The clip's start and end points are dynamically adjusted based on the excitement score. 8 Analysis unit containing narrative arc unit (220) which creates narrative flow by optimizing (200), natural language vocalization of the data from the mentioned narrative arc unit (220) and Large language model module (310) that converts dialogue texts, high resolution By identifying the relevant supporter from wide-angle images and performing a clarity analysis, Using quad video models that perform automatic framing, visual consistency is 5 video synthesis unit (320) which protects and creates personalized summary videos synthesis and production unit (300), personalized content to fans' mobile devices The transmitting distribution unit (400) transmits biometric data and user preferences to the relevant device. On-device privacy vault (410) that provides data security by processing it locally. Hyper-10 ensures that each fan is presented with content tailored only to their interests. Personalized broadcast unit (420), for fans to use specific physical locations within the stadium. access to augmented reality-based content and digital collectible items at various points AR interaction unit (510) that enables fans to get to the events in the match a gamification module that allows players to earn points by making predictions. It includes interaction module (500) containing (520). 15 The system works on the following principle: The invention involves a method that initially involves placing high-rise buildings inside the stadium. Image streams from high-resolution cameras (101) are received from external data providers 20 (For example, Opta) statistical data (102) and public domain texts (103) It starts with the collection of data through a data collection unit (100). At this stage, for example, fan reactions via high resolution cameras (101), on-field While visual events and tribune images are taken; goals are recorded through statistical data (102), Live data streams relating to fouls, substitutions, assists, shots, or similar match events 25 is included in the system. Public domain texts (103) are match, team, player or as a textual data source that can be used in the production of narratives about historical events is being transferred to the system. The collected raw data is semantically categorized via an automatic labeling unit (110) 30 Metadata is being generated based on a large language model and is being marked. At this stage... Video streams, statistical events, and textual sources; player name, event type, factors such as sense of time, crowd reaction, emotional intensity, stage context, and narrative value. It is enriched with semantic tags. Thus, raw data from different sources is collected and enriched with semantic tags. 9 The data provides meaningful information that can be used in later analysis and content production stages. They are being transformed into structures. Then different data streams are normalized in milliseconds via a normalization unit (120). They are synchronized by matching them via timestamps with high precision. This 5 thanks to the process, the image captured by high-resolution cameras (101) Statistical data (102) including clues such as goals, fouls or player substitutions Events are combined on the same time axis. Thus, for example, statistical data... (102) the moment of actual fan reaction within the stream of images with an incoming goal signal. is being harmonized. 10 On synchronized data, visual motion intensity and acoustics in the images... Using energy levels and semantic significance data, within the analysis unit (200) The excitement score is calculated through the calculation module (210) located thereon. The excitement score is determined by the formula; here it is visualized. Motion (M), acoustic energy (A), and semantic significance (S) are the main components that are collected. The excitement score mentioned refers to visual motion, sound intensity, and the semantic significance of the event. They are determined by evaluating them together. In this context, visual movement and acoustics are considered. Which moments are content 20 by primarily bringing together energy and semantic significance It is found to be more valuable in terms of production. The clip's start and end points are analyzed based on the calculated excitement score. through the narrative arc unit (220) which is located within unit (200) in a dynamic way The narrative flow is created by optimizing the video. At this stage, it is determined how much of the clip relates to the event. The time it starts and how long after the event it ends is dynamic, depending on the type of data. The time is adjusted using the specified parameters (Tbefore, Tafter). Thus, the data... Delay issues in the flow are reduced, and not only at the moment of the event, but also in the events leading up to it are addressed. The preparation, the reaction during the event, and the emotional intensity after the event are also described. It is captured in its entirety. This allows, for example, a goal signal from Opta to be detected within 30 seconds. The image is matched in perfect sync with the fan's actual reaction moment. Data from the aforementioned narrative arc unit (220) constitute a synthesis and production unit (300) Natural language speech through the large language model module (310) included within it and are converted into dialogue texts. Subsequently, synthesis and production unit (300) 35 high resolution through the video synthesis unit (320) located inside Identifying the relevant fan from wide-angle images, according to clarity analysis. Automatic framing, preferably in 9:16 vertical format, and quad video. Personalized summary videos that maintain visual consistency using models The creation of the video synthesis unit (320) is ensured. At this stage, the video synthesis unit (320) is high 5 from among the wide-angle images taken by high-resolution cameras (101) determining the audience or key stage area, for example, in a 9:16 vertical format. Quadal Video automatically crops content in social media-compatible formats. Through these models, image consistency is maintained, scene transitions are smoothed, and Professional-quality personalized summary videos are produced. 10 The created personalized content is distributed to fans via the distribution unit (400). is transmitted to mobile devices. At this stage, the distribution unit (400), the produced video, the narrative, augmented reality content or gamification data with the relevant fan profile It enables the transfer of data to mobile devices by pairing them. Thus, the fan can watch the match on 15... directly on their own device, content tailored specifically to their experience It can be accessed via [link / website]. Biometric data and user preferences are accessible via the on-device privacy vault (410). Data is processed locally on the device, thus ensuring data security. This 20 In this system, biometric data and personal preferences relating to fans are directly transmitted to the server. Before being sent, the files are divided into shards and processed on the device. This allows for the creation of thousands of shards. In an environment where the fan's image is processed, the risk to privacy is reduced and personalization is achieved. The process is carried out within a secure architecture. Each fan receives content tailored solely to their interests, a hyper-personalized broadcast. It is presented through the unit (420). In this context, the fan's favorite player, his own Content selection is made taking into account information such as image quality. This ensures that every fan... Instead of general match summaries, choose something that is relevant to your interests, contextual, and has personal value. It displays the content. 30 In addition, an AR interaction unit located within an interaction module (500) (510) increased access for fans to certain physical points within the stadium access to reality-based content and acquisition of digital collectible items This is provided. At this stage, 35 geolocation-based surveys are conducted at specific locations within the stadium. 11 The trigger is activated and the fan can view augmented reality content via their mobile device. By viewing them, users can collect digital collectible items. Finally, the gamification module (520) located within an interaction module (500) Through this platform, fans can make predictions about events during the match and earn 5 points. This ensures that the player wins. For example, fans predict the direction of the next penalty, the player They can log into the system by predicting changes or specific events within the match. and can earn points for correct predictions. These points are obtained from these interactions. data includes fan sentiment, social media sharing trends, interests, and content. Their preferences will be considered and reflected back in subsequent personalized content cycles. It is used as a notification.

Claims

12 REQUESTS 1. Genetic methods that create narrative structures by analyzing multimodal data streams. producing personalized media content through artificial intelligence models and content through augmented reality and gamification elements It is a system that offers users / fans; its feature is 5  from high-resolution cameras (101) placed inside the stadium video streams, statistical data obtained from external data providers (102) and data collection unit (100) collecting public domain texts (103),  semantically annotating the aforementioned raw data and based on a large language model Metadata generating automatic tagging unit (110), 10  Different data streams are captured with millisecond-level timestamps. normalization unit that synchronizes by matching (120),  the visual motion intensity, acoustic energy levels, and in those images 15 that calculate excitement score using semantic significance data calculation module (210), The clip starts and ends depending on the calculated excitement score. a narrative that creates a narrative flow by dynamically optimizing its points arc unit (220) containing analysis unit (200), 20  data from the mentioned narrative arc unit (220) in natural language A large language model module that converts voiceovers and dialogue texts. (310), Identify the relevant supporter from those high-resolution wide-angle images. 25 By performing clarity analysis, it automatically crops the image. Maintaining visual consistency and providing personalized solutions by using quaddal video models. Video synthesis unit that creates summary videos (320) Synthesis and production unit containing (300),  Distribution unit 30 that delivers personalized content to fans' mobile devices (400),  Biometric data and user preferences are stored locally on the device. On-device privacy vault (410) that provides data security by processing data. 13  ensuring that each fan is presented with content tailored only to their interests. hyper-personalized broadcast unit (420),  those fans were increased in certain physical points within the stadium 5 Access to reality-based content and acquisition of digital collectible items AR interaction unit (510) that enables it, those fans making predictions about events during the match Gamification module that enables earning points (520) Interaction module containing (500) It includes. 10 2. Genetic methods that create narrative structures by analyzing multimodal data streams. producing personalized media content through artificial intelligence models and content through augmented reality and gamification elements It is a method that offers this to users / fans, and its feature is; 15  from high-resolution cameras (101) placed inside the stadium video streams, statistical data obtained from external data providers (102) and public domain texts (103) through a data collection unit (100) gathering,  The raw data mentioned are processed through an automatic labeling unit (110) 20 semantically tagging and large language model-based metadata creation,  normalization of different data streams in milliseconds through a normalization unit (120) Synchronization by matching timestamps with high precision,  visual motion intensity, acoustic energy levels and semantics in the images 25 Using the important data, the calculation included in the analysis unit (200) Calculation of excitement score via module (210),  The start and end points of the clip, depending on the calculated excitement score, through the narrative arc unit (220) located within the analysis unit (200) Creating a narrative flow by dynamically optimizing it, 30  data from the mentioned narrative arc unit (220) a synthesis and production through the large language model module (310) located within unit (300) natural voiceovers and conversion of dialogue texts into the language, 14  Identifying the relevant fan from high-resolution wide-angle images automatic framing based on clarity analysis, and Personalized summaries that maintain visual consistency using quadal video models. video synthesis located within the synthesis and production unit (300) creation through unit (320), 5  Personalized content created, through the distribution unit (400) transmitted to fans' mobile devices,  biometric data and user preferences, on-device privacy vault (410) data security is ensured by processing data locally on the relevant device. ensuring, 10  Each fan receives content tailored only to their interests, in a hyper-personalized way. presented through the broadcast unit (420),  AR interaction unit (510) located within an interaction module (500) through increased fan engagement at specific physical points within the stadium access to reality-based content and obtaining digital collectible items 15 ensuring that it does so,  Gamification module (520) located within an interaction module (500) through which fans make predictions about events during the match enabling them to earn points by doing so It includes the steps of the process. 20 3. The method that complies with Claim 2 is characterized by its ratio of excitement score to visual movement. density, acoustic energy level and semantic significance parameters together The calculation process is performed by the calculation module (210) by evaluating it. It includes step 25.

4. The method is in accordance with Request 2 and its feature is; the clip by the narrative arc unit (220). Start and end points are dynamically determined depending on the data type. Optimizing the process step using the specified time parameters It includes. 30 5. The method is compliant with Request-2 and its feature is; video synthesis unit (320) automatic framing performed via 9:16 vertical format It includes the creation process step. 35

Images