system

Abstract

We provide the system. [Solution] A means of acquiring and analyzing match data in real time to identify important events, A notification generation method for notifying users of important moments, A means of receiving questions from users during a match, generating appropriate answers, and sending them back. A method for extracting important scenes after a match and automatically generating highlight videos, A means of providing a communication function for exchanging opinions with other users, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern sports watching, there are problems that it is difficult to understand technical terms and tactics, and many viewers cannot fully enjoy the progress of the game. In addition, communication among fans during watching is scarce, and it often ends up with one-way information reception, so activation as a community is required. Furthermore, there is a problem that the satisfaction of the viewing experience decreases because there is a possibility of missing important moments or notable scenes of the game.

Means for Solving the Problems

[0005] This invention provides a means for acquiring and analyzing match data in real time, thereby aiding in the understanding of specialized terminology and tactics. Furthermore, a notification generation means automatically identifies important events and notifies users, providing an environment where crucial moments can be reliably grasped. In addition, it enables two-way communication and promotes interaction among users by accepting user questions and automatically generating and returning appropriate answers. The automatically generated highlight videos after matches allow viewers to watch important scenes in a consolidated manner, further enhancing the viewing experience. In this way, a system is realized that allows viewers to understand and enjoy sports viewing more deeply.

[0006] "Match data" refers to information that occurs during a sporting event, including player movements, ball position, scores, fouls, and other related data.

[0007] "Analysis" is the process of processing acquired match data and identifying important events and patterns according to a specific purpose.

[0008] A "notification generation means" is a process and mechanism for automatically sending information to a user based on analyzed data.

[0009] "A method for automatically generating highlight videos" refers to a system that selects important scenes after a match has ended and compiles them into a video in a format that is easy for viewers to understand.

[0010] The "communication function" is an interactive platform for users to exchange opinions and share their thoughts. [Brief explanation of the drawing]

[0011] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3]This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0012] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0013] First, let's explain the terminology used in the following explanation.

[0014] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0015] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0016] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0017] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F manages communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0019] [First Embodiment]

[0020] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0021] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0024] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0027] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0031] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0032] This invention is a system designed to make watching sports more interactive and easier to understand, where a server, terminal, and user cooperate to share information and enrich the experience.

[0033] First, the server acquires and analyzes match data in real time. The information collected during the match includes player positions, scores, ball movement, fouls, and more. The server processes this data quickly and applies algorithms to identify important events.

[0034] Next, identified important events are sent to the device and notified to the user. The device receives information from the server and displays notifications so that the user can understand the flow of the match in real time. In addition, when technical terms appear, the system automatically provides explanations of those terms to help the user easily understand the match.

[0035] When a question arises during a match, users can send it to the server via their device. The server receives the question, searches for the relevant information in its database, and sends back the appropriate answer, further enhancing the user's understanding. For example, if a user asks, "What are player A's stats for this season?", the server retrieves the stats information and sends it to the device, allowing the user to check it immediately.

[0036] After the match ends, the server re-analyzes the match data, selects important scenes, and automatically generates a highlight video. This highlight video is quickly delivered to the user via their device, allowing them to quickly review the key points of the match.

[0037] Furthermore, the device provides communication features that allow users to exchange opinions and share their thoughts. Users can share their thoughts and insights with other fans and enjoy the game more deeply through discussion.

[0038] In this way, this system aims to provide a better sports viewing experience through the provision of appropriate information and user interaction according to the progress of the match.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The server retrieves match data in real time from sporting events. It analyzes the data stream provided during the match and extracts information such as player positions, scores, fouls, and ball movement.

[0042] Step 2:

[0043] The server analyzes the acquired match data to identify important events. Here, specific algorithms are used to identify goal scenes, player substitutions, and other key events, and this information is then organized.

[0044] Step 3:

[0045] The server generates notifications based on identified important events. These notifications include information viewers should know, such as when a player scores a goal or when a player receives a warning.

[0046] Step 4:

[0047] The device receives notifications from the server and displays them to the user in real time. Notifications are provided in the form of pop-ups or alerts to ensure the user doesn't miss a match.

[0048] Step 5:

[0049] The device automatically explains technical terms included in notifications. For example, it helps users understand the game more deeply by explaining terms like "offside" and "foul" in an easy-to-understand way.

[0050] Step 6:

[0051] Users can ask questions that arise during a match through their device. For example, they can enter questions about a specific player, such as, "How many assists has player A had this season?"

[0052] Step 7:

[0053] The server receives a question from the user, searches the relevant database, generates an appropriate answer, and sends it back to the terminal. This allows the user to obtain information instantly.

[0054] Step 8:

[0055] The server re-analyzes the match data after the game ends, selects important scenes, and automatically generates a highlight video. It then cuts out and edits scoring scenes and other noteworthy plays.

[0056] Step 9:

[0057] The device delivers the generated highlight video to the user, allowing them to quickly review the key points of the match. Users will then view the match comprehensively through this video.

[0058] Step 10:

[0059] The device provides communication features that allow users to exchange opinions with other fans. Users can share their thoughts after a match or discuss memorable plays during the game.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] In modern sports spectating, spectators demand a deeper understanding and immediate access to information. There is also a need for rapid information sharing regarding in-game events and effective post-game reviews, but integrated systems to achieve these are lacking. In particular, new methods are needed for real-time information analysis, improved understanding of technical terms, and smoother communication among users.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes means for instantly collecting and analyzing competition data and identifying important events, means for generating notifications to inform individuals of important moments, and means for receiving inquiries from individuals during a competition and generating and responding with appropriate responses. This makes it possible to provide important information in real time and to quickly resolve user questions.

[0065] "Competition data" refers to information related to sports and games, including location data, scores, player movements, rule violations, and all other data related to the progress of the match.

[0066] "Immediately" means that the operation or process in question is executed immediately without delay at the location where it takes place.

[0067] "Analysis" refers to the process of processing collected data and deriving useful information or patterns from it.

[0068] A "significant event" refers to any event in the course of a match or game that is particularly noteworthy to spectators and those involved, such as a change in the score, a rule violation, or a strategic turning point.

[0069] "To distinguish" refers to the process of selecting specific elements from multiple data or pieces of information based on particular conditions or criteria.

[0070] "Notification generation means" refers to a function that creates and sends pre-configured messages or alerts within a system to convey specific information to the user.

[0071] "Individual" is a general term used to refer to a user of a system and to identify each user.

[0072] "Receiving" refers to taking in transmitted data or messages and making them ready for processing.

[0073] "Generating a response and sending a reply" refers to the action of creating an appropriate answer or information in response to a received question or request and sending it back to the sender.

[0074] This invention is a system that enables spectators to receive abundant information and gain a deeper understanding of sports and games. It primarily functions through the coordinated interaction of servers, terminals, and users.

[0075] The server uses multiple sensors, cameras, and external data provision APIs to collect and analyze data in real time during the game. The real-time data obtained from this hardware and APIs is processed using machine learning algorithms and statistical analysis techniques. Specific examples include GPS sensors that provide location information to track player movements and image recognition technology to record the movement of the ball. Based on the analysis results, the server identifies important events and proceeds to the next step.

[0076] Next, the device receives notifications from the server and displays important information to the user in real time. To achieve this, the device utilizes push notification technology to directly import the information onto the screen. Furthermore, when technical terms appear during a match, a pre-built glossary function provides accurate information. This allows users to enjoy the match without confusion.

[0077] Furthermore, users can input questions that arise during a match via their device and query the server. The server searches the relevant information infrastructure and sends back an appropriate response. For example, if a user inputs "I want to know the past match record of player B," the server will quickly search the database, retrieve the record, and send the information to the device.

[0078] This system utilizes a generative AI model to support user understanding and quickly resolve inquiries during matches. Examples of specific prompts include, "Please tell me the important events of today's match," and "Please explain player A's performance in detail."

[0079] This advanced system allows individual users to enjoy a deeper sports viewing experience and resolve any concerns immediately.

[0080] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0081] Step 1:

[0082] The server collects competition data in real time. Inputs include real-time data provided by sensors and cameras placed at the match venue, as well as external APIs. This data includes player positions, speeds, scores, and information about events during the match. The server uses data processing software to format this data and convert it into an analyzable state.

[0083] Step 2:

[0084] The server analyzes the collected data and identifies important events. This process uses the formatted data obtained in step 1 as input. The server applies machine learning algorithms to extract important patterns such as player strategies, scoring opportunities, and foul occurrences. The output provides information about these important events.

[0085] Step 3:

[0086] The server prepares to notify the terminal of the identified important event. It uses the information about the important event obtained in step 2 as input. The server uses a notification generation mechanism to generate a message in a user-friendly format. The output is the message sent to the terminal.

[0087] Step 4:

[0088] The device receives notifications about important events sent from the server and displays them to the user. The message obtained in step 3 is used as input. The device displays push notifications on the screen and may also emit audio alerts. The output is communicated to the user visually and aurally.

[0089] Step 5:

[0090] If a user has a question during a match, they send it to the server via their terminal. The input is the question the user enters into their terminal. The output is the question received by the server.

[0091] Step 6:

[0092] The server searches the information infrastructure based on the question received from the user and generates an appropriate response. The input is the user question received in step 5. A generative AI model is used to analyze the relevant data and create the optimal answer. The output is the detailed answer returned to the user.

[0093] Step 7:

[0094] After the match ends, the server re-analyzes the match data, extracts key moments, and generates a highlight video. The input is data collected from the entire match. The server uses a video editing algorithm to automatically select important scenes and create the highlight video. The output is a video format viewable on the user's device.

[0095] Step 8:

[0096] The terminal provides users with generated digest videos and creates a platform for exchanging opinions with other fans. Inputs include video data sent from the server and comments from users. While playing the video, the terminal facilitates communication between users using chat and comment functions. Outputs include the video viewed by users and their shared opinions and comments.

[0097] (Application Example 1)

[0098] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0099] Modern sports viewing demands a real-time, detailed, and interactive experience for spectators. However, typical sports viewing often makes it difficult to immediately grasp information about the progress of the game and technical aspects of the play, and there are limited ways to efficiently review important moments after the game. Furthermore, there is a lack of opportunities for spectators to exchange opinions and receive quick answers to their questions. Thus, the challenge lies in providing means to enjoy and understand sports more deeply.

[0100] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0101] In this invention, the server includes means for acquiring and analyzing match data in real time, means for generating notifications to inform users of important information, and means for receiving questions from users during a match, generating appropriate answers, and sending them back. This enables real-time understanding of the match, comprehension of technical terms, and quick review of important moments after the match. Furthermore, by providing means for users to exchange opinions with each other, it is possible to make the sports viewing experience richer and more interactive.

[0102] "Match data" refers to information that shows the progress of a sporting event in real time, including the positions of participants, the score, the movement of the ball, and any fouls.

[0103] A "notification generation means" is a system or function that transmits information to a user quickly and accurately based on identified important events.

[0104] "Means for receiving questions and generating and sending appropriate answers" refers to a system or function for processing user questions, creating answers by referring to relevant databases, and sending them to the user.

[0105] "Methods for automatically generating highlight videos" refer to automated processes that select important moments after a match has ended, combine them, and convert them into a format that can be viewed in a short amount of time.

[0106] "Means of providing communication functions" refers to an online platform or system for multiple users to exchange opinions and share information.

[0107] "Means of providing glossary" refers to a system or function that provides explanations or descriptions of technical terms or specific plays used during a match to help users understand them.

[0108] The system implementing this invention mainly consists of three components: a server, a terminal, and a user.

[0109] server

[0110] The server acquires and analyzes match data in real time during the game. The analysis uses the Python pandas library to efficiently track player positions and ball movement. When important events are identified, the server generates data to notify the terminal. Furthermore, if a user asks a question during the match, the server uses an artificial intelligence model to generate an answer, searching relevant data storage to reconcile the information.

[0111] terminal

[0112] The terminal displays received data to the user in real time. It immediately notifies users of important events and automatically displays explanations of match terminology and specific plays. After the match ends, it distributes highlight videos generated by the server for easy viewing by users. The terminal also enriches the viewing experience by providing a communication platform where users can exchange opinions.

[0113] User

[0114] Users can watch matches via their devices and receive real-time notifications. If questions arise, they can send questions from their devices to the server and receive immediate answers. They can also exchange opinions about the matches with other fans.

[0115] Specifically, for example, if a user enters a question like "What is offside?" into the terminal, the AI ​​model will process this information and provide a detailed explanation. Furthermore, by using prompts such as "What is player C's goal tally this season?", the system can instantly retrieve player performance information and provide a response to the user. In this way, the system enhances real-time information provision and communication among participants, making sports viewing more interactive and easier to understand.

[0116] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0117] Step 1:

[0118] The server receives match data. Specifically, it inputs player location information, scores, and ball movement in real time from external sensors and streaming APIs. This data is analyzed using the Python pandas library to identify important events (such as goals and fouls). The output of this step is a list of the identified important events.

[0119] Step 2:

[0120] The server generates notification data using a notification generation mechanism based on a list of identified important events. This data processing involves converting the data into a format easily understood by the user. For example, it might create a text notification stating, "Player A scored a goal." This notification data is then sent to the device.

[0121] Step 3:

[0122] The terminal receives notification data sent from the server and displays it on the user's screen as a real-time notification. The input is the notification data, and the output is a pop-up notification generated based on it. In addition, explanations of technical terms included in the notification are displayed using a pre-prepared database.

[0123] Step 4:

[0124] When a user has a question during a match, they type it into their terminal. The entered prompt (for example, "What is offside?") is sent to the server.

[0125] Step 5:

[0126] The server receives a prompt from the user and processes it using a generative AI model. The AI ​​model generates the most appropriate explanation from the relevant data. The output is a text explanation for the question, which is then sent back to the terminal.

[0127] Step 6:

[0128] After the match ends, the server re-analyzes all the match data and extracts the most important moments. The input is the data for the entire match, and the output is a list of highlight scenes based on that data. The automatically generated highlight video is then sent to the terminal.

[0129] Step 7:

[0130] The device receives highlight videos and plays them on the device for the user to watch. Furthermore, users can access a communication platform for exchanging opinions and share their thoughts with other users.

[0131] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0132] This invention is a system that incorporates an emotion engine, in addition to conventional data analysis functions, to recognize the user's emotions and optimize the system's operation, in order to make watching sports more emotionally enriching.

[0133] First, the server acquires and analyzes match data in real time. Here, detailed information about the progress of the match, such as player positions, scores, ball movement, and fouls, is constantly updated, and important events are quickly identified.

[0134] Based on this data, the server uses a notification generation system to notify terminals of important match events. Notifications are made in real time to ensure users stay informed about the progress of the match.

[0135] Furthermore, the device's built-in emotion engine recognizes the user's emotions by analyzing their behavioral data and reactions. This information is combined with match data and used to personalize notifications and feedback based on the user's emotions. For example, if a user reacts with joy to a particular team's score, information about that team will be prioritized in notifications.

[0136] Users can ask questions to the server via their device if they have any doubts during a match. The server searches relevant data, generates a response that takes sentiment into account, and sends it back to the user, thereby improving the user experience.

[0137] After the match ends, the server re-analyzes key moments and, based on feedback from the emotion engine, automatically generates a highlight video focusing on the moments that viewers found particularly interesting. This video is then quickly delivered to the user via their device.

[0138] Furthermore, the emotion engine utilizes users' emotional information to make communication with other fans more effective. For example, it introduces mechanisms to facilitate discussions about sharing emotions such as joy and sadness, deepening interaction among users.

[0139] Thus, this system, which incorporates an emotion engine, goes beyond conventional sports viewing systems to create a personalized information delivery and interaction platform that matches the user's unique emotions.

[0140] The following describes the processing flow.

[0141] Step 1:

[0142] The server retrieves match data from sports events in real time. This data includes player positions, scores, fouls, and the progress of the match, and analysis is performed based on this data.

[0143] Step 2:

[0144] The server analyzes the acquired match data and uses specific algorithms to identify important events, such as scoring opportunities and turning points in the game, making immediate decisions.

[0145] Step 3:

[0146] Based on the analysis results, the server uses a notification generation mechanism to notify the user's terminal of important events in real time. For example, it sends a notification the moment a score is scored.

[0147] Step 4:

[0148] The device uses an emotion engine to analyze user emotions from their reactions and operation history. It utilizes cameras and sensors to capture emotions such as surprise, joy, and tension.

[0149] Step 5:

[0150] Based on the emotional data analyzed by the emotion engine, the device personalizes the content of notifications and displays them to the user. For example, it may adjust the content by providing additional match information when the user is excited.

[0151] Step 6:

[0152] Users can input questions via their devices while watching a match and send them to the server. They can ask about their favorite player's performance during the match.

[0153] Step 7:

[0154] The server receives a question from the user, generates an answer by referring to the relevant database, and sends the answer back to the terminal in the most appropriate format, taking sentiment information into consideration.

[0155] Step 8:

[0156] After the match ends, the server re-analyzes the data accumulated during the match and automatically generates a highlight video, focusing on moments that evoked particularly strong emotional reactions. This video emphasizes the most exciting scenes of the match.

[0157] Step 9:

[0158] The device provides the user with a generated highlight video, allowing for a recap that reflects the flow of emotions. Users relive the excitement of the match while watching the video.

[0159] Step 10:

[0160] The device proposes communication functions based on sentiment analysis, facilitating the exchange of opinions among users. Users can share their impressions and feelings about the match with other fans and enjoy interacting with them.

[0161] (Example 2)

[0162] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0163] In recent years, sports viewing has demanded more than just the results and information provided; it has been sought after for enriching the experience to match the individual emotions and interests of the viewer. However, conventional systems have struggled to recognize viewers' emotions in real time and personalize the information they receive. Furthermore, there is a lack of efficient means to provide viewers with moments of interest during and after a match, and sufficient communication functions to facilitate the sharing of emotions with other fans.

[0164] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0165] In this invention, the server includes means for acquiring and analyzing match information in real time and identifying important events, means for recognizing the user's emotions and personalizing the information based on the user's emotional state, and means for facilitating interaction with other users. This makes it possible to provide information that responds to the individual emotions of viewers, resulting in a more emotionally rich and personalized sports viewing experience.

[0166] "Match information" refers to all data related to the progress of a sports match, such as scores, player positions, and fouls.

[0167] "Real-time" refers to the instantaneous processing of information as the event or process in question unfolds.

[0168] "Analysis" refers to the process used to analyze acquired data and identify important patterns or events.

[0169] A "significant event" refers to an action or outcome in the course of a match that is likely to attract particular attention from viewers.

[0170] "Information generation means" refers to a notification mechanism designed to transmit information obtained from analysis results to users.

[0171] "Emotion recognition" refers to the process of estimating and identifying a user's emotional state based on their behavior and reactions.

[0172] "Personalization" refers to optimizing information according to each user's emotions and interests, and providing it in a way that is tailored to the individual.

[0173] "Means of promoting interaction" refers to platforms and features that help users share their feelings and opinions with one another.

[0174] This invention is a system designed to enhance the experience of watching sports, and it has the function of analyzing the progress of a match in real time and providing information based on the user's emotional state.

[0175] The server acquires match information in real time using a dedicated data acquisition API. This information includes player positions, scoring movements, fouls, and more. The server uses this data to evaluate player performance and employs data processing software (e.g., Pandas) to identify key events in the match. In this process, the server can identify important events more quickly and optimize its operation.

[0176] The device is equipped with an emotion recognition engine that analyzes the user's facial expressions and voice tone. This engine acquires data from the camera and microphone and analyzes it using an emotion recognition service in the cloud (e.g., a general-purpose emotion recognition API). After identifying the user's emotional state, this information is used on the server to generate responses and provide information.

[0177] For example, if a user's expression of joy is recognized when a particular team scores a goal during a match, the server will use that information to send a personalized notification to the device such as, "A fantastic goal by Player A! They're now at the top of the scoring rankings this season!"

[0178] If a user has a question during a match, they can send it to the server via their device. The server uses a generative AI model with a natural language processing engine to analyze the user's inquiry and construct a response based on relevant match information and sentiment data. For example, by entering a prompt such as "What information would you prefer to receive when your favorite team scores?", the server can provide information that meets the user's expectations.

[0179] This system also has a feature that automatically generates highlight videos by re-analyzing important scenes after the match ends. The generated videos incorporate points of interest based on the user's emotions, providing valuable entertainment for the user. It also includes communication features that facilitate sharing emotions and exchanging opinions to deepen interaction among users.

[0180] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0181] Step 1:

[0182] The server receives real-time match information from a data acquisition API during the progress of a sports match. This input includes player positions, scores, foul information, and more. The server processes this data using a Python script and transforms it into a data frame to identify important events. As a result of the analysis, it outputs data that allows for a real-time understanding of the match's progress.

[0183] Step 2:

[0184] The server identifies important events based on the data analyzed in Step 1 and activates the notification generation algorithm. The input is the events deemed important (for example, scoring plays or player substitutions). The server determines the priority of events from the data, creates a message to notify the user's terminal, and outputs it. This message is sent to the terminal immediately.

[0185] Step 3:

[0186] The device uses its built-in camera and microphone to collect the user's facial expressions and voice tone in real time. The input data consists of video and audio, which are then analyzed by an emotion recognition engine. The output generates quantified results of emotional states such as joy, surprise, and sadness, and sends them back to the server.

[0187] Step 4:

[0188] The server analyzes the emotional data received from the terminal using the Emotion API to determine the user's current emotions in detail. The input is the emotional state data obtained in step 3. Based on the analyzed emotional data, it outputs parameters to personalize subsequent notifications and information provision.

[0189] Step 5:

[0190] Users can ask questions to the server via their terminal during a match. This question serves as input, and the server analyzes the question using a generative AI model. It searches relevant match information and sentiment databases, generates an appropriate response, and outputs it. The generated response is then sent to the user.

[0191] Step 6:

[0192] After the match ends, the server analyzes the match data again and automatically generates a highlight video using FFmpeg, focusing on the moments that attracted the user's attention. The input data consists of all the data collected during the match, and the highlight video is output based on this data and quickly delivered to the user via their terminal.

[0193] Step 7:

[0194] The emotion engine-powered platform provides a discussion forum for users to share their joys and sorrows. Using users' emotional data as input, topics that facilitate interaction are automatically suggested, allowing users to deepen their connections with other fans. The output is lively exchange of opinions and interaction within the discussion space.

[0195] (Application Example 2)

[0196] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0197] Current sports viewing systems have the problem of providing uniform information when watching a match, failing to adequately respond to the individual interests and emotions of users. In particular, there is a need to quickly provide important information from large amounts of data and optimize information delivery according to the user's emotions. Furthermore, there is a lack of means to make sports viewing more emotionally enriching and to promote active communication with other viewers.

[0198] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0199] In this invention, the server includes means for acquiring and analyzing match information in real time and identifying important events, means for immediately notifying the user's terminal of important events using notification generation means, and means for recognizing the user's emotions and optimizing the displayed content based on those emotions. This makes it possible to provide users with important match information in real time and to provide information that is tailored to their emotions.

[0200] "Match information" refers to data that shows the progress of a sports competition in real time, including the movements of players, scores, fouls, and other related events.

[0201] "Analysis" is the process of thoroughly analyzing acquired match information to identify important events.

[0202] A "significant event" refers to any event that significantly impacts the progress of a match, such as scoring, fouls, or player substitutions.

[0203] A "notification generation method" is a technique for notifying users of important events in real time on their devices based on information obtained from a server.

[0204] "User's device" refers to an electronic device used by the user to receive notifications, and includes smartphones, smart glasses, and other similar devices.

[0205] "User emotions" refers to the emotional reactions that users exhibit while watching a match, such as joy, sadness, and surprise.

[0206] "Optimizing displayed content" is a process that adjusts the information presented to match the user's interests and emotions based on the acquired user sentiment information.

[0207] The system of the present invention is designed to enhance the sports viewing experience. This system primarily consists of a server, terminals, and an emotion recognition engine. Specific embodiments are described below.

[0208] First, the server acquires and analyzes match information in real time. Specifically, it collects data such as player movements, scores, and ball position during the match to identify important events. This analysis process utilizes reinforcement learning and data mining techniques to enable rapid and accurate data processing.

[0209] Next, the server uses a notification generation mechanism to send information about the identified important events to the terminal in real time. This terminal includes smart glasses worn by the user or a portable communication device, allowing the user to instantly grasp the important moments of the match.

[0210] Furthermore, an emotion recognition engine runs on the device to analyze user behavior and reactions. This allows the system to recognize what emotions the user is experiencing and optimize the displayed content based on those emotions. For example, if a user shows a joyful reaction, match highlights and related information tailored to that emotion will be prioritized.

[0211] As a concrete example, consider a soccer match. If a user cheers during a goal scored by a particular team while watching the match, the emotion recognition engine will recognize this action as "joy." The server will then display a replay of the relevant goal and project it onto the smart glasses. Another example of a prompt message is, "If a user claps or makes other actions while watching a sporting event, please tell us how to recognize emotions such as joy based on those actions and how to provide notifications or information during specific match events."

[0212] Thus, the system of the present invention provides users with a more emotionally resonant sports viewing experience based on real-time analysis and emotional feedback.

[0213] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0214] Step 1:

[0215] The server acquires match information in real time. Specifically, it aggregates data obtained from sensors and cameras at sports stadiums. Raw match data is used as input, and from this, information such as player positions, scores, and fouls is obtained. This input data is converted into a specific format and prepared for analysis in the next step.

[0216] Step 2:

[0217] The server analyzes the match data acquired in Step 1. The analysis utilizes reinforcement learning algorithms and data mining techniques to identify important events from the input match data. Examples of these important events include goals, penalties, and player substitutions. The output generates information about the identified important events.

[0218] Step 3:

[0219] The server sends information to the terminal using a notification generation mechanism based on identified important events. The notification includes important moments in the progress of the match, allowing the user (terminal) to receive information in real time. The input is the information of the important events output in step 2, and the output is a notification message generated for the user's terminal.

[0220] Step 4:

[0221] After receiving a notification, the device uses an emotion recognition engine to analyze the user's emotions. The user's actions, facial expressions, and voice data are input into the emotion recognition engine, and as a result, emotions such as joy, surprise, and tension are output. This emotion information is used to optimize the displayed content in the next step.

[0222] Step 5:

[0223] The device optimizes its display content using output data based on an emotion recognition engine. Specifically, it prioritizes displaying match highlights and related information according to the emotions expressed by the user. This process involves the operation of the visual display on the device, where the input is the emotion information from step 4, and the output is user-tailored content.

[0224] Step 6:

[0225] Based on the information displayed on the device, users can input further questions into the server. The server receives these questions as input, searches its relevant database to generate appropriate answers, and sends those answers back to the device. As output, users are provided with answers that satisfy them, enriching their viewing experience.

[0226] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0227] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0228] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0229] [Second Embodiment]

[0230] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0231] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0232] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0233] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0234] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0235] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0236] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0237] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0238] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0239] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0240] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0241] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0242] This invention is a system designed to make watching sports more interactive and easier to understand, where a server, terminal, and user cooperate to share information and enrich the experience.

[0243] First, the server acquires and analyzes match data in real time. The information collected during the match includes player positions, scores, ball movement, fouls, and more. The server processes this data quickly and applies algorithms to identify important events.

[0244] Next, identified important events are sent to the device and notified to the user. The device receives information from the server and displays notifications so that the user can understand the flow of the match in real time. In addition, when technical terms appear, the system automatically provides explanations of those terms to help the user easily understand the match.

[0245] When a question arises during a match, users can send it to the server via their device. The server receives the question, searches for the relevant information in its database, and sends back the appropriate answer, further enhancing the user's understanding. For example, if a user asks, "What are player A's stats for this season?", the server retrieves the stats information and sends it to the device, allowing the user to check it immediately.

[0246] After the match ends, the server re-analyzes the match data, selects important scenes, and automatically generates a highlight video. This highlight video is quickly delivered to the user via their device, allowing them to quickly review the key points of the match.

[0247] Furthermore, the device provides communication features that allow users to exchange opinions and share their thoughts. Users can share their thoughts and insights with other fans and enjoy the game more deeply through discussion.

[0248] In this way, this system aims to provide a better sports viewing experience through the provision of appropriate information and user interaction according to the progress of the match.

[0249] The following describes the processing flow.

[0250] Step 1:

[0251] The server retrieves match data in real time from sporting events. It analyzes the data stream provided during the match and extracts information such as player positions, scores, fouls, and ball movement.

[0252] Step 2:

[0253] The server analyzes the acquired match data to identify important events. Here, specific algorithms are used to identify goal scenes, player substitutions, and other key events, and this information is then organized.

[0254] Step 3:

[0255] The server generates notifications based on identified important events. These notifications include information viewers should know, such as when a player scores a goal or when a player receives a warning.

[0256] Step 4:

[0257] The device receives notifications from the server and displays them to the user in real time. Notifications are provided in the form of pop-ups or alerts to ensure the user doesn't miss a match.

[0258] Step 5:

[0259] The device automatically explains technical terms included in notifications. For example, it helps users understand the game more deeply by explaining terms like "offside" and "foul" in an easy-to-understand way.

[0260] Step 6:

[0261] Users can ask questions that arise during a match through their device. For example, they can enter questions about a specific player, such as, "How many assists has player A had this season?"

[0262] Step 7:

[0263] The server receives a question from the user, searches the relevant database, generates an appropriate answer, and sends it back to the terminal. This allows the user to obtain information instantly.

[0264] Step 8:

[0265] The server re-analyzes the match data after the game ends, selects important scenes, and automatically generates a highlight video. It then cuts out and edits scoring scenes and other noteworthy plays.

[0266] Step 9:

[0267] The device delivers the generated highlight video to the user, allowing them to quickly review the key points of the match. Users will then view the match comprehensively through this video.

[0268] Step 10:

[0269] The device provides communication features that allow users to exchange opinions with other fans. Users can share their thoughts after a match or discuss memorable plays during the game.

[0270] (Example 1)

[0271] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0272] In modern sports spectating, spectators demand a deeper understanding and immediate access to information. There is also a need for rapid information sharing regarding in-game events and effective post-game reviews, but integrated systems to achieve these are lacking. In particular, new methods are needed for real-time information analysis, improved understanding of technical terms, and smoother communication among users.

[0273] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0274] In this invention, the server includes means for instantly collecting and analyzing competition data and identifying important events, means for generating notifications to inform individuals of important moments, and means for receiving inquiries from individuals during a competition and generating and responding with appropriate responses. This makes it possible to provide important information in real time and to quickly resolve user questions.

[0275] "Competition data" refers to information related to sports and games, including location data, scores, player movements, rule violations, and all other data related to the progress of the match.

[0276] "Immediately" means that the operation or process in question is executed immediately without delay at the location where it takes place.

[0277] "Analysis" refers to the process of processing collected data and deriving useful information or patterns from it.

[0278] A "significant event" refers to any event in the course of a match or game that is particularly noteworthy to spectators and those involved, such as a change in the score, a rule violation, or a strategic turning point.

[0279] "To distinguish" refers to the process of selecting specific elements from multiple data or pieces of information based on particular conditions or criteria.

[0280] "Notification generation means" refers to a function that creates and sends pre-configured messages or alerts within a system to convey specific information to the user.

[0281] "Individual" is a general term used to refer to a user of a system and to identify each user.

[0282] "Receiving" refers to taking in transmitted data or messages and making them ready for processing.

[0283] "Generating and returning a response" refers to the operation of creating an appropriate answer or information for the received question or request and sending it back to the sender.

[0284] This invention is a system that enables rich information provision and deeper understanding for spectators in sports and games. It mainly functions through the mutual cooperation of a server, a terminal, and a user.

[0285] The server uses multiple sensors, cameras, and external data - providing APIs to immediately collect and analyze data during the competition. The real - time data obtained from these hardware and APIs is processed using machine - learning algorithms and statistical analysis methods. Specific examples include a GPS sensor that provides position information for tracking the movements of players and image - recognition technology for recording the movement of the ball. Based on the analysis results, the server discriminates important events and proceeds to the next step.

[0286] Next, the terminal receives notifications from the server and displays important information to the user in real - time. For this purpose, the terminal utilizes push - notification technology and directly imports it onto the screen. Also, when technical terms appear during the game, it uses a pre - incorporated term - explanation function to provide accurate information. This enables the user to enjoy the game situation without confusion.

[0287] Furthermore, the user can input questions that arise during the game via the terminal and inquire with the server. The server searches the relevant information base and returns an appropriate response. For example, when the user inputs "I want to know the past match record of Player B", the server quickly investigates the database, obtains the record, and transmits the information to the terminal.

[0288] This system utilizes a generative AI model to support the user's understanding and quickly solve inquiries during the game. Examples of specific prompt texts include "Please tell me the important events of today's game." and "Please explain in detail about Player A's performance."

[0289] This advanced system allows individual users to enjoy a deeper sports viewing experience and resolve any concerns immediately.

[0290] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0291] Step 1:

[0292] The server collects competition data in real time. Inputs include real-time data provided by sensors and cameras placed at the match venue, as well as external APIs. This data includes player positions, speeds, scores, and information about events during the match. The server uses data processing software to format this data and convert it into an analyzable state.

[0293] Step 2:

[0294] The server analyzes the collected data and identifies important events. This process uses the formatted data obtained in step 1 as input. The server applies machine learning algorithms to extract important patterns such as player strategies, scoring opportunities, and foul occurrences. The output provides information about these important events.

[0295] Step 3:

[0296] The server prepares to notify the terminal of the identified important event. It uses the information about the important event obtained in step 2 as input. The server uses a notification generation mechanism to generate a message in a user-friendly format. The output is the message sent to the terminal.

[0297] Step 4:

[0298] The device receives notifications about important events sent from the server and displays them to the user. The message obtained in step 3 is used as input. The device displays push notifications on the screen and may also emit audio alerts. The output is communicated to the user visually and aurally.

[0299] Step 5:

[0300] If a user has a question during a match, they send it to the server via their terminal. The input is the question the user enters into their terminal. The output is the question received by the server.

[0301] Step 6:

[0302] The server searches the information infrastructure based on the question received from the user and generates an appropriate response. The input is the user question received in step 5. A generative AI model is used to analyze the relevant data and create the optimal answer. The output is the detailed answer returned to the user.

[0303] Step 7:

[0304] After the match ends, the server re-analyzes the match data, extracts key moments, and generates a highlight video. The input is data collected from the entire match. The server uses a video editing algorithm to automatically select important scenes and create the highlight video. The output is a video format viewable on the user's device.

[0305] Step 8:

[0306] The terminal provides users with generated digest videos and creates a platform for exchanging opinions with other fans. Inputs include video data sent from the server and comments from users. While playing the video, the terminal facilitates communication between users using chat and comment functions. Outputs include the video viewed by users and their shared opinions and comments.

[0307] (Application Example 1)

[0308] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0309] Modern sports watching is required to provide a detailed and interactive experience to viewers in real time. However, in normal sports watching, it is difficult to immediately grasp information regarding the progress of the game and specialized plays, and there are also limited means to efficiently review important scenes after the game. Furthermore, there is a lack of opinion exchange among viewers and quick answers to questions. Thus, it is an issue to provide means to more deeply enjoy and understand sports.

[0310] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following respective means.

[0311] In this invention, the server includes means for acquiring and analyzing game data in real time, notification generation means for notifying users of important information, and means for receiving questions from users during the game, generating appropriate answers, and returning them. Thereby, it becomes possible to grasp the content of the game in real time, understand technical terms, and quickly review important scenes after the game. Also, by providing means for users to exchange opinions, it is possible to make the sports watching experience richer and more interactive.

[0312] "Game data" is information that shows in real time the progress of a sports competition, such as the positions of participants in a sports competition, scores, the movement of the ball, and fouls.

[0313] "Notification generation means" is a system or function for quickly and accurately transmitting information to users based on specified important events.

[0314] "Means for receiving questions and generating and sending appropriate answers" refers to a system or function for processing user questions, creating answers by referring to relevant databases, and sending them to the user.

[0315] "Methods for automatically generating highlight videos" refer to automated processes that select important moments after a match has ended, combine them, and convert them into a format that can be viewed in a short amount of time.

[0316] "Means of providing communication functions" refers to an online platform or system for multiple users to exchange opinions and share information.

[0317] "Means of providing glossary" refers to a system or function that provides explanations or descriptions of technical terms or specific plays used during a match to help users understand them.

[0318] The system implementing this invention mainly consists of three components: a server, a terminal, and a user.

[0319] server

[0320] The server acquires and analyzes match data in real time during the game. The analysis uses the Python pandas library to efficiently track player positions and ball movement. When important events are identified, the server generates data to notify the terminal. Furthermore, if a user asks a question during the match, the server uses an artificial intelligence model to generate an answer, searching relevant data storage to reconcile the information.

[0321] terminal

[0322] The terminal displays received data to the user in real time. It immediately notifies users of important events and automatically displays explanations of match terminology and specific plays. After the match ends, it distributes highlight videos generated by the server for easy viewing by users. The terminal also enriches the viewing experience by providing a communication platform where users can exchange opinions.

[0323] User

[0324] Users can watch matches via their devices and receive real-time notifications. If questions arise, they can send questions from their devices to the server and receive immediate answers. They can also exchange opinions about the matches with other fans.

[0325] Specifically, for example, if a user enters a question like "What is offside?" into the terminal, the AI ​​model will process this information and provide a detailed explanation. Furthermore, by using prompts such as "What is player C's goal tally this season?", the system can instantly retrieve player performance information and provide a response to the user. In this way, the system enhances real-time information provision and communication among participants, making sports viewing more interactive and easier to understand.

[0326] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0327] Step 1:

[0328] The server receives match data. Specifically, it inputs player location information, scores, and ball movement in real time from external sensors and streaming APIs. This data is analyzed using the Python pandas library to identify important events (such as goals and fouls). The output of this step is a list of the identified important events.

[0329] Step 2:

[0330] The server generates notification data using a notification generation mechanism based on a list of identified important events. This data processing involves converting the data into a format easily understood by the user. For example, it might create a text notification stating, "Player A scored a goal." This notification data is then sent to the device.

[0331] Step 3:

[0332] The terminal receives notification data sent from the server and displays it on the user's screen as a real-time notification. The input is the notification data, and the output is a pop-up notification generated based on it. In addition, explanations of technical terms included in the notification are displayed using a pre-prepared database.

[0333] Step 4:

[0334] When a user has a question during a match, they type it into their terminal. The entered prompt (for example, "What is offside?") is sent to the server.

[0335] Step 5:

[0336] The server receives a prompt from the user and processes it using a generative AI model. The AI ​​model generates the most appropriate explanation from the relevant data. The output is a text explanation for the question, which is then sent back to the terminal.

[0337] Step 6:

[0338] After the match ends, the server re-analyzes all the match data and extracts the most important moments. The input is the data for the entire match, and the output is a list of highlight scenes based on that data. The automatically generated highlight video is then sent to the terminal.

[0339] Step 7:

[0340] The device receives highlight videos and plays them on the device for the user to watch. Furthermore, users can access a communication platform for exchanging opinions and share their thoughts with other users.

[0341] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0342] This invention is a system that incorporates an emotion engine, in addition to conventional data analysis functions, to recognize the user's emotions and optimize the system's operation, in order to make watching sports more emotionally enriching.

[0343] First, the server acquires and analyzes match data in real time. Here, detailed information about the progress of the match, such as player positions, scores, ball movement, and fouls, is constantly updated, and important events are quickly identified.

[0344] Based on this data, the server uses a notification generation system to notify terminals of important match events. Notifications are made in real time to ensure users stay informed about the progress of the match.

[0345] Furthermore, the device's built-in emotion engine recognizes the user's emotions by analyzing their behavioral data and reactions. This information is combined with match data and used to personalize notifications and feedback based on the user's emotions. For example, if a user reacts with joy to a particular team's score, information about that team will be prioritized in notifications.

[0346] Users can ask questions to the server via their device if they have any doubts during a match. The server searches relevant data, generates a response that takes sentiment into account, and sends it back to the user, thereby improving the user experience.

[0347] After the match ends, the server re-analyzes key moments and, based on feedback from the emotion engine, automatically generates a highlight video focusing on the moments that viewers found particularly interesting. This video is then quickly delivered to the user via their device.

[0348] Furthermore, the emotion engine utilizes users' emotional information to make communication with other fans more effective. For example, it introduces mechanisms to facilitate discussions about sharing emotions such as joy and sadness, deepening interaction among users.

[0349] Thus, this system, which incorporates an emotion engine, goes beyond conventional sports viewing systems to create a personalized information delivery and interaction platform that matches the user's unique emotions.

[0350] The following describes the processing flow.

[0351] Step 1:

[0352] The server retrieves match data from sports events in real time. This data includes player positions, scores, fouls, and the progress of the match, and analysis is performed based on this data.

[0353] Step 2:

[0354] The server analyzes the acquired match data and uses specific algorithms to identify important events, such as scoring opportunities and turning points in the game, making immediate decisions.

[0355] Step 3:

[0356] Based on the analysis results, the server uses a notification generation mechanism to notify the user's terminal of important events in real time. For example, it sends a notification the moment a score is scored.

[0357] Step 4:

[0358] The device uses an emotion engine to analyze user emotions from their reactions and operation history. It utilizes cameras and sensors to capture emotions such as surprise, joy, and tension.

[0359] Step 5:

[0360] Based on the emotional data analyzed by the emotion engine, the device personalizes the content of notifications and displays them to the user. For example, it may adjust the content by providing additional match information when the user is excited.

[0361] Step 6:

[0362] Users can input questions via their devices while watching a match and send them to the server. They can ask about their favorite player's performance during the match.

[0363] Step 7:

[0364] The server receives a question from the user, generates an answer by referring to the relevant database, and sends the answer back to the terminal in the most appropriate format, taking sentiment information into consideration.

[0365] Step 8:

[0366] After the match ends, the server re-analyzes the data accumulated during the match and automatically generates a highlight video, focusing on moments that evoked particularly strong emotional reactions. This video emphasizes the most exciting scenes of the match.

[0367] Step 9:

[0368] The device provides the user with a generated highlight video, allowing for a recap that reflects the flow of emotions. Users relive the excitement of the match while watching the video.

[0369] Step 10:

[0370] The device proposes communication functions based on sentiment analysis, facilitating the exchange of opinions among users. Users can share their impressions and feelings about the match with other fans and enjoy interacting with them.

[0371] (Example 2)

[0372] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0373] In recent years, sports viewing has demanded more than just the results and information provided; it has been sought after for enriching the experience to match the individual emotions and interests of the viewer. However, conventional systems have struggled to recognize viewers' emotions in real time and personalize the information they receive. Furthermore, there is a lack of efficient means to provide viewers with moments of interest during and after a match, and sufficient communication functions to facilitate the sharing of emotions with other fans.

[0374] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0375] In this invention, the server includes means for acquiring and analyzing match information in real time and identifying important events, means for recognizing the user's emotions and personalizing the information based on the user's emotional state, and means for facilitating interaction with other users. This makes it possible to provide information that responds to the individual emotions of viewers, resulting in a more emotionally rich and personalized sports viewing experience.

[0376] "Match information" refers to all data related to the progress of a sports match, such as scores, player positions, and fouls.

[0377] "Real-time" refers to the instantaneous processing of information as the event or process in question unfolds.

[0378] "Analysis" refers to the process used to analyze acquired data and identify important patterns or events.

[0379] A "significant event" refers to an action or outcome in the course of a match that is likely to attract particular attention from viewers.

[0380] "Information generation means" refers to a notification mechanism designed to transmit information obtained from analysis results to users.

[0381] "Emotion recognition" refers to the process of estimating and identifying a user's emotional state based on their behavior and reactions.

[0382] "Personalization" refers to optimizing information according to each user's emotions and interests, and providing it in a way that is tailored to the individual.

[0383] "Means of promoting interaction" refers to platforms and features that help users share their feelings and opinions with one another.

[0384] This invention is a system designed to enhance the experience of watching sports, and it has the function of analyzing the progress of a match in real time and providing information based on the user's emotional state.

[0385] The server acquires match information in real time using a dedicated data acquisition API. This information includes player positions, scoring movements, fouls, and more. The server uses this data to evaluate player performance and employs data processing software (e.g., Pandas) to identify key events in the match. In this process, the server can identify important events more quickly and optimize its operation.

[0386] The device is equipped with an emotion recognition engine that analyzes the user's facial expressions and voice tone. This engine acquires data from the camera and microphone and analyzes it using an emotion recognition service in the cloud (e.g., a general-purpose emotion recognition API). After identifying the user's emotional state, this information is used on the server to generate responses and provide information.

[0387] For example, if a user's expression of joy is recognized when a particular team scores a goal during a match, the server will use that information to send a personalized notification to the device such as, "A fantastic goal by Player A! They're now at the top of the scoring rankings this season!"

[0388] If a user has a question during a match, they can send it to the server via their device. The server uses a generative AI model with a natural language processing engine to analyze the user's inquiry and construct a response based on relevant match information and sentiment data. For example, by entering a prompt such as "What information would you prefer to receive when your favorite team scores?", the server can provide information that meets the user's expectations.

[0389] This system also has a feature that automatically generates highlight videos by re-analyzing important scenes after the match ends. The generated videos incorporate points of interest based on the user's emotions, providing valuable entertainment for the user. It also includes communication features that facilitate sharing emotions and exchanging opinions to deepen interaction among users.

[0390] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0391] Step 1:

[0392] The server receives real-time match information from a data acquisition API during the progress of a sports match. This input includes player positions, scores, foul information, and more. The server processes this data using a Python script and transforms it into a data frame to identify important events. As a result of the analysis, it outputs data that allows for a real-time understanding of the match's progress.

[0393] Step 2:

[0394] The server identifies important events based on the data analyzed in Step 1 and activates the notification generation algorithm. The input is the events deemed important (for example, scoring plays or player substitutions). The server determines the priority of events from the data, creates a message to notify the user's terminal, and outputs it. This message is sent to the terminal immediately.

[0395] Step 3:

[0396] The device uses its built-in camera and microphone to collect the user's facial expressions and voice tone in real time. The input data consists of video and audio, which are then analyzed by an emotion recognition engine. The output generates quantified results of emotional states such as joy, surprise, and sadness, and sends them back to the server.

[0397] Step 4:

[0398] The server analyzes the emotional data received from the terminal using the Emotion API to determine the user's current emotions in detail. The input is the emotional state data obtained in step 3. Based on the analyzed emotional data, it outputs parameters to personalize subsequent notifications and information provision.

[0399] Step 5:

[0400] Users can ask questions to the server via their terminal during a match. This question serves as input, and the server analyzes the question using a generative AI model. It searches relevant match information and sentiment databases, generates an appropriate response, and outputs it. The generated response is then sent to the user.

[0401] Step 6:

[0402] After the match ends, the server analyzes the match data again and automatically generates a highlight video using FFmpeg, focusing on the moments that attracted the user's attention. The input data consists of all the data collected during the match, and the highlight video is output based on this data and quickly delivered to the user via their terminal.

[0403] Step 7:

[0404] The emotion engine-powered platform provides a discussion forum for users to share their joys and sorrows. Using users' emotional data as input, topics that facilitate interaction are automatically suggested, allowing users to deepen their connections with other fans. The output is lively exchange of opinions and interaction within the discussion space.

[0405] (Application Example 2)

[0406] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0407] Current sports viewing systems have the problem of providing uniform information when watching a match, failing to adequately respond to the individual interests and emotions of users. In particular, there is a need to quickly provide important information from large amounts of data and optimize information delivery according to the user's emotions. Furthermore, there is a lack of means to make sports viewing more emotionally enriching and to promote active communication with other viewers.

[0408] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0409] In this invention, the server includes means for acquiring and analyzing match information in real time and identifying important events, means for immediately notifying the user's terminal of important events using notification generation means, and means for recognizing the user's emotions and optimizing the displayed content based on those emotions. This makes it possible to provide users with important match information in real time and to provide information that is tailored to their emotions.

[0410] "Match information" refers to data that shows the progress of a sports competition in real time, including the movements of players, scores, fouls, and other related events.

[0411] "Analysis" is the process of thoroughly analyzing acquired match information to identify important events.

[0412] A "significant event" refers to any event that significantly impacts the progress of a match, such as scoring, fouls, or player substitutions.

[0413] A "notification generation method" is a technique for notifying users of important events in real time on their devices based on information obtained from a server.

[0414] "User's device" refers to an electronic device used by the user to receive notifications, and includes smartphones, smart glasses, and other similar devices.

[0415] "User emotions" refers to the emotional reactions that users exhibit while watching a match, such as joy, sadness, and surprise.

[0416] "Optimizing displayed content" is a process that adjusts the information presented to match the user's interests and emotions based on the acquired user sentiment information.

[0417] The system of the present invention is designed to enhance the sports viewing experience. This system primarily consists of a server, terminals, and an emotion recognition engine. Specific embodiments are described below.

[0418] First, the server acquires and analyzes match information in real time. Specifically, it collects data such as player movements, scores, and ball position during the match to identify important events. This analysis process utilizes reinforcement learning and data mining techniques to enable rapid and accurate data processing.

[0419] Next, the server uses a notification generation mechanism to send information about the identified important events to the terminal in real time. This terminal includes smart glasses worn by the user or a portable communication device, allowing the user to instantly grasp the important moments of the match.

[0420] Furthermore, an emotion recognition engine runs on the device to analyze user behavior and reactions. This allows the system to recognize what emotions the user is experiencing and optimize the displayed content based on those emotions. For example, if a user shows a joyful reaction, match highlights and related information tailored to that emotion will be prioritized.

[0421] As a concrete example, consider a soccer match. If a user cheers during a goal scored by a particular team while watching the match, the emotion recognition engine will recognize this action as "joy." The server will then display a replay of the relevant goal and project it onto the smart glasses. Another example of a prompt message is, "If a user claps or makes other actions while watching a sporting event, please tell us how to recognize emotions such as joy based on those actions and how to provide notifications or information during specific match events."

[0422] Thus, the system of the present invention provides users with a more emotionally resonant sports viewing experience based on real-time analysis and emotional feedback.

[0423] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0424] Step 1:

[0425] The server acquires match information in real time. Specifically, it aggregates data obtained from sensors and cameras at sports stadiums. Raw match data is used as input, and from this, information such as player positions, scores, and fouls is obtained. This input data is converted into a specific format and prepared for analysis in the next step.

[0426] Step 2:

[0427] The server analyzes the match data acquired in Step 1. The analysis utilizes reinforcement learning algorithms and data mining techniques to identify important events from the input match data. Examples of these important events include goals, penalties, and player substitutions. The output generates information about the identified important events.

[0428] Step 3:

[0429] The server sends information to the terminal using a notification generation mechanism based on identified important events. The notification includes important moments in the progress of the match, allowing the user (terminal) to receive information in real time. The input is the information of the important events output in step 2, and the output is a notification message generated for the user's terminal.

[0430] Step 4:

[0431] After receiving a notification, the device uses an emotion recognition engine to analyze the user's emotions. The user's actions, facial expressions, and voice data are input into the emotion recognition engine, and as a result, emotions such as joy, surprise, and tension are output. This emotion information is used to optimize the displayed content in the next step.

[0432] Step 5:

[0433] The device optimizes its display content using output data based on an emotion recognition engine. Specifically, it prioritizes displaying match highlights and related information according to the emotions expressed by the user. This process involves the operation of the visual display on the device, where the input is the emotion information from step 4, and the output is user-tailored content.

[0434] Step 6:

[0435] Based on the information displayed on the device, users can input further questions into the server. The server receives these questions as input, searches its relevant database to generate appropriate answers, and sends those answers back to the device. As output, users are provided with answers that satisfy them, enriching their viewing experience.

[0436] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0437] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0438] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0439] [Third Embodiment]

[0440] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0441] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0442] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0443] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0444] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0445] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0446] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0447] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0448] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0449] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0450] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0451] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0452] This invention is a system designed to make watching sports more interactive and easier to understand, where a server, terminal, and user cooperate to share information and enrich the experience.

[0453] First, the server acquires and analyzes match data in real time. The information collected during the match includes player positions, scores, ball movement, fouls, and more. The server processes this data quickly and applies algorithms to identify important events.

[0454] Next, identified important events are sent to the device and notified to the user. The device receives information from the server and displays notifications so that the user can understand the flow of the match in real time. In addition, when technical terms appear, the system automatically provides explanations of those terms to help the user easily understand the match.

[0455] When a question arises during a match, users can send it to the server via their device. The server receives the question, searches for the relevant information in its database, and sends back the appropriate answer, further enhancing the user's understanding. For example, if a user asks, "What are player A's stats for this season?", the server retrieves the stats information and sends it to the device, allowing the user to check it immediately.

[0456] After the match ends, the server re-analyzes the match data, selects important scenes, and automatically generates a highlight video. This highlight video is quickly delivered to the user via their device, allowing them to quickly review the key points of the match.

[0457] Furthermore, the device provides communication features that allow users to exchange opinions and share their thoughts. Users can share their thoughts and insights with other fans and enjoy the game more deeply through discussion.

[0458] In this way, this system aims to provide a better sports viewing experience through the provision of appropriate information and user interaction according to the progress of the match.

[0459] The following describes the processing flow.

[0460] Step 1:

[0461] The server retrieves match data in real time from sporting events. It analyzes the data stream provided during the match and extracts information such as player positions, scores, fouls, and ball movement.

[0462] Step 2:

[0463] The server analyzes the acquired match data to identify important events. Here, specific algorithms are used to identify goal scenes, player substitutions, and other key events, and this information is then organized.

[0464] Step 3:

[0465] The server generates notifications based on identified important events. These notifications include information viewers should know, such as when a player scores a goal or when a player receives a warning.

[0466] Step 4:

[0467] The device receives notifications from the server and displays them to the user in real time. Notifications are provided in the form of pop-ups or alerts to ensure the user doesn't miss a match.

[0468] Step 5:

[0469] The device automatically explains technical terms included in notifications. For example, it helps users understand the game more deeply by explaining terms like "offside" and "foul" in an easy-to-understand way.

[0470] Step 6:

[0471] Users can ask questions that arise during a match through their device. For example, they can enter questions about a specific player, such as, "How many assists has player A had this season?"

[0472] Step 7:

[0473] The server receives a question from the user, searches the relevant database, generates an appropriate answer, and sends it back to the terminal. This allows the user to obtain information instantly.

[0474] Step 8:

[0475] The server re-analyzes the match data after the game ends, selects important scenes, and automatically generates a highlight video. It then cuts out and edits scoring scenes and other noteworthy plays.

[0476] Step 9:

[0477] The device delivers the generated highlight video to the user, allowing them to quickly review the key points of the match. Users will then view the match comprehensively through this video.

[0478] Step 10:

[0479] The device provides communication features that allow users to exchange opinions with other fans. Users can share their thoughts after a match or discuss memorable plays during the game.

[0480] (Example 1)

[0481] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0482] In modern sports spectating, spectators demand a deeper understanding and immediate access to information. There is also a need for rapid information sharing regarding in-game events and effective post-game reviews, but integrated systems to achieve these are lacking. In particular, new methods are needed for real-time information analysis, improved understanding of technical terms, and smoother communication among users.

[0483] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0484] In this invention, the server includes means for instantly collecting and analyzing competition data and identifying important events, means for generating notifications to inform individuals of important moments, and means for receiving inquiries from individuals during a competition and generating and responding with appropriate responses. This makes it possible to provide important information in real time and to quickly resolve user questions.

[0485] "Competition data" refers to information related to sports and games, including location data, scores, player movements, rule violations, and all other data related to the progress of the match.

[0486] "Immediately" means that the operation or process in question is executed immediately without delay at the location where it takes place.

[0487] "Analysis" refers to the process of processing collected data and deriving useful information or patterns from it.

[0488] A "significant event" refers to any event in the course of a match or game that is particularly noteworthy to spectators and those involved, such as a change in the score, a rule violation, or a strategic turning point.

[0489] "To distinguish" refers to the process of selecting specific elements from multiple data or pieces of information based on particular conditions or criteria.

[0490] "Notification generation means" refers to a function that creates and sends pre-configured messages or alerts within a system to convey specific information to the user.

[0491] "Individual" is a general term used to refer to a user of a system and to identify each user.

[0492] "Receiving" refers to taking in transmitted data or messages and making them ready for processing.

[0493] "Generating a response and sending a reply" refers to the action of creating an appropriate answer or information in response to a received question or request and sending it back to the sender.

[0494] This invention is a system that enables spectators to receive abundant information and gain a deeper understanding of sports and games. It primarily functions through the coordinated interaction of servers, terminals, and users.

[0495] The server uses multiple sensors, cameras, and external data provision APIs to collect and analyze data in real time during the game. The real-time data obtained from this hardware and APIs is processed using machine learning algorithms and statistical analysis techniques. Specific examples include GPS sensors that provide location information to track player movements and image recognition technology to record the movement of the ball. Based on the analysis results, the server identifies important events and proceeds to the next step.

[0496] Next, the device receives notifications from the server and displays important information to the user in real time. To achieve this, the device utilizes push notification technology to directly import the information onto the screen. Furthermore, when technical terms appear during a match, a pre-built glossary function provides accurate information. This allows users to enjoy the match without confusion.

[0497] Furthermore, users can input questions that arise during a match via their device and query the server. The server searches the relevant information infrastructure and sends back an appropriate response. For example, if a user inputs "I want to know the past match record of player B," the server will quickly search the database, retrieve the record, and send the information to the device.

[0498] This system utilizes a generative AI model to support user understanding and quickly resolve inquiries during matches. Examples of specific prompts include, "Please tell me the important events of today's match," and "Please explain player A's performance in detail."

[0499] This advanced system allows individual users to enjoy a deeper sports viewing experience and resolve any concerns immediately.

[0500] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0501] Step 1:

[0502] The server collects competition data in real time. Inputs include real-time data provided by sensors and cameras placed at the match venue, as well as external APIs. This data includes player positions, speeds, scores, and information about events during the match. The server uses data processing software to format this data and convert it into an analyzable state.

[0503] Step 2:

[0504] The server analyzes the collected data and identifies important events. This process uses the formatted data obtained in step 1 as input. The server applies machine learning algorithms to extract important patterns such as player strategies, scoring opportunities, and foul occurrences. The output provides information about these important events.

[0505] Step 3:

[0506] The server prepares to notify the terminal of the identified important event. It uses the information about the important event obtained in step 2 as input. The server uses a notification generation mechanism to generate a message in a user-friendly format. The output is the message sent to the terminal.

[0507] Step 4:

[0508] The device receives notifications about important events sent from the server and displays them to the user. The message obtained in step 3 is used as input. The device displays push notifications on the screen and may also emit audio alerts. The output is communicated to the user visually and aurally.

[0509] Step 5:

[0510] If a user has a question during a match, they send it to the server via their terminal. The input is the question the user enters into their terminal. The output is the question received by the server.

[0511] Step 6:

[0512] The server searches the information infrastructure based on the question received from the user and generates an appropriate response. The input is the user question received in step 5. A generative AI model is used to analyze the relevant data and create the optimal answer. The output is the detailed answer returned to the user.

[0513] Step 7:

[0514] After the match ends, the server re-analyzes the match data, extracts key moments, and generates a highlight video. The input is data collected from the entire match. The server uses a video editing algorithm to automatically select important scenes and create the highlight video. The output is a video format viewable on the user's device.

[0515] Step 8:

[0516] The terminal provides users with generated digest videos and creates a platform for exchanging opinions with other fans. Inputs include video data sent from the server and comments from users. While playing the video, the terminal facilitates communication between users using chat and comment functions. Outputs include the video viewed by users and their shared opinions and comments.

[0517] (Application Example 1)

[0518] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0519] Modern sports viewing demands a real-time, detailed, and interactive experience for spectators. However, typical sports viewing often makes it difficult to immediately grasp information about the progress of the game and technical aspects of the play, and there are limited ways to efficiently review important moments after the game. Furthermore, there is a lack of opportunities for spectators to exchange opinions and receive quick answers to their questions. Thus, the challenge lies in providing means to enjoy and understand sports more deeply.

[0520] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0521] In this invention, the server includes means for acquiring and analyzing match data in real time, means for generating notifications to inform users of important information, and means for receiving questions from users during a match, generating appropriate answers, and sending them back. This enables real-time understanding of the match, comprehension of technical terms, and quick review of important moments after the match. Furthermore, by providing means for users to exchange opinions with each other, it is possible to make the sports viewing experience richer and more interactive.

[0522] "Match data" refers to information that shows the progress of a sporting event in real time, including the positions of participants, the score, the movement of the ball, and any fouls.

[0523] A "notification generation means" is a system or function that transmits information to a user quickly and accurately based on identified important events.

[0524] "Means for receiving questions and generating and sending appropriate answers" refers to a system or function for processing user questions, creating answers by referring to relevant databases, and sending them to the user.

[0525] "Methods for automatically generating highlight videos" refer to automated processes that select important moments after a match has ended, combine them, and convert them into a format that can be viewed in a short amount of time.

[0526] "Means of providing communication functions" refers to an online platform or system for multiple users to exchange opinions and share information.

[0527] "Means of providing glossary" refers to a system or function that provides explanations or descriptions of technical terms or specific plays used during a match to help users understand them.

[0528] The system implementing this invention mainly consists of three components: a server, a terminal, and a user.

[0529] server

[0530] The server acquires and analyzes match data in real time during the game. The analysis uses the Python pandas library to efficiently track player positions and ball movement. When important events are identified, the server generates data to notify the terminal. Furthermore, if a user asks a question during the match, the server uses an artificial intelligence model to generate an answer, searching relevant data storage to reconcile the information.

[0531] terminal

[0532] The terminal displays received data to the user in real time. It immediately notifies users of important events and automatically displays explanations of match terminology and specific plays. After the match ends, it distributes highlight videos generated by the server for easy viewing by users. The terminal also enriches the viewing experience by providing a communication platform where users can exchange opinions.

[0533] User

[0534] Users can watch matches via their devices and receive real-time notifications. If questions arise, they can send questions from their devices to the server and receive immediate answers. They can also exchange opinions about the matches with other fans.

[0535] Specifically, for example, if a user enters a question like "What is offside?" into the terminal, the AI ​​model will process this information and provide a detailed explanation. Furthermore, by using prompts such as "What is player C's goal tally this season?", the system can instantly retrieve player performance information and provide a response to the user. In this way, the system enhances real-time information provision and communication among participants, making sports viewing more interactive and easier to understand.

[0536] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0537] Step 1:

[0538] The server receives match data. Specifically, it inputs player location information, scores, and ball movement in real time from external sensors and streaming APIs. This data is analyzed using the Python pandas library to identify important events (such as goals and fouls). The output of this step is a list of the identified important events.

[0539] Step 2:

[0540] The server generates notification data using a notification generation mechanism based on a list of identified important events. This data processing involves converting the data into a format easily understood by the user. For example, it might create a text notification stating, "Player A scored a goal." This notification data is then sent to the device.

[0541] Step 3:

[0542] The terminal receives notification data sent from the server and displays it on the user's screen as a real-time notification. The input is the notification data, and the output is a pop-up notification generated based on it. In addition, explanations of technical terms included in the notification are displayed using a pre-prepared database.

[0543] Step 4:

[0544] When a user has a question during a match, they type it into their terminal. The entered prompt (for example, "What is offside?") is sent to the server.

[0545] Step 5:

[0546] The server receives a prompt from the user and processes it using a generative AI model. The AI ​​model generates the most appropriate explanation from the relevant data. The output is a text explanation for the question, which is then sent back to the terminal.

[0547] Step 6:

[0548] After the match ends, the server re-analyzes all the match data and extracts the most important moments. The input is the data for the entire match, and the output is a list of highlight scenes based on that data. The automatically generated highlight video is then sent to the terminal.

[0549] Step 7:

[0550] The device receives highlight videos and plays them on the device for the user to watch. Furthermore, users can access a communication platform for exchanging opinions and share their thoughts with other users.

[0551] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0552] This invention is a system that incorporates an emotion engine, in addition to conventional data analysis functions, to recognize the user's emotions and optimize the system's operation, in order to make watching sports more emotionally enriching.

[0553] First, the server acquires and analyzes match data in real time. Here, detailed information about the progress of the match, such as player positions, scores, ball movement, and fouls, is constantly updated, and important events are quickly identified.

[0554] Based on this data, the server uses a notification generation system to notify terminals of important match events. Notifications are made in real time to ensure users stay informed about the progress of the match.

[0555] Furthermore, the device's built-in emotion engine recognizes the user's emotions by analyzing their behavioral data and reactions. This information is combined with match data and used to personalize notifications and feedback based on the user's emotions. For example, if a user reacts with joy to a particular team's score, information about that team will be prioritized in notifications.

[0556] Users can ask questions to the server via their device if they have any doubts during a match. The server searches relevant data, generates a response that takes sentiment into account, and sends it back to the user, thereby improving the user experience.

[0557] After the match ends, the server re-analyzes key moments and, based on feedback from the emotion engine, automatically generates a highlight video focusing on the moments that viewers found particularly interesting. This video is then quickly delivered to the user via their device.

[0558] Furthermore, the emotion engine utilizes users' emotional information to make communication with other fans more effective. For example, it introduces mechanisms to facilitate discussions about sharing emotions such as joy and sadness, deepening interaction among users.

[0559] Thus, this system, which incorporates an emotion engine, goes beyond conventional sports viewing systems to create a personalized information delivery and interaction platform that matches the user's unique emotions.

[0560] The following describes the processing flow.

[0561] Step 1:

[0562] The server retrieves match data from sports events in real time. This data includes player positions, scores, fouls, and the progress of the match, and analysis is performed based on this data.

[0563] Step 2:

[0564] The server analyzes the acquired match data and uses specific algorithms to identify important events, such as scoring opportunities and turning points in the game, making immediate decisions.

[0565] Step 3:

[0566] Based on the analysis results, the server uses a notification generation mechanism to notify the user's terminal of important events in real time. For example, it sends a notification the moment a score is scored.

[0567] Step 4:

[0568] The device uses an emotion engine to analyze user emotions from their reactions and operation history. It utilizes cameras and sensors to capture emotions such as surprise, joy, and tension.

[0569] Step 5:

[0570] Based on the emotional data analyzed by the emotion engine, the device personalizes the content of notifications and displays them to the user. For example, it may adjust the content by providing additional match information when the user is excited.

[0571] Step 6:

[0572] Users can input questions via their devices while watching a match and send them to the server. They can ask about their favorite player's performance during the match.

[0573] Step 7:

[0574] The server receives a question from the user, generates an answer by referring to the relevant database, and sends the answer back to the terminal in the most appropriate format, taking sentiment information into consideration.

[0575] Step 8:

[0576] After the match ends, the server re-analyzes the data accumulated during the match and automatically generates a highlight video, focusing on moments that evoked particularly strong emotional reactions. This video emphasizes the most exciting scenes of the match.

[0577] Step 9:

[0578] The device provides the user with a generated highlight video, allowing for a recap that reflects the flow of emotions. Users relive the excitement of the match while watching the video.

[0579] Step 10:

[0580] The device proposes communication functions based on sentiment analysis, facilitating the exchange of opinions among users. Users can share their impressions and feelings about the match with other fans and enjoy interacting with them.

[0581] (Example 2)

[0582] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0583] In recent years, sports viewing has demanded more than just the results and information provided; it has been sought after for enriching the experience to match the individual emotions and interests of the viewer. However, conventional systems have struggled to recognize viewers' emotions in real time and personalize the information they receive. Furthermore, there is a lack of efficient means to provide viewers with moments of interest during and after a match, and sufficient communication functions to facilitate the sharing of emotions with other fans.

[0584] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0585] In this invention, the server includes means for acquiring and analyzing match information in real time and identifying important events, means for recognizing the user's emotions and personalizing the information based on the user's emotional state, and means for facilitating interaction with other users. This makes it possible to provide information that responds to the individual emotions of viewers, resulting in a more emotionally rich and personalized sports viewing experience.

[0586] "Match information" refers to all data related to the progress of a sports match, such as scores, player positions, and fouls.

[0587] "Real-time" refers to the instantaneous processing of information as the event or process in question unfolds.

[0588] "Analysis" refers to the process used to analyze acquired data and identify important patterns or events.

[0589] A "significant event" refers to an action or outcome in the course of a match that is likely to attract particular attention from viewers.

[0590] "Information generation means" refers to a notification mechanism designed to transmit information obtained from analysis results to users.

[0591] "Emotion recognition" refers to the process of estimating and identifying a user's emotional state based on their behavior and reactions.

[0592] "Personalization" refers to optimizing information according to each user's emotions and interests, and providing it in a way that is tailored to the individual.

[0593] "Means of promoting interaction" refers to platforms and features that help users share their feelings and opinions with one another.

[0594] This invention is a system designed to enhance the experience of watching sports, and it has the function of analyzing the progress of a match in real time and providing information based on the user's emotional state.

[0595] The server acquires match information in real time using a dedicated data acquisition API. This information includes player positions, scoring movements, fouls, and more. The server uses this data to evaluate player performance and employs data processing software (e.g., Pandas) to identify key events in the match. In this process, the server can identify important events more quickly and optimize its operation.

[0596] The device is equipped with an emotion recognition engine that analyzes the user's facial expressions and voice tone. This engine acquires data from the camera and microphone and analyzes it using an emotion recognition service in the cloud (e.g., a general-purpose emotion recognition API). After identifying the user's emotional state, this information is used on the server to generate responses and provide information.

[0597] For example, if a user's expression of joy is recognized when a particular team scores a goal during a match, the server will use that information to send a personalized notification to the device such as, "A fantastic goal by Player A! They're now at the top of the scoring rankings this season!"

[0598] If a user has a question during a match, they can send it to the server via their device. The server uses a generative AI model with a natural language processing engine to analyze the user's inquiry and construct a response based on relevant match information and sentiment data. For example, by entering a prompt such as "What information would you prefer to receive when your favorite team scores?", the server can provide information that meets the user's expectations.

[0599] This system also has a feature that automatically generates highlight videos by re-analyzing important scenes after the match ends. The generated videos incorporate points of interest based on the user's emotions, providing valuable entertainment for the user. It also includes communication features that facilitate sharing emotions and exchanging opinions to deepen interaction among users.

[0600] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0601] Step 1:

[0602] The server receives real-time match information from a data acquisition API during the progress of a sports match. This input includes player positions, scores, foul information, and more. The server processes this data using a Python script and transforms it into a data frame to identify important events. As a result of the analysis, it outputs data that allows for a real-time understanding of the match's progress.

[0603] Step 2:

[0604] The server identifies important events based on the data analyzed in Step 1 and activates the notification generation algorithm. The input is the events deemed important (for example, scoring plays or player substitutions). The server determines the priority of events from the data, creates a message to notify the user's terminal, and outputs it. This message is sent to the terminal immediately.

[0605] Step 3:

[0606] The device uses its built-in camera and microphone to collect the user's facial expressions and voice tone in real time. The input data consists of video and audio, which are then analyzed by an emotion recognition engine. The output generates quantified results of emotional states such as joy, surprise, and sadness, and sends them back to the server.

[0607] Step 4:

[0608] The server analyzes the emotional data received from the terminal using the Emotion API to determine the user's current emotions in detail. The input is the emotional state data obtained in step 3. Based on the analyzed emotional data, it outputs parameters to personalize subsequent notifications and information provision.

[0609] Step 5:

[0610] Users can ask questions to the server via their terminal during a match. This question serves as input, and the server analyzes the question using a generative AI model. It searches relevant match information and sentiment databases, generates an appropriate response, and outputs it. The generated response is then sent to the user.

[0611] Step 6:

[0612] After the match ends, the server analyzes the match data again and automatically generates a highlight video using FFmpeg, focusing on the moments that attracted the user's attention. The input data consists of all the data collected during the match, and the highlight video is output based on this data and quickly delivered to the user via their terminal.

[0613] Step 7:

[0614] The emotion engine-powered platform provides a discussion forum for users to share their joys and sorrows. Using users' emotional data as input, topics that facilitate interaction are automatically suggested, allowing users to deepen their connections with other fans. The output is lively exchange of opinions and interaction within the discussion space.

[0615] (Application Example 2)

[0616] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0617] Current sports viewing systems have the problem of providing uniform information when watching a match, failing to adequately respond to the individual interests and emotions of users. In particular, there is a need to quickly provide important information from large amounts of data and optimize information delivery according to the user's emotions. Furthermore, there is a lack of means to make sports viewing more emotionally enriching and to promote active communication with other viewers.

[0618] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0619] In this invention, the server includes means for acquiring and analyzing match information in real time and identifying important events, means for immediately notifying the user's terminal of important events using notification generation means, and means for recognizing the user's emotions and optimizing the displayed content based on those emotions. This makes it possible to provide users with important match information in real time and to provide information that is tailored to their emotions.

[0620] "Match information" refers to data that shows the progress of a sports competition in real time, including the movements of players, scores, fouls, and other related events.

[0621] "Analysis" is the process of thoroughly analyzing acquired match information to identify important events.

[0622] A "significant event" refers to any event that significantly impacts the progress of a match, such as scoring, fouls, or player substitutions.

[0623] A "notification generation method" is a technique for notifying users of important events in real time on their devices based on information obtained from a server.

[0624] "User's device" refers to an electronic device used by the user to receive notifications, and includes smartphones, smart glasses, and other similar devices.

[0625] "User emotions" refers to the emotional reactions that users exhibit while watching a match, such as joy, sadness, and surprise.

[0626] "Optimizing displayed content" is a process that adjusts the information presented to match the user's interests and emotions based on the acquired user sentiment information.

[0627] The system of the present invention is designed to enhance the sports viewing experience. This system primarily consists of a server, terminals, and an emotion recognition engine. Specific embodiments are described below.

[0628] First, the server acquires and analyzes match information in real time. Specifically, it collects data such as player movements, scores, and ball position during the match to identify important events. This analysis process utilizes reinforcement learning and data mining techniques to enable rapid and accurate data processing.

[0629] Next, the server uses a notification generation mechanism to send information about the identified important events to the terminal in real time. This terminal includes smart glasses worn by the user or a portable communication device, allowing the user to instantly grasp the important moments of the match.

[0630] Furthermore, an emotion recognition engine runs on the device to analyze user behavior and reactions. This allows the system to recognize what emotions the user is experiencing and optimize the displayed content based on those emotions. For example, if a user shows a joyful reaction, match highlights and related information tailored to that emotion will be prioritized.

[0631] As a concrete example, consider a soccer match. If a user cheers during a goal scored by a particular team while watching the match, the emotion recognition engine will recognize this action as "joy." The server will then display a replay of the relevant goal and project it onto the smart glasses. Another example of a prompt message is, "If a user claps or makes other actions while watching a sporting event, please tell us how to recognize emotions such as joy based on those actions and how to provide notifications or information during specific match events."

[0632] Thus, the system of the present invention provides users with a more emotionally resonant sports viewing experience based on real-time analysis and emotional feedback.

[0633] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0634] Step 1:

[0635] The server acquires match information in real time. Specifically, it aggregates data obtained from sensors and cameras at sports stadiums. Raw match data is used as input, and from this, information such as player positions, scores, and fouls is obtained. This input data is converted into a specific format and prepared for analysis in the next step.

[0636] Step 2:

[0637] The server analyzes the match data acquired in Step 1. The analysis utilizes reinforcement learning algorithms and data mining techniques to identify important events from the input match data. Examples of these important events include goals, penalties, and player substitutions. The output generates information about the identified important events.

[0638] Step 3:

[0639] The server sends information to the terminal using a notification generation mechanism based on identified important events. The notification includes important moments in the progress of the match, allowing the user (terminal) to receive information in real time. The input is the information of the important events output in step 2, and the output is a notification message generated for the user's terminal.

[0640] Step 4:

[0641] After receiving a notification, the device uses an emotion recognition engine to analyze the user's emotions. The user's actions, facial expressions, and voice data are input into the emotion recognition engine, and as a result, emotions such as joy, surprise, and tension are output. This emotion information is used to optimize the displayed content in the next step.

[0642] Step 5:

[0643] The device optimizes its display content using output data based on an emotion recognition engine. Specifically, it prioritizes displaying match highlights and related information according to the emotions expressed by the user. This process involves the operation of the visual display on the device, where the input is the emotion information from step 4, and the output is user-tailored content.

[0644] Step 6:

[0645] Based on the information displayed on the device, users can input further questions into the server. The server receives these questions as input, searches its relevant database to generate appropriate answers, and sends those answers back to the device. As output, users are provided with answers that satisfy them, enriching their viewing experience.

[0646] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0647] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0648] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0649] [Fourth Embodiment]

[0650] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0651] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0652] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0653] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0654] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0655] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0656] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0657] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0658] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0659] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0660] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0661] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0662] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0663] This invention is a system designed to make watching sports more interactive and easier to understand, where a server, terminal, and user cooperate to share information and enrich the experience.

[0664] First, the server acquires and analyzes match data in real time. The information collected during the match includes player positions, scores, ball movement, fouls, and more. The server processes this data quickly and applies algorithms to identify important events.

[0665] Next, identified important events are sent to the device and notified to the user. The device receives information from the server and displays notifications so that the user can understand the flow of the match in real time. In addition, when technical terms appear, the system automatically provides explanations of those terms to help the user easily understand the match.

[0666] When a question arises during a match, users can send it to the server via their device. The server receives the question, searches for the relevant information in its database, and sends back the appropriate answer, further enhancing the user's understanding. For example, if a user asks, "What are player A's stats for this season?", the server retrieves the stats information and sends it to the device, allowing the user to check it immediately.

[0667] After the match ends, the server re-analyzes the match data, selects important scenes, and automatically generates a highlight video. This highlight video is quickly delivered to the user via their device, allowing them to quickly review the key points of the match.

[0668] Furthermore, the device provides communication features that allow users to exchange opinions and share their thoughts. Users can share their thoughts and insights with other fans and enjoy the game more deeply through discussion.

[0669] In this way, this system aims to provide a better sports viewing experience through the provision of appropriate information and user interaction according to the progress of the match.

[0670] The following describes the processing flow.

[0671] Step 1:

[0672] The server retrieves match data in real time from sporting events. It analyzes the data stream provided during the match and extracts information such as player positions, scores, fouls, and ball movement.

[0673] Step 2:

[0674] The server analyzes the acquired match data to identify important events. Here, specific algorithms are used to identify goal scenes, player substitutions, and other key events, and this information is then organized.

[0675] Step 3:

[0676] The server generates notifications based on identified important events. These notifications include information viewers should know, such as when a player scores a goal or when a player receives a warning.

[0677] Step 4:

[0678] The device receives notifications from the server and displays them to the user in real time. Notifications are provided in the form of pop-ups or alerts to ensure the user doesn't miss a match.

[0679] Step 5:

[0680] The device automatically explains technical terms included in notifications. For example, it helps users understand the game more deeply by explaining terms like "offside" and "foul" in an easy-to-understand way.

[0681] Step 6:

[0682] Users can ask questions that arise during a match through their device. For example, they can enter questions about a specific player, such as, "How many assists has player A had this season?"

[0683] Step 7:

[0684] The server receives a question from the user, searches the relevant database, generates an appropriate answer, and sends it back to the terminal. This allows the user to obtain information instantly.

[0685] Step 8:

[0686] The server re-analyzes the match data after the game ends, selects important scenes, and automatically generates a highlight video. It then cuts out and edits scoring scenes and other noteworthy plays.

[0687] Step 9:

[0688] The device delivers the generated highlight video to the user, allowing them to quickly review the key points of the match. Users will then view the match comprehensively through this video.

[0689] Step 10:

[0690] The device provides communication features that allow users to exchange opinions with other fans. Users can share their thoughts after a match or discuss memorable plays during the game.

[0691] (Example 1)

[0692] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0693] In modern sports spectating, spectators demand a deeper understanding and immediate access to information. There is also a need for rapid information sharing regarding in-game events and effective post-game reviews, but integrated systems to achieve these are lacking. In particular, new methods are needed for real-time information analysis, improved understanding of technical terms, and smoother communication among users.

[0694] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0695] In this invention, the server includes means for instantly collecting and analyzing competition data and identifying important events, means for generating notifications to inform individuals of important moments, and means for receiving inquiries from individuals during a competition and generating and responding with appropriate responses. This makes it possible to provide important information in real time and to quickly resolve user questions.

[0696] "Competition data" refers to information related to sports and games, including location data, scores, player movements, rule violations, and all other data related to the progress of the match.

[0697] "Immediately" means that the operation or process in question is executed immediately without delay at the location where it takes place.

[0698] "Analysis" refers to the process of processing collected data and deriving useful information or patterns from it.

[0699] A "significant event" refers to any event in the course of a match or game that is particularly noteworthy to spectators and those involved, such as a change in the score, a rule violation, or a strategic turning point.

[0700] "To distinguish" refers to the process of selecting specific elements from multiple data or pieces of information based on particular conditions or criteria.

[0701] "Notification generation means" refers to a function that creates and sends pre-configured messages or alerts within a system to convey specific information to the user.

[0702] "Individual" is a general term used to refer to a user of a system and to identify each user.

[0703] "Receiving" refers to taking in transmitted data or messages and making them ready for processing.

[0704] "Generating a response and sending a reply" refers to the action of creating an appropriate answer or information in response to a received question or request and sending it back to the sender.

[0705] This invention is a system that enables spectators to receive abundant information and gain a deeper understanding of sports and games. It primarily functions through the coordinated interaction of servers, terminals, and users.

[0706] The server uses multiple sensors, cameras, and external data provision APIs to collect and analyze data in real time during the game. The real-time data obtained from this hardware and APIs is processed using machine learning algorithms and statistical analysis techniques. Specific examples include GPS sensors that provide location information to track player movements and image recognition technology to record the movement of the ball. Based on the analysis results, the server identifies important events and proceeds to the next step.

[0707] Next, the device receives notifications from the server and displays important information to the user in real time. To achieve this, the device utilizes push notification technology to directly import the information onto the screen. Furthermore, when technical terms appear during a match, a pre-built glossary function provides accurate information. This allows users to enjoy the match without confusion.

[0708] Furthermore, users can input questions that arise during a match via their device and query the server. The server searches the relevant information infrastructure and sends back an appropriate response. For example, if a user inputs "I want to know the past match record of player B," the server will quickly search the database, retrieve the record, and send the information to the device.

[0709] This system utilizes a generative AI model to support user understanding and quickly resolve inquiries during matches. Examples of specific prompts include, "Please tell me the important events of today's match," and "Please explain player A's performance in detail."

[0710] This advanced system allows individual users to enjoy a deeper sports viewing experience and resolve any concerns immediately.

[0711] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0712] Step 1:

[0713] The server collects competition data in real time. Inputs include real-time data provided by sensors and cameras placed at the match venue, as well as external APIs. This data includes player positions, speeds, scores, and information about events during the match. The server uses data processing software to format this data and convert it into an analyzable state.

[0714] Step 2:

[0715] The server analyzes the collected data and identifies important events. This process uses the formatted data obtained in step 1 as input. The server applies machine learning algorithms to extract important patterns such as player strategies, scoring opportunities, and foul occurrences. The output provides information about these important events.

[0716] Step 3:

[0717] The server prepares to notify the terminal of the identified important event. It uses the information about the important event obtained in step 2 as input. The server uses a notification generation mechanism to generate a message in a user-friendly format. The output is the message sent to the terminal.

[0718] Step 4:

[0719] The device receives notifications about important events sent from the server and displays them to the user. The message obtained in step 3 is used as input. The device displays push notifications on the screen and may also emit audio alerts. The output is communicated to the user visually and aurally.

[0720] Step 5:

[0721] If a user has a question during a match, they send it to the server via their terminal. The input is the question the user enters into their terminal. The output is the question received by the server.

[0722] Step 6:

[0723] The server searches the information infrastructure based on the question received from the user and generates an appropriate response. The input is the user question received in step 5. A generative AI model is used to analyze the relevant data and create the optimal answer. The output is the detailed answer returned to the user.

[0724] Step 7:

[0725] After the match ends, the server re-analyzes the match data, extracts key moments, and generates a highlight video. The input is data collected from the entire match. The server uses a video editing algorithm to automatically select important scenes and create the highlight video. The output is a video format viewable on the user's device.

[0726] Step 8:

[0727] The terminal provides users with generated digest videos and creates a platform for exchanging opinions with other fans. Inputs include video data sent from the server and comments from users. While playing the video, the terminal facilitates communication between users using chat and comment functions. Outputs include the video viewed by users and their shared opinions and comments.

[0728] (Application Example 1)

[0729] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0730] Modern sports viewing demands a real-time, detailed, and interactive experience for spectators. However, typical sports viewing often makes it difficult to immediately grasp information about the progress of the game and technical aspects of the play, and there are limited ways to efficiently review important moments after the game. Furthermore, there is a lack of opportunities for spectators to exchange opinions and receive quick answers to their questions. Thus, the challenge lies in providing means to enjoy and understand sports more deeply.

[0731] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0732] In this invention, the server includes means for acquiring and analyzing match data in real time, means for generating notifications to inform users of important information, and means for receiving questions from users during a match, generating appropriate answers, and sending them back. This enables real-time understanding of the match, comprehension of technical terms, and quick review of important moments after the match. Furthermore, by providing means for users to exchange opinions with each other, it is possible to make the sports viewing experience richer and more interactive.

[0733] "Match data" refers to information that shows the progress of a sporting event in real time, including the positions of participants, the score, the movement of the ball, and any fouls.

[0734] A "notification generation means" is a system or function that transmits information to a user quickly and accurately based on identified important events.

[0735] "Means for receiving questions and generating and sending appropriate answers" refers to a system or function for processing user questions, creating answers by referring to relevant databases, and sending them to the user.

[0736] "Methods for automatically generating highlight videos" refer to automated processes that select important moments after a match has ended, combine them, and convert them into a format that can be viewed in a short amount of time.

[0737] "Means of providing communication functions" refers to an online platform or system for multiple users to exchange opinions and share information.

[0738] "Means of providing glossary" refers to a system or function that provides explanations or descriptions of technical terms or specific plays used during a match to help users understand them.

[0739] The system implementing this invention mainly consists of three components: a server, a terminal, and a user.

[0740] server

[0741] The server acquires and analyzes match data in real time during the game. The analysis uses the Python pandas library to efficiently track player positions and ball movement. When important events are identified, the server generates data to notify the terminal. Furthermore, if a user asks a question during the match, the server uses an artificial intelligence model to generate an answer, searching relevant data storage to reconcile the information.

[0742] terminal

[0743] The terminal displays received data to the user in real time. It immediately notifies users of important events and automatically displays explanations of match terminology and specific plays. After the match ends, it distributes highlight videos generated by the server for easy viewing by users. The terminal also enriches the viewing experience by providing a communication platform where users can exchange opinions.

[0744] User

[0745] Users can watch matches via their devices and receive real-time notifications. If questions arise, they can send questions from their devices to the server and receive immediate answers. They can also exchange opinions about the matches with other fans.

[0746] Specifically, for example, if a user enters a question like "What is offside?" into the terminal, the AI ​​model will process this information and provide a detailed explanation. Furthermore, by using prompts such as "What is player C's goal tally this season?", the system can instantly retrieve player performance information and provide a response to the user. In this way, the system enhances real-time information provision and communication among participants, making sports viewing more interactive and easier to understand.

[0747] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0748] Step 1:

[0749] The server receives match data. Specifically, it inputs player location information, scores, and ball movement in real time from external sensors and streaming APIs. This data is analyzed using the Python pandas library to identify important events (such as goals and fouls). The output of this step is a list of the identified important events.

[0750] Step 2:

[0751] The server generates notification data using a notification generation mechanism based on a list of identified important events. This data processing involves converting the data into a format easily understood by the user. For example, it might create a text notification stating, "Player A scored a goal." This notification data is then sent to the device.

[0752] Step 3:

[0753] The terminal receives notification data sent from the server and displays it on the user's screen as a real-time notification. The input is the notification data, and the output is a pop-up notification generated based on it. In addition, explanations of technical terms included in the notification are displayed using a pre-prepared database.

[0754] Step 4:

[0755] When a user has a question during a match, they type it into their terminal. The entered prompt (for example, "What is offside?") is sent to the server.

[0756] Step 5:

[0757] The server receives a prompt from the user and processes it using a generative AI model. The AI ​​model generates the most appropriate explanation from the relevant data. The output is a text explanation for the question, which is then sent back to the terminal.

[0758] Step 6:

[0759] After the match ends, the server re-analyzes all the match data and extracts the most important moments. The input is the data for the entire match, and the output is a list of highlight scenes based on that data. The automatically generated highlight video is then sent to the terminal.

[0760] Step 7:

[0761] The device receives highlight videos and plays them on the device for the user to watch. Furthermore, users can access a communication platform for exchanging opinions and share their thoughts with other users.

[0762] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0763] This invention is a system that incorporates an emotion engine, in addition to conventional data analysis functions, to recognize the user's emotions and optimize the system's operation, in order to make watching sports more emotionally enriching.

[0764] First, the server acquires and analyzes match data in real time. Here, detailed information about the progress of the match, such as player positions, scores, ball movement, and fouls, is constantly updated, and important events are quickly identified.

[0765] Based on this data, the server uses a notification generation system to notify terminals of important match events. Notifications are made in real time to ensure users stay informed about the progress of the match.

[0766] Furthermore, the device's built-in emotion engine recognizes the user's emotions by analyzing their behavioral data and reactions. This information is combined with match data and used to personalize notifications and feedback based on the user's emotions. For example, if a user reacts with joy to a particular team's score, information about that team will be prioritized in notifications.

[0767] Users can ask questions to the server via their device if they have any doubts during a match. The server searches relevant data, generates a response that takes sentiment into account, and sends it back to the user, thereby improving the user experience.

[0768] After the match ends, the server re-analyzes key moments and, based on feedback from the emotion engine, automatically generates a highlight video focusing on the moments that viewers found particularly interesting. This video is then quickly delivered to the user via their device.

[0769] Furthermore, the emotion engine utilizes users' emotional information to make communication with other fans more effective. For example, it introduces mechanisms to facilitate discussions about sharing emotions such as joy and sadness, deepening interaction among users.

[0770] Thus, this system, which incorporates an emotion engine, goes beyond conventional sports viewing systems to create a personalized information delivery and interaction platform that matches the user's unique emotions.

[0771] The following describes the processing flow.

[0772] Step 1:

[0773] The server retrieves match data from sports events in real time. This data includes player positions, scores, fouls, and the progress of the match, and analysis is performed based on this data.

[0774] Step 2:

[0775] The server analyzes the acquired match data and uses specific algorithms to identify important events, such as scoring opportunities and turning points in the game, making immediate decisions.

[0776] Step 3:

[0777] Based on the analysis results, the server uses a notification generation mechanism to notify the user's terminal of important events in real time. For example, it sends a notification the moment a score is scored.

[0778] Step 4:

[0779] The device uses an emotion engine to analyze user emotions from their reactions and operation history. It utilizes cameras and sensors to capture emotions such as surprise, joy, and tension.

[0780] Step 5:

[0781] Based on the emotional data analyzed by the emotion engine, the device personalizes the content of notifications and displays them to the user. For example, it may adjust the content by providing additional match information when the user is excited.

[0782] Step 6:

[0783] Users can input questions via their devices while watching a match and send them to the server. They can ask about their favorite player's performance during the match.

[0784] Step 7:

[0785] The server receives a question from the user, generates an answer by referring to the relevant database, and sends the answer back to the terminal in the most appropriate format, taking sentiment information into consideration.

[0786] Step 8:

[0787] After the match ends, the server re-analyzes the data accumulated during the match and automatically generates a highlight video, focusing on moments that evoked particularly strong emotional reactions. This video emphasizes the most exciting scenes of the match.

[0788] Step 9:

[0789] The device provides the user with a generated highlight video, allowing for a recap that reflects the flow of emotions. Users relive the excitement of the match while watching the video.

[0790] Step 10:

[0791] The device proposes communication functions based on sentiment analysis, facilitating the exchange of opinions among users. Users can share their impressions and feelings about the match with other fans and enjoy interacting with them.

[0792] (Example 2)

[0793] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0794] In recent years, sports viewing has demanded more than just the results and information provided; it has been sought after for enriching the experience to match the individual emotions and interests of the viewer. However, conventional systems have struggled to recognize viewers' emotions in real time and personalize the information they receive. Furthermore, there is a lack of efficient means to provide viewers with moments of interest during and after a match, and sufficient communication functions to facilitate the sharing of emotions with other fans.

[0795] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0796] In this invention, the server includes means for acquiring and analyzing match information in real time and identifying important events, means for recognizing the user's emotions and personalizing the information based on the user's emotional state, and means for facilitating interaction with other users. This makes it possible to provide information that responds to the individual emotions of viewers, resulting in a more emotionally rich and personalized sports viewing experience.

[0797] "Match information" refers to all data related to the progress of a sports match, such as scores, player positions, and fouls.

[0798] "Real-time" refers to the instantaneous processing of information as the event or process in question unfolds.

[0799] "Analysis" refers to the process used to analyze acquired data and identify important patterns or events.

[0800] A "significant event" refers to an action or outcome in the course of a match that is likely to attract particular attention from viewers.

[0801] "Information generation means" refers to a notification mechanism designed to transmit information obtained from analysis results to users.

[0802] "Emotion recognition" refers to the process of estimating and identifying a user's emotional state based on their behavior and reactions.

[0803] "Personalization" refers to optimizing information according to each user's emotions and interests, and providing it in a way that is tailored to the individual.

[0804] "Means of promoting interaction" refers to platforms and features that help users share their feelings and opinions with one another.

[0805] This invention is a system designed to enhance the experience of watching sports, and it has the function of analyzing the progress of a match in real time and providing information based on the user's emotional state.

[0806] The server acquires match information in real time using a dedicated data acquisition API. This information includes player positions, scoring movements, fouls, and more. The server uses this data to evaluate player performance and employs data processing software (e.g., Pandas) to identify key events in the match. In this process, the server can identify important events more quickly and optimize its operation.

[0807] The device is equipped with an emotion recognition engine that analyzes the user's facial expressions and voice tone. This engine acquires data from the camera and microphone and analyzes it using an emotion recognition service in the cloud (e.g., a general-purpose emotion recognition API). After identifying the user's emotional state, this information is used on the server to generate responses and provide information.

[0808] For example, if a user's expression of joy is recognized when a particular team scores a goal during a match, the server will use that information to send a personalized notification to the device such as, "A fantastic goal by Player A! They're now at the top of the scoring rankings this season!"

[0809] If a user has a question during a match, they can send it to the server via their device. The server uses a generative AI model with a natural language processing engine to analyze the user's inquiry and construct a response based on relevant match information and sentiment data. For example, by entering a prompt such as "What information would you prefer to receive when your favorite team scores?", the server can provide information that meets the user's expectations.

[0810] This system also has a feature that automatically generates highlight videos by re-analyzing important scenes after the match ends. The generated videos incorporate points of interest based on the user's emotions, providing valuable entertainment for the user. It also includes communication features that facilitate sharing emotions and exchanging opinions to deepen interaction among users.

[0811] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0812] Step 1:

[0813] The server receives real-time match information from a data acquisition API during the progress of a sports match. This input includes player positions, scores, foul information, and more. The server processes this data using a Python script and transforms it into a data frame to identify important events. As a result of the analysis, it outputs data that allows for a real-time understanding of the match's progress.

[0814] Step 2:

[0815] The server identifies important events based on the data analyzed in Step 1 and activates the notification generation algorithm. The input is the events deemed important (for example, scoring plays or player substitutions). The server determines the priority of events from the data, creates a message to notify the user's terminal, and outputs it. This message is sent to the terminal immediately.

[0816] Step 3:

[0817] The device uses its built-in camera and microphone to collect the user's facial expressions and voice tone in real time. The input data consists of video and audio, which are then analyzed by an emotion recognition engine. The output generates quantified results of emotional states such as joy, surprise, and sadness, and sends them back to the server.

[0818] Step 4:

[0819] The server analyzes the emotional data received from the terminal using the Emotion API to determine the user's current emotions in detail. The input is the emotional state data obtained in step 3. Based on the analyzed emotional data, it outputs parameters to personalize subsequent notifications and information provision.

[0820] Step 5:

[0821] Users can ask questions to the server via their terminal during a match. This question serves as input, and the server analyzes the question using a generative AI model. It searches relevant match information and sentiment databases, generates an appropriate response, and outputs it. The generated response is then sent to the user.

[0822] Step 6:

[0823] After the match ends, the server analyzes the match data again and automatically generates a highlight video using FFmpeg, focusing on the moments that attracted the user's attention. The input data consists of all the data collected during the match, and the highlight video is output based on this data and quickly delivered to the user via their terminal.

[0824] Step 7:

[0825] The emotion engine-powered platform provides a discussion forum for users to share their joys and sorrows. Using users' emotional data as input, topics that facilitate interaction are automatically suggested, allowing users to deepen their connections with other fans. The output is lively exchange of opinions and interaction within the discussion space.

[0826] (Application Example 2)

[0827] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0828] Current sports viewing systems have the problem of providing uniform information when watching a match, failing to adequately respond to the individual interests and emotions of users. In particular, there is a need to quickly provide important information from large amounts of data and optimize information delivery according to the user's emotions. Furthermore, there is a lack of means to make sports viewing more emotionally enriching and to promote active communication with other viewers.

[0829] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0830] In this invention, the server includes means for acquiring and analyzing match information in real time and identifying important events, means for immediately notifying the user's terminal of important events using notification generation means, and means for recognizing the user's emotions and optimizing the displayed content based on those emotions. This makes it possible to provide users with important match information in real time and to provide information that is tailored to their emotions.

[0831] "Match information" refers to data that shows the progress of a sports competition in real time, including the movements of players, scores, fouls, and other related events.

[0832] "Analysis" is the process of thoroughly analyzing acquired match information to identify important events.

[0833] A "significant event" refers to any event that significantly impacts the progress of a match, such as scoring, fouls, or player substitutions.

[0834] A "notification generation method" is a technique for notifying users of important events in real time on their devices based on information obtained from a server.

[0835] "User's device" refers to an electronic device used by the user to receive notifications, and includes smartphones, smart glasses, and other similar devices.

[0836] "User emotions" refers to the emotional reactions that users exhibit while watching a match, such as joy, sadness, and surprise.

[0837] "Optimizing displayed content" is a process that adjusts the information presented to match the user's interests and emotions based on the acquired user sentiment information.

[0838] The system of the present invention is designed to enhance the sports viewing experience. This system primarily consists of a server, terminals, and an emotion recognition engine. Specific embodiments are described below.

[0839] First, the server acquires and analyzes match information in real time. Specifically, it collects data such as player movements, scores, and ball position during the match to identify important events. This analysis process utilizes reinforcement learning and data mining techniques to enable rapid and accurate data processing.

[0840] Next, the server uses a notification generation mechanism to send information about the identified important events to the terminal in real time. This terminal includes smart glasses worn by the user or a portable communication device, allowing the user to instantly grasp the important moments of the match.

[0841] Furthermore, an emotion recognition engine runs on the device to analyze user behavior and reactions. This allows the system to recognize what emotions the user is experiencing and optimize the displayed content based on those emotions. For example, if a user shows a joyful reaction, match highlights and related information tailored to that emotion will be prioritized.

[0842] As a concrete example, consider a soccer match. If a user cheers during a goal scored by a particular team while watching the match, the emotion recognition engine will recognize this action as "joy." The server will then display a replay of the relevant goal and project it onto the smart glasses. Another example of a prompt message is, "If a user claps or makes other actions while watching a sporting event, please tell us how to recognize emotions such as joy based on those actions and how to provide notifications or information during specific match events."

[0843] Thus, the system of the present invention provides users with a more emotionally resonant sports viewing experience based on real-time analysis and emotional feedback.

[0844] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0845] Step 1:

[0846] The server acquires match information in real time. Specifically, it aggregates data obtained from sensors and cameras at sports stadiums. Raw match data is used as input, and from this, information such as player positions, scores, and fouls is obtained. This input data is converted into a specific format and prepared for analysis in the next step.

[0847] Step 2:

[0848] The server analyzes the match data acquired in Step 1. The analysis utilizes reinforcement learning algorithms and data mining techniques to identify important events from the input match data. Examples of these important events include goals, penalties, and player substitutions. The output generates information about the identified important events.

[0849] Step 3:

[0850] The server sends information to the terminal using a notification generation mechanism based on identified important events. The notification includes important moments in the progress of the match, allowing the user (terminal) to receive information in real time. The input is the information of the important events output in step 2, and the output is a notification message generated for the user's terminal.

[0851] Step 4:

[0852] After receiving a notification, the device uses an emotion recognition engine to analyze the user's emotions. The user's actions, facial expressions, and voice data are input into the emotion recognition engine, and as a result, emotions such as joy, surprise, and tension are output. This emotion information is used to optimize the displayed content in the next step.

[0853] Step 5:

[0854] The device optimizes its display content using output data based on an emotion recognition engine. Specifically, it prioritizes displaying match highlights and related information according to the emotions expressed by the user. This process involves the operation of the visual display on the device, where the input is the emotion information from step 4, and the output is user-tailored content.

[0855] Step 6:

[0856] Based on the information displayed on the device, users can input further questions into the server. The server receives these questions as input, searches its relevant database to generate appropriate answers, and sends those answers back to the device. As output, users are provided with answers that satisfy them, enriching their viewing experience.

[0857] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0858] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0859] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0860] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0861] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0862] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0863] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0864] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0865] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0866] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0867] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0868] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0869] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0870] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0871] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0872] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0873] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0874] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0875] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0876] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0877] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0878] The following is further disclosed regarding the embodiments described above.

[0879] (Claim 1)

[0880] A means of acquiring and analyzing match data in real time to identify important events,

[0881] A notification generation method for notifying users of important moments,

[0882] A means of receiving questions from users during a match, generating appropriate answers, and sending them back.

[0883] A method for extracting important scenes after a match and automatically generating highlight videos,

[0884] A means of providing a communication function for exchanging opinions with other users,

[0885] A system that includes this.

[0886] (Claim 2)

[0887] The system according to claim 1, wherein the server analyzes match data and a notification generation means immediately notifies the user's terminal of important events.

[0888] (Claim 3)

[0889] The system according to claim 1, wherein the server searches a relevant database in response to a user's question and generates an answer.

[0890] "Example 1"

[0891] (Claim 1)

[0892] A means to instantly collect and analyze competition data and identify important events,

[0893] A means for generating notifications to inform individuals of important moments,

[0894] A means of receiving inquiries from individuals during a competition, generating appropriate responses, and sending back.

[0895] A method for extracting important moments after a competition and automatically generating a digest video,

[0896] A means of providing a conversational function for exchanging opinions with other individuals,

[0897] A system that includes this.

[0898] (Claim 2)

[0899] The system according to claim 1, wherein a computing device analyzes competition data and a notification generation means quickly notifies individual terminals of important events.

[0900] (Claim 3)

[0901] The system according to claim 1, wherein a computing device searches a relevant information infrastructure in response to an individual query and generates a response.

[0902] "Application Example 1"

[0903] (Claim 1)

[0904] A means of acquiring and analyzing match data in real time to identify important events,

[0905] A notification generation means for notifying users of important information,

[0906] A means of receiving questions from users during a match, generating appropriate answers, and sending them back.

[0907] A method for extracting important moments after the match and automatically generating highlight videos,

[0908] A means of providing a communication function for exchanging opinions with other users,

[0909] A means of providing a glossary of terms using match data,

[0910] A system that includes this.

[0911] (Claim 2)

[0912] The system according to claim 1, wherein the server analyzes match data and a notification generation means immediately notifies the user's information device of important events.

[0913] (Claim 3)

[0914] The system according to claim 1, wherein the server searches relevant data storage in response to a user's question and generates an answer.

[0915] "Example 2 of combining an emotion engine"

[0916] (Claim 1)

[0917] A means of acquiring and analyzing match information in real time to identify important events,

[0918] Information generation means for notifying users of important moments,

[0919] A means of receiving inquiries from users during a match, generating appropriate responses, and sending them back.

[0920] A method for automatically generating summary videos by extracting important scenes after a match,

[0921] A means of providing a communication function for exchanging opinions with other users,

[0922] A means of recognizing the user's emotions and personalizing information based on the user's emotional state,

[0923] A means of promoting interaction with other users based on users' emotional information,

[0924] A system that includes this.

[0925] (Claim 2)

[0926] The system according to claim 1, wherein the server analyzes match information and the information generation means immediately notifies the user's terminal of important events.

[0927] (Claim 3)

[0928] The system according to claim 1, wherein the server searches for relevant information sources and generates a response to a user's inquiry.

[0929] "Application example 2 when combining with an emotional engine"

[0930] (Claim 1)

[0931] A means of acquiring and analyzing match information in real time to identify important events,

[0932] A notification generation method for notifying users of important moments,

[0933] A means of receiving questions from users during a match, generating appropriate answers, and sending them back.

[0934] A method for extracting important moments after a match and automatically generating highlight videos,

[0935] A means of providing a communication function for exchanging opinions with other users,

[0936] A means of recognizing the user's emotions and optimizing the displayed content based on those emotions,

[0937] A system that includes this.

[0938] (Claim 2)

[0939] The system according to claim 1, wherein the server analyzes match information and a notification generation means immediately notifies the user's terminal of important events.

[0940] (Claim 3)

[0941] The system according to claim 1, wherein the server searches a relevant database in response to a user's question and generates an answer. [Explanation of symbols]

[0942] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring and analyzing match data in real time to identify important events, A notification generation method for notifying users of important moments, A means of receiving questions from users during a match, generating appropriate answers, and sending them back. A method for extracting important scenes after a match and automatically generating highlight videos, A means of providing a communication function for exchanging opinions with other users, A system that includes this.

2. The system according to claim 1, wherein the server analyzes match data and a notification generation means immediately notifies the user's terminal of important events.

3. The system according to claim 1, wherein the server searches a relevant database in response to a user's question and generates an answer.

Images