System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving input data of a user and setting a preference or a degree of understanding of the user for a sport; means for receiving a plurality of camera videos and analyzing the videos in real time; means for generating a running commentary customized according to a status of the user based on a result of the analysis; and means for providing the generated running commentary to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The sports viewing experience is greatly influenced by the announcer's knowledge and expressiveness, but typical commentary is limited to one pattern, making it difficult to satisfy all spectators. Furthermore, the announcer's lack of knowledge or bias may prevent the commentary from being delivered flawlessly. For this reason, it is necessary to improve the viewing experience by providing live sports commentary that is tailored to each user's individual knowledge level and preferences. However, there is currently no effective method to achieve this. [Means for solving the problem]

[0005] The present invention is a system that includes a means for receiving user input data and setting the user's sports preferences and level of understanding, a means for receiving and analyzing video footage from multiple cameras in real time, a means for generating a customized commentary tailored to the user's status based on the analysis results, and a means for providing the generated commentary to the user. Specifically, based on the user's level of sports understanding, the system provides beginners with detailed explanations of basic rules and plays, intermediate players with explanations of tactics and player characteristics, and advanced players with advanced analysis based on statistical data and past match data. It also provides information specific to the team or player the user selects as their favorite. This allows for personalized, high-quality sports commentary to be provided to each spectator in real time.

[0006] "User" refers to a person who uses the sports viewing system.

[0007] "Input data" refers to information provided by users to the system, including their favorite sports, the teams and players they support, and their level of understanding of the rules.

[0008] "Status" is a collection of information that indicates a user's preferences and level of understanding.

[0009] "Camera footage" refers to video data captured of a sports game.

[0010] "Analyzing in real time" means processing camera footage immediately upon receiving it and analyzing its contents.

[0011] "Analysis results" refers to the data and information obtained after analyzing the contents of camera footage.

[0012] "Play-by-play commentary" refers to the provision of commentary on the progress of a sports game and the actions of the players.

[0013] "Generate" refers to creating new information or data based on specific conditions.

[0014] "Providing" refers to providing the generated commentary to the user.

[0015] "Basic rules" refers to the basic rules and methods of the sport.

[0016] A "detailed explanation of the play" is a detailed explanation of specific plays and actions.

[0017] "Tactics" refers to the strategies and methods that a team or player employs in a game.

[0018] "Player characteristics" refers to the personality and technical attributes of a particular player.

[0019] "Statistical data" refers to numerical information based on past matches and plays.

[0020] "Past Match Data" refers to records and information relating to matches that have previously taken place.

[0021] "Personalized" means that it is individually optimized for each user. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] This invention relates to a sports viewing system that generates and provides live commentary customized for each user using user input data and real-time video analysis. This system is mainly composed of three elements: a server, a terminal, and a user.

[0044] Program processing explanation

[0045] Program Overview

[0046] 1. User status settings

[0047] User: Opens a sports viewing app and enters their favorite sport, the team and player they support, and their level of understanding of the rules.

[0048] Terminal: Collects user input data and sends it to the server.

[0049] Server: Analyzes the user's input data, generates a user profile, and stores it in a database.

[0050] 2. Real-time video analysis

[0051] Terminal: Receives multiple camera images in real time and streams them to the server.

[0052] Server: Sends video data to the analysis module for object recognition and motion analysis. Extracts play data (shots, passes, goals, etc.) and updates each player and the game situation in real time.

[0053] 3. AI commentary voice generation

[0054] Server: Compares extracted play data with user profile information.

[0055] Server: Generates customized commentary for beginner, intermediate and advanced players, including information specific to the team or player you support.

[0056] 4. Providing customized commentary

[0057] Server: Sends the generated commentary audio data to the device.

[0058] Device: Plays audio data, allowing users to enjoy real-time, customized commentary.

[0059] Specific examples

[0060] (Example 1) Beginner user

[0061] User: A beginner who barely knows the rules of soccer.

[0062] Device: Select "Soccer" as your favorite sport and set your rule comprehension level to "Beginner."

[0063] Server: Three minutes into the game, the ball is in the hands of the goalkeeper.

[0064] Server: Generates the explanation "The goalkeeper has caught the ball. The goalkeeper is the only position that can catch the ball with their hands."

[0065] Device: Provide this description to the user aloud.

[0066] (Example 2) Intermediate user

[0067] User: An intermediate player who enjoys watching soccer. His favorite team is Barcelona.

[0068] Device: Select "Soccer" as your favorite sport, set your favorite team to "Barcelona," and set your rule comprehension level to "Intermediate."

[0069] Server: A Barcelona player takes the shot.

[0070] Server: Generates commentary like, "The Barcelona player took a powerful shot. This player is known for his forward movement and is particularly good at free kicks."

[0071] Device: Provide this description to the user aloud.

[0072] (Example 3) Advanced user

[0073] User: Experienced soccer player with a thorough understanding of tactics and player data.

[0074] Device: Select "Soccer" as your favorite sport and set your rule understanding level to "Advanced."

[0075] Sarver: Midway through the match, Barcelona changed to a 4-3-3 formation.

[0076] Server: Generates commentary: "Barcelona have now switched to a 4-3-3 formation. This tactical change is designed to strengthen their midfield press, and Messi in particular could be a game-changer from now on. His data from his last 15 games shows that he has a 2% higher goal success rate when using this formation."

[0077] Device: Provide this description to the user aloud.

[0078] The system of the present invention significantly improves the viewing experience by providing customizable live sports commentary tailored to the individual needs of users. This system is a new platform that allows for more enjoyment and a deeper understanding of sports viewing.

[0079] The processing flow will be explained below.

[0080] Step 1:

[0081] User: Opens a sports viewing app and enters their favorite sport, the team / player they support, and their level of understanding of the rules.

[0082] Step 2:

[0083] Terminal: Collects user input data and sends it to the server.

[0084] Step 3:

[0085] Server: Analyzes the received user input data and generates a profile that reflects the user's preferences, level of understanding, etc.

[0086] Step 4:

[0087] Server: Stores user profile information in a database.

[0088] Step 5:

[0089] Terminal: As soon as the match starts, it receives multiple camera images in real time and streams them to the server.

[0090] Step 6:

[0091] Server: Sends the received video data to the analysis module, where object recognition and motion analysis are performed.

[0092] Step 7:

[0093] Server: Based on the results of video analysis, it extracts play data (shots, passes, goals, etc.) and updates the status of each player and the match in real time.

[0094] Step 8:

[0095] Server: Compares extracted play data with user profile information.

[0096] Step 9:

[0097] Server: Generates customized explanations based on the user's level of understanding. For example, beginners can learn basic rules, intermediate players can learn tactics and player characteristics, and advanced players can learn advanced statistics and analysis using past match data.

[0098] Step 10:

[0099] Server: Generates commentary that includes information specific to the team or player you support.

[0100] Step 11:

[0101] Server: Sends the generated commentary audio data to the user's device.

[0102] Step 12:

[0103] Device: By playing back the received audio data, users can enjoy customized commentary in real time.

[0104] Step 13:

[0105] Users: Gain a deeper understanding and enjoyment of watching sports.

[0106] According to the above steps, the system of the present invention provides personalized sports commentary to enhance the viewing experience.

[0107] Example 1

[0108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] Conventional sports viewing systems were unable to provide customized commentary tailored to the user's preferences and level of understanding, resulting in a uniform viewing experience that could not meet the needs of individual users. Furthermore, it was difficult to provide real-time video analysis or information specialized for specific teams or players, preventing users from gaining a deeper understanding or enjoyment.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0111] In this invention, the server includes means for receiving user input data and setting the user's preferences and level of understanding of sports, means for receiving and analyzing images from multiple camera devices in real time, and means for generating audio commentary customized according to the user's status based on the analysis results, thereby enabling the provision of play-by-play commentary customized to meet the individual needs of each user.

[0112] "User input data" refers to information that users input when using a sports viewing app, such as their favorite sport, the team or player they support, and their understanding of the rules.

[0113] The "multiple image capturing devices" refer to multiple cameras or other image capturing devices installed within the venue of a sporting event.

[0114] "Means for analyzing in real time" refers to a means for instantly analyzing the video data received from the filming device and recognizing the player's movements and type of play.

[0115] "Customized audio commentary according to user status" refers to audio data in which the content of the live commentary to be provided is adjusted according to the preferences and level of understanding set for each user.

[0116] A "means for providing to a user" is a means for delivering the generated audio description to a user device so that the user can listen to the description in real time.

[0117] "Means for storing and using user profile information" refers to the means for storing the information entered by the user and the results of analysis by the server in a database and using that information the next time the service is used.

[0118] "Means for generating explanations using an AI language model" refers to a means for generating explanations based on the analysis results using a language model based on machine learning.

[0119] "Means for converting a text-format commentary into an audio format" refers to means for converting the generated commentary into audio data using speech synthesis technology.

[0120] This invention relates to a sports viewing system that generates and provides live commentary customized for each user using user input data and real-time video analysis. This system is mainly composed of three elements: a server, a terminal, and a user.

[0121] The system's main hardware includes terminals for collecting user input information, camera equipment used in stadiums, etc., and a server for processing and analyzing the data. The software used includes an analysis module that analyzes input data and generates profiles, a video analysis module that analyzes video data in real time, and software that generates audio commentary using a generative AI model.

[0122] In operation, the user first accesses a sports viewing app using their device and inputs their favorite sport, favorite team, players, and level of understanding of the rules. The device collects this input data and sends it to the server. The server analyzes the received data and generates a profile for each user. For example, information such as favorite sport being "soccer," favorite team being "Barcelona," and rule understanding level being "intermediate" is saved as a profile.

[0123] When the match begins, the device receives real-time video feeds from multiple cameras installed in the stadium and streams the data to a server, where a video analysis module analyzes the video data to recognize player movements and types of plays, such as when a Barcelona player is about to take a shot.

[0124] The server then uses the analysis results to generate a customized commentary based on the user's profile information. Using a generative AI model, the server creates a commentary tailored to each user. For example, the server might generate a commentary such as, "A Barcelona player took a powerful shot. This player is known for his forward movement and particularly for his accuracy in free kicks."

[0125] The generated text commentary is converted into audio data by a speech synthesis module and sent from the server to the device, where it is played back, allowing the user to enjoy a customized live commentary in real time.

[0126] For example, the following prompt sentences are used:

[0127] "Generate commentary when a Barcelona player takes a shot. Intermediate level commentary."

[0128] The system of this invention significantly improves the viewing experience by providing customizable live sports commentary tailored to the individual needs of users. This system is a new platform that allows for more enjoyment and a deeper understanding of sports viewing.

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] Step 1:

[0131] User status settings

[0132] User: Opens a sports viewing app and enters their favorite sport, the team and player they support, and their level of understanding of the rules.

[0133] Input: Favorite sport, team / player you support, level of understanding of the rules

[0134] Output: Input data

[0135] Terminal: Collects user input data and sends it to the server.

[0136] Input: Data entered by the user

[0137] Output: Data sent to the server

[0138] Server: Analyzes the received input data, generates a user profile, and stores it in a database.

[0139] Input: The input data received.

[0140] Data processing: Organizing user preferences and understanding into profiles

[0141] Output: User profile information (stored in database)

[0142] Specific working example:

[0143] The user selects "soccer" as their "favorite sport," sets their favorite team to "Barcelona," and sets their level of understanding of the rules to "intermediate." The device sends this information to the server, which then generates profile data and stores it in a database.

[0144] Step 2:

[0145] Real-time video analysis

[0146] Terminal: Receives images from multiple imaging devices in real time and streams the data to the server.

[0147] Input: Image data from imaging device

[0148] Output: Streaming data (to server)

[0149] Server: Sends video data to the analysis module for object recognition and motion analysis. Extracts play data (shots, passes, goals, etc.) and updates each player and the game situation in real time.

[0150] Input: Video data received in real time

[0151] Data processing: Extraction of play data through video analysis

[0152] Output: Updated play data

[0153] Specific working example:

[0154] At the start of the match, the device connects to the stadium's camera system to capture video in real time. This data is then sent to a server, where an analysis module detects when a Barcelona player is about to take a shot.

[0155] Step 3:

[0156] AI live commentary voice generation

[0157] Server: The extracted play data is compared with each user's profile information to generate customized commentary content.

[0158] Input: Play data, user profile information

[0159] Data manipulation: Generating customized commentary using generative AI models

[0160] Output: explanatory text

[0161] Server: Converts the generated textual commentary into audio.

[0162] Input: explanatory text

[0163] Data processing: Converting data into voice data using voice synthesis technology

[0164] Output: Audio data

[0165] Specific working example:

[0166] The system analyzes the situation when a Barcelona player takes a shot and generates a commentary aimed at intermediate users, such as, "A Barcelona player took a powerful shot. This player is known for his movement up front, and is particularly good at taking free kicks." This is then converted into audio data.

[0167] Step 4:

[0168] Providing customized commentary

[0169] Server: Sends the generated audio description data to the user's device.

[0170] Input: Audio data

[0171] Output: Data sent to the terminal

[0172] Device: Plays back the received audio data and provides the user with real-time, customized commentary.

[0173] Input: Received audio data

[0174] Output: The audio to be played

[0175] Specific working example:

[0176] The server generates audio data and sends it to the device of the intermediate user, who then instantly plays it back, allowing the user to enjoy customized commentary while watching the game.

[0177] (Application example 1)

[0178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0179] In virtual stores, users have difficulty receiving customized information based on their preferences and level of understanding in real time. Furthermore, the lack of systems that provide voice guidance and advanced analytical information to understand product details limits the user experience.

[0180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0181] In this invention, the server includes a means for acquiring user profile data, a means for capturing camera images in real time and performing preprocessing, and a means for analyzing the images using a deep learning model and interpreting the prediction results, thereby enabling the provision of customized product information and explanations based on the user's preferences and level of understanding in real time.

[0182] "User Input Data" refers to information provided by users, such as preferences and comprehension levels.

[0183] "Multiple camera footage" refers to real-time video data acquired from multiple cameras.

[0184] "Means for analyzing in real time" refers to means for processing and analyzing acquired video data in real time.

[0185] "User Status" refers to the current state or profile information of a User based on their preferences and understanding.

[0186] "Customized Commentary" refers to specific information or explanations generated based on a user's status.

[0187] "User profile data" refers to information about a user, such as certain attributes, preferences, and behavioral history.

[0188] The "means for pre-processing" refers to a means for performing processing to make the acquired video data easier to analyze.

[0189] A "deep learning model" refers to a machine learning model based on a large dataset and is used for video analysis.

[0190] "Means for interpreting prediction results" refers to the means for interpreting the output of a deep learning model and making specific judgments.

[0191] "Product data" refers to detailed information about each product in the virtual store.

[0192] "Voice data" refers to data that provides generated product information and explanations to users as voice information.

[0193] This invention relates to a system that provides customized product information and explanations based on the user's preferences and level of understanding within a virtual store. The system is mainly composed of three elements: a server, a terminal, and a user.

[0194] 1. Program Overview

[0195] Users access the virtual store using a terminal and input their purchase history and preferences. The server collects this data and analyzes the camera footage in real time to provide the user with the most appropriate information. A specific embodiment of this will be described below.

[0196] 2. Hardware and Software Used

[0197] Device: Smartphone, smart glasses, or head-mounted display.

[0198] Camera: A camera that captures video in real time.

[0199] Server: A server for processing and storing user data and video data.

[0200] software:

[0201] OpenCV: A library for capturing and preprocessing camera footage.

[0202] TensorFlow & Keras: Libraries for implementing deep learning models used in video analysis.

[0203] REST API: Interface for retrieving user profile data and product information.

[0204] gTTS (Google Text-to-Speech): A tool for converting customized product information into audio.

[0205] pygame: A library for playing generated audio data.

[0206] 3. Program flow and data processing

[0207] Retrieving a user profile

[0208] The server uses a REST API to retrieve user profile data, including user preferences and past purchase history. When a user accesses a virtual store, the system uses this data to provide optimal information.

[0209] Camera image capture and pre-processing

[0210] The device's camera captures video in real time, preprocessing is performed using OpenCV, and the video data is fed into a deep learning model using TensorFlow and Keras for real-time analysis.

[0211] Video analysis and interpretation of prediction results

[0212] The server analyzes the video data input in real time using a deep learning model, interprets the prediction results, and generates product information and explanations based on the user's status (preferences and level of understanding).

[0213] Product data acquisition and voice generation

[0214] Based on the analysis results, the server retrieves product data using the REST API. Next, it converts the product information generated using gTTS into voice data. Finally, it plays the voice data generated using pygame on the device and provides it to the user.

[0215] 4. Specific Examples

[0216] Example 1: While a user is browsing the fruit section, the camera recognizes an apple. The system retrieves the "detailed apple information" and provides a voice prompt such as "This product is an apple from Aomori Prefecture, priced at 100 yen each. It is currently on sale, with a set of three for 250 yen."

[0217] 5. Examples of prompts

[0218] When a user is browsing the fruit section of a virtual store, generate customized voice guidance based on product data recognized through camera video analysis.

[0219] Thus, the present invention provides a system for improving the user experience within a virtual store.

[0220] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0221] Step 1:

[0222] The server retrieves the user's profile data using a REST API. It receives the user ID as input and returns the profile data, including past purchase history and preferences, as output. Specifically, it sends the user ID to the API endpoint and receives the profile data in JSON format as a response.

[0223] Step 2:

[0224] The device uses a camera to capture video in real time. It receives video data from the camera as input and obtains the captured video data as output. The specific operation is to start the camera device and continuously acquire frames.

[0225] Step 3:

[0226] The device uses OpenCV to preprocess the captured video data. It receives video frames from the camera as input and converts them into an easily analyzable format as output. Specific operations include resizing and normalizing the video frames.

[0227] Step 4:

[0228] The device uses TensorFlow and Keras to analyze preprocessed video data in real time. It receives preprocessed video data as input and obtains prediction results as output. Specifically, it inputs the data into a deep learning model and makes predictions about product types and features.

[0229] Step 5:

[0230] The server interprets the prediction results and retrieves the product data. It receives the analysis results (such as the predicted product ID) as input and obtains detailed information about the product as output. Specifically, it uses the REST API again based on the product ID to retrieve the product data, including the product name, price, description, etc.

[0231] Step 6:

[0232] The server uses gTTS to convert the product description into voice data. It receives product data (in text format) as input and generates a voice file as output. The specific operation is to input text information into gTTS, generate and save the voice file.

[0233] Step 7:

[0234] The terminal plays audio data generated using pygame. It receives an audio file as input and provides audio information to the user as output. The specific operation is to load an audio file and play the audio through the speaker.

[0235] Step 8:

[0236] Users use their devices to receive customized product information and explanations in real time. While listening to the audio guidance, they can browse products in a virtual store. Specific actions include checking the audio information through the device and taking purchasing action as needed.

[0237] This allows users to receive real-time, individually customized product information within a virtual store.

[0238] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0239] This invention relates to a system that recognizes and analyzes user input data and emotional state, and customizes live sports commentary through real-time video analysis. The system consists of three main components: a server, a terminal, and a user.

[0240] Program processing explanation

[0241] Setting user status and emotions

[0242] 1. User: Opens a sports viewing app and enters their favorite sport, the team / player they support, and their level of understanding of the rules.

[0243] 2. Terminal: Collects user input data and sends it to the server.

[0244] 3. Server: Analyzes the received user input data and generates a profile that reflects the user's preferences and level of understanding.

[0245] 4. Server: Stores user profile information in a database.

[0246] Emotional Engine Activation

[0247] 1. Device: Collects the user's facial expressions and tone of voice in real time through a camera and microphone.

[0248] 2. Server: Sends the collected data to the emotion engine to recognize and analyze the user's emotional state.

[0249] 3. Server: Reflects the results of the emotion engine into the user profile and stores the analysis results in a database.

[0250] Real-time video analysis

[0251] 1. Terminal: As soon as the match starts, it receives images from multiple cameras in real time and streams them to the server.

[0252] 2. Server: Sends the received video data to the analysis module for object recognition and motion analysis.

[0253] 3. Server: Based on the results of video analysis, it extracts play data (shots, passes, goals, etc.) and updates the status of each player and the match in real time.

[0254] AI live commentary voice generation

[0255] 1. Server: Compares extracted play data with the user's profile information and emotional state.

[0256] 2. Server: Generates customized commentary based on the user's level of understanding and emotions. For example, it provides beginners with basic rule explanations, intermediate players with tactics and player characteristics, and advanced players with advanced statistics and analysis using past match data.

[0257] 3. Server: Depending on the user's emotional state, the server adjusts the commentary to be more uplifting when excited, or more detailed when relaxed.

[0258] 4. Server: Generates commentary that includes information specific to the team or player you support.

[0259] Providing customized commentary

[0260] 1. Server: Sends the generated commentary audio data to the user's device.

[0261] 2. Terminal: By playing back the received audio data, users can enjoy customized commentary in real time.

[0262] Specific examples

[0263] (Example 1) Beginner user

[0264] User: A beginner who barely knows the rules of soccer.

[0265] Device: Select "Soccer" as your favorite sport and set your rule comprehension level to "Beginner."

[0266] Server: Three minutes into the game, the ball is in the hands of the goalkeeper.

[0267] Server: Generates the explanation "The goalkeeper has caught the ball. The goalkeeper is the only position that can catch the ball with their hands."

[0268] Device: Provide this description to the user aloud.

[0269] (Example 2) Intermediate user

[0270] User: An intermediate player who enjoys watching soccer. His favorite team is a specific team.

[0271] Device: Select "Soccer" as your favorite sport, set your favorite team to a specific team, and set your rule comprehension level to "Intermediate."

[0272] Server: A player on a particular team takes the shot.

[0273] Server: Generates an explanation such as, "A player from a particular team took a powerful shot. This player is known for his forward movement and is particularly good at free kicks."

[0274] Device: Provide this description to the user aloud.

[0275] (Example 3) Advanced user

[0276] User: Experienced soccer player with a thorough understanding of tactics and player data.

[0277] Device: Select "Soccer" as your favorite sport and set your rule understanding level to "Advanced."

[0278] Server: Midway through the match, certain teams change to a 4-3-3 formation.

[0279] Server: "Currently, a certain team has switched to a 4-3-3 formation. This tactical change is to strengthen the midfield press, and a certain player could be a game changer from now on. His data from the last 15 games shows that he has a 2% higher goal success rate when using this formation."

[0280] Device: Provide this description to the user aloud.

[0281] (Example 4) User using emotion engine

[0282] User: Supports a particular team and gets excited during the game.

[0283] Device: The camera and microphone collect the user's facial expressions and tone of voice, which are then analyzed by the emotion engine.

[0284] Server: Detects when the user is in an excited state and generates a commentary that corresponds to that state.

[0285] Server: Generate commentary to further excite users, such as "A certain team has a chance to score! What a spectacular play!"

[0286] Device: Provide this description to the user aloud.

[0287] The system of the present invention significantly improves the viewing experience by customizing live sports commentary to suit the individual needs and emotions of each user. This system is a new platform that allows for more enjoyment and deeper understanding of sports viewing.

[0288] The processing flow will be explained below.

[0289] Step 1:

[0290] User: Launches a sports viewing app and enters their favorite sport (e.g., soccer), the team or player they support (e.g., Team A or Player X), and their level of understanding of the rules (e.g., beginner, intermediate, advanced).

[0291] Step 2:

[0292] Terminal: Collects data entered by the user and sends it to the server.

[0293] Step 3:

[0294] Server: Analyzes the received user input data and generates a user profile, which includes the user's preferences and understanding.

[0295] Step 4:

[0296] Server: The generated user profile information is saved in a database, which manages the individual information of each user.

[0297] Step 5:

[0298] Terminal: As soon as the match starts, it receives images from multiple cameras in real time and streams the data to the server.

[0299] Step 6:

[0300] Server: The received video data is sent to the analysis module, where object recognition and motion analysis are performed, thereby identifying the movements of players and the position of the ball.

[0301] Step 7:

[0302] Server: Based on the analysis results, it extracts play data (e.g., shots, passes, goals, etc.) and updates each player's data and the match situation in real time.

[0303] Step 8:

[0304] On the device: Cameras and microphones collect the user's facial expressions, tone of voice, and other emotion-related data in real time.

[0305] Step 9:

[0306] Terminal: Sends collected emotion-related data to the server.

[0307] Step 10:

[0308] Server: The emotion engine in the server recognizes and analyzes the user's emotional state based on their facial expressions, tone of voice, and other data.

[0309] Step 11:

[0310] Server: Updates the user's emotional state based on the user profile information and stores the information in a database.

[0311] Step 12:

[0312] Server: Generates customized commentary based on extracted play data, user profile information, and emotional state.

[0313] Step 13:

[0314] Server: Generates explanations using basic rules for beginners, tactics and player characteristics for intermediate players, and advanced statistics and past match data for advanced players.

[0315] Step 14:

[0316] Server: Generates commentary with highlights and exciting information from the match if the user is excited, and provides detailed information and expert commentary if the user is calm.

[0317] Step 15:

[0318] Server: Sends the generated commentary audio data to the user's device.

[0319] Step 16:

[0320] Device: Plays back the received audio data and provides the user with real-time, customized commentary.

[0321] Step 17:

[0322] Users: Enjoy watching sports more deeply and with personalized commentary.

[0323] According to the above steps, the system of the present invention provides personalized sports commentary and enhances the viewing experience through real-time adjustments according to the emotional state.

[0324] Example 2

[0325] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0326] Conventional sports viewing systems typically provide uniform live commentary without considering each user's individual needs or emotional state. This results in a failure to provide optimal commentary tailored to the user's level of understanding and preferences, resulting in a poor viewing experience. Furthermore, because commentary does not reflect the user's real-time emotional state during the game, it is not possible to fully evoke the user's excitement and satisfaction. This leads to a decline in the user's motivation to watch the game.

[0327] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving input data from a user and setting the user's preferences and understanding of sports, a means for receiving images from multiple camera devices and analyzing them in real time, a means for generating a customized commentary based on the analysis results in accordance with the user's status and emotional state, a means for collecting the user's facial expressions and tone of voice via a camera and microphone and recognizing and analyzing the emotional state, and a means for providing the generated commentary to the user. This makes it possible to provide a customized commentary that reflects the user's individual needs and emotional state in real time.

[0328] A "user" is an individual who inputs their favorite sports, the teams and players they support, and their understanding of the rules through a sports viewing app.

[0329] The "server" is a central processing unit that analyzes user input data, generates and stores user profiles, and uses emotion engines and video analysis modules to generate and provide customized live commentary.

[0330] A "terminal" is a device on which a sports viewing app used by a user is installed, and is a device that transmits user input data and real-time video, audio, and facial expression data to a server.

[0331] A "user profile" is a data set that reflects a user's sports preferences and understanding, the teams and players they support, and their real-time emotional state.

[0332] An "emotion engine" is a machine learning algorithm and software that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[0333] The "video analysis module" is an algorithm and software that analyzes video data received from multiple camera devices in real time and detects and tags play events such as shots, passes, and goals.

[0334] "Customized Commentary" means personalized sports commentary generated based on a user's profile information and real-time emotional state.

[0335] "Sports understanding" refers to the level of knowledge and understanding of the rules of a particular sport that a user has, and is classified into beginner, intermediate, and advanced levels.

[0336] "Filming equipment" refers to equipment such as a camera used to capture video of the match in real time and transmit it to a server via a terminal.

[0337] "Real-time analysis" is the process of instantly processing video data and user emotional data collected during a match to generate results.

[0338] This invention relates to a system that recognizes and analyzes user input data and emotional states, and customizes live sports commentary using real-time video analysis. This system consists of three main components: a server, a terminal, and a user.

[0339] Hardware and Software Configuration

[0340] Server: A high-performance data analysis server is used, which is installed with software that analyzes user input data, generates and saves profiles, an emotion engine, a video analysis module, and software that generates AI commentary voices.

[0341] Device: A device used by a user, such as a smartphone, tablet, or PC. The device is equipped with a camera and microphone and has a sports viewing app installed.

[0342] User: An individual who enjoys watching sports, inputs data and watches the game through the app.

[0343] User input data and status settings

[0344] A user launches a sports viewing app and inputs their favorite sport, the team and player they support, and their level of understanding. For example, they may select soccer, support Team X, and set their level of understanding as "Beginner." This data is collected by the device and sent to the server. The server analyzes the received data and generates a user profile. The generated profile information is stored in a database.

[0345] Emotional Engine Activation

[0346] During a match, the device's camera and microphone are used to collect the user's facial expressions and tone of voice in real time. This data is sent to a server and analyzed by an emotion engine. The server then reflects the results of this analysis in the user's profile and stores them in a database.

[0347] Real-time video analysis

[0348] When the match begins, the device receives real-time video feeds from multiple cameras and streams them to the server. The server then sends the video data to an analysis module for object recognition and motion analysis. Based on the analysis results, play data (e.g., shots, passes, goals) is extracted and the status of each player and the match is updated in real time.

[0349] AI live commentary voice generation

[0350] The server compares the play data with the user's profile information and emotional state to generate customized commentary. For example, beginners are provided with basic rule explanations, intermediate players with explanations of tactics and player characteristics, and advanced players with advanced statistical data and analysis of past matches. Also, depending on the user's emotional state, the server provides an uplifting commentary when excited, and a detailed commentary when relaxed. The server also generates commentary that includes information specific to the team or player the user is rooting for.

[0351] Providing customized commentary

[0352] The server transmits the generated live commentary audio data to the user's device, which then plays the audio data, allowing the user to enjoy the customized live commentary in real time.

[0353] Specific examples

[0354] (Example 1) Beginner user

[0355] User: A beginner who barely knows the rules of soccer.

[0356] Device: Select "Soccer" as your favorite sport and set your rule comprehension level to "Beginner."

[0357] Server: Three minutes into the game, the ball is in the hands of the goalkeeper.

[0358] Server: Generates the explanation "The goalkeeper has caught the ball. The goalkeeper is the only position that can catch the ball with their hands."

[0359] Device: Provide this description to the user aloud.

[0360] (Example 2) Intermediate user

[0361] User: An intermediate player who enjoys watching soccer. His favorite team is a specific team.

[0362] Device: Select "Soccer" as your favorite sport, set your favorite team to a specific team, and set your rule comprehension level to "Intermediate."

[0363] Server: A player on a particular team takes the shot.

[0364] Server: Generates an explanation such as, "A player from a particular team took a powerful shot. This player is known for his forward movement and is particularly good at free kicks."

[0365] Device: Provide this description to the user aloud.

[0366] (Example 3) Advanced user

[0367] User: Experienced soccer player with a thorough understanding of tactics and player data.

[0368] Device: Select "Soccer" as your favorite sport and set your rule understanding level to "Advanced."

[0369] Server: Midway through the match, certain teams change to a 4-3-3 formation.

[0370] Server: "Currently, a certain team has switched to a 4-3-3 formation. This tactical change is to strengthen the midfield press, and a certain player could be a game changer from now on. His data from the last 15 games shows that he has a 2% higher goal success rate when using this formation."

[0371] Device: Provide this description to the user aloud.

[0372] (Example 4) User using emotion engine

[0373] User: Supports a particular team and gets excited during the game.

[0374] Device: The camera and microphone collect the user's facial expressions and tone of voice, which are then analyzed by the emotion engine.

[0375] Server: Detects when the user is in an excited state and generates a commentary that corresponds to that state.

[0376] Server: Generate commentary to further excite users, such as "A certain team has a chance to score! What a spectacular play!"

[0377] Device: Provide this description to the user aloud.

[0378] The system dramatically improves the sports viewing experience by providing optimized commentary in real time based on the user's status and emotional state.

[0379] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0380] Step 1: Enter and submit user information

[0381] User: Launches a sports viewing app and inputs their favorite sport, the team and player they support, and their level of understanding of the rules. Specifically, they input information such as "Soccer," "Team X," "Player Y," and "Beginner" into the input fields on the screen.

[0382] Device: The data entered by the user is temporarily stored on the device and then sent to the server. JSON format data containing the input data (sport, team, player, and level of understanding) is generated and sent to the server.

[0383] Input: User input data (sport, team, player, understanding).

[0384] Output: User information data in JSON format sent to the server.

[0385] Step 2: Generate a user profile

[0386] Server: Analyzes the received user input data and generates a profile that reflects the user's preferences and level of understanding. Specifically, the input data is processed by the analysis module to generate a user profile object.

[0387] Server: Stores the generated user profile information in a database.

[0388] Input: User information data sent from the device (JSON format).

[0389] Output: A user profile object is saved to the database.

[0390] Step 3: Collect and analyze emotion data

[0391] Device: During the match, a camera and microphone are used to collect the user's facial expressions and tone of voice in real time. Specifically, the camera captures facial expression data and the microphone captures audio data.

[0392] Terminal: Sends collected data to the server in real time.

[0393] Server: Sends the received data to the emotion engine to analyze the user's emotional state. The emotion engine uses machine learning algorithms to analyze the data and generate emotion tags.

[0394] Server: Reflects the analysis results of the emotion engine into the user profile and saves it in the database.

[0395] Input: Facial expression and voice data collected by camera and microphone.

[0396] Output: An updated user profile reflecting their emotional state.

[0397] Step 4: Receiving and analyzing video data

[0398] Terminal: Receives multiple camera images in real time as soon as the match starts and streams them to the server. Specifically, the camera image data is sent using RTSP (Real Time Streaming Protocol).

[0399] Server: The server sends the received video data to the analysis module, where it performs object recognition and motion analysis, such as detecting and tagging play events such as shots, passes, and goals.

[0400] Server: Stores the analysis results in a database and updates the match progress data.

[0401] Input: Real-time video data from each camera.

[0402] Output: Match progress data tagged with play events.

[0403] Step 5: Generate and serve AI commentary

[0404] Server: Compares play data with the user's profile information and emotional state to generate customized commentary. For example, beginners can receive basic commentary such as "The goalkeeper has caught the ball."

[0405] Server: It also generates commentary based on the user's emotional state. For an excited user, it provides emotional commentary such as "A specific team has a chance to score a goal soon!"

[0406] Server: Also generates commentary including information related to the team and players you support.

[0407] Server: Sends the generated commentary audio data to the user's device.

[0408] Terminal: Plays back the received audio data, allowing users to enjoy the commentary in real time.

[0409] Input: User profile data, emotional state data, match progress data.

[0410] Output: Customized commentary audio data.

[0411] (Application example 2)

[0412] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0413] In modern sports viewing, it is difficult to provide real-time commentary tailored to the user's skill level and interests, and general commentary alone is not enough to provide a satisfying experience for some users. Furthermore, there are limited means to provide customized commentary that takes into account the user's emotional state. In particular, to enhance the viewing experience on smart devices, flexible responses that respond to the user's emotions are required.

[0414] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0415] In this invention, the server includes means for receiving user input data and setting the user's sports preferences and understanding level, means for receiving and analyzing video from multiple cameras in real time, means for generating a commentary customized according to the user's status based on the analysis results, means for providing the generated commentary to the user, means for recognizing the user's emotional state, means for adaptively customizing the commentary according to the emotional state, and means for providing the commentary through an augmented reality device, thereby enabling a more personalized sports viewing experience according to the user's skill level and emotional state.

[0416] "User input data" refers to data that users input into a sports viewing app, including information about their preferences and understanding of sports, and the teams and players they support.

[0417] "Means for recognizing emotional states" refers to a system or technology that collects and analyzes biometric data such as a user's facial expressions and tone of voice to determine the user's emotional state in real time.

[0418] "Means for receiving footage from multiple cameras and analyzing it in real time" refers to a technology that receives footage from multiple cameras taken from different perspectives and analyzes it in real time to understand the situation and play of the game.

[0419] A "means for generating customized commentary" is a system or technology that generates sports commentary optimized for an individual user based on the user's input data and emotional state.

[0420] The "means for providing the generated commentary to the user" is a technology for transmitting the generated customized commentary to the user's terminal and providing it as audio, text, or video.

[0421] "Augmented reality devices" are devices such as smart glasses and head-mounted displays that display images and information in real time, enhancing the user's viewing experience.

[0422] "User's level of sports understanding" indicates the extent to which a user understands the rules, tactics, and characteristics of players of a sport, and is classified into beginner, intermediate, and advanced levels.

[0423] "User status" refers to the user's overall profile information, such as their sports viewing preferences, understanding, and emotional state.

[0424] "Means for adaptively customizing live commentary" refers to a technology that adjusts the content and tone of live commentary in real time according to the user's emotional state and status.

[0425] The "means for setting preferences and level of understanding" refers to a system in which users input their preferences and level of understanding of sports into an application, and a user profile is generated based on that information.

[0426] This system recognizes and analyzes the user's input data and emotional state, and customizes live sports commentary using real-time video analysis. This system consists of three main components: a server, a device, and a user.

[0427] System configuration

[0428] Setting user status and emotions

[0429] The server receives a user's input data when the user opens a sports viewing app, including their favorite sport, the team or player they support, and their level of understanding of the rules. The device then collects the user's input data and sends it to the server. The server then analyzes the received user input data and generates a profile that reflects the user's preferences and level of understanding. The server then stores the generated user profile information in a database.

[0430] Emotional Engine Activation

[0431] The device collects the user's facial expressions and tone of voice in real time through a camera and microphone. This biometric data is sent to a server, where an emotion engine recognizes and analyzes the user's emotional state. The emotion engine's results are reflected in the user's profile, and the analysis results are stored in a database.

[0432] Real-time video analysis

[0433] As soon as the match begins, the device receives real-time video from multiple cameras and streams it to the server. The server then sends the video data to an analysis module for object recognition and motion analysis. This extracts play data (shots, passes, goals, etc.) and updates the status of each player and the match in real time.

[0434] AI commentary generation

[0435] The server compares the extracted play data with the user's profile information and emotional state. For example, it provides beginners with basic rule explanations, intermediate players with explanations of tactics and player characteristics, and advanced players with analysis using advanced statistics and past match data. It also adjusts the commentary according to the user's emotional state, focusing on uplifting commentary when excited and detailed commentary when relaxed. It also generates commentary that includes information specific to the team or player the user is supporting.

[0436] Providing customized commentary

[0437] The generated commentary audio data is sent from the server to the user's device, which then plays it back. Users can enjoy customized commentary in real time. This system enables a more personalized sports viewing experience that reflects the user's skill level and emotional state.

[0438] Hardware and software used

[0439] Hardware:

[0440] Camera (captures the user's facial expressions)

[0441] Microphone (captures the tone of your voice)

[0442] Augmented reality devices (smart glasses, head-mounted displays)

[0443] software:

[0444] opencv-python (real-time video analysis)

[0445] requests (communication with the server)

[0446] TensorFlow (machine learning model)

[0447] Flask and Django (API server construction)

[0448] Transformers (models for text generation)

[0449] Specific examples

[0450] Beginner users:

[0451] Prompt: Generate an explanatory sentence that explains the role of a goalkeeper for beginners.

[0452] Advanced users:

[0453] Prompt: For advanced players, please generate an explanatory statement based on the purpose of the 4-3-3 formation change and historical statistics.

[0454] In this way, the system of the present invention significantly enhances the viewing experience by providing a customized commentary in real time that is tailored to the user's preferences, level of understanding, and emotional state.

[0455] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0456] Step 1:

[0457] The user opens a sports viewing app and inputs their favorite sport, the team / player they support, and their level of understanding of the rules. The device receives this input data and sends it to the server. The input data includes their favorite sport (e.g., soccer), the team / player they support (e.g., Team A), and their level of understanding of the rules (e.g., beginner) in text format. The device sends this data to the server in JSON format.

[0458] Step 2:

[0459] The server analyzes the received input data and generates a profile that reflects the user's preferences and understanding. Specifically, it stores the input data in a database and applies algorithms (e.g., normalization and clustering of user information) to generate a profile. This profile information is stored in the database and used for further analysis. The generated user profile is saved as an output.

[0460] Step 3:

[0461] The device collects the user's facial expressions and tone of voice in real time through a camera and microphone. The collected data is temporarily stored on the device and periodically sent to a server. Specifically, facial expressions are captured using the OpenCV library, and audio data is collected from the microphone. Input data includes video frames and audio data.

[0462] Step 4:

[0463] The server sends the collected facial expression data and voice data to an emotion engine, which recognizes and analyzes the user's emotional state. This emotion engine uses machine learning models such as TensorFlow to analyze the emotional state (e.g., joy, excitement, sadness, etc.) using a pre-trained emotion classification model. The server reflects the analysis results in the user profile and stores them in a database. The input data is facial expression data and voice data, and the output is a classification result of the emotional state.

[0464] Step 5:

[0465] As soon as the match starts, the device receives multiple camera images in real time and streams them to the server. The camera images are transmitted using a streaming protocol (e.g., RTSP). The input data is the camera image data, and the output is the streaming data transmitted to the server in real time.

[0466] Step 6:

[0467] The server sends the received video data to an analysis module for object recognition and motion analysis. This analysis module incorporates an image recognition algorithm (e.g., YOLO) and extracts play data (shots, passes, goals, etc.). The input data is video data, and the output is analyzed play data.

[0468] Step 7:

[0469] The server compares the extracted play data with the user's profile information and emotional state to generate a customized commentary. For example, it provides basic rule explanations for beginners, tactics and player characteristics for intermediate players, and advanced statistics and analysis using past match data for advanced players. The generated commentary is generated using a natural language processing library (e.g., Transformers). The input data are play data, user profile, and emotional state, and the output is a customized commentary text.

[0470] Step 8:

[0471] The generated commentary audio data is sent from the server to the user's device. The device plays the received audio data and provides the user with a customized commentary in real time. The input data is the customized commentary text, which is converted into audio data using speech synthesis technology (e.g., Text-to-Speech). The output is the audio commentary that is played on the user's device.

[0472] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0473] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0474] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0475] [Second embodiment]

[0476] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0477] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0478] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0479] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0480] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0481] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0482] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0483] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0484] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0485] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0486] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0487] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0488] This invention relates to a sports viewing system that generates and provides live commentary customized for each user using user input data and real-time video analysis. This system is mainly composed of three elements: a server, a terminal, and a user.

[0489] Program processing explanation

[0490] Program Overview

[0491] 1. User status settings

[0492] User: Opens a sports viewing app and enters their favorite sport, the team and player they support, and their level of understanding of the rules.

[0493] Terminal: Collects user input data and sends it to the server.

[0494] Server: Analyzes the user's input data, generates a user profile, and stores it in a database.

[0495] 2. Real-time video analysis

[0496] Terminal: Receives multiple camera images in real time and streams them to the server.

[0497] Server: Sends video data to the analysis module for object recognition and motion analysis. Extracts play data (shots, passes, goals, etc.) and updates each player and the game situation in real time.

[0498] 3. AI commentary voice generation

[0499] Server: Compares extracted play data with user profile information.

[0500] Server: Generates customized commentary for beginner, intermediate and advanced players, including information specific to the team or player you support.

[0501] 4. Providing customized commentary

[0502] Server: Sends the generated commentary audio data to the device.

[0503] Device: Plays audio data, allowing users to enjoy real-time, customized commentary.

[0504] Specific examples

[0505] (Example 1) Beginner user

[0506] User: A beginner who barely knows the rules of soccer.

[0507] Device: Select "Soccer" as your favorite sport and set your rule comprehension level to "Beginner."

[0508] Server: Three minutes into the game, the ball is in the hands of the goalkeeper.

[0509] Server: Generates the explanation "The goalkeeper has caught the ball. The goalkeeper is the only position that can catch the ball with their hands."

[0510] Device: Provide this description to the user aloud.

[0511] (Example 2) Intermediate user

[0512] User: An intermediate player who enjoys watching soccer. His favorite team is Barcelona.

[0513] Device: Select "Soccer" as your favorite sport, set your favorite team to "Barcelona," and set your rule comprehension level to "Intermediate."

[0514] Server: A Barcelona player takes the shot.

[0515] Server: Generates commentary like, "The Barcelona player took a powerful shot. This player is known for his forward movement and is particularly good at free kicks."

[0516] Device: Provide this description to the user aloud.

[0517] (Example 3) Advanced user

[0518] User: Experienced soccer player with a thorough understanding of tactics and player data.

[0519] Device: Select "Soccer" as your favorite sport and set your rule understanding level to "Advanced."

[0520] Sarver: Midway through the match, Barcelona changed to a 4-3-3 formation.

[0521] Server: Generates commentary: "Barcelona have now switched to a 4-3-3 formation. This tactical change is designed to strengthen their midfield press, and Messi in particular could be a game-changer from now on. His data from his last 15 games shows that he has a 2% higher goal success rate when using this formation."

[0522] Device: Provide this description to the user aloud.

[0523] The system of the present invention significantly improves the viewing experience by providing customizable live sports commentary tailored to the individual needs of users. This system is a new platform that allows for more enjoyment and a deeper understanding of sports viewing.

[0524] The processing flow will be explained below.

[0525] Step 1:

[0526] User: Opens a sports viewing app and enters their favorite sport, the team / player they support, and their level of understanding of the rules.

[0527] Step 2:

[0528] Terminal: Collects user input data and sends it to the server.

[0529] Step 3:

[0530] Server: Analyzes the received user input data and generates a profile that reflects the user's preferences, level of understanding, etc.

[0531] Step 4:

[0532] Server: Stores user profile information in a database.

[0533] Step 5:

[0534] Terminal: As soon as the match starts, it receives multiple camera images in real time and streams them to the server.

[0535] Step 6:

[0536] Server: Sends the received video data to the analysis module, where object recognition and motion analysis are performed.

[0537] Step 7:

[0538] Server: Based on the results of video analysis, it extracts play data (shots, passes, goals, etc.) and updates the status of each player and the match in real time.

[0539] Step 8:

[0540] Server: Compares extracted play data with user profile information.

[0541] Step 9:

[0542] Server: Generates customized explanations based on the user's level of understanding. For example, beginners can learn basic rules, intermediate players can learn tactics and player characteristics, and advanced players can learn advanced statistics and past match data.

[0543] Step 10:

[0544] Server: Generates commentary that includes information specific to the team or player you support.

[0545] Step 11:

[0546] Server: Sends the generated commentary audio data to the user's device.

[0547] Step 12:

[0548] Device: By playing back the received audio data, users can enjoy customized commentary in real time.

[0549] Step 13:

[0550] Users: Gain a deeper understanding and enjoyment of watching sports.

[0551] According to the above steps, the system of the present invention provides personalized sports commentary to enhance the viewing experience.

[0552] Example 1

[0553] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0554] Conventional sports viewing systems were unable to provide customized commentary tailored to the user's preferences and level of understanding, resulting in a uniform viewing experience that could not meet the needs of individual users. Furthermore, it was difficult to provide real-time video analysis or information specialized for specific teams or players, preventing users from gaining a deeper understanding or enjoyment.

[0555] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0556] In this invention, the server includes means for receiving user input data and setting the user's preferences and level of understanding of sports, means for receiving and analyzing images from multiple camera devices in real time, and means for generating audio commentary customized according to the user's status based on the analysis results, thereby enabling the provision of play-by-play commentary customized to meet the individual needs of each user.

[0557] "User input data" refers to information that users input when using a sports viewing app, such as their favorite sports, the teams and players they support, and their understanding of the rules.

[0558] The "multiple image capturing devices" refer to multiple cameras or other image capturing devices installed within the venue of a sporting event.

[0559] "Means for analyzing in real time" refers to a means for instantly analyzing the video data received from the filming device and recognizing the player's movements and type of play.

[0560] "Customized audio commentary according to user status" refers to audio data in which the content of the live commentary to be provided is adjusted according to the preferences and level of understanding set for each user.

[0561] A "means for providing to a user" is a means for delivering the generated audio description to a user device so that the user can listen to the description in real time.

[0562] "Means for storing and using user profile information" refers to the means for storing the information entered by the user and the results of analysis by the server in a database and using that information the next time the service is used.

[0563] "Means for generating explanations using an AI language model" refers to a means for generating explanations based on the analysis results using a language model based on machine learning.

[0564] "Means for converting a text-format commentary into an audio format" refers to means for converting the generated commentary into audio data using speech synthesis technology.

[0565] This invention relates to a sports viewing system that generates and provides live commentary customized for each user using user input data and real-time video analysis. This system is mainly composed of three elements: a server, a terminal, and a user.

[0566] The system's main hardware includes terminals for collecting user input information, camera equipment used in stadiums, etc., and a server for processing and analyzing the data. The software used includes an analysis module that analyzes input data and generates profiles, a video analysis module that analyzes video data in real time, and software that generates audio commentary using a generative AI model.

[0567] In operation, the user first accesses a sports viewing app using their device and inputs their favorite sport, favorite team, players, and level of understanding of the rules. The device collects this input data and sends it to the server. The server analyzes the received data and generates a profile for each user. For example, information such as favorite sport being "soccer," favorite team being "Barcelona," and rule understanding level being "intermediate" is saved as a profile.

[0568] When the match begins, the device receives real-time video feeds from multiple cameras installed in the stadium and streams the data to a server, where a video analysis module analyzes the video data to recognize player movements and types of plays, such as when a Barcelona player is about to take a shot.

[0569] The server then uses the analysis results to generate a customized commentary based on the user's profile information. Using a generative AI model, the server creates a commentary tailored to each user. For example, the server might generate a commentary such as, "A Barcelona player took a powerful shot. This player is known for his forward movement and particularly for his accuracy in free kicks."

[0570] The generated commentary in text format is converted into audio data by a speech synthesis module and sent from the server to the device, where it is played back, allowing the user to enjoy a customized commentary in real time.

[0571] For example, the following prompt sentences are used:

[0572] "Generate commentary when a Barcelona player takes a shot. Intermediate level commentary."

[0573] The system of this invention significantly improves the viewing experience by providing customizable live sports commentary tailored to the individual needs of users. This system is a new platform that allows for more enjoyment and a deeper understanding of sports viewing.

[0574] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0575] Step 1:

[0576] User status settings

[0577] User: Opens a sports viewing app and enters their favorite sport, the team and player they support, and their level of understanding of the rules.

[0578] Input: Favorite sport, team / player you support, level of understanding of the rules

[0579] Output: Input data

[0580] Terminal: Collects user input data and sends it to the server.

[0581] Input: Data entered by the user

[0582] Output: Data sent to the server

[0583] Server: Analyzes the received input data, generates a user profile, and stores it in a database.

[0584] Input: The input data received.

[0585] Data processing: Organizing user preferences and understanding into profiles

[0586] Output: User profile information (stored in database)

[0587] Specific working example:

[0588] The user selects "soccer" as their "favorite sport," sets their favorite team to "Barcelona," and sets their level of understanding of the rules to "intermediate." The device sends this information to the server, which then generates profile data and stores it in a database.

[0589] Step 2:

[0590] Real-time video analysis

[0591] Terminal: Receives images from multiple imaging devices in real time and streams the data to the server.

[0592] Input: Image data from imaging device

[0593] Output: Streaming data (to server)

[0594] Server: Sends video data to the analysis module for object recognition and motion analysis. Extracts play data (shots, passes, goals, etc.) and updates each player and the game situation in real time.

[0595] Input: Video data received in real time

[0596] Data processing: Extraction of play data through video analysis

[0597] Output: Updated play data

[0598] Specific working example:

[0599] At the start of the match, the device connects to the stadium's camera system to capture video in real time. This data is then sent to a server, where an analysis module detects when a Barcelona player is about to take a shot.

[0600] Step 3:

[0601] AI live commentary voice generation

[0602] Server: The extracted play data is compared with each user's profile information to generate customized commentary content.

[0603] Input: Play data, user profile information

[0604] Data manipulation: Generating customized commentary using generative AI models

[0605] Output: explanatory text

[0606] Server: Converts the generated textual commentary into audio.

[0607] Input: explanatory text

[0608] Data processing: Converting data into voice data using voice synthesis technology

[0609] Output: Audio data

[0610] Specific working example:

[0611] The system analyzes the situation when a Barcelona player takes a shot and generates a commentary aimed at intermediate users, such as, "A Barcelona player took a powerful shot. This player is known for his movement up front, and is particularly good at taking free kicks." This is then converted into audio data.

[0612] Step 4:

[0613] Providing customized commentary

[0614] Server: Sends the generated audio description data to the user's device.

[0615] Input: Audio data

[0616] Output: Data sent to the terminal

[0617] Device: Plays back the received audio data and provides the user with real-time, customized commentary.

[0618] Input: Received audio data

[0619] Output: The audio to be played

[0620] Specific working example:

[0621] The server generates audio data and sends it to the device of the intermediate user, who then instantly plays it back, allowing the user to enjoy customized commentary while watching the game.

[0622] (Application example 1)

[0623] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0624] In virtual stores, users have difficulty receiving customized information based on their preferences and level of understanding in real time. Furthermore, the lack of systems that provide voice guidance and advanced analytical information to understand product details limits the user experience.

[0625] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0626] In this invention, the server includes a means for acquiring user profile data, a means for capturing camera images in real time and performing preprocessing, and a means for analyzing the images using a deep learning model and interpreting the prediction results, thereby enabling the provision of customized product information and explanations based on the user's preferences and level of understanding in real time.

[0627] "User Input Data" refers to information provided by users, such as preferences and comprehension levels.

[0628] "Multiple camera footage" refers to real-time video data acquired from multiple cameras.

[0629] "Means for analyzing in real time" refers to means for processing and analyzing acquired video data in real time.

[0630] "User Status" refers to the current state or profile information of a User based on their preferences and understanding.

[0631] "Customized Commentary" refers to specific information or explanations generated based on a user's status.

[0632] "User profile data" refers to information about a user, such as certain attributes, preferences, and behavioral history.

[0633] The "means for pre-processing" refers to a means for performing processing to make the acquired video data easier to analyze.

[0634] A "deep learning model" refers to a machine learning model based on a large dataset and is used for video analysis.

[0635] "Means for interpreting prediction results" refers to the means for interpreting the output of a deep learning model and making specific judgments.

[0636] "Product data" refers to detailed information about each product in the virtual store.

[0637] "Voice data" refers to data that provides generated product information and explanations to users as voice information.

[0638] This invention relates to a system that provides customized product information and explanations based on the user's preferences and level of understanding within a virtual store. The system is mainly composed of three elements: a server, a terminal, and a user.

[0639] 1. Program Overview

[0640] Users access the virtual store using a terminal and input their purchase history and preferences. The server collects this data and analyzes the camera footage in real time to provide the user with the most appropriate information. A specific embodiment of this will be described below.

[0641] 2. Hardware and Software Used

[0642] Device: Smartphone, smart glasses, or head-mounted display.

[0643] Camera: A camera that captures video in real time.

[0644] Server: A server for processing and storing user data and video data.

[0645] software:

[0646] OpenCV: A library for capturing and preprocessing camera footage.

[0647] TensorFlow & Keras: Libraries for implementing deep learning models used in video analysis.

[0648] REST API: Interface for retrieving user profile data and product information.

[0649] gTTS (Google Text-to-Speech): A tool for converting customized product information into audio.

[0650] pygame: A library for playing generated audio data.

[0651] 3. Program flow and data processing

[0652] Retrieving a user profile

[0653] The server uses a REST API to retrieve user profile data, including user preferences and past purchase history. When a user accesses a virtual store, the system uses this data to provide optimal information.

[0654] Camera image capture and pre-processing

[0655] The device's camera captures video in real time, preprocessing is performed using OpenCV, and the video data is fed into a deep learning model using TensorFlow and Keras for real-time analysis.

[0656] Video analysis and interpretation of prediction results

[0657] The server analyzes the video data input in real time using a deep learning model, interprets the prediction results, and generates product information and explanations based on the user's status (preferences and level of understanding).

[0658] Product data acquisition and voice generation

[0659] Based on the analysis results, the server retrieves product data using the REST API. Next, it converts the product information generated using gTTS into voice data. Finally, it plays the voice data generated using pygame on the device and provides it to the user.

[0660] 4. Specific Examples

[0661] Example 1: While a user is browsing the fruit section, the camera recognizes an apple. The system retrieves the "detailed apple information" and provides a voice prompt such as "This product is an apple from Aomori Prefecture, priced at 100 yen each. It is currently on sale, with a set of three for 250 yen."

[0662] 5. Examples of prompts

[0663] When a user is browsing the fruit section of a virtual store, generate customized voice guidance based on product data recognized through camera video analysis.

[0664] Thus, the present invention provides a system for improving the user experience within a virtual store.

[0665] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0666] Step 1:

[0667] The server retrieves the user's profile data using a REST API. It receives the user ID as input and returns the profile data, including past purchase history and preferences, as output. Specifically, it sends the user ID to the API endpoint and receives the profile data in JSON format as a response.

[0668] Step 2:

[0669] The device uses a camera to capture video in real time. It receives video data from the camera as input and obtains the captured video data as output. The specific operation is to start the camera device and continuously acquire frames.

[0670] Step 3:

[0671] The device uses OpenCV to preprocess the captured video data. It receives video frames from the camera as input and converts them into an easily analyzable format as output. Specific operations include resizing and normalizing the video frames.

[0672] Step 4:

[0673] The device uses TensorFlow and Keras to analyze preprocessed video data in real time. It receives preprocessed video data as input and obtains prediction results as output. Specifically, it inputs the data into a deep learning model and makes predictions about product types and features.

[0674] Step 5:

[0675] The server interprets the prediction results and retrieves the product data. It receives the analysis results (such as the predicted product ID) as input and obtains detailed information about the product as output. Specifically, it uses the REST API again based on the product ID to retrieve the product data, including the product name, price, description, etc.

[0676] Step 6:

[0677] The server uses gTTS to convert the product description into voice data. It receives product data (in text format) as input and generates a voice file as output. The specific operation is to input text information into gTTS, generate and save the voice file.

[0678] Step 7:

[0679] The terminal plays audio data generated using pygame. It receives an audio file as input and provides audio information to the user as output. The specific operation is to load an audio file and play the audio through the speaker.

[0680] Step 8:

[0681] Users use their devices to receive customized product information and explanations in real time. While listening to the audio guidance, they can browse products in a virtual store. Specific actions include checking the audio information through the device and taking purchasing action as needed.

[0682] This allows users to receive real-time, individually customized product information within a virtual store.

[0683] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0684] This invention relates to a system that recognizes and analyzes user input data and emotional state, and customizes live sports commentary through real-time video analysis. The system consists of three main components: a server, a terminal, and a user.

[0685] Program processing explanation

[0686] Setting user status and emotions

[0687] 1. User: Opens a sports viewing app and enters their favorite sport, the team / player they support, and their level of understanding of the rules.

[0688] 2. Terminal: Collects user input data and sends it to the server.

[0689] 3. Server: Analyzes the received user input data and generates a profile that reflects the user's preferences and level of understanding.

[0690] 4. Server: Stores user profile information in a database.

[0691] Emotional Engine Activation

[0692] 1. Device: Collects the user's facial expressions and tone of voice in real time through a camera and microphone.

[0693] 2. Server: Sends the collected data to the emotion engine to recognize and analyze the user's emotional state.

[0694] 3. Server: Reflects the results of the emotion engine into the user profile and stores the analysis results in a database.

[0695] Real-time video analysis

[0696] 1. Terminal: As soon as the match starts, it receives images from multiple cameras in real time and streams them to the server.

[0697] 2. Server: Sends the received video data to the analysis module for object recognition and motion analysis.

[0698] 3. Server: Based on the results of video analysis, it extracts play data (shots, passes, goals, etc.) and updates the status of each player and the match in real time.

[0699] AI live commentary voice generation

[0700] 1. Server: Compares extracted play data with the user's profile information and emotional state.

[0701] 2. Server: Generates customized commentary based on the user's level of understanding and emotions. For example, it provides beginners with basic rule explanations, intermediate players with tactics and player characteristics, and advanced players with advanced statistics and analysis using past match data.

[0702] 3. Server: Depending on the user's emotional state, the server adjusts the commentary to be more uplifting when excited, or more detailed when relaxed.

[0703] 4. Server: Generates commentary that includes information specific to the team or player you support.

[0704] Providing customized commentary

[0705] 1. Server: Sends the generated commentary audio data to the user's device.

[0706] 2. Terminal: By playing back the received audio data, users can enjoy customized commentary in real time.

[0707] Specific examples

[0708] (Example 1) Beginner user

[0709] User: A beginner who barely knows the rules of soccer.

[0710] Device: Select "Soccer" as your favorite sport and set your rule comprehension level to "Beginner."

[0711] Server: Three minutes into the game, the ball is in the hands of the goalkeeper.

[0712] Server: Generates the explanation "The goalkeeper has caught the ball. The goalkeeper is the only position that can catch the ball with their hands."

[0713] Device: Provide this description to the user aloud.

[0714] (Example 2) Intermediate user

[0715] User: An intermediate player who enjoys watching soccer. His favorite team is a specific team.

[0716] Device: Select "Soccer" as your favorite sport, set your favorite team to a specific team, and set your rule comprehension level to "Intermediate."

[0717] Server: A player on a particular team takes the shot.

[0718] Server: Generates an explanation such as, "A player from a particular team took a powerful shot. This player is known for his forward movement and is particularly good at free kicks."

[0719] Device: Provide this description to the user aloud.

[0720] (Example 3) Advanced user

[0721] User: Experienced soccer player with a thorough understanding of tactics and player data.

[0722] Device: Select "Soccer" as your favorite sport and set your rule understanding level to "Advanced."

[0723] Server: Midway through the match, certain teams change to a 4-3-3 formation.

[0724] Server: "Currently, a certain team has switched to a 4-3-3 formation. This tactical change is to strengthen the midfield press, and a certain player could be a game changer from now on. His data from the last 15 games shows that he has a 2% higher goal success rate when using this formation."

[0725] Device: Provide this description to the user aloud.

[0726] (Example 4) User using emotion engine

[0727] User: Supports a particular team and gets excited during the game.

[0728] Device: The camera and microphone collect the user's facial expressions and tone of voice, which are then analyzed by the emotion engine.

[0729] Server: Detects when the user is in an excited state and generates a commentary that corresponds to that state.

[0730] Server: Generate commentary to further excite users, such as "A certain team has a chance to score! What a spectacular play!"

[0731] Device: Provide this description to the user aloud.

[0732] The system of the present invention significantly improves the viewing experience by customizing live sports commentary to suit the individual needs and emotions of each user. This system is a new platform that allows for more enjoyment and deeper understanding of sports viewing.

[0733] The processing flow will be explained below.

[0734] Step 1:

[0735] User: Launches a sports viewing app and enters their favorite sport (e.g., soccer), the team or player they support (e.g., Team A or Player X), and their level of understanding of the rules (e.g., beginner, intermediate, advanced).

[0736] Step 2:

[0737] Terminal: Collects data entered by the user and sends it to the server.

[0738] Step 3:

[0739] Server: Analyzes the received user input data and generates a user profile, which includes the user's preferences and understanding.

[0740] Step 4:

[0741] Server: The generated user profile information is saved in a database, which manages the individual information of each user.

[0742] Step 5:

[0743] Terminal: As soon as the match starts, it receives images from multiple cameras in real time and streams the data to the server.

[0744] Step 6:

[0745] Server: The received video data is sent to the analysis module, where object recognition and motion analysis are performed, thereby identifying the movements of players and the position of the ball.

[0746] Step 7:

[0747] Server: Based on the analysis results, it extracts play data (e.g., shots, passes, goals, etc.) and updates each player's data and the match situation in real time.

[0748] Step 8:

[0749] On the device: Cameras and microphones collect the user's facial expressions, tone of voice, and other emotion-related data in real time.

[0750] Step 9:

[0751] Terminal: Sends collected emotion-related data to the server.

[0752] Step 10:

[0753] Server: The emotion engine in the server recognizes and analyzes the user's emotional state based on their facial expressions, tone of voice, and other data.

[0754] Step 11:

[0755] Server: Updates the user's emotional state based on the user profile information and stores the information in a database.

[0756] Step 12:

[0757] Server: Generates customized commentary based on extracted play data, user profile information, and emotional state.

[0758] Step 13:

[0759] Server: Generates explanations using basic rules for beginners, tactics and player characteristics for intermediate players, and advanced statistics and past match data for advanced players.

[0760] Step 14:

[0761] Server: Generates commentary with highlights and exciting information from the match if the user is excited, and provides detailed information and expert commentary if the user is calm.

[0762] Step 15:

[0763] Server: Sends the generated commentary audio data to the user's device.

[0764] Step 16:

[0765] Device: Plays back the received audio data and provides the user with real-time, customized commentary.

[0766] Step 17:

[0767] Users: Enjoy watching sports more deeply and with personalized commentary.

[0768] According to the above steps, the system of the present invention provides personalized sports commentary and enhances the viewing experience through real-time adjustments according to the emotional state.

[0769] Example 2

[0770] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0771] Conventional sports viewing systems typically provide uniform live commentary without considering each user's individual needs or emotional state. This results in a failure to provide optimal commentary tailored to the user's level of understanding and preferences, resulting in a poor viewing experience. Furthermore, because commentary does not reflect the user's real-time emotional state during the game, it is not possible to fully evoke the user's excitement and satisfaction. This leads to a decline in the user's motivation to watch the game.

[0772] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving input data from a user and setting the user's preferences and understanding of sports, a means for receiving images from multiple camera devices and analyzing them in real time, a means for generating a customized commentary based on the analysis results in accordance with the user's status and emotional state, a means for collecting the user's facial expressions and tone of voice via a camera and microphone and recognizing and analyzing the emotional state, and a means for providing the generated commentary to the user. This makes it possible to provide a customized commentary that reflects the user's individual needs and emotional state in real time.

[0773] A "user" is an individual who inputs their favorite sports, the teams and players they support, and their understanding of the rules through a sports viewing app.

[0774] The "server" is a central processing unit that analyzes user input data, generates and stores user profiles, and uses emotion engines and video analysis modules to generate and provide customized live commentary.

[0775] A "terminal" is a device on which a sports viewing app used by a user is installed, and is a device that transmits user input data and real-time video, audio, and facial expression data to a server.

[0776] A "user profile" is a data set that reflects a user's sports preferences and understanding, the teams and players they support, and their real-time emotional state.

[0777] An "emotion engine" is a machine learning algorithm and software that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[0778] The "video analysis module" is an algorithm and software that analyzes video data received from multiple camera devices in real time and detects and tags play events such as shots, passes, and goals.

[0779] "Customized Commentary" means personalized sports commentary generated based on a user's profile information and real-time emotional state.

[0780] "Sports understanding" refers to the level of knowledge and understanding of the rules of a particular sport that a user has, and is classified into beginner, intermediate, and advanced levels.

[0781] "Filming equipment" refers to equipment such as a camera used to capture video of the match in real time and transmit it to a server via a terminal.

[0782] "Real-time analysis" is the process of instantly processing video data and user emotional data collected during a match to generate results.

[0783] This invention relates to a system that recognizes and analyzes user input data and emotional states, and customizes live sports commentary using real-time video analysis. This system consists of three main components: a server, a terminal, and a user.

[0784] Hardware and Software Configuration

[0785] Server: A high-performance data analysis server is used, which is installed with software that analyzes user input data, generates and saves profiles, an emotion engine, a video analysis module, and software that generates AI commentary voices.

[0786] Device: A device used by a user, such as a smartphone, tablet, or PC. The device is equipped with a camera and microphone and has a sports viewing app installed.

[0787] User: An individual who enjoys watching sports, inputs data and watches the game through the app.

[0788] User input data and status settings

[0789] A user launches a sports viewing app and inputs their favorite sport, the team and player they support, and their level of understanding. For example, they may select soccer, support Team X, and set their level of understanding as "Beginner." This data is collected by the device and sent to the server. The server analyzes the received data and generates a user profile. The generated profile information is stored in a database.

[0790] Emotional Engine Activation

[0791] During a match, the device's camera and microphone are used to collect the user's facial expressions and tone of voice in real time. This data is sent to a server and analyzed by an emotion engine. The server then reflects the results of this analysis in the user's profile and stores them in a database.

[0792] Real-time video analysis

[0793] When the match begins, the device receives real-time video feeds from multiple cameras and streams them to the server. The server then sends the video data to an analysis module for object recognition and motion analysis. Based on the analysis results, play data (e.g., shots, passes, goals) is extracted and the status of each player and the match is updated in real time.

[0794] AI live commentary voice generation

[0795] The server compares the play data with the user's profile information and emotional state to generate customized commentary. For example, beginners are provided with basic rule explanations, intermediate players with explanations of tactics and player characteristics, and advanced players with advanced statistical data and analysis of past matches. Also, depending on the user's emotional state, the server provides an uplifting commentary when excited, and a detailed commentary when relaxed. The server also generates commentary that includes information specific to the team or player the user is rooting for.

[0796] Providing customized commentary

[0797] The server transmits the generated live commentary audio data to the user's device, which then plays the audio data, allowing the user to enjoy the customized live commentary in real time.

[0798] Specific examples

[0799] (Example 1) Beginner user

[0800] User: A beginner who barely knows the rules of soccer.

[0801] Device: Select "Soccer" as your favorite sport and set your rule comprehension level to "Beginner."

[0802] Server: Three minutes into the game, the ball is in the hands of the goalkeeper.

[0803] Server: Generates the explanation "The goalkeeper has caught the ball. The goalkeeper is the only position that can catch the ball with their hands."

[0804] Device: Provide this description to the user aloud.

[0805] (Example 2) Intermediate user

[0806] User: An intermediate player who enjoys watching soccer. His favorite team is a specific team.

[0807] Device: Select "Soccer" as your favorite sport, set your favorite team to a specific team, and set your rule comprehension level to "Intermediate."

[0808] Server: A player on a particular team takes the shot.

[0809] Server: Generates an explanation such as, "A player from a particular team took a powerful shot. This player is known for his forward movement and is particularly good at free kicks."

[0810] Device: Provide this description to the user aloud.

[0811] (Example 3) Advanced user

[0812] User: Experienced soccer player with a thorough understanding of tactics and player data.

[0813] Device: Select "Soccer" as your favorite sport and set your rule understanding level to "Advanced."

[0814] Server: Midway through the match, certain teams change to a 4-3-3 formation.

[0815] Server: "Currently, a certain team has switched to a 4-3-3 formation. This tactical change is to strengthen the midfield press, and a certain player could be a game changer from now on. His data from the last 15 games shows that he has a 2% higher goal success rate when using this formation."

[0816] Device: Provide this description to the user aloud.

[0817] (Example 4) User using emotion engine

[0818] User: Supports a particular team and gets excited during the game.

[0819] Device: The camera and microphone collect the user's facial expressions and tone of voice, which are then analyzed by the emotion engine.

[0820] Server: Detects when the user is in an excited state and generates a commentary that corresponds to that state.

[0821] Server: Generate commentary to further excite users, such as "A certain team has a chance to score! What a spectacular play!"

[0822] Device: Provide this description to the user aloud.

[0823] The system dramatically improves the sports viewing experience by providing optimized commentary in real time based on the user's status and emotional state.

[0824] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0825] Step 1: Enter and submit user information

[0826] User: Launches a sports viewing app and inputs their favorite sport, the team and player they support, and their level of understanding of the rules. Specifically, they input information such as "Soccer," "Team X," "Player Y," and "Beginner" into the input fields on the screen.

[0827] Device: The data entered by the user is temporarily stored on the device and then sent to the server. JSON format data containing the input data (sport, team, player, and level of understanding) is generated and sent to the server.

[0828] Input: User input data (sport, team, player, understanding).

[0829] Output: User information data in JSON format sent to the server.

[0830] Step 2: Generate a user profile

[0831] Server: Analyzes the received user input data and generates a profile that reflects the user's preferences and level of understanding. Specifically, the input data is processed by the analysis module to generate a user profile object.

[0832] Server: Stores the generated user profile information in a database.

[0833] Input: User information data sent from the device (JSON format).

[0834] Output: A user profile object is saved to the database.

[0835] Step 3: Collect and analyze emotion data

[0836] Device: During the match, a camera and microphone are used to collect the user's facial expressions and tone of voice in real time. Specifically, the camera captures facial expression data and the microphone captures audio data.

[0837] Terminal: Sends collected data to the server in real time.

[0838] Server: Sends the received data to the emotion engine to analyze the user's emotional state. The emotion engine uses machine learning algorithms to analyze the data and generate emotion tags.

[0839] Server: Reflects the analysis results of the emotion engine into the user profile and saves it in the database.

[0840] Input: Facial expression and voice data collected by camera and microphone.

[0841] Output: An updated user profile reflecting their emotional state.

[0842] Step 4: Receiving and analyzing video data

[0843] Terminal: Receives multiple camera images in real time as soon as the match starts and streams them to the server. Specifically, the camera image data is sent using RTSP (Real Time Streaming Protocol).

[0844] Server: The server sends the received video data to the analysis module, where it performs object recognition and motion analysis, such as detecting and tagging play events such as shots, passes, and goals.

[0845] Server: Stores the analysis results in a database and updates the match progress data.

[0846] Input: Real-time video data from each camera.

[0847] Output: Match progress data tagged with play events.

[0848] Step 5: Generate and serve AI commentary

[0849] Server: Compares play data with the user's profile information and emotional state to generate customized commentary. For example, beginners can receive basic commentary such as "The goalkeeper has caught the ball."

[0850] Server: It also generates commentary based on the user's emotional state. For an excited user, it provides emotional commentary such as "A specific team has a chance to score a goal soon!"

[0851] Server: Also generates commentary including information related to the team and players you support.

[0852] Server: Sends the generated commentary audio data to the user's device.

[0853] Terminal: Plays back the received audio data, allowing users to enjoy the commentary in real time.

[0854] Input: User profile data, emotional state data, match progress data.

[0855] Output: Customized commentary audio data.

[0856] (Application example 2)

[0857] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0858] In modern sports viewing, it is difficult to provide real-time commentary tailored to the user's skill level and interests, and general commentary alone is not enough to provide a satisfying experience for some users. Furthermore, there are limited means to provide customized commentary that takes into account the user's emotional state. In particular, to enhance the viewing experience on smart devices, flexible responses that respond to the user's emotions are required.

[0859] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0860] In this invention, the server includes means for receiving user input data and setting the user's sports preferences and understanding level, means for receiving and analyzing video from multiple cameras in real time, means for generating a commentary customized according to the user's status based on the analysis results, means for providing the generated commentary to the user, means for recognizing the user's emotional state, means for adaptively customizing the commentary according to the emotional state, and means for providing the commentary through an augmented reality device, thereby enabling a more personalized sports viewing experience according to the user's skill level and emotional state.

[0861] "User input data" refers to data that users input into a sports viewing app, including information about their preferences and understanding of sports, and the teams and players they support.

[0862] "Means for recognizing emotional states" refers to a system or technology that collects and analyzes biometric data such as a user's facial expressions and tone of voice to determine the user's emotional state in real time.

[0863] "Means for receiving footage from multiple cameras and analyzing it in real time" refers to a technology that receives footage from multiple cameras taken from different perspectives and analyzes it in real time to understand the situation and play of the game.

[0864] A "means for generating customized commentary" is a system or technology that generates sports commentary optimized for an individual user based on the user's input data and emotional state.

[0865] The "means for providing the generated commentary to the user" is a technology for transmitting the generated customized commentary to the user's terminal and providing it as audio, text, or video.

[0866] "Augmented reality devices" are devices such as smart glasses and head-mounted displays that display images and information in real time, enhancing the user's viewing experience.

[0867] "User's level of sports understanding" indicates the extent to which a user understands the rules, tactics, and characteristics of players of a sport, and is classified into beginner, intermediate, and advanced levels.

[0868] "User status" refers to the user's overall profile information, such as their sports viewing preferences, understanding, and emotional state.

[0869] "Means for adaptively customizing live commentary" refers to a technology that adjusts the content and tone of live commentary in real time according to the user's emotional state and status.

[0870] The "means for setting preferences and level of understanding" refers to a system in which users input their preferences and level of understanding of sports into an application, and a user profile is generated based on that information.

[0871] This system recognizes and analyzes the user's input data and emotional state, and customizes live sports commentary using real-time video analysis. This system consists of three main components: a server, a device, and a user.

[0872] System configuration

[0873] Setting user status and emotions

[0874] The server receives a user's input data when the user opens a sports viewing app, including their favorite sport, the team or player they support, and their level of understanding of the rules. The device then collects the user's input data and sends it to the server. The server then analyzes the received user input data and generates a profile that reflects the user's preferences and level of understanding. The server then stores the generated user profile information in a database.

[0875] Emotional Engine Activation

[0876] The device collects the user's facial expressions and tone of voice in real time through a camera and microphone. This biometric data is sent to a server, where an emotion engine recognizes and analyzes the user's emotional state. The emotion engine's results are reflected in the user's profile, and the analysis results are stored in a database.

[0877] Real-time video analysis

[0878] As soon as the match begins, the device receives real-time video from multiple cameras and streams it to the server. The server then sends the video data to an analysis module for object recognition and motion analysis. This extracts play data (shots, passes, goals, etc.) and updates the status of each player and the match in real time.

[0879] AI commentary generation

[0880] The server compares the extracted play data with the user's profile information and emotional state. For example, it provides beginners with basic rule explanations, intermediate players with explanations of tactics and player characteristics, and advanced players with analysis using advanced statistics and past match data. It also adjusts the commentary according to the user's emotional state, focusing on uplifting commentary when excited and detailed commentary when relaxed. It also generates commentary that includes information specific to the team or player the user is supporting.

[0881] Providing customized commentary

[0882] The generated commentary audio data is sent from the server to the user's device, which then plays it back. Users can enjoy customized commentary in real time. This system enables a more personalized sports viewing experience that reflects the user's skill level and emotional state.

[0883] Hardware and software used

[0884] Hardware:

[0885] Camera (captures the user's facial expressions)

[0886] Microphone (captures the tone of your voice)

[0887] Augmented reality devices (smart glasses, head-mounted displays)

[0888] software:

[0889] opencv-python (real-time video analysis)

[0890] requests (communication with the server)

[0891] TensorFlow (machine learning model)

[0892] Flask and Django (API server construction)

[0893] Transformers (models for text generation)

[0894] Specific examples

[0895] Beginner users:

[0896] Prompt: Generate an explanatory sentence that explains the role of a goalkeeper for beginners.

[0897] Advanced users:

[0898] Prompt: For advanced players, please generate an explanatory statement based on the purpose of the 4-3-3 formation change and historical statistics.

[0899] In this way, the system of the present invention significantly enhances the viewing experience by providing a customized commentary in real time that is tailored to the user's preferences, level of understanding, and emotional state.

[0900] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0901] Step 1:

[0902] The user opens a sports viewing app and inputs their favorite sport, the team / player they support, and their level of understanding of the rules. The device receives this input data and sends it to the server. The input data includes their favorite sport (e.g., soccer), the team / player they support (e.g., Team A), and their level of understanding of the rules (e.g., beginner) in text format. The device sends this data to the server in JSON format.

[0903] Step 2:

[0904] The server analyzes the received input data and generates a profile that reflects the user's preferences and understanding. Specifically, it stores the input data in a database and applies algorithms (e.g., normalization and clustering of user information) to generate a profile. This profile information is stored in the database and used for further analysis. The generated user profile is saved as an output.

[0905] Step 3:

[0906] The device collects the user's facial expressions and tone of voice in real time through a camera and microphone. The collected data is temporarily stored on the device and periodically sent to a server. Specifically, facial expressions are captured using the OpenCV library, and audio data is collected from the microphone. Input data includes video frames and audio data.

[0907] Step 4:

[0908] The server sends the collected facial expression data and voice data to an emotion engine, which recognizes and analyzes the user's emotional state. This emotion engine uses machine learning models such as TensorFlow to analyze the emotional state (e.g., joy, excitement, sadness, etc.) using a pre-trained emotion classification model. The server reflects the analysis results in the user profile and stores them in a database. The input data is facial expression data and voice data, and the output is a classification result of the emotional state.

[0909] Step 5:

[0910] As soon as the match starts, the device receives multiple camera images in real time and streams them to the server. The camera images are transmitted using a streaming protocol (e.g., RTSP). The input data is the camera image data, and the output is the streaming data transmitted to the server in real time.

[0911] Step 6:

[0912] The server sends the received video data to an analysis module for object recognition and motion analysis. This analysis module incorporates an image recognition algorithm (e.g., YOLO) and extracts play data (shots, passes, goals, etc.). The input data is video data, and the output is analyzed play data.

[0913] Step 7:

[0914] The server compares the extracted play data with the user's profile information and emotional state to generate a customized commentary. For example, it provides basic rule explanations for beginners, tactics and player characteristics for intermediate players, and advanced statistics and analysis using past match data for advanced players. The generated commentary is generated using a natural language processing library (e.g., Transformers). The input data are play data, user profile, and emotional state, and the output is a customized commentary text.

[0915] Step 8:

[0916] The generated commentary audio data is sent from the server to the user's device. The device plays the received audio data and provides the user with a customized commentary in real time. The input data is the customized commentary text, which is converted into audio data using speech synthesis technology (e.g., Text-to-Speech). The output is the audio commentary that is played on the user's device.

[0917] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0918] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0919] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0920] [Third embodiment]

[0921] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0922] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0923] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0924] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0925] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0926] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0927] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0928] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0929] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0930] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0931] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0932] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0933] This invention relates to a sports viewing system that generates and provides live commentary customized for each user using user input data and real-time video analysis. This system is mainly composed of three elements: a server, a terminal, and a user.

[0934] Program processing explanation

[0935] Program Overview

[0936] 1. User status settings

[0937] User: Opens a sports viewing app and enters their favorite sport, the team and player they support, and their level of understanding of the rules.

[0938] Terminal: Collects user input data and sends it to the server.

[0939] Server: Analyzes the user's input data, generates a user profile, and stores it in a database.

[0940] 2. Real-time video analysis

[0941] Terminal: Receives multiple camera images in real time and streams them to the server.

[0942] Server: Sends video data to the analysis module for object recognition and motion analysis. Extracts play data (shots, passes, goals, etc.) and updates each player and the game situation in real time.

[0943] 3. AI commentary voice generation

[0944] Server: Compares extracted play data with user profile information.

[0945] Server: Generates customized commentary for beginner, intermediate and advanced players, including information specific to the team or player you support.

[0946] 4. Providing customized commentary

[0947] Server: Sends the generated commentary audio data to the device.

[0948] Device: Plays audio data, allowing users to enjoy real-time, customized commentary.

[0949] Specific examples

[0950] (Example 1) Beginner user

[0951] User: A beginner who barely knows the rules of soccer.

[0952] Device: Select "Soccer" as your favorite sport and set your rule comprehension level to "Beginner."

[0953] Server: Three minutes into the game, the ball is in the hands of the goalkeeper.

[0954] Server: Generates the explanation "The goalkeeper has caught the ball. The goalkeeper is the only position that can catch the ball with their hands."

[0955] Device: Provide this description to the user aloud.

[0956] (Example 2) Intermediate user

[0957] User: An intermediate player who enjoys watching soccer. His favorite team is Barcelona.

[0958] Device: Select "Soccer" as your favorite sport, set your favorite team to "Barcelona," and set your rule comprehension level to "Intermediate."

[0959] Server: A Barcelona player takes the shot.

[0960] Server: Generates commentary like, "The Barcelona player took a powerful shot. This player is known for his forward movement and is particularly good at free kicks."

[0961] Device: Provide this description to the user aloud.

[0962] (Example 3) Advanced user

[0963] User: Experienced soccer player with a thorough understanding of tactics and player data.

[0964] Device: Select "Soccer" as your favorite sport and set your rule understanding level to "Advanced."

[0965] Sarver: Midway through the match, Barcelona changed to a 4-3-3 formation.

[0966] Server: Generates commentary: "Barcelona have now switched to a 4-3-3 formation. This tactical change is designed to strengthen their midfield press, and Messi in particular could be a game-changer from now on. His data from his last 15 games shows that he has a 2% higher goal success rate when using this formation."

[0967] Device: Provide this description to the user aloud.

[0968] The system of the present invention significantly improves the viewing experience by providing customizable live sports commentary tailored to the individual needs of users. This system is a new platform that allows for more enjoyment and a deeper understanding of sports viewing.

[0969] The processing flow will be explained below.

[0970] Step 1:

[0971] User: Opens a sports viewing app and enters their favorite sport, the team / player they support, and their level of understanding of the rules.

[0972] Step 2:

[0973] Terminal: Collects user input data and sends it to the server.

[0974] Step 3:

[0975] Server: Analyzes the received user input data and generates a profile that reflects the user's preferences, level of understanding, etc.

[0976] Step 4:

[0977] Server: Stores user profile information in a database.

[0978] Step 5:

[0979] Terminal: As soon as the match starts, it receives multiple camera images in real time and streams them to the server.

[0980] Step 6:

[0981] Server: Sends the received video data to the analysis module, where object recognition and motion analysis are performed.

[0982] Step 7:

[0983] Server: Based on the results of video analysis, it extracts play data (shots, passes, goals, etc.) and updates the status of each player and the match in real time.

[0984] Step 8:

[0985] Server: Compares extracted play data with user profile information.

[0986] Step 9:

[0987] Server: Generates customized explanations based on the user's level of understanding. For example, beginners can learn basic rules, intermediate players can learn tactics and player characteristics, and advanced players can learn advanced statistics and past match data.

[0988] Step 10:

[0989] Server: Generates commentary that includes information specific to the team or player you support.

[0990] Step 11:

[0991] Server: Sends the generated commentary audio data to the user's device.

[0992] Step 12:

[0993] Device: By playing back the received audio data, users can enjoy customized commentary in real time.

[0994] Step 13:

[0995] Users: Gain a deeper understanding and enjoyment of watching sports.

[0996] According to the above steps, the system of the present invention provides personalized sports commentary to enhance the viewing experience.

[0997] Example 1

[0998] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0999] Conventional sports viewing systems were unable to provide customized commentary tailored to the user's preferences and level of understanding, resulting in a uniform viewing experience that could not meet the needs of individual users. Furthermore, it was difficult to provide real-time video analysis or information specialized for specific teams or players, preventing users from gaining a deeper understanding or enjoyment.

[1000] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1001] In this invention, the server includes means for receiving user input data and setting the user's preferences and level of understanding of sports, means for receiving and analyzing images from multiple camera devices in real time, and means for generating audio commentary customized according to the user's status based on the analysis results, thereby enabling the provision of play-by-play commentary customized to meet the individual needs of each user.

[1002] "User input data" refers to information that users input when using a sports viewing app, such as their favorite sports, the teams and players they support, and their understanding of the rules.

[1003] The "multiple image capturing devices" refer to multiple cameras or other image capturing devices installed within the venue of a sporting event.

[1004] "Means for analyzing in real time" refers to a means for instantly analyzing the video data received from the filming device and recognizing the player's movements and type of play.

[1005] "Customized audio commentary according to user status" refers to audio data in which the content of the live commentary to be provided is adjusted according to the preferences and level of understanding set for each user.

[1006] A "means for providing to a user" is a means for delivering the generated audio description to a user device so that the user can listen to the description in real time.

[1007] "Means for storing and using user profile information" refers to the means for storing the information entered by the user and the results of analysis by the server in a database and using that information the next time the service is used.

[1008] "Means for generating explanations using an AI language model" refers to a means for generating explanations based on the analysis results using a language model based on machine learning.

[1009] "Means for converting a text-format commentary into an audio format" refers to means for converting the generated commentary into audio data using speech synthesis technology.

[1010] This invention relates to a sports viewing system that generates and provides live commentary customized for each user using user input data and real-time video analysis. This system is mainly composed of three elements: a server, a terminal, and a user.

[1011] The system's main hardware includes terminals for collecting user input information, camera equipment used in stadiums, etc., and a server for processing and analyzing the data. The software used includes an analysis module that analyzes input data and generates profiles, a video analysis module that analyzes video data in real time, and software that generates audio commentary using a generative AI model.

[1012] In operation, the user first accesses a sports viewing app using their device and inputs their favorite sport, favorite team, players, and level of understanding of the rules. The device collects this input data and sends it to the server. The server analyzes the received data and generates a profile for each user. For example, information such as favorite sport being "soccer," favorite team being "Barcelona," and rule understanding level being "intermediate" is saved as a profile.

[1013] When the match begins, the device receives real-time video feeds from multiple cameras installed in the stadium and streams the data to a server, where a video analysis module analyzes the video data to recognize player movements and types of plays, such as when a Barcelona player is about to take a shot.

[1014] The server then uses the analysis results to generate a customized commentary based on the user's profile information. Using a generative AI model, the server creates a commentary tailored to each user. For example, the server might generate a commentary such as, "A Barcelona player took a powerful shot. This player is known for his forward movement and particularly for his accuracy in free kicks."

[1015] The generated commentary in text format is converted into audio data by a speech synthesis module and sent from the server to the device, where it is played back, allowing the user to enjoy a customized commentary in real time.

[1016] For example, the following prompt sentences are used:

[1017] "Generate commentary when a Barcelona player takes a shot. Intermediate level commentary."

[1018] The system of this invention significantly improves the viewing experience by providing customizable live sports commentary tailored to the individual needs of users. This system is a new platform that allows for more enjoyment and a deeper understanding of sports viewing.

[1019] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1020] Step 1:

[1021] User status settings

[1022] User: Opens a sports viewing app and enters their favorite sport, the team and player they support, and their level of understanding of the rules.

[1023] Input: Favorite sport, team / player you support, level of understanding of the rules

[1024] Output: Input data

[1025] Terminal: Collects user input data and sends it to the server.

[1026] Input: Data entered by the user

[1027] Output: Data sent to the server

[1028] Server: Analyzes the received input data, generates a user profile, and stores it in a database.

[1029] Input: The input data received.

[1030] Data processing: Organizing user preferences and understanding into profiles

[1031] Output: User profile information (stored in database)

[1032] Specific working example:

[1033] The user selects "soccer" as their "favorite sport," sets their favorite team to "Barcelona," and sets their level of understanding of the rules to "intermediate." The device sends this information to the server, which then generates profile data and stores it in a database.

[1034] Step 2:

[1035] Real-time video analysis

[1036] Terminal: Receives images from multiple imaging devices in real time and streams the data to the server.

[1037] Input: Image data from imaging device

[1038] Output: Streaming data (to server)

[1039] Server: Sends video data to the analysis module for object recognition and motion analysis. Extracts play data (shots, passes, goals, etc.) and updates each player and the game situation in real time.

[1040] Input: Video data received in real time

[1041] Data processing: Extraction of play data through video analysis

[1042] Output: Updated play data

[1043] Specific working example:

[1044] At the start of the match, the device connects to the stadium's camera system to capture video in real time. This data is then sent to a server, where an analysis module detects when a Barcelona player is about to take a shot.

[1045] Step 3:

[1046] AI live commentary voice generation

[1047] Server: The extracted play data is compared with each user's profile information to generate customized commentary content.

[1048] Input: Play data, user profile information

[1049] Data manipulation: Generating customized commentary using generative AI models

[1050] Output: explanatory text

[1051] Server: Converts the generated textual commentary into audio.

[1052] Input: explanatory text

[1053] Data processing: Converting data into voice data using voice synthesis technology

[1054] Output: Audio data

[1055] Specific working example:

[1056] The system analyzes the situation when a Barcelona player takes a shot and generates a commentary aimed at intermediate users, such as, "A Barcelona player took a powerful shot. This player is known for his movement up front, and is particularly good at taking free kicks." This is then converted into audio data.

[1057] Step 4:

[1058] Providing customized commentary

[1059] Server: Sends the generated audio description data to the user's device.

[1060] Input: Audio data

[1061] Output: Data sent to the terminal

[1062] Device: Plays back the received audio data and provides the user with real-time, customized commentary.

[1063] Input: Received audio data

[1064] Output: The audio to be played

[1065] Specific working example:

[1066] The server generates audio data and sends it to the device of the intermediate user, who then instantly plays it back, allowing the user to enjoy customized commentary while watching the game.

[1067] (Application example 1)

[1068] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1069] In virtual stores, users have difficulty receiving customized information based on their preferences and level of understanding in real time. Furthermore, the lack of systems that provide voice guidance and advanced analytical information to understand product details limits the user experience.

[1070] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1071] In this invention, the server includes a means for acquiring user profile data, a means for capturing camera images in real time and performing preprocessing, and a means for analyzing the images using a deep learning model and interpreting the prediction results, thereby enabling the provision of customized product information and explanations based on the user's preferences and level of understanding in real time.

[1072] "User Input Data" refers to information provided by users, such as preferences and comprehension levels.

[1073] "Multiple camera footage" refers to real-time video data acquired from multiple cameras.

[1074] "Means for analyzing in real time" refers to means for processing and analyzing acquired video data in real time.

[1075] "User Status" refers to the current state or profile information of a User based on their preferences and understanding.

[1076] "Customized Commentary" refers to specific information or explanations generated based on a user's status.

[1077] "User profile data" refers to information about a user, such as certain attributes, preferences, and behavioral history.

[1078] The "means for pre-processing" refers to a means for performing processing to make the acquired video data easier to analyze.

[1079] A "deep learning model" refers to a machine learning model based on a large dataset and is used for video analysis.

[1080] "Means for interpreting prediction results" refers to the means for interpreting the output of a deep learning model and making specific judgments.

[1081] "Product data" refers to detailed information about each product in the virtual store.

[1082] "Voice data" refers to data that provides generated product information and explanations to users as voice information.

[1083] This invention relates to a system that provides customized product information and explanations based on the user's preferences and level of understanding within a virtual store. The system is mainly composed of three elements: a server, a terminal, and a user.

[1084] 1. Program Overview

[1085] Users access the virtual store using a terminal and input their purchase history and preferences. The server collects this data and analyzes the camera footage in real time to provide the user with the most appropriate information. A specific embodiment of this will be described below.

[1086] 2. Hardware and Software Used

[1087] Device: Smartphone, smart glasses, or head-mounted display.

[1088] Camera: A camera that captures video in real time.

[1089] Server: A server for processing and storing user data and video data.

[1090] software:

[1091] OpenCV: A library for capturing and preprocessing camera footage.

[1092] TensorFlow & Keras: Libraries for implementing deep learning models used in video analysis.

[1093] REST API: Interface for retrieving user profile data and product information.

[1094] gTTS (Google Text-to-Speech): A tool for converting customized product information into audio.

[1095] pygame: A library for playing generated audio data.

[1096] 3. Program flow and data processing

[1097] Retrieving a user profile

[1098] The server uses a REST API to retrieve user profile data, including user preferences and past purchase history. When a user accesses a virtual store, the system uses this data to provide optimal information.

[1099] Camera image capture and pre-processing

[1100] The device's camera captures video in real time, preprocessing is performed using OpenCV, and the video data is fed into a deep learning model using TensorFlow and Keras for real-time analysis.

[1101] Video analysis and interpretation of prediction results

[1102] The server analyzes the video data input in real time using a deep learning model, interprets the prediction results, and generates product information and explanations based on the user's status (preferences and level of understanding).

[1103] Product data acquisition and voice generation

[1104] Based on the analysis results, the server retrieves product data using the REST API. Next, it converts the product information generated using gTTS into voice data. Finally, it plays the voice data generated using pygame on the device and provides it to the user.

[1105] 4. Specific Examples

[1106] Example 1: While a user is browsing the fruit section, the camera recognizes an apple. The system retrieves the "detailed apple information" and provides a voice prompt such as "This product is an apple from Aomori Prefecture, priced at 100 yen each. It is currently on sale, with a set of three for 250 yen."

[1107] 5. Examples of prompts

[1108] When a user is browsing the fruit section of a virtual store, generate customized voice guidance based on product data recognized through camera video analysis.

[1109] Thus, the present invention provides a system for improving the user experience within a virtual store.

[1110] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1111] Step 1:

[1112] The server retrieves the user's profile data using a REST API. It receives the user ID as input and returns the profile data, including past purchase history and preferences, as output. Specifically, it sends the user ID to the API endpoint and receives the profile data in JSON format as a response.

[1113] Step 2:

[1114] The device uses a camera to capture video in real time. It receives video data from the camera as input and obtains the captured video data as output. The specific operation is to start the camera device and continuously acquire frames.

[1115] Step 3:

[1116] The device uses OpenCV to preprocess the captured video data. It receives video frames from the camera as input and converts them into an easily analyzable format as output. Specific operations include resizing and normalizing the video frames.

[1117] Step 4:

[1118] The device uses TensorFlow and Keras to analyze preprocessed video data in real time. It receives preprocessed video data as input and obtains prediction results as output. Specifically, it inputs the data into a deep learning model and makes predictions about product types and features.

[1119] Step 5:

[1120] The server interprets the prediction results and retrieves the product data. It receives the analysis results (such as the predicted product ID) as input and obtains detailed information about the product as output. Specifically, it uses the REST API again based on the product ID to retrieve the product data, including the product name, price, description, etc.

[1121] Step 6:

[1122] The server uses gTTS to convert the product description into voice data. It receives product data (in text format) as input and generates a voice file as output. The specific operation is to input text information into gTTS, generate and save the voice file.

[1123] Step 7:

[1124] The terminal plays audio data generated using pygame. It receives an audio file as input and provides audio information to the user as output. The specific operation is to read the audio file and play the audio through the speaker.

[1125] Step 8:

[1126] Users use their devices to receive customized product information and explanations in real time. While listening to the audio guidance, they can browse products in a virtual store. Specific actions include checking the audio information through the device and taking purchasing action as needed.

[1127] This allows users to receive real-time, individually customized product information within a virtual store.

[1128] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1129] This invention relates to a system that recognizes and analyzes user input data and emotional states, and customizes live sports commentary through real-time video analysis. The system consists of three main components: a server, a terminal, and a user.

[1130] Program processing explanation

[1131] Setting user status and emotions

[1132] 1. User: Opens a sports viewing app and enters their favorite sport, the team / player they support, and their level of understanding of the rules.

[1133] 2. Terminal: Collects user input data and sends it to the server.

[1134] 3. Server: Analyzes the received user input data and generates a profile that reflects the user's preferences and level of understanding.

[1135] 4. Server: Stores user profile information in a database.

[1136] Emotional Engine Activation

[1137] 1. Device: Collects the user's facial expressions and tone of voice in real time through a camera and microphone.

[1138] 2. Server: Sends the collected data to the emotion engine to recognize and analyze the user's emotional state.

[1139] 3. Server: Reflects the results of the emotion engine into the user profile and stores the analysis results in a database.

[1140] Real-time video analysis

[1141] 1. Terminal: As soon as the match starts, it receives images from multiple cameras in real time and streams them to the server.

[1142] 2. Server: Sends the received video data to the analysis module for object recognition and motion analysis.

[1143] 3. Server: Based on the results of video analysis, it extracts play data (shots, passes, goals, etc.) and updates the status of each player and the match in real time.

[1144] AI live commentary voice generation

[1145] 1. Server: Compares extracted play data with the user's profile information and emotional state.

[1146] 2. Server: Generates customized commentary based on the user's level of understanding and emotions. For example, it provides beginners with basic rule explanations, intermediate players with tactics and player characteristics, and advanced players with advanced statistics and analysis using past match data.

[1147] 3. Server: Depending on the user's emotional state, the server adjusts the commentary to elevate the user's emotions, such as providing more detailed commentary when the user is excited, or more detailed commentary when the user is relaxed.

[1148] 4. Server: Generates commentary that includes information specific to the team or player you support.

[1149] Providing customized commentary

[1150] 1. Server: Sends the generated commentary audio data to the user's device.

[1151] 2. Terminal: By playing back the received audio data, users can enjoy customized commentary in real time.

[1152] Specific examples

[1153] (Example 1) Beginner user

[1154] User: A beginner who barely knows the rules of soccer.

[1155] Device: Select "Soccer" as your favorite sport and set your rule comprehension level to "Beginner."

[1156] Server: Three minutes into the game, the ball is in the hands of the goalkeeper.

[1157] Server: Generates the explanation "The goalkeeper has caught the ball. The goalkeeper is the only position that can catch the ball with their hands."

[1158] Device: Provide this description to the user aloud.

[1159] (Example 2) Intermediate user

[1160] User: An intermediate player who enjoys watching soccer. His favorite team is a specific team.

[1161] Device: Select "Soccer" as your favorite sport, set your favorite team to a specific team, and set your rule comprehension level to "Intermediate."

[1162] Server: A player on a particular team takes the shot.

[1163] Server: Generates an explanation such as, "A player from a particular team took a powerful shot. This player is known for his forward movement and is particularly good at free kicks."

[1164] Device: Provide this description to the user aloud.

[1165] (Example 3) Advanced user

[1166] User: Experienced soccer player with a thorough understanding of tactics and player data.

[1167] Device: Select "Soccer" as your favorite sport and set your rule understanding level to "Advanced."

[1168] Server: Midway through the match, certain teams change to a 4-3-3 formation.

[1169] Server: "Currently, a certain team has switched to a 4-3-3 formation. This tactical change is to strengthen the midfield press, and a certain player could be a game changer from now on. His data from the last 15 games shows that he has a 2% higher goal success rate when using this formation."

[1170] Device: Provide this description to the user aloud.

[1171] (Example 4) User using emotion engine

[1172] User: Supports a particular team and gets excited during the game.

[1173] Device: The camera and microphone collect the user's facial expressions and tone of voice, which are then analyzed by the emotion engine.

[1174] Server: Detects when the user is in an excited state and generates a commentary that corresponds to that state.

[1175] Server: Generate commentary to further excite users, such as "A certain team has a chance to score! What a spectacular play!"

[1176] Device: Provide this description to the user aloud.

[1177] The system of the present invention significantly improves the viewing experience by customizing live sports commentary to suit the individual needs and emotions of each user. This system is a new platform that allows for more enjoyment and deeper understanding of sports viewing.

[1178] The processing flow will be explained below.

[1179] Step 1:

[1180] User: Launches a sports viewing app and enters their favorite sport (e.g., soccer), the team or player they support (e.g., Team A or Player X), and their level of understanding of the rules (e.g., beginner, intermediate, advanced).

[1181] Step 2:

[1182] Terminal: Collects data entered by the user and sends it to the server.

[1183] Step 3:

[1184] Server: Analyzes the received user input data and generates a user profile, which includes the user's preferences and understanding.

[1185] Step 4:

[1186] Server: The generated user profile information is saved in a database, which manages the individual information of each user.

[1187] Step 5:

[1188] Terminal: As soon as the match starts, it receives images from multiple cameras in real time and streams the data to the server.

[1189] Step 6:

[1190] Server: The received video data is sent to the analysis module, where object recognition and motion analysis are performed, thereby identifying the movements of players and the position of the ball.

[1191] Step 7:

[1192] Server: Based on the analysis results, it extracts play data (e.g., shots, passes, goals, etc.) and updates each player's data and the match situation in real time.

[1193] Step 8:

[1194] On the device: Cameras and microphones collect the user's facial expressions, tone of voice, and other emotion-related data in real time.

[1195] Step 9:

[1196] Terminal: Sends collected emotion-related data to the server.

[1197] Step 10:

[1198] Server: The emotion engine in the server recognizes and analyzes the user's emotional state based on their facial expressions, tone of voice, and other data.

[1199] Step 11:

[1200] Server: Updates the user's emotional state based on the user profile information and stores the information in a database.

[1201] Step 12:

[1202] Server: Generates customized commentary based on extracted play data, user profile information, and emotional state.

[1203] Step 13:

[1204] Server: Generates explanations using basic rules for beginners, tactics and player characteristics for intermediate players, and advanced statistics and past match data for advanced players.

[1205] Step 14:

[1206] Server: Generates commentary with highlights and exciting information from the match if the user is excited, and provides detailed information and expert commentary if the user is calm.

[1207] Step 15:

[1208] Server: Sends the generated commentary audio data to the user's device.

[1209] Step 16:

[1210] Device: Plays back the received audio data and provides the user with real-time, customized commentary.

[1211] Step 17:

[1212] Users: Enjoy watching sports more deeply and with personalized commentary.

[1213] According to the above steps, the system of the present invention provides personalized sports commentary and enhances the viewing experience through real-time adjustments according to the emotional state.

[1214] Example 2

[1215] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1216] Conventional sports viewing systems typically provide uniform live commentary without considering each user's individual needs or emotional state. This results in a failure to provide optimal commentary tailored to the user's level of understanding and preferences, resulting in a poor viewing experience. Furthermore, because commentary does not reflect the user's real-time emotional state during the game, it is not possible to fully evoke the user's excitement and satisfaction. This leads to a decline in the user's motivation to watch the game.

[1217] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving input data from a user and setting the user's preferences and understanding of sports, a means for receiving images from multiple camera devices and analyzing them in real time, a means for generating a customized commentary based on the analysis results in accordance with the user's status and emotional state, a means for collecting the user's facial expressions and tone of voice via a camera and microphone and recognizing and analyzing the emotional state, and a means for providing the generated commentary to the user. This makes it possible to provide a customized commentary that reflects the user's individual needs and emotional state in real time.

[1218] A "user" is an individual who inputs their favorite sports, the teams and players they support, and their understanding of the rules through a sports viewing app.

[1219] The "server" is a central processing unit that analyzes user input data, generates and stores user profiles, and uses emotion engines and video analysis modules to generate and provide customized live commentary.

[1220] A "terminal" is a device on which a sports viewing app used by a user is installed, and is a device that transmits user input data and real-time video, audio, and facial expression data to a server.

[1221] A "user profile" is a data set that reflects a user's sports preferences and understanding, the teams and players they support, and their real-time emotional state.

[1222] An "emotion engine" is a machine learning algorithm and software that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[1223] The "video analysis module" is an algorithm and software that analyzes video data received from multiple camera devices in real time and detects and tags play events such as shots, passes, and goals.

[1224] "Customized Commentary" means personalized sports commentary generated based on a user's profile information and real-time emotional state.

[1225] "Sports understanding" refers to the level of knowledge and understanding of the rules of a particular sport that a user has, and is classified into beginner, intermediate, and advanced levels.

[1226] "Filming equipment" refers to equipment such as a camera used to capture video of the match in real time and transmit it to a server via a terminal.

[1227] "Real-time analysis" is the process of instantly processing video data and user emotional data collected during a match to generate results.

[1228] This invention relates to a system that recognizes and analyzes user input data and emotional states, and customizes live sports commentary using real-time video analysis. This system consists of three main components: a server, a terminal, and a user.

[1229] Hardware and Software Configuration

[1230] Server: A high-performance data analysis server is used, which is installed with software that analyzes user input data, generates and saves profiles, an emotion engine, a video analysis module, and software that generates AI commentary voices.

[1231] Device: A device used by a user, such as a smartphone, tablet, or PC. The device is equipped with a camera and microphone and has a sports viewing app installed.

[1232] User: An individual who enjoys watching sports, inputs data and watches the game through the app.

[1233] User input data and status settings

[1234] A user launches a sports viewing app and inputs their favorite sport, the team and player they support, and their level of understanding. For example, they may select soccer, support Team X, and set their level of understanding as "Beginner." This data is collected by the device and sent to the server. The server analyzes the received data and generates a user profile. The generated profile information is stored in a database.

[1235] Emotional Engine Activation

[1236] During a match, the device's camera and microphone are used to collect the user's facial expressions and tone of voice in real time. This data is sent to a server and analyzed by an emotion engine. The server then reflects the results of this analysis in the user's profile and stores them in a database.

[1237] Real-time video analysis

[1238] When the match begins, the device receives real-time video feeds from multiple cameras and streams them to the server. The server then sends the video data to an analysis module for object recognition and motion analysis. Based on the analysis results, play data (e.g., shots, passes, goals) is extracted and the status of each player and the match is updated in real time.

[1239] AI live commentary voice generation

[1240] The server compares the play data with the user's profile information and emotional state to generate customized commentary. For example, beginners are provided with basic rule explanations, intermediate players with explanations of tactics and player characteristics, and advanced players with advanced statistical data and analysis of past matches. Also, depending on the user's emotional state, the server provides an uplifting commentary when excited, and a detailed commentary when relaxed. The server also generates commentary that includes information specific to the team or player the user is rooting for.

[1241] Providing customized commentary

[1242] The server transmits the generated live commentary audio data to the user's device, which then plays the audio data, allowing the user to enjoy the customized live commentary in real time.

[1243] Specific examples

[1244] (Example 1) Beginner user

[1245] User: A beginner who barely knows the rules of soccer.

[1246] Device: Select "Soccer" as your favorite sport and set your rule comprehension level to "Beginner."

[1247] Server: Three minutes into the game, the ball is in the hands of the goalkeeper.

[1248] Server: Generates the explanation "The goalkeeper has caught the ball. The goalkeeper is the only position that can catch the ball with their hands."

[1249] Device: Provide this description to the user aloud.

[1250] (Example 2) Intermediate user

[1251] User: An intermediate player who enjoys watching soccer. His favorite team is a specific team.

[1252] Device: Select "Soccer" as your favorite sport, set your favorite team to a specific team, and set your rule comprehension level to "Intermediate."

[1253] Server: A player on a particular team takes the shot.

[1254] Server: Generates an explanation such as, "A player from a particular team took a powerful shot. This player is known for his forward movement and is particularly good at free kicks."

[1255] Device: Provide this description to the user aloud.

[1256] (Example 3) Advanced user

[1257] User: Experienced soccer player with a thorough understanding of tactics and player data.

[1258] Device: Select "Soccer" as your favorite sport and set your rule understanding level to "Advanced."

[1259] Server: Midway through the match, certain teams change to a 4-3-3 formation.

[1260] Server: "Currently, a certain team has switched to a 4-3-3 formation. This tactical change is to strengthen the midfield press, and a certain player could be a game changer from now on. His data from the last 15 games shows that he has a 2% higher goal success rate when using this formation."

[1261] Device: Provide this description to the user aloud.

[1262] (Example 4) User using emotion engine

[1263] User: Supports a particular team and gets excited during the game.

[1264] Device: The camera and microphone collect the user's facial expressions and tone of voice, which are then analyzed by the emotion engine.

[1265] Server: Detects when the user is in an excited state and generates a commentary that corresponds to that state.

[1266] Server: Generate commentary to further excite users, such as "A certain team has a chance to score! What a spectacular play!"

[1267] Device: Provide this description to the user aloud.

[1268] The system dramatically improves the sports viewing experience by providing optimized commentary in real time based on the user's status and emotional state.

[1269] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1270] Step 1: Enter and submit user information

[1271] User: Launches a sports viewing app and inputs their favorite sport, the team and player they support, and their level of understanding of the rules. Specifically, they input information such as "Soccer," "Team X," "Player Y," and "Beginner" into the input fields on the screen.

[1272] Device: The data entered by the user is temporarily stored on the device and then sent to the server. JSON format data containing the input data (sport, team, player, and level of understanding) is generated and sent to the server.

[1273] Input: User input data (sport, team, player, understanding).

[1274] Output: User information data in JSON format sent to the server.

[1275] Step 2: Generate a user profile

[1276] Server: Analyzes the received user input data and generates a profile that reflects the user's preferences and level of understanding. Specifically, the input data is processed by the analysis module to generate a user profile object.

[1277] Server: Stores the generated user profile information in a database.

[1278] Input: User information data sent from the device (JSON format).

[1279] Output: A user profile object is saved to the database.

[1280] Step 3: Collect and analyze emotion data

[1281] Device: During the match, a camera and microphone are used to collect the user's facial expressions and tone of voice in real time. Specifically, the camera captures facial expression data and the microphone captures audio data.

[1282] Terminal: Sends collected data to the server in real time.

[1283] Server: Sends the received data to the emotion engine to analyze the user's emotional state. The emotion engine uses machine learning algorithms to analyze the data and generate emotion tags.

[1284] Server: Reflects the analysis results of the emotion engine into the user profile and saves it in the database.

[1285] Input: Facial expression and voice data collected by camera and microphone.

[1286] Output: An updated user profile reflecting their emotional state.

[1287] Step 4: Receiving and analyzing video data

[1288] Terminal: Receives multiple camera images in real time as soon as the match starts and streams them to the server. Specifically, the camera image data is sent using RTSP (Real Time Streaming Protocol).

[1289] Server: The server sends the received video data to the analysis module, where it performs object recognition and motion analysis, such as detecting and tagging play events such as shots, passes, and goals.

[1290] Server: Stores the analysis results in a database and updates the match progress data.

[1291] Input: Real-time video data from each camera.

[1292] Output: Match progress data tagged with play events.

[1293] Step 5: Generate and serve AI commentary

[1294] Server: Compares play data with the user's profile information and emotional state to generate customized commentary. For example, beginners can receive basic commentary such as "The goalkeeper has caught the ball."

[1295] Server: It also generates commentary based on the user's emotional state. For an excited user, it provides emotional commentary such as "A specific team has a chance to score a goal soon!"

[1296] Server: Also generates commentary including information related to the team and players you support.

[1297] Server: Sends the generated commentary audio data to the user's device.

[1298] Terminal: Plays back the received audio data, allowing users to enjoy the commentary in real time.

[1299] Input: User profile data, emotional state data, match progress data.

[1300] Output: Customized commentary audio data.

[1301] (Application example 2)

[1302] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1303] In modern sports viewing, it is difficult to provide real-time commentary tailored to the user's skill level and interests, and general commentary alone is not enough to provide a satisfying experience for some users. Furthermore, there are limited means to provide customized commentary that takes into account the user's emotional state. In particular, to enhance the viewing experience on smart devices, flexible responses that respond to the user's emotions are required.

[1304] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1305] In this invention, the server includes means for receiving user input data and setting the user's sports preferences and understanding level, means for receiving and analyzing video from multiple cameras in real time, means for generating a commentary customized according to the user's status based on the analysis results, means for providing the generated commentary to the user, means for recognizing the user's emotional state, means for adaptively customizing the commentary according to the emotional state, and means for providing the commentary through an augmented reality device, thereby enabling a more personalized sports viewing experience according to the user's skill level and emotional state.

[1306] "User input data" refers to data that users input into a sports viewing app, including information about their preferences and understanding of sports, and the teams and players they support.

[1307] "Means for recognizing emotional states" refers to a system or technology that collects and analyzes biometric data such as a user's facial expressions and tone of voice to determine the user's emotional state in real time.

[1308] "Means for receiving footage from multiple cameras and analyzing it in real time" refers to a technology that receives footage from multiple cameras taken from different perspectives and analyzes it in real time to understand the situation and play of the game.

[1309] A "means for generating customized commentary" is a system or technology that generates sports commentary optimized for an individual user based on the user's input data and emotional state.

[1310] The "means for providing the generated commentary to the user" is a technology for transmitting the generated customized commentary to the user's terminal and providing it as audio, text, or video.

[1311] "Augmented reality devices" are devices such as smart glasses and head-mounted displays that display images and information in real time, enhancing the user's viewing experience.

[1312] "User's level of sports understanding" indicates the extent to which a user understands the rules, tactics, and characteristics of players of a sport, and is classified into beginner, intermediate, and advanced levels.

[1313] "User status" refers to the user's overall profile information, such as their sports viewing preferences, understanding, and emotional state.

[1314] "Means for adaptively customizing live commentary" refers to a technology that adjusts the content and tone of live commentary in real time according to the user's emotional state and status.

[1315] The "means for setting preferences and level of understanding" refers to a system in which users input their preferences and level of understanding of sports into an application, and a user profile is generated based on that information.

[1316] This system recognizes and analyzes the user's input data and emotional state, and customizes live sports commentary using real-time video analysis. This system consists of three main components: a server, a device, and a user.

[1317] System configuration

[1318] Setting user status and emotions

[1319] The server receives a user's input data when the user opens a sports viewing app, including their favorite sport, the team or player they support, and their level of understanding of the rules. The device then collects the user's input data and sends it to the server. The server then analyzes the received user input data and generates a profile that reflects the user's preferences and level of understanding. The server then stores the generated user profile information in a database.

[1320] Emotional Engine Activation

[1321] The device collects the user's facial expressions and tone of voice in real time through a camera and microphone. This biometric data is sent to a server, where an emotion engine recognizes and analyzes the user's emotional state. The emotion engine's results are reflected in the user's profile, and the analysis results are stored in a database.

[1322] Real-time video analysis

[1323] As soon as the match begins, the device receives real-time video from multiple cameras and streams it to the server. The server then sends the video data to an analysis module for object recognition and motion analysis. This extracts play data (shots, passes, goals, etc.) and updates the status of each player and the match in real time.

[1324] AI commentary generation

[1325] The server compares the extracted play data with the user's profile information and emotional state. For example, it provides beginners with basic rule explanations, intermediate players with explanations of tactics and player characteristics, and advanced players with analysis using advanced statistics and past match data. It also adjusts the commentary according to the user's emotional state, focusing on uplifting commentary when excited and detailed commentary when relaxed. It also generates commentary that includes information specific to the team or player the user is supporting.

[1326] Providing customized commentary

[1327] The generated commentary audio data is sent from the server to the user's device, which then plays it back. Users can enjoy customized commentary in real time. This system enables a more personalized sports viewing experience that reflects the user's skill level and emotional state.

[1328] Hardware and software used

[1329] Hardware:

[1330] Camera (captures the user's facial expressions)

[1331] Microphone (captures the tone of your voice)

[1332] Augmented reality devices (smart glasses, head-mounted displays)

[1333] software:

[1334] opencv-python (real-time video analysis)

[1335] requests (communication with the server)

[1336] TensorFlow (machine learning model)

[1337] Flask and Django (API server construction)

[1338] Transformers (models for text generation)

[1339] Specific examples

[1340] Beginner users:

[1341] Prompt: Generate an explanatory sentence that explains the role of a goalkeeper for beginners.

[1342] Advanced users:

[1343] Prompt: For advanced players, please generate an explanatory statement based on the purpose of the 4-3-3 formation change and historical statistics.

[1344] In this way, the system of the present invention significantly enhances the viewing experience by providing a customized commentary in real time that is tailored to the user's preferences, level of understanding, and emotional state.

[1345] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1346] Step 1:

[1347] The user opens a sports viewing app and inputs their favorite sport, the team / player they support, and their level of understanding of the rules. The device receives this input data and sends it to the server. The input data includes their favorite sport (e.g., soccer), the team / player they support (e.g., Team A), and their level of understanding of the rules (e.g., beginner) in text format. The device sends this data to the server in JSON format.

[1348] Step 2:

[1349] The server analyzes the received input data and generates a profile that reflects the user's preferences and understanding. Specifically, it stores the input data in a database and applies algorithms (e.g., normalization and clustering of user information) to generate a profile. This profile information is stored in the database and used for further analysis. The generated user profile is saved as an output.

[1350] Step 3:

[1351] The device collects the user's facial expressions and tone of voice in real time through a camera and microphone. The collected data is temporarily stored on the device and periodically sent to a server. Specifically, facial expressions are captured using the OpenCV library, and audio data is collected from the microphone. Input data includes video frames and audio data.

[1352] Step 4:

[1353] The server sends the collected facial expression data and voice data to an emotion engine, which recognizes and analyzes the user's emotional state. This emotion engine uses machine learning models such as TensorFlow to analyze the emotional state (e.g., joy, excitement, sadness, etc.) using a pre-trained emotion classification model. The server reflects the analysis results in the user profile and stores them in a database. The input data is facial expression data and voice data, and the output is a classification result of the emotional state.

[1354] Step 5:

[1355] As soon as the match starts, the device receives multiple camera images in real time and streams them to the server. The camera images are transmitted using a streaming protocol (e.g., RTSP). The input data is the camera image data, and the output is the streaming data transmitted to the server in real time.

[1356] Step 6:

[1357] The server sends the received video data to an analysis module for object recognition and motion analysis. This analysis module incorporates an image recognition algorithm (e.g., YOLO) and extracts play data (shots, passes, goals, etc.). The input data is video data, and the output is analyzed play data.

[1358] Step 7:

[1359] The server compares the extracted play data with the user's profile information and emotional state to generate a customized commentary. For example, it provides basic rule explanations for beginners, tactics and player characteristics for intermediate players, and advanced statistics and analysis using past match data for advanced players. The generated commentary is generated using a natural language processing library (e.g., Transformers). The input data are play data, user profile, and emotional state, and the output is a customized commentary text.

[1360] Step 8:

[1361] The generated commentary audio data is sent from the server to the user's device. The device plays the received audio data and provides the user with a customized commentary in real time. The input data is the customized commentary text, which is converted into audio data using speech synthesis technology (e.g., Text-to-Speech). The output is the audio commentary that is played on the user's device.

[1362] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1363] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1364] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1365] [Fourth embodiment]

[1366] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1367] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1368] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1369] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1370] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1371] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1372] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1373] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1374] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1375] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1376] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1377] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1378] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1379] This invention relates to a sports viewing system that generates and provides live commentary customized for each user using user input data and real-time video analysis. This system is mainly composed of three elements: a server, a terminal, and a user.

[1380] Program processing explanation

[1381] Program Overview

[1382] 1. User status settings

[1383] User: Opens a sports viewing app and enters their favorite sport, the team and player they support, and their level of understanding of the rules.

[1384] Terminal: Collects user input data and sends it to the server.

[1385] Server: Analyzes the user's input data, generates a user profile, and stores it in a database.

[1386] 2. Real-time video analysis

[1387] Terminal: Receives multiple camera images in real time and streams them to the server.

[1388] Server: Sends video data to the analysis module for object recognition and motion analysis. Extracts play data (shots, passes, goals, etc.) and updates each player and the game situation in real time.

[1389] 3. AI commentary voice generation

[1390] Server: Compares extracted play data with user profile information.

[1391] Server: Generates customized commentary for beginner, intermediate and advanced players, including information specific to the team or player you support.

[1392] 4. Providing customized commentary

[1393] Server: Sends the generated commentary audio data to the device.

[1394] Device: Plays audio data, allowing users to enjoy real-time, customized commentary.

[1395] Specific examples

[1396] (Example 1) Beginner user

[1397] User: A beginner who barely knows the rules of soccer.

[1398] Device: Select "Soccer" as your favorite sport and set your rule comprehension level to "Beginner."

[1399] Server: Three minutes into the game, the ball is in the hands of the goalkeeper.

[1400] Server: Generates the explanation "The goalkeeper has caught the ball. The goalkeeper is the only position that can catch the ball with their hands."

[1401] Device: Provide this description to the user aloud.

[1402] (Example 2) Intermediate user

[1403] User: An intermediate player who enjoys watching soccer. His favorite team is Barcelona.

[1404] Device: Select "Soccer" as your favorite sport, set your favorite team to "Barcelona," and set your rule comprehension level to "Intermediate."

[1405] Server: A Barcelona player takes the shot.

[1406] Server: Generates commentary like, "The Barcelona player took a powerful shot. This player is known for his forward movement and is particularly good at free kicks."

[1407] Device: Provide this description to the user aloud.

[1408] (Example 3) Advanced user

[1409] User: Experienced soccer player with a thorough understanding of tactics and player data.

[1410] Device: Select "Soccer" as your favorite sport and set your rule understanding level to "Advanced."

[1411] Sarver: Midway through the match, Barcelona changed to a 4-3-3 formation.

[1412] Server: Generates commentary: "Barcelona have now switched to a 4-3-3 formation. This tactical change is designed to strengthen their midfield press, and Messi in particular could be a game-changer from now on. His data from his last 15 games shows that he has a 2% higher goal success rate when using this formation."

[1413] Device: Provide this description to the user aloud.

[1414] The system of the present invention significantly improves the viewing experience by providing customizable live sports commentary tailored to the individual needs of users. This system is a new platform that allows for more enjoyment and a deeper understanding of sports viewing.

[1415] The processing flow will be explained below.

[1416] Step 1:

[1417] User: Opens a sports viewing app and enters their favorite sport, the team / player they support, and their level of understanding of the rules.

[1418] Step 2:

[1419] Terminal: Collects user input data and sends it to the server.

[1420] Step 3:

[1421] Server: Analyzes the received user input data and generates a profile that reflects the user's preferences, level of understanding, etc.

[1422] Step 4:

[1423] Server: Stores user profile information in a database.

[1424] Step 5:

[1425] Terminal: As soon as the match starts, it receives multiple camera images in real time and streams them to the server.

[1426] Step 6:

[1427] Server: Sends the received video data to the analysis module, where object recognition and motion analysis are performed.

[1428] Step 7:

[1429] Server: Based on the results of video analysis, it extracts play data (shots, passes, goals, etc.) and updates the status of each player and the match in real time.

[1430] Step 8:

[1431] Server: Compares extracted play data with user profile information.

[1432] Step 9:

[1433] Server: Generates customized explanations based on the user's level of understanding. For example, beginners can learn basic rules, intermediate players can learn tactics and player characteristics, and advanced players can learn advanced statistics and past match data.

[1434] Step 10:

[1435] Server: Generates commentary that includes information specific to the team or player you support.

[1436] Step 11:

[1437] Server: Sends the generated commentary audio data to the user's device.

[1438] Step 12:

[1439] Device: By playing back the received audio data, users can enjoy customized commentary in real time.

[1440] Step 13:

[1441] Users: Gain a deeper understanding and enjoyment of watching sports.

[1442] According to the above steps, the system of the present invention provides personalized sports commentary to enhance the viewing experience.

[1443] Example 1

[1444] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1445] Conventional sports viewing systems were unable to provide customized commentary tailored to the user's preferences and level of understanding, resulting in a uniform viewing experience that could not meet the needs of individual users. Furthermore, it was difficult to provide real-time video analysis or information specialized for specific teams or players, preventing users from gaining a deeper understanding or enjoyment.

[1446] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1447] In this invention, the server includes means for receiving user input data and setting the user's preferences and level of understanding of sports, means for receiving and analyzing images from multiple camera devices in real time, and means for generating audio commentary customized according to the user's status based on the analysis results, thereby enabling the provision of play-by-play commentary customized to meet the individual needs of each user.

[1448] "User input data" refers to information that users input when using a sports viewing app, such as their favorite sports, the teams and players they support, and their understanding of the rules.

[1449] The "multiple image capturing devices" refer to multiple cameras or other image capturing devices installed within the venue of a sporting event.

[1450] "Means for analyzing in real time" refers to a means for instantly analyzing the video data received from the filming device and recognizing the player's movements and type of play.

[1451] "Customized audio commentary according to user status" refers to audio data in which the content of the live commentary to be provided is adjusted according to the preferences and level of understanding set for each user.

[1452] A "means for providing to a user" is a means for delivering the generated audio description to a user device so that the user can listen to the description in real time.

[1453] "Means for storing and using user profile information" refers to the means for storing the information entered by the user and the results of analysis by the server in a database and using that information the next time the service is used.

[1454] "Means for generating explanations using an AI language model" refers to a means for generating explanations based on the analysis results using a language model based on machine learning.

[1455] "Means for converting a text-format commentary into an audio format" refers to means for converting the generated commentary into audio data using speech synthesis technology.

[1456] This invention relates to a sports viewing system that generates and provides live commentary customized for each user using user input data and real-time video analysis. This system is mainly composed of three elements: a server, a terminal, and a user.

[1457] The system's main hardware includes terminals for collecting user input information, camera equipment used in stadiums, etc., and a server for processing and analyzing the data. The software used includes an analysis module that analyzes input data and generates profiles, a video analysis module that analyzes video data in real time, and software that generates audio commentary using a generative AI model.

[1458] In operation, the user first accesses a sports viewing app using their device and inputs their favorite sport, favorite team, players, and level of understanding of the rules. The device collects this input data and sends it to the server. The server analyzes the received data and generates a profile for each user. For example, information such as favorite sport being "soccer," favorite team being "Barcelona," and rule understanding level being "intermediate" is saved as a profile.

[1459] When the match begins, the device receives real-time video feeds from multiple cameras installed in the stadium and streams the data to a server, where a video analysis module analyzes the video data to recognize player movements and types of plays, such as when a Barcelona player is about to take a shot.

[1460] The server then uses the analysis results to generate a customized commentary based on the user's profile information. Using a generative AI model, the server creates a commentary tailored to each user. For example, the server might generate a commentary such as, "A Barcelona player took a powerful shot. This player is known for his forward movement and particularly for his accuracy in free kicks."

[1461] The generated commentary in text format is converted into audio data by a speech synthesis module and sent from the server to the device, where it is played back, allowing the user to enjoy a customized commentary in real time.

[1462] For example, the following prompt sentences are used:

[1463] "Generate commentary when a Barcelona player takes a shot. Intermediate level commentary."

[1464] The system of this invention significantly improves the viewing experience by providing customizable live sports commentary tailored to the individual needs of users. This system is a new platform that allows for more enjoyment and a deeper understanding of sports viewing.

[1465] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1466] Step 1:

[1467] User status settings

[1468] User: Opens a sports viewing app and enters their favorite sport, the team and player they support, and their level of understanding of the rules.

[1469] Input: Favorite sport, team / player you support, level of understanding of the rules

[1470] Output: Input data

[1471] Terminal: Collects user input data and sends it to the server.

[1472] Input: Data entered by the user

[1473] Output: Data sent to the server

[1474] Server: Analyzes the received input data, generates a user profile, and stores it in a database.

[1475] Input: The input data received.

[1476] Data processing: Organizing user preferences and understanding into profiles

[1477] Output: User profile information (stored in database)

[1478] Specific working example:

[1479] The user selects "soccer" as their "favorite sport," sets their favorite team to "Barcelona," and sets their level of understanding of the rules to "intermediate." The device sends this information to the server, which then generates profile data and stores it in a database.

[1480] Step 2:

[1481] Real-time video analysis

[1482] Terminal: Receives images from multiple imaging devices in real time and streams the data to the server.

[1483] Input: Image data from imaging device

[1484] Output: Streaming data (to server)

[1485] Server: Sends video data to the analysis module for object recognition and motion analysis. Extracts play data (shots, passes, goals, etc.) and updates each player and the game situation in real time.

[1486] Input: Video data received in real time

[1487] Data processing: Extraction of play data through video analysis

[1488] Output: Updated play data

[1489] Specific working example:

[1490] At the start of the match, the device connects to the stadium's camera system to capture video in real time. This data is then sent to a server, where an analysis module detects when a Barcelona player is about to take a shot.

[1491] Step 3:

[1492] AI live commentary voice generation

[1493] Server: The extracted play data is compared with each user's profile information to generate customized commentary content.

[1494] Input: Play data, user profile information

[1495] Data manipulation: Generating customized commentary using generative AI models

[1496] Output: explanatory text

[1497] Server: Converts the generated textual commentary into audio.

[1498] Input: explanatory text

[1499] Data processing: Converting data into voice data using voice synthesis technology

[1500] Output: Audio data

[1501] Specific working example:

[1502] The system analyzes the situation when a Barcelona player takes a shot and generates a commentary aimed at intermediate users, such as, "A Barcelona player took a powerful shot. This player is known for his movement up front, and is particularly good at taking free kicks." This is then converted into audio data.

[1503] Step 4:

[1504] Providing customized commentary

[1505] Server: Sends the generated audio description data to the user's device.

[1506] Input: Audio data

[1507] Output: Data sent to the terminal

[1508] Device: Plays back the received audio data and provides the user with real-time, customized commentary.

[1509] Input: Received audio data

[1510] Output: The audio to be played

[1511] Specific working example:

[1512] The server generates audio data and sends it to the device of the intermediate user, who then instantly plays it back, allowing the user to enjoy customized commentary while watching the game.

[1513] (Application example 1)

[1514] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1515] In virtual stores, users have difficulty receiving customized information based on their preferences and level of understanding in real time. Furthermore, the lack of systems that provide voice guidance and advanced analytical information to understand product details limits the user experience.

[1516] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1517] In this invention, the server includes a means for acquiring user profile data, a means for capturing camera images in real time and performing preprocessing, and a means for analyzing the images using a deep learning model and interpreting the prediction results, thereby enabling the provision of customized product information and explanations based on the user's preferences and level of understanding in real time.

[1518] "User Input Data" refers to information provided by users, such as preferences and comprehension levels.

[1519] "Multiple camera footage" refers to real-time video data acquired from multiple cameras.

[1520] "Means for analyzing in real time" refers to means for processing and analyzing acquired video data in real time.

[1521] "User Status" refers to the current state or profile information of a User based on their preferences and understanding.

[1522] "Customized Commentary" refers to specific information or explanations generated based on a user's status.

[1523] "User profile data" refers to information about a user, such as certain attributes, preferences, and behavioral history.

[1524] The "means for pre-processing" refers to a means for performing processing to make the acquired video data easier to analyze.

[1525] A "deep learning model" refers to a machine learning model based on a large dataset and is used for video analysis.

[1526] "Means for interpreting prediction results" refers to the means for interpreting the output of a deep learning model and making specific judgments.

[1527] "Product data" refers to detailed information about each product in the virtual store.

[1528] "Voice data" refers to data that provides generated product information and explanations to users as voice information.

[1529] This invention relates to a system that provides customized product information and explanations based on the user's preferences and level of understanding within a virtual store. The system is mainly composed of three elements: a server, a terminal, and a user.

[1530] 1. Program Overview

[1531] Users access the virtual store using a terminal and input their purchase history and preferences. The server collects this data and analyzes the camera footage in real time to provide the user with the most appropriate information. A specific embodiment of this will be described below.

[1532] 2. Hardware and Software Used

[1533] Device: Smartphone, smart glasses, or head-mounted display.

[1534] Camera: A camera that captures video in real time.

[1535] Server: A server for processing and storing user data and video data.

[1536] software:

[1537] OpenCV: A library for capturing and preprocessing camera footage.

[1538] TensorFlow & Keras: Libraries for implementing deep learning models used in video analysis.

[1539] REST API: Interface for retrieving user profile data and product information.

[1540] gTTS (Google Text-to-Speech): A tool for converting customized product information into audio.

[1541] pygame: A library for playing generated audio data.

[1542] 3. Program flow and data processing

[1543] Retrieving a user profile

[1544] The server uses a REST API to retrieve user profile data, including user preferences and past purchase history. When a user accesses a virtual store, the system uses this data to provide optimal information.

[1545] Camera image capture and pre-processing

[1546] The device's camera captures video in real time, preprocessing is performed using OpenCV, and the video data is fed into a deep learning model using TensorFlow and Keras for real-time analysis.

[1547] Video analysis and interpretation of prediction results

[1548] The server analyzes the video data input in real time using a deep learning model, interprets the prediction results, and generates product information and explanations based on the user's status (preferences and level of understanding).

[1549] Product data acquisition and voice generation

[1550] Based on the analysis results, the server retrieves product data using the REST API. Next, it converts the product information generated using gTTS into voice data. Finally, it plays the voice data generated using pygame on the device and provides it to the user.

[1551] 4. Specific Examples

[1552] Example 1: While a user is browsing the fruit section, the camera recognizes an apple. The system retrieves the "detailed apple information" and provides a voice prompt such as "This product is an apple from Aomori Prefecture, priced at 100 yen each. It is currently on sale, with a set of three for 250 yen."

[1553] 5. Examples of prompts

[1554] When a user is browsing the fruit section of a virtual store, generate customized voice guidance based on product data recognized through camera video analysis.

[1555] Thus, the present invention provides a system for improving the user experience within a virtual store.

[1556] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1557] Step 1:

[1558] The server retrieves the user's profile data using a REST API. It receives the user ID as input and returns the profile data, including past purchase history and preferences, as output. Specifically, it sends the user ID to the API endpoint and receives the profile data in JSON format as a response.

[1559] Step 2:

[1560] The device uses a camera to capture video in real time. It receives video data from the camera as input and obtains the captured video data as output. The specific operation is to start the camera device and continuously acquire frames.

[1561] Step 3:

[1562] The device uses OpenCV to preprocess the captured video data. It receives video frames from the camera as input and converts them into an easily analyzable format as output. Specific operations include resizing and normalizing the video frames.

[1563] Step 4:

[1564] The device uses TensorFlow and Keras to analyze preprocessed video data in real time. It receives preprocessed video data as input and obtains prediction results as output. Specifically, it inputs the data into a deep learning model and makes predictions about product types and features.

[1565] Step 5:

[1566] The server interprets the prediction results and retrieves the product data. It receives the analysis results (such as the predicted product ID) as input and obtains detailed information about the product as output. Specifically, it uses the REST API again based on the product ID to retrieve the product data, including the product name, price, description, etc.

[1567] Step 6:

[1568] The server uses gTTS to convert the product description into voice data. It receives product data (in text format) as input and generates a voice file as output. The specific operation is to input text information into gTTS, generate and save the voice file.

[1569] Step 7:

[1570] The terminal plays audio data generated using pygame. It receives an audio file as input and provides audio information to the user as output. The specific operation is to read the audio file and play the audio through the speaker.

[1571] Step 8:

[1572] Users use their devices to receive customized product information and explanations in real time. While listening to the audio guidance, they can browse products in a virtual store. Specific actions include checking the audio information through the device and taking purchasing action as needed.

[1573] This allows users to receive real-time, individually customized product information within a virtual store.

[1574] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1575] This invention relates to a system that recognizes and analyzes user input data and emotional states, and customizes live sports commentary through real-time video analysis. The system consists of three main components: a server, a terminal, and a user.

[1576] Program processing explanation

[1577] Setting user status and emotions

[1578] 1. User: Opens a sports viewing app and enters their favorite sport, the team / player they support, and their level of understanding of the rules.

[1579] 2. Terminal: Collects user input data and sends it to the server.

[1580] 3. Server: Analyzes the received user input data and generates a profile that reflects the user's preferences and level of understanding.

[1581] 4. Server: Stores user profile information in a database.

[1582] Emotional Engine Activation

[1583] 1. Device: Collects the user's facial expressions and tone of voice in real time through a camera and microphone.

[1584] 2. Server: Sends the collected data to the emotion engine to recognize and analyze the user's emotional state.

[1585] 3. Server: Reflects the results of the emotion engine into the user profile and stores the analysis results in a database.

[1586] Real-time video analysis

[1587] 1. Terminal: As soon as the match starts, it receives images from multiple cameras in real time and streams them to the server.

[1588] 2. Server: Sends the received video data to the analysis module for object recognition and motion analysis.

[1589] 3. Server: Based on the results of video analysis, it extracts play data (shots, passes, goals, etc.) and updates the status of each player and the match in real time.

[1590] AI live commentary voice generation

[1591] 1. Server: Compares extracted play data with the user's profile information and emotional state.

[1592] 2. Server: Generates customized commentary based on the user's level of understanding and emotions. For example, it provides beginners with basic rule explanations, intermediate players with tactics and player characteristics, and advanced players with advanced statistics and analysis using past match data.

[1593] 3. Server: Depending on the user's emotional state, the server adjusts the commentary to elevate the user's emotions, such as providing more detailed commentary when the user is excited, or more detailed commentary when the user is relaxed.

[1594] 4. Server: Generates commentary that includes information specific to the team or player you support.

[1595] Providing customized commentary

[1596] 1. Server: Sends the generated commentary audio data to the user's device.

[1597] 2. Terminal: By playing back the received audio data, users can enjoy customized commentary in real time.

[1598] Specific examples

[1599] (Example 1) Beginner user

[1600] User: A beginner who barely knows the rules of soccer.

[1601] Device: Select "Soccer" as your favorite sport and set your rule comprehension level to "Beginner."

[1602] Server: Three minutes into the game, the ball is in the hands of the goalkeeper.

[1603] Server: Generates the explanation "The goalkeeper has caught the ball. The goalkeeper is the only position that can catch the ball with their hands."

[1604] Device: Provide this description to the user aloud.

[1605] (Example 2) Intermediate user

[1606] User: An intermediate player who enjoys watching soccer. His favorite team is a specific team.

[1607] Device: Select "Soccer" as your favorite sport, set your favorite team to a specific team, and set your rule comprehension level to "Intermediate."

[1608] Server: A player on a particular team takes the shot.

[1609] Server: Generates an explanation such as, "A player from a particular team took a powerful shot. This player is known for his forward movement and is particularly good at free kicks."

[1610] Device: Provide this description to the user aloud.

[1611] (Example 3) Advanced user

[1612] User: Experienced soccer player with a thorough understanding of tactics and player data.

[1613] Device: Select "Soccer" as your favorite sport and set your rule understanding level to "Advanced."

[1614] Server: Midway through the match, certain teams change to a 4-3-3 formation.

[1615] Server: "Currently, a certain team has switched to a 4-3-3 formation. This tactical change is to strengthen the midfield press, and a certain player could be a game changer from now on. His data from the last 15 games shows that he has a 2% higher goal success rate when using this formation."

[1616] Device: Provide this description to the user aloud.

[1617] (Example 4) User using emotion engine

[1618] User: Supports a particular team and gets excited during the game.

[1619] Device: The camera and microphone collect the user's facial expressions and tone of voice, which are then analyzed by the emotion engine.

[1620] Server: Detects when the user is in an excited state and generates a commentary that corresponds to that state.

[1621] Server: Generate commentary to further excite users, such as "A certain team has a chance to score! What a spectacular play!"

[1622] Device: Provide this description to the user aloud.

[1623] The system of the present invention significantly improves the viewing experience by customizing live sports commentary to suit the individual needs and emotions of each user. This system is a new platform that allows for more enjoyment and deeper understanding of sports viewing.

[1624] The processing flow will be explained below.

[1625] Step 1:

[1626] User: Launches a sports viewing app and enters their favorite sport (e.g., soccer), the team or player they support (e.g., Team A or Player X), and their level of understanding of the rules (e.g., beginner, intermediate, advanced).

[1627] Step 2:

[1628] Terminal: Collects data entered by the user and sends it to the server.

[1629] Step 3:

[1630] Server: Analyzes the received user input data and generates a user profile, which includes the user's preferences and understanding.

[1631] Step 4:

[1632] Server: The generated user profile information is saved in a database, which manages the individual information of each user.

[1633] Step 5:

[1634] Terminal: As soon as the match starts, it receives images from multiple cameras in real time and streams the data to the server.

[1635] Step 6:

[1636] Server: The received video data is sent to the analysis module, where object recognition and motion analysis are performed, thereby identifying the movements of players and the position of the ball.

[1637] Step 7:

[1638] Server: Based on the analysis results, it extracts play data (e.g., shots, passes, goals, etc.) and updates each player's data and the match situation in real time.

[1639] Step 8:

[1640] On the device: Cameras and microphones collect the user's facial expressions, tone of voice, and other emotion-related data in real time.

[1641] Step 9:

[1642] Terminal: Sends collected emotion-related data to the server.

[1643] Step 10:

[1644] Server: The emotion engine in the server recognizes and analyzes the user's emotional state based on their facial expressions, tone of voice, and other data.

[1645] Step 11:

[1646] Server: Updates the user's emotional state based on the user profile information and stores the information in a database.

[1647] Step 12:

[1648] Server: Generates customized commentary based on extracted play data, user profile information, and emotional state.

[1649] Step 13:

[1650] Server: Generates explanations using basic rules for beginners, tactics and player characteristics for intermediate players, and advanced statistics and past match data for advanced players.

[1651] Step 14:

[1652] Server: Generates commentary with highlights and exciting information from the match if the user is excited, and provides detailed information and expert commentary if the user is calm.

[1653] Step 15:

[1654] Server: Sends the generated commentary audio data to the user's device.

[1655] Step 16:

[1656] Device: Plays back the received audio data and provides the user with real-time, customized commentary.

[1657] Step 17:

[1658] Users: Enjoy watching sports more deeply and with personalized commentary.

[1659] According to the above steps, the system of the present invention provides personalized sports commentary and enhances the viewing experience through real-time adjustments according to the emotional state.

[1660] Example 2

[1661] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1662] Conventional sports viewing systems typically provide uniform live commentary without considering each user's individual needs or emotional state. This results in a failure to provide optimal commentary tailored to the user's level of understanding and preferences, resulting in a poor viewing experience. Furthermore, because commentary does not reflect the user's real-time emotional state during the game, it is not possible to fully evoke the user's excitement and satisfaction. This leads to a decline in the user's motivation to watch the game.

[1663] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving input data from a user and setting the user's preferences and understanding of sports, a means for receiving images from multiple camera devices and analyzing them in real time, a means for generating a customized commentary based on the analysis results in accordance with the user's status and emotional state, a means for collecting the user's facial expressions and tone of voice via a camera and microphone and recognizing and analyzing the emotional state, and a means for providing the generated commentary to the user. This makes it possible to provide a customized commentary that reflects the user's individual needs and emotional state in real time.

[1664] A "user" is an individual who inputs their favorite sports, the teams and players they support, and their understanding of the rules through a sports viewing app.

[1665] The "server" is a central processing unit that analyzes user input data, generates and stores user profiles, and uses emotion engines and video analysis modules to generate and provide customized live commentary.

[1666] A "terminal" is a device on which a sports viewing app used by a user is installed, and is a device that transmits user input data and real-time video, audio, and facial expression data to a server.

[1667] A "user profile" is a data set that reflects a user's sports preferences and understanding, the teams and players they support, and their real-time emotional state.

[1668] An "emotion engine" is a machine learning algorithm and software that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[1669] The "video analysis module" is an algorithm and software that analyzes video data received from multiple camera devices in real time and detects and tags play events such as shots, passes, and goals.

[1670] "Customized Commentary" means personalized sports commentary generated based on a user's profile information and real-time emotional state.

[1671] "Sports understanding" refers to the level of knowledge and understanding of the rules of a particular sport that a user has, and is classified into beginner, intermediate, and advanced levels.

[1672] "Filming equipment" refers to equipment such as a camera used to capture video of the match in real time and transmit it to a server via a terminal.

[1673] "Real-time analysis" is the process of instantly processing video data and user emotional data collected during a match to generate results.

[1674] This invention relates to a system that recognizes and analyzes user input data and emotional states, and customizes live sports commentary using real-time video analysis. This system consists of three main components: a server, a terminal, and a user.

[1675] Hardware and Software Configuration

[1676] Server: A high-performance data analysis server is used, which is installed with software that analyzes user input data, generates and saves profiles, an emotion engine, a video analysis module, and software that generates AI commentary voices.

[1677] Device: A device used by a user, such as a smartphone, tablet, or PC. The device is equipped with a camera and microphone and has a sports viewing app installed.

[1678] User: An individual who enjoys watching sports, inputs data and watches the game through the app.

[1679] User input data and status settings

[1680] A user launches a sports viewing app and inputs their favorite sport, the team and player they support, and their level of understanding. For example, they may select soccer, support Team X, and set their level of understanding as "Beginner." This data is collected by the device and sent to the server. The server analyzes the received data and generates a user profile. The generated profile information is stored in a database.

[1681] Emotional Engine Activation

[1682] During a match, the device's camera and microphone are used to collect the user's facial expressions and tone of voice in real time. This data is sent to a server and analyzed by an emotion engine. The server then reflects the results of this analysis in the user's profile and stores them in a database.

[1683] Real-time video analysis

[1684] When the match begins, the device receives real-time video feeds from multiple cameras and streams them to the server. The server then sends the video data to an analysis module for object recognition and motion analysis. Based on the analysis results, play data (e.g., shots, passes, goals) is extracted and the status of each player and the match is updated in real time.

[1685] AI live commentary voice generation

[1686] The server compares the play data with the user's profile information and emotional state to generate customized commentary. For example, beginners are provided with basic rule explanations, intermediate players with explanations of tactics and player characteristics, and advanced players with advanced statistical data and analysis of past matches. Also, depending on the user's emotional state, the server provides an uplifting commentary when excited, and a detailed commentary when relaxed. The server also generates commentary that includes information specific to the team or player the user is rooting for.

[1687] Providing customized commentary

[1688] The server transmits the generated live commentary audio data to the user's device, which then plays the audio data, allowing the user to enjoy the customized live commentary in real time.

[1689] Specific examples

[1690] (Example 1) Beginner user

[1691] User: A beginner who barely knows the rules of soccer.

[1692] Device: Select "Soccer" as your favorite sport and set your rule comprehension level to "Beginner."

[1693] Server: Three minutes into the game, the ball is in the hands of the goalkeeper.

[1694] Server: Generates the explanation "The goalkeeper has caught the ball. The goalkeeper is the only position that can catch the ball with their hands."

[1695] Device: Provide this description to the user aloud.

[1696] (Example 2) Intermediate user

[1697] User: An intermediate player who enjoys watching soccer. His favorite team is a specific team.

[1698] Device: Select "Soccer" as your favorite sport, set your favorite team to a specific team, and set your rule comprehension level to "Intermediate."

[1699] Server: A player on a particular team takes the shot.

[1700] Server: Generates an explanation such as, "A player from a particular team took a powerful shot. This player is known for his forward movement and is particularly good at free kicks."

[1701] Device: Provide this description to the user aloud.

[1702] (Example 3) Advanced user

[1703] User: Experienced soccer player with a thorough understanding of tactics and player data.

[1704] Device: Select "Soccer" as your favorite sport and set your rule understanding level to "Advanced."

[1705] Server: Midway through the match, certain teams change to a 4-3-3 formation.

[1706] Server: "Currently, a certain team has switched to a 4-3-3 formation. This tactical change is to strengthen the midfield press, and a certain player could be a game changer from now on. His data from the last 15 games shows that he has a 2% higher goal success rate when using this formation."

[1707] Device: Provide this description to the user aloud.

[1708] (Example 4) User using emotion engine

[1709] User: Supports a particular team and gets excited during the game.

[1710] Device: The camera and microphone collect the user's facial expressions and tone of voice, which are then analyzed by the emotion engine.

[1711] Server: Detects when the user is in an excited state and generates a commentary that corresponds to that state.

[1712] Server: Generate commentary to further excite users, such as "A certain team has a chance to score! What a spectacular play!"

[1713] Device: Provide this description to the user aloud.

[1714] The system dramatically improves the sports viewing experience by providing optimized commentary in real time based on the user's status and emotional state.

[1715] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1716] Step 1: Enter and submit user information

[1717] User: Launches a sports viewing app and inputs their favorite sport, the team and player they support, and their level of understanding of the rules. Specifically, they input information such as "Soccer," "Team X," "Player Y," and "Beginner" into the input fields on the screen.

[1718] Device: The data entered by the user is temporarily stored on the device and then sent to the server. JSON format data containing the input data (sport, team, player, and level of understanding) is generated and sent to the server.

[1719] Input: User input data (sport, team, player, understanding).

[1720] Output: User information data in JSON format sent to the server.

[1721] Step 2: Generate a user profile

[1722] Server: Analyzes the received user input data and generates a profile that reflects the user's preferences and level of understanding. Specifically, the input data is processed by the analysis module to generate a user profile object.

[1723] Server: Stores the generated user profile information in a database.

[1724] Input: User information data sent from the device (JSON format).

[1725] Output: A user profile object is saved to the database.

[1726] Step 3: Collect and analyze emotion data

[1727] Device: During the match, a camera and microphone are used to collect the user's facial expressions and tone of voice in real time. Specifically, the camera captures facial expression data and the microphone captures audio data.

[1728] Terminal: Sends collected data to the server in real time.

[1729] Server: Sends the received data to the emotion engine to analyze the user's emotional state. The emotion engine uses machine learning algorithms to analyze the data and generate emotion tags.

[1730] Server: Reflects the analysis results of the emotion engine into the user profile and saves it in the database.

[1731] Input: Facial expression and voice data collected by camera and microphone.

[1732] Output: An updated user profile reflecting their emotional state.

[1733] Step 4: Receiving and analyzing video data

[1734] Terminal: Receives multiple camera images in real time as soon as the match starts and streams them to the server. Specifically, the camera image data is sent using RTSP (Real Time Streaming Protocol).

[1735] Server: The server sends the received video data to the analysis module, where it performs object recognition and motion analysis, such as detecting and tagging play events such as shots, passes, and goals.

[1736] Server: Stores the analysis results in a database and updates the match progress data.

[1737] Input: Real-time video data from each camera.

[1738] Output: Match progress data tagged with play events.

[1739] Step 5: Generate and serve AI commentary

[1740] Server: Compares play data with the user's profile information and emotional state to generate customized commentary. For example, beginners can receive basic commentary such as "The goalkeeper has caught the ball."

[1741] Server: It also generates commentary based on the user's emotional state. For an excited user, it provides emotional commentary such as "A specific team has a chance to score a goal soon!"

[1742] Server: Also generates commentary including information related to the team and players you support.

[1743] Server: Sends the generated commentary audio data to the user's device.

[1744] Terminal: Plays back the received audio data, allowing users to enjoy the commentary in real time.

[1745] Input: User profile data, emotional state data, match progress data.

[1746] Output: Customized commentary audio data.

[1747] (Application example 2)

[1748] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1749] In modern sports viewing, it is difficult to provide real-time commentary tailored to the user's skill level and interests, and general commentary alone is not enough to provide a satisfying experience for some users. Furthermore, there are limited means to provide customized commentary that takes into account the user's emotional state. In particular, to enhance the viewing experience on smart devices, flexible responses that respond to the user's emotions are required.

[1750] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1751] In this invention, the server includes means for receiving user input data and setting the user's sports preferences and understanding level, means for receiving and analyzing video from multiple cameras in real time, means for generating a commentary customized according to the user's status based on the analysis results, means for providing the generated commentary to the user, means for recognizing the user's emotional state, means for adaptively customizing the commentary according to the emotional state, and means for providing the commentary through an augmented reality device, thereby enabling a more personalized sports viewing experience according to the user's skill level and emotional state.

[1752] "User input data" refers to data that users input into a sports viewing app, including information about their preferences and understanding of sports, and the teams and players they support.

[1753] "Means for recognizing emotional states" refers to a system or technology that collects and analyzes biometric data such as a user's facial expressions and tone of voice to determine the user's emotional state in real time.

[1754] "Means for receiving footage from multiple cameras and analyzing it in real time" refers to a technology that receives footage from multiple cameras taken from different perspectives and analyzes it in real time to understand the situation and play of the game.

[1755] A "means for generating customized commentary" is a system or technology that generates sports commentary optimized for an individual user based on the user's input data and emotional state.

[1756] The "means for providing the generated commentary to the user" is a technology for transmitting the generated customized commentary to the user's terminal and providing it as audio, text, or video.

[1757] "Augmented reality devices" are devices such as smart glasses and head-mounted displays that display images and information in real time, enhancing the user's viewing experience.

[1758] "User's level of sports understanding" indicates the extent to which a user understands the rules, tactics, and characteristics of players of a sport, and is classified into beginner, intermediate, and advanced levels.

[1759] "User status" refers to the user's overall profile information, such as their sports viewing preferences, understanding, and emotional state.

[1760] "Means for adaptively customizing live commentary" refers to a technology that adjusts the content and tone of live commentary in real time according to the user's emotional state and status.

[1761] The "means for setting preferences and level of understanding" refers to a system in which users input their preferences and level of understanding of sports into an application, and a user profile is generated based on that information.

[1762] This system recognizes and analyzes the user's input data and emotional state, and customizes live sports commentary using real-time video analysis. This system consists of three main components: a server, a device, and a user.

[1763] System configuration

[1764] Setting user status and emotions

[1765] The server receives a user's input data when the user opens a sports viewing app, including their favorite sport, the team or player they support, and their level of understanding of the rules. The device then collects the user's input data and sends it to the server. The server then analyzes the received user input data and generates a profile that reflects the user's preferences and level of understanding. The server then stores the generated user profile information in a database.

[1766] Emotional Engine Activation

[1767] The device collects the user's facial expressions and tone of voice in real time through a camera and microphone. This biometric data is sent to a server, where an emotion engine recognizes and analyzes the user's emotional state. The emotion engine's results are reflected in the user's profile, and the analysis results are stored in a database.

[1768] Real-time video analysis

[1769] As soon as the match begins, the device receives real-time video from multiple cameras and streams it to the server. The server then sends the video data to an analysis module for object recognition and motion analysis. This extracts play data (shots, passes, goals, etc.) and updates the status of each player and the match in real time.

[1770] AI commentary generation

[1771] The server compares the extracted play data with the user's profile information and emotional state. For example, it provides beginners with basic rule explanations, intermediate players with explanations of tactics and player characteristics, and advanced players with analysis using advanced statistics and past match data. It also adjusts the commentary according to the user's emotional state, focusing on uplifting commentary when excited and detailed commentary when relaxed. It also generates commentary that includes information specific to the team or player the user is supporting.

[1772] Providing customized commentary

[1773] The generated commentary audio data is sent from the server to the user's device, which then plays it back. Users can enjoy customized commentary in real time. This system enables a more personalized sports viewing experience that reflects the user's skill level and emotional state.

[1774] Hardware and software used

[1775] Hardware:

[1776] Camera (captures the user's facial expressions)

[1777] Microphone (captures the tone of your voice)

[1778] Augmented reality devices (smart glasses, head-mounted displays)

[1779] software:

[1780] opencv-python (real-time video analysis)

[1781] requests (communication with the server)

[1782] TensorFlow (machine learning model)

[1783] Flask and Django (API server construction)

[1784] Transformers (models for text generation)

[1785] Specific examples

[1786] Beginner users:

[1787] Prompt: Generate an explanatory sentence that explains the role of a goalkeeper for beginners.

[1788] Advanced users:

[1789] Prompt: For advanced players, please generate an explanatory statement based on the purpose of the 4-3-3 formation change and historical statistics.

[1790] In this way, the system of the present invention significantly enhances the viewing experience by providing a customized commentary in real time that is tailored to the user's preferences, level of understanding, and emotional state.

[1791] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1792] Step 1:

[1793] The user opens a sports viewing app and inputs their favorite sport, the team / player they support, and their level of understanding of the rules. The device receives this input data and sends it to the server. The input data includes their favorite sport (e.g., soccer), the team / player they support (e.g., Team A), and their level of understanding of the rules (e.g., beginner) in text format. The device sends this data to the server in JSON format.

[1794] Step 2:

[1795] The server analyzes the received input data and generates a profile that reflects the user's preferences and understanding. Specifically, it stores the input data in a database and applies algorithms (e.g., normalization and clustering of user information) to generate a profile. This profile information is stored in the database and used for further analysis. The generated user profile is saved as an output.

[1796] Step 3:

[1797] The device collects the user's facial expressions and tone of voice in real time through a camera and microphone. The collected data is temporarily stored on the device and periodically sent to a server. Specifically, facial expressions are captured using the OpenCV library, and audio data is collected from the microphone. Input data includes video frames and audio data.

[1798] Step 4:

[1799] The server sends the collected facial expression data and voice data to an emotion engine, which recognizes and analyzes the user's emotional state. This emotion engine uses machine learning models such as TensorFlow to analyze the emotional state (e.g., joy, excitement, sadness, etc.) using a pre-trained emotion classification model. The server reflects the analysis results in the user profile and stores them in a database. The input data is facial expression data and voice data, and the output is a classification result of the emotional state.

[1800] Step 5:

[1801] As soon as the match starts, the device receives multiple camera images in real time and streams them to the server. The camera images are transmitted using a streaming protocol (e.g., RTSP). The input data is the camera image data, and the output is the streaming data transmitted to the server in real time.

[1802] Step 6:

[1803] The server sends the received video data to an analysis module for object recognition and motion analysis. This analysis module incorporates an image recognition algorithm (e.g., YOLO) and extracts play data (shots, passes, goals, etc.). The input data is video data, and the output is analyzed play data.

[1804] Step 7:

[1805] The server compares the extracted play data with the user's profile information and emotional state to generate a customized commentary. For example, it provides basic rule explanations for beginners, tactics and player characteristics for intermediate players, and advanced statistics and analysis using past match data for advanced players. The generated commentary is generated using a natural language processing library (e.g., Transformers). The input data are play data, user profile, and emotional state, and the output is a customized commentary text.

[1806] Step 8:

[1807] The generated commentary audio data is sent from the server to the user's device. The device plays the received audio data and provides the user with a customized commentary in real time. The input data is the customized commentary text, which is converted into audio data using speech synthesis technology (e.g., Text-to-Speech). The output is the audio commentary that is played on the user's device.

[1808] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1809] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1810] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1811] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1812] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1813] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1814] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1815] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1816] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1817] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1818] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1819] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1820] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1821] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1822] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1823] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1824] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1825] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1826] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1827] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1828] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1829] The following is further disclosed regarding the above embodiment.

[1830] (Claim 1)

[1831] A means for receiving user input data and setting the user's sports preferences and understanding level;

[1832] A means of receiving and analyzing images from multiple cameras in real time;

[1833] means for generating a commentary customized according to the user's status based on the analysis result;

[1834] means for providing the generated commentary to the user;

[1835] A system including:

[1836] (Claim 2)

[1837] The system of claim 1 includes a means for providing beginners with detailed explanations of basic rules and plays, intermediate players with explanations of tactics and player characteristics, and advanced players with advanced analysis based on statistical data and past match data, based on the user's level of understanding of sports.

[1838] (Claim 3)

[1839] The system according to claim 1, further comprising means for providing information specific to a team or player that a user selects as a favorite.

[1840] "Example 1"

[1841] (Claim 1)

[1842] A means for receiving user input data and setting the user's sports preferences and understanding level;

[1843] means for receiving and analyzing images from a plurality of imaging devices in real time;

[1844] means for generating a customized audio commentary according to the user's status based on the analysis result;

[1845] means for providing the generated audio description to the user;

[1846] a means for storing and using user profile information;

[1847] a means for generating commentary using an AI language model;

[1848] means for converting the generated textual description into an audio format;

[1849] A system including:

[1850] (Claim 2)

[1851] The system of claim 1 includes a means for providing beginners with detailed explanations of basic rules and plays, intermediate players with explanations of tactics and player characteristics, and advanced players with advanced analysis based on statistical data and past match data, based on the user's level of understanding of sports.

[1852] (Claim 3)

[1853] The system according to claim 1, further comprising means for providing information specific to a team or player that a user selects as a favorite.

[1854] "Application Example 1"

[1855] (Claim 1)

[1856] means for receiving user input data and setting the user's preferences and comprehension level;

[1857] A means of receiving and analyzing images from multiple cameras in real time;

[1858] means for generating a customized commentary according to the user's status based on the analysis result;

[1859] means for providing the generated commentary to the user;

[1860] a means for obtaining user profile data;

[1861] A means of capturing and pre-processing camera footage in real time;

[1862] A means of analyzing video using deep learning models and interpreting the prediction results;

[1863] A means for acquiring product data from a server;

[1864] means for playing the generated audio data to the user;

[1865] A system including:

[1866] (Claim 2)

[1867] The system of claim 1 includes a means for providing basic information and detailed product descriptions to beginners, explanations of the characteristics and background of specific products to intermediate users, and advanced analysis based on statistical data and past trends to advanced users, based on the user's level of understanding.

[1868] (Claim 3)

[1869] 10. The system of claim 1, further comprising means for providing information specific to a product or category preferred by the user.

[1870] "Example 2: Combining Emotion Engines"

[1871] (Claim 1)

[1872] A means for receiving user input data and setting the user's sports preferences and understanding level;

[1873] means for receiving and analyzing images from a plurality of imaging devices in real time;

[1874] means for generating a customized commentary according to the user's status and emotional state based on the analysis results;

[1875] A means of recognizing and analyzing the user's emotional state by collecting facial expressions and tone of voice through a camera and microphone;

[1876] means for providing the generated commentary to the user;

[1877] A system including:

[1878] (Claim 2)

[1879] The system of claim 1 includes a means for providing beginners with detailed explanations of basic rules and plays, intermediate players with explanations of tactics and player characteristics, and advanced players with advanced analysis based on statistical data and past match data, based on the user's level of understanding of sports.

[1880] (Claim 3)

[1881] The system according to claim 1, further comprising means for providing information specific to a team or player that a user selects as a favorite.

[1882] "Application example 2 when combining emotion engines"

[1883] (Claim 1)

[1884] A means for receiving user input data and setting the user's sports preferences and understanding level;

[1885] A means of receiving and analyzing images from multiple cameras in real time;

[1886] means for generating a commentary customized according to the user's status based on the analysis result;

[1887] means for providing the generated commentary to the user;

[1888] a means for recognizing the emotional state of a user;

[1889] means for adaptively customizing the commentary according to the emotional state;

[1890] means for providing said commentary through an augmented reality device;

[1891] A system including:

[1892] (Claim 2)

[1893] The system of claim 1 includes a means for providing beginners with detailed explanations of basic rules and plays, intermediate players with explanations of tactics and player characteristics, and advanced players with advanced analysis based on statistical data and past match data, based on the user's level of understanding of sports.

[1894] (Claim 3)

[1895] The system according to claim 1, further comprising means for providing information specific to a team or player that a user selects as a favorite. [Explanation of symbols]

[1896] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving user input data and setting the user's sports preferences and understanding level; A means of receiving and analyzing images from multiple cameras in real time; means for generating a commentary customized according to the user's status based on the analysis result; means for providing the generated commentary to the user; A system including:

2. The system of claim 1 includes a means for providing beginners with detailed explanations of basic rules and plays, intermediate players with explanations of tactics and player characteristics, and advanced players with advanced analysis based on statistical data and past match data, based on the user's level of understanding of sports.

3. 2. The system according to claim 1, further comprising means for providing information specific to a team or player that a user selects as a favorite.

Images