system

Abstract

Provide a system. [Solution] means for receiving a match request from a user; means for retrieving match-related data from a knowledge base based on the received request; means for dynamically generating commentary text based on the acquired data; means for converting the generated text into audio data; a means for providing the converted voice data to a user terminal in real time in cooperation with a distribution system; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In sports game broadcasts, compelling live commentary is an important factor in increasing viewer satisfaction. However, arranging for skilled commentators or professional commentators is costly and difficult, especially for less common games or smaller leagues. Furthermore, providing accurate commentary in real time requires extensive knowledge and experience, making securing the necessary personnel a challenge. To address this issue, a system capable of efficiently providing live commentary in real time is needed. [Means for solving the problem]

[0005] To solve this problem, the present invention provides a system including: means for receiving a request about a match from a user, acquiring match-related data from a knowledge base based on the received request, means for dynamically generating commentary text based on the acquired data, means for converting the generated text into audio data, and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system. In particular, the means for dynamically generating commentary text adjusts the content of the text to match the personality of a specific commentator based on the acquired data, and converts the generated text into audio data based on a pre-set audio profile and provides it to the user, thereby enabling commentary that is as realistic as that of a commentator.

[0006] The "means for receiving requests from users regarding a game" refers to a device or software function that allows a user to input information about a sports game that the user desires to be commentated on, and transmits that information to the server side.

[0007] "Means for retrieving match-related data from a knowledge base" refers to the functions or processes that search and retrieve data about a particular match (such as player performance, past match results, etc.) from a match database or external information source.

[0008] "Means for dynamically generating commentary text" refers to algorithms or software functions that utilize acquired match-related data to generate appropriate commentary text in real time.

[0009] "Means for converting the generated text into audio data" refers to technology for converting commentary expressed in text format into audio format, such as Text-to-Speech (TTS) technology.

[0010] "Means for providing converted voice data to user terminals in real time in cooperation with a distribution system" refers to the infrastructure and software functions for encoding the generated voice data in real time and distributing it to user terminals via a network.

[0011] "Adjusting the content of the text to match the personality of a specific commentator" refers to natural language processing technology that adjusts the generated commentary text to make it sound as if a specific person is speaking.

[0012] A "predefined voice profile" is a profile that defines the characteristics of a particular voice quality or speaking style, and is used during speech synthesis to reproduce a particular voice or emotion. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] MODE FOR CARRYING OUT THE INVENTION

[0035] Program processing flow explanation

[0036] This invention is a system that allows users to send requests about sports matches and provides real-time commentary based on the requests. The program processing of this system will be specifically described below.

[0037] Request received

[0038] User

[0039] A user uses the game broadcasting application to input and send a request for live commentary of a particular game, for example, "I would like live commentary of this game."

[0040] Terminal

[0041] The terminal receives the user's request and sends data including detailed information (e.g., match ID, team name, player name) to the server.

[0042] Data acquisition and preparation

[0043] server

[0044] The server analyzes the request received from the terminal and searches for relevant match data from a knowledge base (database).

[0045] This match data includes players' past performances, team statistics, current match status, past match results, and more.

[0046] Organize the data you obtain and convert it into a format that can be used in the next step.

[0047] Text generation

[0048] server

[0049] The server uses a RAG (Search Augmentation Generation) module to generate commentary text appropriate for a particular scene in real time based on the acquired data.

[0050] For example, it generates text such as "Player A hit a home run" or "Team B added three points."

[0051] The generated text is tailored to a particular commentator's personality, such as a calm, detailed commentator saying, "Great hit! Player A hit a home run and Team B added three runs."

[0052] Audio conversion

[0053] server

[0054] The server sends the generated commentary text to a TTS (Text-to-Speech) module, which converts it into voice data.

[0055] The TTS module generates voice data based on a pre-defined voice profile (e.g., a deep male voice).

[0056] For example, voice data such as "Great hit! Player A hit a home run, and Team B added three points" is generated.

[0057] Linking with distribution systems and distribution

[0058] server

[0059] The generated audio data is transferred to a distribution system and encoded in real time.

[0060] The server supports the distribution system in integrating audio data into game footage.

[0061] Terminal

[0062] The terminal receives encoded audio data from the distribution system and plays it back in synchronization with the game video.

[0063] Users can watch immersive real-time commentary on their devices.

[0064] Specific examples

[0065] Scenario: A live commentary of a decisive scoring moment in a baseball game

[0066] Request received

[0067] The user sends a request via the terminal saying, "I would like a live commentary of this baseball game."

[0068] Data Acquisition

[0069] The server analyzes the user's request and retrieves the player's performance, the team's past match results, and the current match situation from the match database.

[0070] Text Generation

[0071] The server uses the RAG module to generate the text "Player B hits a home run and Team C adds 3 runs."

[0072] This text can be tailored to fit the personality of a particular commentator, such as "Great hit! Team C scores three runs with a home run by Player B."

[0073] Audio conversion

[0074] The server sends this adjusted text to the TTS module, which converts it into voice data.

[0075] Generates audio data such as, "Great hit! Player B's home run gave Team C three more runs."

[0076] delivery

[0077] The server transfers the generated audio data to a distribution system in real time and distributes it to user terminals together with the game video.

[0078] The device plays audio and video in sync, allowing users to enjoy live commentary in real time.

[0079] This invention allows users to receive live commentary in real time with a sense of presence, making it possible to maximize the appeal of the match.

[0080] The processing flow will be explained below.

[0081] Step 1:

[0082] Using the game broadcasting application, a user inputs and submits a request for live commentary on a particular game, for example, "I would like live commentary on this game."

[0083] Step 2:

[0084] The device receives the user's request, analyzes the request details (match ID, team name, player name, etc.), and sends the analyzed request data to the server.

[0085] Step 3:

[0086] The server analyzes the requests received from the devices and searches for relevant match data in a knowledge base, which includes information such as players' past performances, team statistics, and the current match situation.

[0087] Step 4:

[0088] The server uses a RAG (Search Augmentation and Generation) module to organize the acquired game data and generate commentary text appropriate for a specific scene, for example, "Player A hit a home run and Team B scored three runs."

[0089] Step 5:

[0090] The server-generated text can be tailored to fit a specific commentator's personality, such as "Great hit! Player A hit a home run, and Team B added three runs."

[0091] Step 6:

[0092] The server sends the adjusted commentary text to a Text-to-Speech (TTS) module, which converts the text into speech data, such as "Great hit! Player A hit a home run, and Team B added three points."

[0093] Step 7:

[0094] The server transfers the generated audio data to a real-time streaming system, which encodes the audio data and synchronizes it with the game video.

[0095] Step 8:

[0096] The device plays back the encoded audio data received from the distribution system, synchronizing the audio with the game footage to provide users with immersive, real-time commentary.

[0097] Example 1

[0098] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0099] In modern sports management, real-time commentary is crucial to enhancing the appeal of matches. However, many systems struggle to provide fast and appropriate commentary for specific matches or scenes requested by users. Furthermore, problems arise with adjusting the commentary to suit the individual styles of commentators and converting the commentary into audio data. Therefore, there is a need for a more flexible system that can respond to individual user requests.

[0100] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0101] In this invention, the server includes means for receiving requests about a match from a user, means for retrieving match-related data from a knowledge base based on the received requests, means for using a generative AI model to generate commentary text in real time based on the retrieved data, means for tailoring the generated text to the personality of a specific commentator, means for converting the tailored text into audio data, and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system, thereby enabling flexible and appropriate commentary to be provided in real time based on user requests.

[0102] A "user" is a person who utilizes the system's game broadcast application to request live commentary about a particular game or team.

[0103] A "request for a match" is information that a user uses to inform the system that they would like live commentary on a specific match or scene.

[0104] A "knowledge base" is a database or collection of information containing game-related data, such as player performance, team statistics, current game situation, and past match results.

[0105] "Match-related data" refers to any information related to a match, such as player performance, team statistics, the current match situation, and past match results.

[0106] A "generative AI model" is an artificial intelligence model that generates live commentary text in real time based on acquired data.

[0107] "Live commentary text" is commentary text that follows the progress of the match and is generated based on the content requested by the user.

[0108] "The personality of a particular commentator" refers to the commentator's unique speaking style or manner of delivery, including, for example, calm commentary or enthusiastic commentary styles.

[0109] "Audio data" refers to an audio file that has been converted from the generated commentary text using text-to-speech (TTS) technology.

[0110] The "distribution system" is an infrastructure for synchronizing the generated audio data with the game video and providing it to user terminals in real time.

[0111] A "user terminal" is a device that a user uses to view live commentary of a match, and includes a smartphone, tablet, PC, etc.

[0112] "Flexible and appropriate live commentary" means responding quickly to individual user requests and generating and providing commentary that matches the progress of the game.

[0113] "Real-time" refers to the immediacy of time in which commentary is provided as the game progresses.

[0114] "Tuning" refers to the process of converting and modifying the generated commentary text to fit the personality and voice profile of a particular commentator.

[0115] "Game Footage" means a live video feed of an actual game or recorded game video.

[0116] MODE FOR CARRYING OUT THE INVENTION

[0117] The present invention is a system that allows users to submit requests for sports matches and provides real-time commentary based on the requests. Details of how to implement this system are described below.

[0118] Basic configuration

[0119] User

[0120] A user uses the game broadcasting application to input and send a request for commentary on a particular game or scene, for example, "I would like commentary on this baseball game."

[0121] Terminal

[0122] The user's device (smartphone, tablet, PC, etc.) receives the user's request and sends detailed information (match ID, team name, player names, etc.) in JSON format to the server.

[0123] server

[0124] The server analyzes the JSON data sent from the device, identifies the request, and then accesses the match database to retrieve relevant match data (player performance, team statistics, current match status, past match results, etc.).

[0125] The server generates commentary text in real time based on the acquired data. It uses a generative AI model to generate appropriate commentary based on the data. For example, it generates text such as "Player A hit a home run" or "Team B added three points."

[0126] The generated text is then tailored to fit the personality of a particular commentator, for example, a calm commentary style might say, "Great hit! Player A hit a home run, and Team B added three runs."

[0127] The adjusted text is sent to the Text-to-Speech (TTS) module, which converts it into speech data. The TTS module generates speech data based on a pre-defined voice profile (e.g., a deep male voice). For example, the generated speech data is, "Great hit! Player A hit a home run, and Team B scored three points."

[0128] The server then transfers the generated audio data to the distribution system, where it is encoded in real time. The encoded audio data is generated in a format that is synchronized with the game video.

[0129] Terminal

[0130] The user's device receives the encoded audio data from the distribution system and plays it in sync with the game footage, allowing the user to enjoy immersive, real-time commentary.

[0131] Specific examples

[0132] For example, consider a scenario where a user requests commentary on a baseball game, saying, "I'd like a commentator for this baseball game." The server receives this request and retrieves data from the game database, such as player performance, past team records, and the current game situation. It then uses a generative AI model to generate text like, "Player B hit a home run, and Team C added three runs," and adjusts it to fit the commentator's personality, such as, "Great hit! Player B hit a home run, and Team C added three runs." Finally, this adjusted text is sent to a TTS module, which converts it into voice data. The voice data is then distributed to the user's device in real time along with the game video.

[0133] Prompt Sentence Examples

[0134] "Please show us the process flow for this system, which generates commentary text in real time as the game progresses based on the match data requested by the user, and converts it into voice data using a TTS module."

[0135] In this way, this invention allows users to enjoy flexible and appropriate live commentary in real time, making it possible to maximize the appeal of the match.

[0136] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0137] Step 1:

[0138] User request submission

[0139] User: Using the match broadcasting application, the user inputs and submits a request for live commentary on a particular match or scene. For example, the user inputs the phrase "I would like live commentary on this match" and presses the submit button.

[0140] Input: The request text entered by the user.

[0141] Output: The request information is sent to the terminal.

[0142] Step 2:

[0143] Terminal request reception and server transmission

[0144] Terminal: The terminal receives the user's request, generates JSON data containing the details (match ID, team name, player name, etc.), and sends it to the server.

[0145] Input: User request information.

[0146] Output: The request data in JSON format that is sent to the server.

[0147] Step 3:

[0148] Server request analysis

[0149] Server: The server parses the received JSON data and identifies the request content. Specifically, it extracts information such as the match ID, team name, and player names.

[0150] Input: JSON data sent from the terminal.

[0151] Output: Analysis results such as match ID, team name, player name, etc.

[0152] Step 4:

[0153] Obtaining match data

[0154] Server: Based on the extracted request, the server searches and retrieves relevant match data (e.g., player performance, team statistics, current match situation, past match results, etc.) from its knowledge base.

[0155] Input: Analysis results (match ID, team name, player name, etc.).

[0156] Output: Match-related data obtained.

[0157] Step 5:

[0158] Data organization and format conversion

[0159] Server: Organizes the match data and converts it into JSON format for use in the next processing step.

[0160] Input: Retrieved match-related data.

[0161] Output: Match data converted to JSON format.

[0162] Step 6:

[0163] Generating commentary text

[0164] Server: The server uses a generative AI model to generate commentary text in real time based on the organized data, such as "Player A hit a home run" or "Team B scored three runs."

[0165] Input: Match data in JSON format.

[0166] Output: Text of the commentary.

[0167] Step 7:

[0168] Text adjustment to suit the commentator's personality

[0169] Server: Tailors the generated text to fit the personality of a particular commentator. For a calm commentary style, the text might be tailored to something like, "Great hit! Player A hit a home run and Team B added three runs."

[0170] Input: The generated commentary text.

[0171] Output: Adjusted commentary text.

[0172] Step 8:

[0173] Text-to-speech

[0174] Server: The adjusted text is sent to the TTS module, which converts it into voice data. The TTS module generates voice data based on the configured voice profile (e.g., a deep male voice).

[0175] Input: Adjusted commentary text.

[0176] Output: The generated audio data.

[0177] Step 9:

[0178] Preparing for audio data distribution

[0179] Server: The generated audio data is transferred to the distribution system and encoded in real time. The encoded audio data is generated in a format that is synchronized with the game video.

[0180] Input: The generated audio data.

[0181] Output: The encoded audio data.

[0182] Step 10:

[0183] Synchronized audio and video playback on devices

[0184] Terminal: The terminal receives the encoded audio data and plays it in sync with the game footage, allowing users to enjoy immersive real-time commentary.

[0185] Input: Encoded audio data

[0186] Output: Real-time commentary with synchronized audio and video.

[0187] (Application example 1)

[0188] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0189] Conventional sports commentary systems lack a way for users to request real-time commentary on specific scenes or players. They also struggle to provide realistic commentary in real time, limiting the user experience. Furthermore, commentary that reflects the commentator's individuality and real-time audio delivery are complex and ineffective, preventing the full appeal of the game.

[0190] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0191] In this invention, the server includes: means for receiving a request about a match from a user; means for acquiring match-related data from a knowledge base based on the received request; means for dynamically generating commentary text based on the acquired data; means for using a generative AI model to generate commentary text appropriate for a specific scene based on the user request; means for converting the generated text into audio data; and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system. This allows users to easily request real-time commentary about specific scenes or players, enabling them to enjoy immersive commentary in real time. Furthermore, customized commentary reflecting the commentator's personality can be provided to maximize the appeal of the match.

[0192] A "user" is an individual or group of people who desires and submits a request for live commentary on a match.

[0193] A "request" is information that a user sends to request commentary on a specific match or scene that the user desires.

[0194] A "knowledge base" is a database containing game-related data such as players' past performances, team statistics, the current game situation, and past match results.

[0195] "Match-related data" refers to all data necessary for live commentary, such as the progress of the match, player and team performance, and statistical information.

[0196] "Live commentary text" is textual information that describes the progress of the match or specific scenes.

[0197] A "generative AI model" is an artificial intelligence model used to generate commentary text appropriate for a specific scene or match situation based on a user request.

[0198] "Audio data" refers to data in audio format that has been converted from the generated live commentary text using a TTS module.

[0199] A "real-time delivery system" is a system for instantly encoding generated audio data and delivering it to a user terminal.

[0200] A "customized commentary" is a commentary that reflects the personality and tone of a particular commentator and is tailored to the user's request.

[0201] MODE FOR CARRYING OUT THE INVENTION

[0202] This invention is a system that allows users to request live commentary on a match in real time, generates commentary text based on the request, and provides it as audio data. An embodiment of this system will be described in detail below.

[0203] Hardware and software used

[0204] Server: Backend system for data acquisition, analysis, text generation, and speech generation

[0205] Database: Stores players' past performances, team statistics, match status, etc.

[0206] RAG module: A generative AI model for generating commentary text based on acquired data.

[0207] TTS module: Converts generated text into speech data

[0208] User devices: Smartphones, smart glasses, head-mounted displays, etc., for sending requests and viewing commentary

[0209] Data processing and calculation flow

[0210] Receiving requests and retrieving data

[0211] A user inputs and sends a request for live commentary of a game through a smartphone application. The user device then sends this request (including the game ID, team name, and player names) to the server.

[0212] The server analyzes the received request and retrieves relevant match data from the database, including player past performances, team statistics, the current match situation, past match results, etc. The retrieved data is then analyzed, organized, and converted into a format for generating play-by-play commentary.

[0213] Text Generation

[0214] The server uses the RAG module to generate commentary text based on the acquired data in real time, tailored to the specific scene, such as "Player X scores a goal, giving team Y the lead."

[0215] Using this generative AI model, we can generate commentary text appropriate for a specific scene based on the user's request. For example, we can use the following prompt sentence from the generative AI model to provide an emotional commentary:

[0216] Generate text to provide the user with real-time sports commentary based on the following data, where Player X scores a goal and Team Y takes the lead:

[0217] Past goal success rate of player X

[0218] Team Y's past win rate when leading

[0219] Current match situation

[0220] For customized commentary, please provide a detailed explanation in an excited tone.

[0221] Audio Conversion and Distribution

[0222] The generated text is sent from the server to the TTS module, which converts it into speech data, such as "Player X scores a goal, giving team Y the lead."

[0223] The server transfers the converted audio data to the distribution system in real time, encodes the audio data, and distributes it to the user's device. The device then plays the audio commentary in real time in sync with the game footage, allowing the user to enjoy a realistic live commentary.

[0224] Specific examples

[0225] Live commentary of decisive goals in soccer matches

[0226] A user launches the app and inputs and sends "I would like live commentary for this soccer match." The device sends the match ID and related data to the server, which then retrieves the match situation and past results from the database. The RAG module generates text such as "Player X scores a goal, giving team Y the lead," and the TTS module converts that text into voice data. The server provides the generated voice data to the user's device in real time, allowing the user to enjoy live commentary along with the match footage.

[0227] This invention allows users to easily request real-time commentary on specific scenes or players, making it possible to maximize the appeal of the match.

[0228] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0229] Step 1:

[0230] A user inputs and submits a request for live commentary on a match.

[0231] Specific behavior:

[0232] Users launch the application on their smartphone and enter the match ID, team name, player names, etc. of a specific match.

[0233] Check the request details and tap the send button.

[0234] Input: Match ID, team name, player name

[0235] Output: Request data

[0236] Step 2:

[0237] The terminal sends the user's request to the server.

[0238] Specific behavior:

[0239] The terminal receives the request data entered by the user and transmits it to the server as an HTTP request.

[0240] Input: Request data

[0241] Output: HTTP request to the server

[0242] Step 3:

[0243] The server analyzes the received request and retrieves the relevant match data from the knowledge base.

[0244] Specific behavior:

[0245] The server analyzes the request and searches and retrieves match-related data from the database based on the match ID, team name, and player name.

[0246] The necessary data to be obtained includes players' past performance, team statistics, current match situation, and past match results.

[0247] Input: Request data

[0248] Output: Match-related data

[0249] Step 4:

[0250] The server generates commentary text based on the acquired data using a generative AI model.

[0251] Specific behavior:

[0252] The server formats the match-related data it receives and inputs it as prompts to the generative AI model.

[0253] Prompt: "Generate text to provide the user with real-time sports commentary based on the following data: Player X scores a goal and Team Y takes the lead; Player X's past goal success rate; Team Y's past win rate when leading; Current game situation. For the customized commentary, please provide detailed commentary in an excited tone."

[0254] A generative AI model dynamically generates appropriate commentary text.

[0255] Input: Match-related data, prompt text

[0256] Output: Commentary text

[0257] Step 5:

[0258] The server sends the generated commentary text to the TTS module, which converts it into voice data.

[0259] Specific behavior:

[0260] The server sends the generated commentary text to the TTS module.

[0261] The TTS module generates speech data based on the text.

[0262] Input: Commentary text

[0263] Output: Audio data

[0264] Step 6:

[0265] The server provides the generated voice data to the user terminal in real time in cooperation with the distribution system.

[0266] Specific behavior:

[0267] The server transfers the audio data to the distribution system and encodes it in real time.

[0268] A distribution system encodes the audio data and distributes it to the user terminal.

[0269] Input: Audio data

[0270] Output: Encoded audio data

[0271] Step 7:

[0272] The terminal plays back the received audio data in synchronization with the game video.

[0273] Specific behavior:

[0274] The user terminal analyzes the audio data received from the distribution system and plays it back in synchronization with the game video.

[0275] Users can enjoy realistic, real-time commentary.

[0276] Input: Encoded audio data

[0277] Output: Synchronized audio and video playback

[0278] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0279] MODE FOR CARRYING OUT THE INVENTION

[0280] Program processing flow explanation

[0281] This invention is a system that allows users to send requests about sports games and provides real-time commentary based on those requests, and by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized commentary. The program processing of this system is explained in detail below.

[0282] Request received

[0283] User

[0284] Using the match broadcasting application, a user inputs and submits a request for live commentary on a particular match, for example, "I would like live commentary on this match."

[0285] Terminal

[0286] The terminal receives the user's request, analyzes the request details (match ID, team name, player name, etc.), and sends the analyzed request data to the server.

[0287] Emotion Recognition and Data Acquisition

[0288] server

[0289] The server analyzes the user's emotion data sent together with the request received from the terminal. The emotion data is generated by an emotion engine installed in the terminal.

[0290] The server searches the knowledge base for relevant game data based on the emotion data. For example, if the user is excited, data on important plays and scoring scenes will be prioritized.

[0291] The knowledge base includes players' past performances, team statistics, and current match situations.

[0292] Text generation

[0293] server

[0294] The server uses a RAG (Search Augmentation and Generation) module to organize the acquired match data and generate commentary text appropriate for a particular scene.

[0295] For example, "Player A hit a home run and Team B added three points."

[0296] The generated text is tailored to the specific commentator's personality and the user's emotions, for example, if the user is excited, it might say, "Great hit! Player A's home run gives Team B three runs!"

[0297] Audio conversion

[0298] server

[0299] The server sends the adjusted commentary text to a Text-to-Speech (TTS) module, which converts the text into voice data.

[0300] The TTS module generates voice data based on a pre-defined voice profile, and also includes the ability to change the voice profile depending on the user's emotions.

[0301] For example, voice data with a lively tone such as "Great hit! Player A's home run gave Team B three more points!" is generated.

[0302] Linking with distribution systems and distribution

[0303] server

[0304] The generated audio data is transferred to a distribution system and encoded in real time.

[0305] The server supports the distribution system in integrating audio data into game footage.

[0306] Terminal

[0307] The device plays back the encoded audio data received from the distribution system, synchronizing it with the game footage to provide users with immersive, real-time commentary.

[0308] Specific examples

[0309] Scenario: A live commentary of a decisive scoring moment in a baseball game

[0310] Request received

[0311] The user sends a request via the terminal saying, "I would like a live commentary of this baseball game."

[0312] The device analyzes the user's emotions (for example, heart rate and excitement level based on facial expression recognition) and sends this data to the server.

[0313] Data Acquisition

[0314] The server analyzes the user's emotional data and retrieves relevant match data (scoring scenes and important plays) from the knowledge base, prioritizing them according to the emotion.

[0315] The data obtained includes player performance, team past performances, and the current game situation.

[0316] Text Generation

[0317] The server uses the RAG module to generate the text "Player B hits a home run and Team C adds 3 runs."

[0318] This text can be adjusted to reflect the user's excitement level, such as "Great hit! Player B's home run gives Team C three more runs!"

[0319] Audio conversion

[0320] The server sends the adjusted text to the TTS module, which converts it into voice data in a lively tone that matches the user's emotions.

[0321] Generates audio data such as "Great hit! Player B's home run gave Team C three more points!"

[0322] delivery

[0323] The server transfers the generated audio data to the distribution system in real time and distributes it to the user's terminal together with the game video.

[0324] The device plays audio and video in sync, allowing users to enjoy personalized, exciting commentary in real time.

[0325] This invention provides a realistic commentary that takes into account the user's emotions, making it possible to maximize the appeal of the match.

[0326] The processing flow will be explained below.

[0327] Step 1:

[0328] Using the game broadcasting application, a user inputs and submits a request for live commentary on a particular game, for example, "I would like live commentary on this game."

[0329] Step 2:

[0330] The device receives the user's request and analyzes the request details (match ID, team name, player name, etc.). After analysis, the emotion engine analyzes the request data and the user's emotional data (e.g., heart rate, facial expression data), and sends them to the server.

[0331] Step 3:

[0332] The server analyzes the request data and emotional data received from the device. The emotional data includes information such as whether the user is excited, nervous, or relaxed.

[0333] Step 4:

[0334] The server searches the knowledge base based on the analyzed data to retrieve data related to the match. For example, if the user is excited, data related to important plays and scoring scenes will be retrieved first.

[0335] Step 5:

[0336] The server uses a RAG (Search Augmentation and Generation) module to organize the acquired game data and generate commentary text appropriate for a specific scene, for example, "Player A hit a home run and Team B scored three runs."

[0337] Step 6:

[0338] Tailor the server-generated text to the personality of a particular commentator and the user's emotions. For example, if the user is excited, tailor the text to "Great hit! Player A's home run gives Team B three runs!"

[0339] Step 7:

[0340] The server sends the adjusted commentary text to the TTS (Text-to-Speech) module, which converts the text into voice data. The TTS module generates voice data based on a pre-defined voice profile and changes the voice profile depending on the user's emotions. For example, it generates voice data with a lively tone, such as "Great hit! Player A's home run gave Team B three points!"

[0341] Step 8:

[0342] The server transfers the generated audio data to a real-time streaming system, which encodes the audio data and synchronizes it with the game video.

[0343] Step 9:

[0344] The device plays back the encoded audio data received from the distribution system, synchronizing the audio with the game footage to provide users with immersive, real-time commentary.

[0345] Specific examples

[0346] Scenario: A live commentary of a decisive scoring moment in a baseball game

[0347] Step 1:

[0348] The user sends a request via the terminal saying, "I would like a live commentary of this baseball game."

[0349] Step 2:

[0350] The device receives the user's request and analyzes details such as the match ID, team name, and player names. At the same time, the device's built-in emotion engine analyzes the user's heart rate and facial expression data, adds emotional information (e.g., excitement level), and sends it to the server.

[0351] Step 3:

[0352] The server receives the request and emotion data from the terminal and confirms that the user is in an excited state.

[0353] Step 4:

[0354] The server searches the knowledge base taking into account the emotional data and prioritizes retrieving match-related data (scoring scenes, important plays).

[0355] Step 5:

[0356] Based on the game data obtained by the server, the RAG module generates the text "Player B hit a home run and Team C added three points."

[0357] Step 6:

[0358] The server adjusts the generated text depending on the user's excitement level, such as "Great hit! Player B's home run gives Team C three more runs!"

[0359] Step 7:

[0360] The server sends the adjusted text to the TTS module, which generates voice data saying in a cheerful tone, "Great hit! Player B's home run gave Team C three more runs!"

[0361] Step 8:

[0362] The server transfers the generated audio data to a real-time distribution system and transmits it to the user's terminal together with the game video.

[0363] Step 9:

[0364] The device receives the audio data and plays it back in sync with the game footage, allowing users to enjoy personalized, exciting commentary in real time.

[0365] Example 2

[0366] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0367] Conventional sports commentary systems have been unable to provide personalized commentary that takes into account the user's emotions, resulting in a lack of realism and a lack of user experience. Specifically, it has been difficult to recognize important moments in a game and provide appropriate commentary in real time that reflects the user's emotional state.

[0368] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for receiving a request related to a game from a user; means for analyzing the received request and the user's emotion data; means for acquiring game-related data from a knowledge base based on the analysis result; means for dynamically generating commentary text based on the acquired data; means for converting the generated text into audio data based on a preset audio profile; and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system. This makes it possible to provide realistic commentary in real time that takes into account the user's emotions.

[0369] "User" refers to an individual who wishes to use the system to provide live commentary of a sports game.

[0370] A "game request" refers to a request sent by a user to the system requesting commentary on a particular sports game.

[0371] "Emotional data" refers to data that indicates the user's current emotional state, and refers to information obtained using, for example, a heart rate sensor or facial expression analysis software.

[0372] "Knowledge base" refers to the database in which the server stores match-related data, including information such as the match situation and player performance.

[0373] "Analyzing" refers to the process of deciphering the content and meaning of data based on received requests and emotional data, and extracting the necessary information.

[0374] "Game-Related Data" refers to data related to a particular sports game, such as statistics, current game status, and player or team performance.

[0375] "Dynamic generation of commentary text" refers to the process of automatically generating appropriate commentary text in real time based on acquired match-related data.

[0376] A "voice profile" is a profile that contains settings for a specific voice synthesis, including pre-defined voice tone, accent, etc.

[0377] "Converting to speech data" refers to the process of converting the generated text into speech using techniques that generate natural language speech from text.

[0378] "Delivery system" refers to the network and software infrastructure for transmitting generated audio data to user terminals in real time.

[0379] "User Device" means the electronic device used by a User to view the live commentary of a match, such as a smartphone, tablet, or PC.

[0380] The present invention provides a system that allows users to submit requests about sports matches and provides personalized commentary in real time. An embodiment of the system will be described in detail below.

[0381] User operation

[0382] When a user wants to receive live commentary of a sports game, he or she uses a game streaming application. The application has a search function, and the user selects a specific game, enters a request such as "I would like live commentary for this game," and submits it. For example, the user might enter "I would like live commentary for this baseball game." This operation also collects the user's emotional data (e.g., heart rate and facial expression data obtained through facial recognition) via the device.

[0383] Device Features

[0384] The device analyzes the request received from the user. The analysis targets the text data in the request and the acquired emotion data. As a result of the analysis, detailed information such as the match ID, team name, and player names is extracted. This information is then sent to the server. A natural language processing engine and an emotion recognition engine are used for the analysis. Specific examples include software such as "SpaCy" for natural language processing and "Affectiva" for emotion recognition.

[0385] Server Processing

[0386] The server analyzes the request data and emotion data received from the device. The emotion recognition engine used for the analysis is API-based emotion recognition software or a machine learning model (e.g., OpenAI's (registered trademark) GPT-based emotion recognition model). The server determines the user's emotional state and, based on that, searches a knowledge base for relevant match data. The knowledge base consists of a sports database, including team and player statistics, past match results, and current match status.

[0387] The acquired data is then organized and a RAG (Search Augmentation and Generation) module is used to generate commentary text. For example, it generates text such as "Player A hits a home run, and Team B scores three runs." This text is tailored to the personality of the specific commentator and the emotions of the user. If the user is excited, it will be tailored to "Great hit! Player A hits a home run, and Team B scores three runs!"

[0388] The generated text is sent to a Text-to-Speech (TTS) module (e.g., Amazon Polly, Google® Cloud Text-to-Speech) and converted into voice data. The TTS module generates voice data based on a pre-defined voice profile. The voice profile is adjusted according to the user's emotions, for example, to set a cheerful tone or a calm tone.

[0389] The generated audio data is transferred to a distribution system and provided to user terminals in real time. The distribution system encodes the audio data in real time and distributes it together with the game video.

[0390] Specific examples

[0391] For example, if a user sends a request saying, "I would like a commentary of a baseball game," the server analyzes the user's emotions (e.g., heart rate and facial expression indicating excitement) and retrieves relevant data about the game. Using data about scoring moments and key plays, the server generates text such as, "Player A hits a home run, and Team B scores three runs." This text is then converted into lively audio data and delivered in real time. Users can enjoy immersive commentary in real time on their devices.

[0392] Prompt Sentence Examples

[0393] A prompt to the generative AI model to explain this system might look like this:

[0394] "If a user requests a live commentary of a sports game and the system recognizes the user's emotions and provides a personalized commentary, what would that be?"

[0395] In this way, the present invention takes into account the user's emotions and can enhance the realism and entertainment value of the match.

[0396] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0397] Step 1: Submitting a request

[0398] User

[0399] A user opens a game broadcasting application and enters a request for live commentary on a particular game.

[0400] Example: "I'd like to provide live commentary for this match."

[0401] Input: Request text for commentary on a specific sports game

[0402] Output: Request sent successfully

[0403] Step 2: Parsing and sending the request

[0404] Terminal

[0405] The terminal receives a request from a user and analyzes the text data and emotional data (e.g., heart rate, facial expression analysis) in the request.

[0406] Through analysis, detailed information such as the match ID, team name, and player name is extracted and sent to the server along with emotion data.

[0407] Example: A natural language processing engine (e.g., "SpaCy") analyzes the request and identifies the match ID and team name.

[0408] Input: Request text, emotion data

[0409] Output: Analysis results including match ID, team name, player name, and user emotion data

[0410] Step 3: Emotion Recognition and Data Acquisition

[0411] server

[0412] The server analyzes the request data and emotion data received from the device. For emotion analysis, it uses API-based emotion recognition software (e.g., "Affectiva").

[0413] The server determines the user's emotional state and retrieves relevant match data from a knowledge base (e.g., a sports database).

[0414] Example: If it is determined that the user is excited, data related to important play scenes and scoring scenes will be prioritized.

[0415] Input: Analyzed request data, emotion data

[0416] Output: Relevant match data (player performance, team past match results, current match status)

[0417] Step 4: Generate commentary text

[0418] server

[0419] The server generates commentary text based on the acquired match data using a RAG (Search Augmentation and Generation) module.

[0420] This text is tailored to the personality of the particular commentator and the emotions of the user.

[0421] Example: Generate "Player A hits a home run, and Team B scores 3 points," then adjust for an excited user to say "Great hit! Player A hits a home run, and Team B scores 3 points!"

[0422] Input: Related match data, user emotion data

[0423] Output: Adjusted commentary text

[0424] Step 5: Convert commentary text to audio

[0425] server

[0426] The server sends the adjusted commentary text to a Text-to-Speech (TTS) module, which converts it into audio data using a TTS service such as Amazon Polly or Google Cloud Text-to-Speech.

[0427] The generated voice data is tone-adjusted to match the user's emotional state.

[0428] Example: For an excited user, generate a voice in an upbeat tone saying, "Great hit! Player A's home run gives Team B three points!"

[0429] Input: Adjusted commentary text

[0430] Output: Voice data that matches the user's emotions

[0431] Step 6: Streaming audio data

[0432] server

[0433] The server transmits the generated audio data to the distribution system in real time.

[0434] The distribution system integrates and encodes the audio data with the game footage and provides it to the user's terminal in real time.

[0435] Example: Integrating audio data and game footage for real-time distribution.

[0436] Input: Audio data, match footage

[0437] Output: Real-time streaming stream

[0438] Step 7: Synchronize audio and video

[0439] Terminal

[0440] The device plays back the encoded audio data sent from the distribution system in real time, synchronizing it with the game footage to provide users with a realistic commentary experience.

[0441] Example: Audio and video are played synchronously on the user's device.

[0442] Input: Real-time streaming stream

[0443] Output: Synchronized audio and video commentary display

[0444] (Application example 2)

[0445] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0446] Conventional sports commentary systems do not take user emotions into account, resulting in a uniform viewing experience and no personalized commentary. Furthermore, they are unable to provide commentary at the appropriate time based on the user's emotions, making it difficult to maximize the sense of realism and excitement. Therefore, there is a demand for a more personalized viewing experience.

[0447] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0448] In this invention, the server includes means for receiving a request about a match from a user, means for acquiring match-related data from a knowledge base based on the received request, means for dynamically generating commentary text based on the acquired data and the user's emotional data, means for adjusting the generated text to match the user's emotional state and converting it into audio data, and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system, thereby enabling the provision of personalized commentary in real time that takes into account the user's emotions.

[0449] The "means for receiving a request from a user regarding a match" is a function that allows a user to input a request for live commentary of a match and for the system to receive the request.

[0450] "Means for obtaining match-related data from a knowledge base" is a function for searching and obtaining information related to a match from a knowledge base (database) based on a request.

[0451] "Means for dynamically generating commentary text based on acquired data and user emotional data" refers to a function for automatically generating appropriate commentary text in real time based on match-related data acquired from the knowledge base and the user's emotional state.

[0452] The "means for adjusting the generated text to match the emotional state and converting it into audio data" is a function for adjusting the content and tone of the live commentary text generated according to the user's emotions and converting the text into audio data.

[0453] "Means for providing converted voice data to a user terminal in real time in cooperation with a distribution system" is a function that cooperates with a distribution system in order to deliver voice data to a user terminal in real time.

[0454] "The personality of a particular commentator" refers to the commentator's unique way of speaking and expressing himself when providing live commentary.

[0455] "Emotion data" is data that represents the user's emotional state, and is information analyzed based on, for example, facial expressions and heart rate.

[0456] A "voice profile" is a set of settings and parameters used to generate voice data to reproduce the voice and tone of a particular speaker.

[0457] The present invention provides a system for providing personalized live sports commentary in real time that takes into account the user's emotions. The system includes the following main components and processing flow:

[0458] The server receives a match request sent from a user's terminal as a means of receiving a match-related request from the user. This request includes detailed information such as the match ID, team name, and player names. Upon receiving the request, the server performs a database search to obtain the relevant match-related data from the knowledge base.

[0459] At the same time, the user's device collects emotional data. This emotional data is analyzed using cameras and sensors built into the smartphone or smart glasses to analyze the user's facial expressions and heart rate. A facial expression recognition engine called "EmotionDetector" is used for the analysis.

[0460] The server generates commentary text in real time based on the acquired match-related data and emotional data. It uses a generative AI model called the "RAG model" to dynamically generate commentary text, which is adjusted to the user's emotional state.

[0461] Next, the generated text is converted into voice data using the "TTS module (pyttsx3)." The TTS module generates voice data based on a pre-set voice profile and adjusts the voice tone according to the user's emotional data. For example, if the user is happy, it creates voice data with a cheerful tone.

[0462] The generated audio data is provided in real time to user devices via a distribution system, where the audio is played back in synchronization with the game footage, providing users with personalized commentary.

[0463] A concrete example of this system is when a user sends a request saying, "I would like live commentary for this game," and the server generates a live commentary based on the user's emotional data, such as, "Great hit! Player A's home run gave Team B three points!" This allows users to enjoy live commentary in real time.

[0464] Example prompt sentence:

[0465] "Prompt for generating commentary when the user is happy while watching a game: 'The user is smiling. Please commentate on the good plays in the game.'"

[0466] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0467] Step 1:

[0468] The user inputs a request for a match and transmits the request from the terminal.

[0469] Input: The user inputs a match request (match ID, team name, player name, etc.).

[0470] Output: The device sends a request to the server.

[0471] Step 2:

[0472] The server receives the user's request and analyzes the request details.

[0473] Input: A match request sent from the device.

[0474] Output: Parsed request data (match ID, team name, player name, etc.).

[0475] Step 3:

[0476] The terminal acquires the user's emotion data using an emotion engine.

[0477] Input: User's facial expressions and heart rate data.

[0478] Output: Parsed user emotion data.

[0479] Specific operation: Using the device's camera and sensors, Emotion Detector analyzes facial expressions and heart rate.

[0480] Step 4:

[0481] The server retrieves relevant match data from a knowledge base based on the parsed request and emotion data.

[0482] Input: Parsed request data and sentiment data.

[0483] Output: Match-related data retrieved from the knowledge base (player performances, team statistics, current match situation, etc.).

[0484] Specific operation: The server performs a database search and extracts relevant match data.

[0485] Step 5:

[0486] The server acquires the match data and emotional data and generates commentary text based on the data.

[0487] Input: Obtained match data and emotion data.

[0488] Output: The generated commentary text.

[0489] Specific operation: The server generates commentary text using the RAG model and adjusts the text content to match the user's emotional state.

[0490] Step 6:

[0491] The server converts the generated commentary text into voice data using a TTS module.

[0492] Input: Generated commentary text.

[0493] Output: The generated audio data.

[0494] Specific operation: The server uses a TTS module (pyttsx3) to generate speech based on pre-configured voice profiles and emotion data.

[0495] Step 7:

[0496] The server provides the converted voice data to the user terminal in real time in cooperation with the distribution system.

[0497] Input: The converted audio data.

[0498] Output: Audio data played on the user's device.

[0499] Specific operation: The server transfers the audio data to the distribution system, and the audio is played on the user's terminal so that it is synchronized with the game video.

[0500] Step 8:

[0501] The user terminal plays the game video and audio commentary in sync.

[0502] Input: Audio data and game footage received from the distribution system.

[0503] Output: Audio commentary synchronized with match footage.

[0504] Specific operation: The user device simultaneously plays the received audio and video, providing personalized commentary in real time.

[0505] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0506] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0507] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0508] [Second embodiment]

[0509] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0510] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0511] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0512] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0513] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0514] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0515] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0516] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0517] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0518] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0519] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0520] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0521] MODE FOR CARRYING OUT THE INVENTION

[0522] Program processing flow explanation

[0523] This invention is a system that allows users to send requests about sports matches and provides real-time commentary based on the requests. The program processing of this system will be specifically described below.

[0524] Request received

[0525] User

[0526] A user uses the game broadcasting application to input and send a request for live commentary of a particular game, for example, "I would like live commentary of this game."

[0527] Terminal

[0528] The terminal receives the user's request and sends data including detailed information (e.g., match ID, team name, player name) to the server.

[0529] Data acquisition and preparation

[0530] server

[0531] The server analyzes the request received from the terminal and searches for relevant match data from a knowledge base (database).

[0532] This match data includes players' past performances, team statistics, current match status, past match results, and more.

[0533] Organize the data you obtain and convert it into a format that can be used in the next step.

[0534] Text generation

[0535] server

[0536] The server uses a RAG (Search Augmentation Generation) module to generate commentary text appropriate for a particular scene in real time based on the acquired data.

[0537] For example, it generates text such as "Player A hit a home run" or "Team B added three points."

[0538] The generated text is tailored to a particular commentator's personality, such as a calm, detailed commentator saying, "Great hit! Player A hit a home run and Team B added three runs."

[0539] Audio conversion

[0540] server

[0541] The server sends the generated commentary text to a TTS (Text-to-Speech) module, which converts it into voice data.

[0542] The TTS module generates voice data based on a pre-defined voice profile (e.g., a deep male voice).

[0543] For example, voice data such as "Great hit! Player A hit a home run, and Team B added three points" is generated.

[0544] Linking with distribution systems and distribution

[0545] server

[0546] The generated audio data is transferred to a distribution system and encoded in real time.

[0547] The server supports the distribution system in integrating audio data into game footage.

[0548] Terminal

[0549] The terminal receives encoded audio data from the distribution system and plays it back in synchronization with the game video.

[0550] Users can watch immersive real-time commentary on their devices.

[0551] Specific examples

[0552] Scenario: A live commentary of a decisive scoring moment in a baseball game

[0553] Request received

[0554] The user sends a request via the terminal saying, "I would like a live commentary of this baseball game."

[0555] Data Acquisition

[0556] The server analyzes the user's request and retrieves the player's performance, the team's past match results, and the current match situation from the match database.

[0557] Text Generation

[0558] The server uses the RAG module to generate the text "Player B hits a home run and Team C adds 3 runs."

[0559] This text can be tailored to fit the personality of a particular commentator, such as "Great hit! Team C scores three runs with a home run by Player B."

[0560] Audio conversion

[0561] The server sends this adjusted text to the TTS module, which converts it into voice data.

[0562] Generates audio data such as, "Great hit! Player B's home run gave Team C three more runs."

[0563] delivery

[0564] The server transfers the generated audio data to a distribution system in real time and distributes it to user terminals together with the game video.

[0565] The device plays audio and video in sync, allowing users to enjoy live commentary in real time.

[0566] This invention allows users to receive live commentary in real time with a sense of presence, making it possible to maximize the appeal of the match.

[0567] The processing flow will be explained below.

[0568] Step 1:

[0569] Using the game broadcasting application, a user inputs and submits a request for live commentary on a particular game, for example, "I would like live commentary on this game."

[0570] Step 2:

[0571] The device receives the user's request, analyzes the request details (match ID, team name, player name, etc.), and sends the analyzed request data to the server.

[0572] Step 3:

[0573] The server analyzes the requests received from the devices and searches for relevant match data in a knowledge base, which includes information such as players' past performances, team statistics, and the current match situation.

[0574] Step 4:

[0575] The server uses a RAG (Search Augmentation and Generation) module to organize the acquired game data and generate commentary text appropriate for a specific scene, for example, "Player A hit a home run and Team B scored three runs."

[0576] Step 5:

[0577] The server-generated text can be tailored to fit a specific commentator's personality, such as "Great hit! Player A hit a home run, and Team B added three runs."

[0578] Step 6:

[0579] The server sends the adjusted commentary text to a Text-to-Speech (TTS) module, which converts the text into speech data, such as "Great hit! Player A hit a home run, and Team B added three points."

[0580] Step 7:

[0581] The server transfers the generated audio data to a real-time streaming system, which encodes the audio data and synchronizes it with the game video.

[0582] Step 8:

[0583] The device plays back the encoded audio data received from the distribution system, synchronizing the audio with the game footage to provide users with immersive, real-time commentary.

[0584] Example 1

[0585] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0586] In modern sports management, real-time commentary is crucial to enhancing the appeal of matches. However, many systems struggle to provide fast and appropriate commentary for specific matches or scenes requested by users. Furthermore, problems arise with adjusting the commentary to suit the individual styles of commentators and converting the commentary into audio data. Therefore, there is a need for a more flexible system that can respond to individual user requests.

[0587] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0588] In this invention, the server includes means for receiving requests about a match from a user, means for retrieving match-related data from a knowledge base based on the received requests, means for using a generative AI model to generate commentary text in real time based on the retrieved data, means for tailoring the generated text to the personality of a specific commentator, means for converting the tailored text into audio data, and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system, thereby enabling flexible and appropriate commentary to be provided in real time based on user requests.

[0589] A "user" is a person who utilizes the system's game broadcast application to request live commentary about a particular game or team.

[0590] A "request for a match" is information that a user uses to inform the system that they would like live commentary on a specific match or scene.

[0591] A "knowledge base" is a database or collection of information containing game-related data, such as player performance, team statistics, current game situation, and past match results.

[0592] "Match-related data" refers to any information related to a match, such as player performance, team statistics, the current match situation, and past match results.

[0593] A "generative AI model" is an artificial intelligence model that generates live commentary text in real time based on acquired data.

[0594] "Live commentary text" is commentary text that follows the progress of the match and is generated based on the content requested by the user.

[0595] "The personality of a particular commentator" refers to the commentator's unique speaking style or manner of delivery, including, for example, calm commentary or enthusiastic commentary styles.

[0596] "Audio data" refers to an audio file that has been converted from the generated commentary text using text-to-speech (TTS) technology.

[0597] The "distribution system" is an infrastructure for synchronizing the generated audio data with the game video and providing it to user terminals in real time.

[0598] A "user terminal" is a device that a user uses to view live commentary of a match, and includes a smartphone, tablet, PC, etc.

[0599] "Flexible and appropriate live commentary" means responding quickly to individual user requests and generating and providing commentary that matches the progress of the game.

[0600] "Real-time" refers to the immediacy of time in which commentary is provided as the game progresses.

[0601] "Tuning" refers to the process of converting and modifying the generated commentary text to fit the personality and voice profile of a particular commentator.

[0602] "Game Footage" means a live video feed of an actual game or recorded game video.

[0603] MODE FOR CARRYING OUT THE INVENTION

[0604] The present invention is a system that allows users to submit requests for sports matches and provides real-time commentary based on the requests. Details of how to implement this system are described below.

[0605] Basic configuration

[0606] User

[0607] A user uses the game broadcasting application to input and send a request for commentary on a particular game or scene, for example, "I would like commentary on this baseball game."

[0608] Terminal

[0609] The user's device (smartphone, tablet, PC, etc.) receives the user's request and sends detailed information (match ID, team name, player names, etc.) in JSON format to the server.

[0610] server

[0611] The server analyzes the JSON data sent from the device, identifies the request, and then accesses the match database to retrieve relevant match data (player performance, team statistics, current match status, past match results, etc.).

[0612] The server generates commentary text in real time based on the acquired data. It uses a generative AI model to generate appropriate commentary based on the data. For example, it generates text such as "Player A hit a home run" or "Team B added three points."

[0613] The generated text is then tailored to fit the personality of a particular commentator, for example, a calm commentary style might say, "Great hit! Player A hit a home run, and Team B added three runs."

[0614] The adjusted text is sent to the Text-to-Speech (TTS) module, which converts it into speech data. The TTS module generates speech data based on a pre-defined voice profile (e.g., a deep male voice). For example, the generated speech data is, "Great hit! Player A hit a home run, and Team B scored three points."

[0615] The server then transfers the generated audio data to the distribution system, where it is encoded in real time. The encoded audio data is generated in a format that is synchronized with the game video.

[0616] Terminal

[0617] The user's device receives the encoded audio data from the distribution system and plays it in sync with the game footage, allowing the user to enjoy immersive, real-time commentary.

[0618] Specific examples

[0619] For example, consider a scenario where a user requests commentary on a baseball game, saying, "I'd like a commentator for this baseball game." The server receives this request and retrieves data from the game database, such as player performance, past team records, and the current game situation. It then uses a generative AI model to generate text like, "Player B hit a home run, and Team C added three runs," and adjusts it to fit the commentator's personality, such as, "Great hit! Player B hit a home run, and Team C added three runs." Finally, this adjusted text is sent to a TTS module, which converts it into voice data. The voice data is then distributed to the user's device in real time along with the game video.

[0620] Prompt Sentence Examples

[0621] "Please show us the process flow for this system, which generates commentary text in real time as the game progresses based on the match data requested by the user, and converts it into voice data using a TTS module."

[0622] In this way, this invention allows users to enjoy flexible and appropriate live commentary in real time, making it possible to maximize the appeal of the match.

[0623] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0624] Step 1:

[0625] User request submission

[0626] User: Using the match broadcasting application, the user inputs and submits a request for live commentary on a particular match or scene. For example, the user inputs the phrase "I would like live commentary on this match" and presses the submit button.

[0627] Input: The request text entered by the user.

[0628] Output: The request information is sent to the terminal.

[0629] Step 2:

[0630] Terminal request reception and server transmission

[0631] Terminal: The terminal receives the user's request, generates JSON data containing the details (match ID, team name, player name, etc.), and sends it to the server.

[0632] Input: User request information.

[0633] Output: The request data in JSON format that is sent to the server.

[0634] Step 3:

[0635] Server request analysis

[0636] Server: The server parses the received JSON data and identifies the request content. Specifically, it extracts information such as the match ID, team name, and player names.

[0637] Input: JSON data sent from the terminal.

[0638] Output: Analysis results such as match ID, team name, player name, etc.

[0639] Step 4:

[0640] Obtaining match data

[0641] Server: Based on the extracted request, the server searches and retrieves relevant match data (e.g., player performance, team statistics, current match situation, past match results, etc.) from its knowledge base.

[0642] Input: Analysis results (match ID, team name, player name, etc.).

[0643] Output: Match-related data obtained.

[0644] Step 5:

[0645] Data organization and format conversion

[0646] Server: Organizes the match data and converts it into JSON format for use in the next processing step.

[0647] Input: Retrieved match-related data.

[0648] Output: Match data converted to JSON format.

[0649] Step 6:

[0650] Generating commentary text

[0651] Server: The server uses a generative AI model to generate commentary text in real time based on the organized data, such as "Player A hit a home run" or "Team B scored three runs."

[0652] Input: Match data in JSON format.

[0653] Output: Text of the commentary.

[0654] Step 7:

[0655] Text adjustment to suit the commentator's personality

[0656] Server: Tailors the generated text to fit the personality of a particular commentator. For a calm commentary style, the text might be tailored to something like, "Great hit! Player A hit a home run and Team B added three runs."

[0657] Input: The generated commentary text.

[0658] Output: Adjusted commentary text.

[0659] Step 8:

[0660] Text-to-speech

[0661] Server: The adjusted text is sent to the TTS module, which converts it into voice data. The TTS module generates voice data based on the configured voice profile (e.g., a deep male voice).

[0662] Input: Adjusted commentary text.

[0663] Output: The generated audio data.

[0664] Step 9:

[0665] Preparing for audio data distribution

[0666] Server: The generated audio data is transferred to the distribution system and encoded in real time. The encoded audio data is generated in a format that is synchronized with the game video.

[0667] Input: The generated audio data.

[0668] Output: The encoded audio data.

[0669] Step 10:

[0670] Synchronized audio and video playback on devices

[0671] Terminal: The terminal receives the encoded audio data and plays it in sync with the game footage, allowing users to enjoy immersive real-time commentary.

[0672] Input: Encoded audio data

[0673] Output: Real-time commentary with synchronized audio and video.

[0674] (Application example 1)

[0675] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0676] Conventional sports commentary systems lack a way for users to request real-time commentary on specific scenes or players. They also struggle to provide realistic commentary in real time, limiting the user experience. Furthermore, commentary that reflects the commentator's individuality and real-time audio delivery are complex and ineffective, preventing the full appeal of the game.

[0677] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0678] In this invention, the server includes: means for receiving a request about a match from a user; means for acquiring match-related data from a knowledge base based on the received request; means for dynamically generating commentary text based on the acquired data; means for using a generative AI model to generate commentary text appropriate for a specific scene based on the user request; means for converting the generated text into audio data; and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system. This allows users to easily request real-time commentary about specific scenes or players, enabling them to enjoy immersive commentary in real time. Furthermore, customized commentary reflecting the commentator's personality can be provided to maximize the appeal of the match.

[0679] A "user" is an individual or group of people who desires and submits a request for live commentary on a match.

[0680] A "request" is information that a user sends to request commentary on a specific match or scene that the user desires.

[0681] A "knowledge base" is a database containing game-related data such as players' past performances, team statistics, the current game situation, and past match results.

[0682] "Match-related data" refers to all data necessary for live commentary, such as the progress of the match, player and team performance, and statistical information.

[0683] "Live commentary text" is textual information that describes the progress of the match or specific scenes.

[0684] A "generative AI model" is an artificial intelligence model used to generate commentary text appropriate for a specific scene or match situation based on a user request.

[0685] "Audio data" refers to data in audio format that has been converted from the generated live commentary text using a TTS module.

[0686] A "real-time delivery system" is a system for instantly encoding generated audio data and delivering it to a user terminal.

[0687] A "customized commentary" is a commentary that reflects the personality and tone of a particular commentator and is tailored to the user's request.

[0688] MODE FOR CARRYING OUT THE INVENTION

[0689] This invention is a system that allows users to request live commentary on a match in real time, generates commentary text based on the request, and provides it as audio data. An embodiment of this system will be described in detail below.

[0690] Hardware and software used

[0691] Server: Backend system for data acquisition, analysis, text generation, and speech generation

[0692] Database: Stores players' past performances, team statistics, match status, etc.

[0693] RAG module: A generative AI model for generating commentary text based on acquired data.

[0694] TTS module: Converts generated text into speech data

[0695] User devices: Smartphones, smart glasses, head-mounted displays, etc., for sending requests and viewing commentary

[0696] Data processing and calculation flow

[0697] Receiving requests and retrieving data

[0698] A user inputs and sends a request for live commentary of a game through a smartphone application. The user device then sends this request (including the game ID, team name, and player names) to the server.

[0699] The server analyzes the received request and retrieves relevant match data from the database, including player past performances, team statistics, the current match situation, past match results, etc. The retrieved data is then analyzed, organized, and converted into a format for generating play-by-play commentary.

[0700] Text Generation

[0701] The server uses the RAG module to generate commentary text based on the acquired data in real time, tailored to the specific scene, such as "Player X scores a goal, giving team Y the lead."

[0702] Using this generative AI model, we can generate commentary text appropriate for a specific scene based on the user's request. For example, we can use the following prompt sentence from the generative AI model to provide an emotional commentary:

[0703] Generate text to provide the user with real-time sports commentary based on the following data, where Player X scores a goal and Team Y takes the lead:

[0704] Past goal success rate of player X

[0705] Team Y's past win rate when leading

[0706] Current match situation

[0707] For customized commentary, please provide a detailed explanation in an excited tone.

[0708] Audio Conversion and Distribution

[0709] The generated text is sent from the server to the TTS module, which converts it into speech data, such as "Player X scores a goal, giving team Y the lead."

[0710] The server transfers the converted audio data to the distribution system in real time, encodes the audio data, and distributes it to the user's device. The device then plays the audio commentary in real time in sync with the game footage, allowing the user to enjoy a realistic live commentary.

[0711] Specific examples

[0712] Live commentary of decisive goals in soccer matches

[0713] A user launches the app and inputs and sends "I would like live commentary for this soccer match." The device sends the match ID and related data to the server, which then retrieves the match situation and past results from the database. The RAG module generates text such as "Player X scores a goal, giving team Y the lead," and the TTS module converts that text into voice data. The server provides the generated voice data to the user's device in real time, allowing the user to enjoy live commentary along with the match footage.

[0714] This invention allows users to easily request real-time commentary on specific scenes or players, making it possible to maximize the appeal of the match.

[0715] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0716] Step 1:

[0717] A user inputs and submits a request for live commentary on a match.

[0718] Specific behavior:

[0719] Users launch the application on their smartphone and enter the match ID, team name, player names, etc. of a specific match.

[0720] Check the request details and tap the send button.

[0721] Input: Match ID, team name, player name

[0722] Output: Request data

[0723] Step 2:

[0724] The terminal sends the user's request to the server.

[0725] Specific behavior:

[0726] The terminal receives the request data entered by the user and transmits it to the server as an HTTP request.

[0727] Input: Request data

[0728] Output: HTTP request to the server

[0729] Step 3:

[0730] The server analyzes the received request and retrieves the relevant match data from the knowledge base.

[0731] Specific behavior:

[0732] The server analyzes the request and searches and retrieves match-related data from the database based on the match ID, team name, and player name.

[0733] The necessary data to be obtained includes players' past performance, team statistics, current match situation, and past match results.

[0734] Input: Request data

[0735] Output: Match-related data

[0736] Step 4:

[0737] The server generates commentary text based on the acquired data using a generative AI model.

[0738] Specific behavior:

[0739] The server formats the match-related data it receives and inputs it as prompts to the generative AI model.

[0740] Prompt: "Generate text to provide the user with real-time sports commentary based on the following data: Player X scores a goal and Team Y takes the lead; Player X's past goal success rate; Team Y's past win rate when leading; Current game situation. For the customized commentary, please provide detailed commentary in an excited tone."

[0741] A generative AI model dynamically generates appropriate commentary text.

[0742] Input: Match-related data, prompt text

[0743] Output: Commentary text

[0744] Step 5:

[0745] The server sends the generated commentary text to the TTS module, which converts it into voice data.

[0746] Specific behavior:

[0747] The server sends the generated commentary text to the TTS module.

[0748] The TTS module generates speech data based on the text.

[0749] Input: Commentary text

[0750] Output: Audio data

[0751] Step 6:

[0752] The server provides the generated voice data to the user terminal in real time in cooperation with the distribution system.

[0753] Specific behavior:

[0754] The server transfers the audio data to the distribution system and encodes it in real time.

[0755] A distribution system encodes the audio data and distributes it to the user terminal.

[0756] Input: Audio data

[0757] Output: Encoded audio data

[0758] Step 7:

[0759] The terminal plays back the received audio data in synchronization with the game video.

[0760] Specific behavior:

[0761] The user terminal analyzes the audio data received from the distribution system and plays it back in synchronization with the game video.

[0762] Users can enjoy realistic, real-time commentary.

[0763] Input: Encoded audio data

[0764] Output: Synchronized audio and video playback

[0765] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0766] MODE FOR CARRYING OUT THE INVENTION

[0767] Program processing flow explanation

[0768] This invention is a system that allows users to send requests about sports games and provides real-time commentary based on those requests, and by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized commentary. The program processing of this system is explained in detail below.

[0769] Request received

[0770] User

[0771] Using the match broadcasting application, a user inputs and submits a request for live commentary on a particular match, for example, "I would like live commentary on this match."

[0772] Terminal

[0773] The terminal receives the user's request, analyzes the request details (match ID, team name, player name, etc.), and sends the analyzed request data to the server.

[0774] Emotion Recognition and Data Acquisition

[0775] server

[0776] The server analyzes the user's emotion data sent together with the request received from the terminal. The emotion data is generated by an emotion engine installed in the terminal.

[0777] The server searches the knowledge base for relevant game data based on the emotion data. For example, if the user is excited, data on important plays and scoring scenes will be prioritized.

[0778] The knowledge base includes players' past performances, team statistics, and current match situations.

[0779] Text generation

[0780] server

[0781] The server uses a RAG (Search Augmentation and Generation) module to organize the acquired match data and generate commentary text appropriate for a particular scene.

[0782] For example, "Player A hit a home run and Team B added three points."

[0783] The generated text is tailored to the specific commentator's personality and the user's emotions, for example, if the user is excited, it might say, "Great hit! Player A's home run gives Team B three runs!"

[0784] Audio conversion

[0785] server

[0786] The server sends the adjusted commentary text to a Text-to-Speech (TTS) module, which converts the text into voice data.

[0787] The TTS module generates voice data based on a pre-defined voice profile, and also includes the ability to change the voice profile depending on the user's emotions.

[0788] For example, voice data with a lively tone such as "Great hit! Player A's home run gave Team B three more points!" is generated.

[0789] Linking with distribution systems and distribution

[0790] server

[0791] The generated audio data is transferred to a distribution system and encoded in real time.

[0792] The server supports the distribution system in integrating audio data into game footage.

[0793] Terminal

[0794] The device plays back the encoded audio data received from the distribution system, synchronizing it with the game footage to provide users with immersive, real-time commentary.

[0795] Specific examples

[0796] Scenario: A live commentary of a decisive scoring moment in a baseball game

[0797] Request received

[0798] The user sends a request via the terminal saying, "I would like a live commentary of this baseball game."

[0799] The device analyzes the user's emotions (for example, heart rate and excitement level based on facial expression recognition) and sends this data to the server.

[0800] Data Acquisition

[0801] The server analyzes the user's emotional data and retrieves relevant match data (scoring scenes and important plays) from the knowledge base, prioritizing them according to the emotion.

[0802] The data obtained includes player performance, team past performances, and the current game situation.

[0803] Text Generation

[0804] The server uses the RAG module to generate the text "Player B hits a home run and Team C adds 3 runs."

[0805] This text can be adjusted to reflect the user's excitement level, such as "Great hit! Player B's home run gives Team C three more runs!"

[0806] Audio conversion

[0807] The server sends the adjusted text to the TTS module, which converts it into voice data in a lively tone that matches the user's emotions.

[0808] Generates audio data such as "Great hit! Player B's home run gave Team C three more points!"

[0809] delivery

[0810] The server transfers the generated audio data to the distribution system in real time and distributes it to the user's terminal together with the game video.

[0811] The device plays audio and video in sync, allowing users to enjoy personalized, exciting commentary in real time.

[0812] This invention provides a realistic commentary that takes into account the user's emotions, making it possible to maximize the appeal of the match.

[0813] The processing flow will be explained below.

[0814] Step 1:

[0815] Using the game broadcasting application, a user inputs and submits a request for live commentary on a particular game, for example, "I would like live commentary on this game."

[0816] Step 2:

[0817] The device receives the user's request and analyzes the request details (match ID, team name, player name, etc.). After analysis, the emotion engine analyzes the request data and the user's emotional data (e.g., heart rate, facial expression data), and sends them to the server.

[0818] Step 3:

[0819] The server analyzes the request data and emotional data received from the device. The emotional data includes information such as whether the user is excited, nervous, or relaxed.

[0820] Step 4:

[0821] The server searches the knowledge base based on the analyzed data to retrieve data related to the match. For example, if the user is excited, data related to important plays and scoring scenes will be retrieved first.

[0822] Step 5:

[0823] The server uses a RAG (Search Augmentation and Generation) module to organize the acquired game data and generate commentary text appropriate for a specific scene, for example, "Player A hit a home run and Team B scored three runs."

[0824] Step 6:

[0825] Tailor the server-generated text to the personality of a particular commentator and the user's emotions. For example, if the user is excited, tailor the text to "Great hit! Player A's home run gives Team B three runs!"

[0826] Step 7:

[0827] The server sends the adjusted commentary text to the TTS (Text-to-Speech) module, which converts the text into voice data. The TTS module generates voice data based on a pre-defined voice profile and changes the voice profile depending on the user's emotions. For example, it generates voice data with a lively tone, such as "Great hit! Player A's home run gave Team B three points!"

[0828] Step 8:

[0829] The server transfers the generated audio data to a real-time streaming system, which encodes the audio data and synchronizes it with the game video.

[0830] Step 9:

[0831] The device plays back the encoded audio data received from the distribution system, synchronizing the audio with the game footage to provide users with immersive, real-time commentary.

[0832] Specific examples

[0833] Scenario: A live commentary of a decisive scoring moment in a baseball game

[0834] Step 1:

[0835] The user sends a request via the terminal saying, "I would like a live commentary of this baseball game."

[0836] Step 2:

[0837] The device receives the user's request and analyzes details such as the match ID, team name, and player names. At the same time, the device's built-in emotion engine analyzes the user's heart rate and facial expression data, adds emotional information (e.g., excitement level), and sends it to the server.

[0838] Step 3:

[0839] The server receives the request and emotion data from the terminal and confirms that the user is in an excited state.

[0840] Step 4:

[0841] The server searches the knowledge base taking into account the emotional data and prioritizes retrieving match-related data (scoring scenes, important plays).

[0842] Step 5:

[0843] Based on the game data obtained by the server, the RAG module generates the text "Player B hit a home run and Team C added three points."

[0844] Step 6:

[0845] The server adjusts the generated text depending on the user's excitement level, such as "Great hit! Player B's home run gives Team C three more runs!"

[0846] Step 7:

[0847] The server sends the adjusted text to the TTS module, which generates voice data saying in a cheerful tone, "Great hit! Player B's home run gave Team C three more runs!"

[0848] Step 8:

[0849] The server transfers the generated audio data to a real-time distribution system and transmits it to the user's terminal together with the game video.

[0850] Step 9:

[0851] The device receives the audio data and plays it back in sync with the game footage, allowing users to enjoy personalized, exciting commentary in real time.

[0852] Example 2

[0853] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0854] Conventional sports commentary systems have been unable to provide personalized commentary that takes into account the user's emotions, resulting in a lack of realism and a lack of user experience. Specifically, it has been difficult to recognize important moments in a game and provide appropriate commentary in real time that reflects the user's emotional state.

[0855] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for receiving a request related to a game from a user; means for analyzing the received request and the user's emotion data; means for acquiring game-related data from a knowledge base based on the analysis result; means for dynamically generating commentary text based on the acquired data; means for converting the generated text into audio data based on a preset audio profile; and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system. This makes it possible to provide realistic commentary in real time that takes into account the user's emotions.

[0856] "User" refers to an individual who wishes to use the system to provide live commentary of a sports game.

[0857] A "game request" refers to a request sent by a user to the system requesting commentary on a particular sports game.

[0858] "Emotional data" refers to data that indicates the user's current emotional state, and refers to information obtained using, for example, a heart rate sensor or facial expression analysis software.

[0859] "Knowledge base" refers to the database in which the server stores match-related data, including information such as the match situation and player performance.

[0860] "Analyzing" refers to the process of deciphering the content and meaning of data based on received requests and emotional data, and extracting the necessary information.

[0861] "Game-Related Data" refers to data related to a particular sports game, such as statistics, current game status, and player or team performance.

[0862] "Dynamic generation of commentary text" refers to the process of automatically generating appropriate commentary text in real time based on acquired match-related data.

[0863] A "voice profile" is a profile that contains settings for a specific voice synthesis, including pre-defined voice tone, accent, etc.

[0864] "Converting to speech data" refers to the process of converting the generated text into speech using techniques that generate natural language speech from text.

[0865] "Delivery system" refers to the network and software infrastructure for transmitting generated audio data to user terminals in real time.

[0866] "User Device" means the electronic device used by a User to view the live commentary of a match, such as a smartphone, tablet, or PC.

[0867] The present invention provides a system that allows users to submit requests about sports matches and provides personalized commentary in real time. An embodiment of the system will be described in detail below.

[0868] User operation

[0869] When a user wants to receive live commentary of a sports game, he or she uses a game streaming application. The application has a search function, and the user selects a specific game, enters a request such as "I would like live commentary for this game," and submits it. For example, the user might enter "I would like live commentary for this baseball game." This operation also collects the user's emotional data (e.g., heart rate and facial expression data obtained through facial recognition) via the device.

[0870] Device Features

[0871] The device analyzes the request received from the user. The analysis targets the text data in the request and the acquired emotion data. As a result of the analysis, detailed information such as the match ID, team name, and player names is extracted. This information is then sent to the server. A natural language processing engine and an emotion recognition engine are used for the analysis. Specific examples include software such as "SpaCy" for natural language processing and "Affectiva" for emotion recognition.

[0872] Server Processing

[0873] The server analyzes the request data and emotion data received from the device. The emotion recognition engine used for the analysis is API-based emotion recognition software or a machine learning model (e.g., OpenAI's GPT-based emotion recognition model). The server determines the user's emotional state and searches a knowledge base for relevant match data based on that. The knowledge base consists of a sports database, including team and player statistics, past match results, and the current match situation.

[0874] The acquired data is then organized and a RAG (Search Augmentation and Generation) module is used to generate commentary text. For example, it generates text such as "Player A hits a home run, and Team B scores three runs." This text is tailored to the personality of the specific commentator and the emotions of the user. If the user is excited, it will be tailored to "Great hit! Player A hits a home run, and Team B scores three runs!"

[0875] The generated text is sent to a Text-to-Speech (TTS) module (e.g., Amazon Polly, Google Cloud Text-to-Speech) and converted into voice data. The TTS module generates voice data based on a pre-configured voice profile. The voice profile is adjusted according to the user's emotions, for example, to set a cheerful or calm tone.

[0876] The generated audio data is transferred to a distribution system and provided to user terminals in real time. The distribution system encodes the audio data in real time and distributes it together with the game video.

[0877] Specific examples

[0878] For example, if a user sends a request saying, "I would like a commentary of a baseball game," the server analyzes the user's emotions (e.g., heart rate and facial expression indicating excitement) and retrieves relevant data about the game. Using data about scoring moments and key plays, the server generates text such as, "Player A hits a home run, and Team B scores three runs." This text is then converted into lively audio data and delivered in real time. Users can enjoy immersive commentary in real time on their devices.

[0879] Prompt Sentence Examples

[0880] A prompt to the generative AI model to explain this system might look like this:

[0881] "If a user requests a live commentary of a sports game and the system recognizes the user's emotions and provides a personalized commentary, what would that be?"

[0882] In this way, the present invention takes into account the user's emotions and can enhance the realism and entertainment value of the match.

[0883] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0884] Step 1: Submitting a request

[0885] User

[0886] A user opens a game broadcasting application and enters a request for live commentary on a particular game.

[0887] Example: "I'd like to provide live commentary for this match."

[0888] Input: Request text for commentary on a specific sports game

[0889] Output: Request sent successfully

[0890] Step 2: Parsing and sending the request

[0891] Terminal

[0892] The terminal receives a request from a user and analyzes the text data and emotional data (e.g., heart rate, facial expression analysis) in the request.

[0893] Through analysis, detailed information such as the match ID, team name, and player name is extracted and sent to the server along with emotion data.

[0894] Example: A natural language processing engine (e.g., "SpaCy") analyzes the request and identifies the match ID and team name.

[0895] Input: Request text, emotion data

[0896] Output: Analysis results including match ID, team name, player name, and user emotion data

[0897] Step 3: Emotion Recognition and Data Acquisition

[0898] server

[0899] The server analyzes the request data and emotion data received from the device. For emotion analysis, it uses API-based emotion recognition software (e.g., "Affectiva").

[0900] The server determines the user's emotional state and retrieves relevant match data from a knowledge base (e.g., a sports database).

[0901] Example: If it is determined that the user is excited, data related to important play scenes and scoring scenes will be prioritized.

[0902] Input: Analyzed request data, emotion data

[0903] Output: Relevant match data (player performance, team past match results, current match status)

[0904] Step 4: Generate commentary text

[0905] server

[0906] The server generates commentary text based on the acquired match data using a RAG (Search Augmentation and Generation) module.

[0907] This text is tailored to the personality of the particular commentator and the emotions of the user.

[0908] Example: Generate "Player A hits a home run, and Team B scores 3 points," then adjust for an excited user to say "Great hit! Player A hits a home run, and Team B scores 3 points!"

[0909] Input: Related match data, user emotion data

[0910] Output: Adjusted commentary text

[0911] Step 5: Convert commentary text to audio

[0912] server

[0913] The server sends the adjusted commentary text to a Text-to-Speech (TTS) module, which converts it into audio data using a TTS service such as Amazon Polly or Google Cloud Text-to-Speech.

[0914] The generated voice data is tone-adjusted to match the user's emotional state.

[0915] Example: For an excited user, generate a voice in an upbeat tone saying, "Great hit! Player A's home run gives Team B three points!"

[0916] Input: Adjusted commentary text

[0917] Output: Voice data that matches the user's emotions

[0918] Step 6: Streaming audio data

[0919] server

[0920] The server transmits the generated audio data to the distribution system in real time.

[0921] The distribution system integrates and encodes the audio data with the game footage and provides it to the user's terminal in real time.

[0922] Example: Integrating audio data and game footage for real-time distribution.

[0923] Input: Audio data, match footage

[0924] Output: Real-time streaming stream

[0925] Step 7: Synchronize audio and video

[0926] Terminal

[0927] The device plays back the encoded audio data sent from the distribution system in real time, synchronizing it with the game footage to provide users with a realistic commentary experience.

[0928] Example: Audio and video are played synchronously on the user's device.

[0929] Input: Real-time streaming stream

[0930] Output: Synchronized audio and video commentary display

[0931] (Application example 2)

[0932] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0933] Conventional sports commentary systems do not take user emotions into account, resulting in a uniform viewing experience and no personalized commentary. Furthermore, they are unable to provide commentary at the appropriate time based on the user's emotions, making it difficult to maximize the sense of realism and excitement. Therefore, there is a demand for a more personalized viewing experience.

[0934] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0935] In this invention, the server includes means for receiving a request about a match from a user, means for acquiring match-related data from a knowledge base based on the received request, means for dynamically generating commentary text based on the acquired data and the user's emotional data, means for adjusting the generated text to match the user's emotional state and converting it into audio data, and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system, thereby enabling the provision of personalized commentary in real time that takes into account the user's emotions.

[0936] The "means for receiving a request from a user regarding a match" is a function that allows a user to input a request for live commentary of a match and for the system to receive the request.

[0937] "Means for obtaining match-related data from a knowledge base" is a function for searching and obtaining information related to a match from a knowledge base (database) based on a request.

[0938] "Means for dynamically generating commentary text based on acquired data and user emotional data" refers to a function for automatically generating appropriate commentary text in real time based on match-related data acquired from the knowledge base and the user's emotional state.

[0939] The "means for adjusting the generated text to match the emotional state and converting it into audio data" is a function for adjusting the content and tone of the live commentary text generated according to the user's emotions and converting the text into audio data.

[0940] "Means for providing converted voice data to a user terminal in real time in cooperation with a distribution system" is a function that cooperates with a distribution system in order to deliver voice data to a user terminal in real time.

[0941] "The personality of a particular commentator" refers to the commentator's unique way of speaking and expressing himself when providing live commentary.

[0942] "Emotion data" is data that represents the user's emotional state, and is information analyzed based on, for example, facial expressions and heart rate.

[0943] A "voice profile" is a set of settings and parameters used to generate voice data to reproduce the voice and tone of a particular speaker.

[0944] The present invention provides a system for providing personalized live sports commentary in real time that takes into account the user's emotions. The system includes the following main components and processing flow:

[0945] The server receives a match request sent from a user's terminal as a means of receiving a match-related request from the user. This request includes detailed information such as the match ID, team name, and player names. Upon receiving the request, the server performs a database search to obtain the relevant match-related data from the knowledge base.

[0946] At the same time, the user's device collects emotional data. This emotional data is analyzed using cameras and sensors built into the smartphone or smart glasses to analyze the user's facial expressions and heart rate. A facial expression recognition engine called "EmotionDetector" is used for the analysis.

[0947] The server generates commentary text in real time based on the acquired match-related data and emotional data. It uses a generative AI model called the "RAG model" to dynamically generate commentary text, which is adjusted to the user's emotional state.

[0948] Next, the generated text is converted into voice data using the "TTS module (pyttsx3)." The TTS module generates voice data based on a pre-set voice profile and adjusts the voice tone according to the user's emotional data. For example, if the user is happy, it creates voice data with a cheerful tone.

[0949] The generated audio data is provided in real time to user devices via a distribution system, where the audio is played back in synchronization with the game footage, providing users with personalized commentary.

[0950] A concrete example of this system is when a user sends a request saying, "I would like live commentary for this game," and the server generates a live commentary based on the user's emotional data, such as, "Great hit! Player A's home run gave Team B three points!" This allows users to enjoy live commentary in real time.

[0951] Example prompt sentence:

[0952] "Prompt for generating commentary when the user is happy while watching a game: 'The user is smiling. Please commentate on the good plays in the game.'"

[0953] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0954] Step 1:

[0955] The user inputs a request for a match and transmits the request from the terminal.

[0956] Input: The user inputs a match request (match ID, team name, player name, etc.).

[0957] Output: The device sends a request to the server.

[0958] Step 2:

[0959] The server receives the user's request and analyzes the request details.

[0960] Input: A match request sent from the device.

[0961] Output: Parsed request data (match ID, team name, player name, etc.).

[0962] Step 3:

[0963] The terminal acquires the user's emotion data using an emotion engine.

[0964] Input: User's facial expressions and heart rate data.

[0965] Output: Parsed user emotion data.

[0966] Specific operation: Using the device's camera and sensors, Emotion Detector analyzes facial expressions and heart rate.

[0967] Step 4:

[0968] The server retrieves relevant match data from a knowledge base based on the parsed request and emotion data.

[0969] Input: Parsed request data and sentiment data.

[0970] Output: Match-related data retrieved from the knowledge base (player performances, team statistics, current match situation, etc.).

[0971] Specific operation: The server performs a database search and extracts relevant match data.

[0972] Step 5:

[0973] The server acquires the match data and emotional data and generates commentary text based on the data.

[0974] Input: Obtained match data and emotion data.

[0975] Output: The generated commentary text.

[0976] Specific operation: The server generates commentary text using the RAG model and adjusts the text content to match the user's emotional state.

[0977] Step 6:

[0978] The server converts the generated commentary text into voice data using a TTS module.

[0979] Input: Generated commentary text.

[0980] Output: The generated audio data.

[0981] Specific operation: The server uses a TTS module (pyttsx3) to generate speech based on pre-configured voice profiles and emotion data.

[0982] Step 7:

[0983] The server provides the converted voice data to the user terminal in real time in cooperation with the distribution system.

[0984] Input: The converted audio data.

[0985] Output: Audio data played on the user's device.

[0986] Specific operation: The server transfers the audio data to the distribution system, and the audio is played on the user's terminal so that it is synchronized with the game video.

[0987] Step 8:

[0988] The user terminal plays the game video and audio commentary in sync.

[0989] Input: Audio data and game footage received from the distribution system.

[0990] Output: Audio commentary synchronized with match footage.

[0991] Specific operation: The user device simultaneously plays the received audio and video, providing personalized commentary in real time.

[0992] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0993] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0994] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0995] [Third embodiment]

[0996] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0997] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0998] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0999] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1000] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1001] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1002] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1003] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1004] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1005] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1006] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1007] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1008] MODE FOR CARRYING OUT THE INVENTION

[1009] Program processing flow explanation

[1010] This invention is a system that allows users to send requests about sports matches and provides real-time commentary based on the requests. The program processing of this system will be specifically described below.

[1011] Request received

[1012] User

[1013] A user uses the game broadcasting application to input and send a request for live commentary of a particular game, for example, "I would like live commentary of this game."

[1014] Terminal

[1015] The terminal receives the user's request and sends data including detailed information (e.g., match ID, team name, player name) to the server.

[1016] Data acquisition and preparation

[1017] server

[1018] The server analyzes the request received from the terminal and searches for relevant match data from a knowledge base (database).

[1019] This match data includes players' past performances, team statistics, current match status, past match results, and more.

[1020] Organize the data you obtain and convert it into a format that can be used in the next step.

[1021] Text generation

[1022] server

[1023] The server uses a RAG (Search Augmentation Generation) module to generate commentary text appropriate for a particular scene in real time based on the acquired data.

[1024] For example, it generates text such as "Player A hit a home run" or "Team B added three points."

[1025] The generated text is tailored to a particular commentator's personality, such as a calm, detailed commentator saying, "Great hit! Player A hit a home run and Team B added three runs."

[1026] Audio conversion

[1027] server

[1028] The server sends the generated commentary text to a TTS (Text-to-Speech) module, which converts it into voice data.

[1029] The TTS module generates voice data based on a pre-defined voice profile (e.g., a deep male voice).

[1030] For example, voice data such as "Great hit! Player A hit a home run, and Team B added three points" is generated.

[1031] Linking with distribution systems and distribution

[1032] server

[1033] The generated audio data is transferred to a distribution system and encoded in real time.

[1034] The server supports the distribution system in integrating audio data into game footage.

[1035] Terminal

[1036] The terminal receives encoded audio data from the distribution system and plays it back in synchronization with the game video.

[1037] Users can watch immersive real-time commentary on their devices.

[1038] Specific examples

[1039] Scenario: A live commentary of a decisive scoring moment in a baseball game

[1040] Request received

[1041] The user sends a request via the terminal saying, "I would like a live commentary of this baseball game."

[1042] Data Acquisition

[1043] The server analyzes the user's request and retrieves the player's performance, the team's past match results, and the current match situation from the match database.

[1044] Text Generation

[1045] The server uses the RAG module to generate the text "Player B hits a home run and Team C adds 3 runs."

[1046] This text can be tailored to fit the personality of a particular commentator, such as "Great hit! Team C scores three runs with a home run by Player B."

[1047] Audio conversion

[1048] The server sends this adjusted text to the TTS module, which converts it into voice data.

[1049] Generates audio data such as, "Great hit! Player B's home run gave Team C three more runs."

[1050] delivery

[1051] The server transfers the generated audio data to a distribution system in real time and distributes it to user terminals together with the game video.

[1052] The device plays audio and video in sync, allowing users to enjoy live commentary in real time.

[1053] This invention allows users to receive live commentary in real time with a sense of presence, making it possible to maximize the appeal of the match.

[1054] The processing flow will be explained below.

[1055] Step 1:

[1056] Using the game broadcasting application, a user inputs and submits a request for live commentary on a particular game, for example, "I would like live commentary on this game."

[1057] Step 2:

[1058] The device receives the user's request, analyzes the request details (match ID, team name, player name, etc.), and sends the analyzed request data to the server.

[1059] Step 3:

[1060] The server analyzes the requests received from the devices and searches for relevant match data in a knowledge base, which includes information such as players' past performances, team statistics, and the current match situation.

[1061] Step 4:

[1062] The server uses a RAG (Search Augmentation and Generation) module to organize the acquired game data and generate commentary text appropriate for a specific scene, for example, "Player A hit a home run and Team B scored three runs."

[1063] Step 5:

[1064] The server-generated text can be tailored to fit a specific commentator's personality, such as "Great hit! Player A hit a home run, and Team B added three runs."

[1065] Step 6:

[1066] The server sends the adjusted commentary text to a Text-to-Speech (TTS) module, which converts the text into speech data, such as "Great hit! Player A hit a home run, and Team B added three points."

[1067] Step 7:

[1068] The server transfers the generated audio data to a real-time streaming system, which encodes the audio data and synchronizes it with the game video.

[1069] Step 8:

[1070] The device plays back the encoded audio data received from the distribution system, synchronizing the audio with the game footage to provide users with immersive, real-time commentary.

[1071] Example 1

[1072] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1073] In modern sports management, real-time commentary is crucial to enhancing the appeal of matches. However, many systems struggle to provide fast and appropriate commentary for specific matches or scenes requested by users. Furthermore, problems arise with adjusting the commentary to suit the individual styles of commentators and converting the commentary into audio data. Therefore, there is a need for a more flexible system that can respond to individual user requests.

[1074] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1075] In this invention, the server includes means for receiving requests about a match from a user, means for retrieving match-related data from a knowledge base based on the received requests, means for using a generative AI model to generate commentary text in real time based on the retrieved data, means for tailoring the generated text to the personality of a specific commentator, means for converting the tailored text into audio data, and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system, thereby enabling flexible and appropriate commentary to be provided in real time based on user requests.

[1076] A "user" is a person who utilizes the system's game broadcast application to request live commentary about a particular game or team.

[1077] A "request for a match" is information that a user uses to inform the system that they would like live commentary on a specific match or scene.

[1078] A "knowledge base" is a database or collection of information containing game-related data, such as player performance, team statistics, current game situation, and past match results.

[1079] "Match-related data" refers to any information related to a match, such as player performance, team statistics, the current match situation, and past match results.

[1080] A "generative AI model" is an artificial intelligence model that generates live commentary text in real time based on acquired data.

[1081] "Live commentary text" is commentary text that follows the progress of the match and is generated based on the content requested by the user.

[1082] "The personality of a particular commentator" refers to the commentator's unique speaking style or manner of delivery, including, for example, calm commentary or enthusiastic commentary styles.

[1083] "Audio data" refers to an audio file that has been converted from the generated commentary text using text-to-speech (TTS) technology.

[1084] The "distribution system" is an infrastructure for synchronizing the generated audio data with the game video and providing it to user terminals in real time.

[1085] A "user terminal" is a device that a user uses to view live commentary of a match, and includes a smartphone, tablet, PC, etc.

[1086] "Flexible and appropriate live commentary" means responding quickly to individual user requests and generating and providing commentary that matches the progress of the game.

[1087] "Real-time" refers to the immediacy of time in which commentary is provided as the game progresses.

[1088] "Tuning" refers to the process of converting and modifying the generated commentary text to fit the personality and voice profile of a particular commentator.

[1089] "Game Footage" means a live video feed of an actual game or recorded game video.

[1090] MODE FOR CARRYING OUT THE INVENTION

[1091] The present invention is a system that allows users to submit requests for sports matches and provides real-time commentary based on the requests. Details of how to implement this system are described below.

[1092] Basic configuration

[1093] User

[1094] A user uses the game broadcasting application to input and send a request for commentary on a particular game or scene, for example, "I would like commentary on this baseball game."

[1095] Terminal

[1096] The user's device (smartphone, tablet, PC, etc.) receives the user's request and sends detailed information (match ID, team name, player names, etc.) in JSON format to the server.

[1097] server

[1098] The server analyzes the JSON data sent from the device, identifies the request, and then accesses the match database to retrieve relevant match data (player performance, team statistics, current match status, past match results, etc.).

[1099] The server generates commentary text in real time based on the acquired data. It uses a generative AI model to generate appropriate commentary based on the data. For example, it generates text such as "Player A hit a home run" or "Team B added three points."

[1100] The generated text is then tailored to fit the personality of a particular commentator, for example, a calm commentary style might say, "Great hit! Player A hit a home run, and Team B added three runs."

[1101] The adjusted text is sent to the Text-to-Speech (TTS) module, which converts it into speech data. The TTS module generates speech data based on a pre-defined voice profile (e.g., a deep male voice). For example, the generated speech data is, "Great hit! Player A hit a home run, and Team B scored three points."

[1102] The server then transfers the generated audio data to the distribution system, where it is encoded in real time. The encoded audio data is generated in a format that is synchronized with the game video.

[1103] Terminal

[1104] The user's device receives the encoded audio data from the distribution system and plays it in sync with the game footage, allowing the user to enjoy immersive, real-time commentary.

[1105] Specific examples

[1106] For example, consider a scenario where a user requests commentary on a baseball game, saying, "I'd like a commentator for this baseball game." The server receives this request and retrieves data from the game database, such as player performance, past team records, and the current game situation. It then uses a generative AI model to generate text like, "Player B hit a home run, and Team C added three runs," and adjusts it to fit the commentator's personality, such as, "Great hit! Player B hit a home run, and Team C added three runs." Finally, this adjusted text is sent to a TTS module, which converts it into voice data. The voice data is then distributed to the user's device in real time along with the game video.

[1107] Prompt Sentence Examples

[1108] "Please show us the process flow for this system, which generates commentary text in real time as the game progresses based on the match data requested by the user, and converts it into voice data using a TTS module."

[1109] In this way, this invention allows users to enjoy flexible and appropriate live commentary in real time, making it possible to maximize the appeal of the match.

[1110] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1111] Step 1:

[1112] User request submission

[1113] User: Using the match broadcasting application, the user inputs and submits a request for live commentary on a particular match or scene. For example, the user inputs the phrase "I would like live commentary on this match" and presses the submit button.

[1114] Input: The request text entered by the user.

[1115] Output: The request information is sent to the terminal.

[1116] Step 2:

[1117] Terminal request reception and server transmission

[1118] Terminal: The terminal receives the user's request, generates JSON data containing the details (match ID, team name, player name, etc.), and sends it to the server.

[1119] Input: User request information.

[1120] Output: The request data in JSON format that is sent to the server.

[1121] Step 3:

[1122] Server request analysis

[1123] Server: The server parses the received JSON data and identifies the request content. Specifically, it extracts information such as the match ID, team name, and player names.

[1124] Input: JSON data sent from the terminal.

[1125] Output: Analysis results such as match ID, team name, player name, etc.

[1126] Step 4:

[1127] Obtaining match data

[1128] Server: Based on the extracted request, the server searches and retrieves relevant match data (e.g., player performance, team statistics, current match situation, past match results, etc.) from its knowledge base.

[1129] Input: Analysis results (match ID, team name, player name, etc.).

[1130] Output: Match-related data obtained.

[1131] Step 5:

[1132] Data organization and format conversion

[1133] Server: Organizes the match data and converts it into JSON format for use in the next processing step.

[1134] Input: Retrieved match-related data.

[1135] Output: Match data converted to JSON format.

[1136] Step 6:

[1137] Generating commentary text

[1138] Server: The server uses a generative AI model to generate commentary text in real time based on the organized data, such as "Player A hit a home run" or "Team B scored three runs."

[1139] Input: Match data in JSON format.

[1140] Output: Text of the commentary.

[1141] Step 7:

[1142] Text adjustment to suit the commentator's personality

[1143] Server: Tailors the generated text to fit the personality of a particular commentator. For a calm commentary style, the text might be tailored to something like, "Great hit! Player A hit a home run and Team B added three runs."

[1144] Input: The generated commentary text.

[1145] Output: Adjusted commentary text.

[1146] Step 8:

[1147] Text-to-speech

[1148] Server: The adjusted text is sent to the TTS module, which converts it into voice data. The TTS module generates voice data based on the configured voice profile (e.g., a deep male voice).

[1149] Input: Adjusted commentary text.

[1150] Output: The generated audio data.

[1151] Step 9:

[1152] Preparing for audio data distribution

[1153] Server: The generated audio data is transferred to the distribution system and encoded in real time. The encoded audio data is generated in a format that is synchronized with the game video.

[1154] Input: The generated audio data.

[1155] Output: The encoded audio data.

[1156] Step 10:

[1157] Synchronized audio and video playback on devices

[1158] Terminal: The terminal receives the encoded audio data and plays it in sync with the game footage, allowing users to enjoy immersive real-time commentary.

[1159] Input: Encoded audio data

[1160] Output: Real-time commentary with synchronized audio and video.

[1161] (Application example 1)

[1162] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1163] Conventional sports commentary systems lack a way for users to request real-time commentary on specific scenes or players. They also struggle to provide realistic commentary in real time, limiting the user experience. Furthermore, commentary that reflects the commentator's individuality and real-time audio delivery are complex and ineffective, preventing the full appeal of the game.

[1164] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1165] In this invention, the server includes: means for receiving a request about a match from a user; means for acquiring match-related data from a knowledge base based on the received request; means for dynamically generating commentary text based on the acquired data; means for using a generative AI model to generate commentary text appropriate for a specific scene based on the user request; means for converting the generated text into audio data; and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system. This allows users to easily request real-time commentary about specific scenes or players, enabling them to enjoy immersive commentary in real time. Furthermore, customized commentary reflecting the commentator's personality can be provided to maximize the appeal of the match.

[1166] A "user" is an individual or group of people who desires and submits a request for live commentary on a match.

[1167] A "request" is information that a user sends to request commentary on a specific match or scene that the user desires.

[1168] A "knowledge base" is a database containing game-related data such as players' past performances, team statistics, the current game situation, and past match results.

[1169] "Match-related data" refers to all data necessary for live commentary, such as the progress of the match, player and team performance, and statistical information.

[1170] "Live commentary text" is textual information that describes the progress of the match or specific scenes.

[1171] A "generative AI model" is an artificial intelligence model used to generate commentary text appropriate for a specific scene or match situation based on a user request.

[1172] "Audio data" refers to data in audio format that has been converted from the generated live commentary text using a TTS module.

[1173] A "real-time delivery system" is a system for instantly encoding generated audio data and delivering it to a user terminal.

[1174] A "customized commentary" is a commentary that reflects the personality and tone of a particular commentator and is tailored to the user's request.

[1175] MODE FOR CARRYING OUT THE INVENTION

[1176] This invention is a system that allows users to request live commentary on a match in real time, generates commentary text based on the request, and provides it as audio data. An embodiment of this system will be described in detail below.

[1177] Hardware and software used

[1178] Server: Backend system for data acquisition, analysis, text generation, and speech generation

[1179] Database: Stores players' past performances, team statistics, match status, etc.

[1180] RAG module: A generative AI model for generating commentary text based on acquired data.

[1181] TTS module: Converts generated text into speech data

[1182] User devices: Smartphones, smart glasses, head-mounted displays, etc., for sending requests and viewing commentary

[1183] Data processing and calculation flow

[1184] Receiving requests and retrieving data

[1185] A user inputs and sends a request for live commentary of a game through a smartphone application. The user device then sends this request (including the game ID, team name, and player names) to the server.

[1186] The server analyzes the received request and retrieves relevant match data from the database, including player past performances, team statistics, the current match situation, past match results, etc. The retrieved data is then analyzed, organized, and converted into a format for generating play-by-play commentary.

[1187] Text Generation

[1188] The server uses the RAG module to generate commentary text based on the acquired data in real time, tailored to the specific scene, such as "Player X scores a goal, giving team Y the lead."

[1189] Using this generative AI model, we can generate commentary text appropriate for a specific scene based on the user's request. For example, we can use the following prompt sentence from the generative AI model to provide an emotional commentary:

[1190] Generate text to provide the user with real-time sports commentary based on the following data, where Player X scores a goal and Team Y takes the lead:

[1191] Past goal success rate of player X

[1192] Team Y's past win rate when leading

[1193] Current match situation

[1194] For customized commentary, please provide a detailed explanation in an excited tone.

[1195] Audio Conversion and Distribution

[1196] The generated text is sent from the server to the TTS module, which converts it into speech data, such as "Player X scores a goal, giving team Y the lead."

[1197] The server transfers the converted audio data to the distribution system in real time, encodes the audio data, and distributes it to the user's device. The device then plays the audio commentary in real time in sync with the game footage, allowing the user to enjoy a realistic live commentary.

[1198] Specific examples

[1199] Live commentary of decisive goals in soccer matches

[1200] A user launches the app and inputs and sends "I would like live commentary for this soccer match." The device sends the match ID and related data to the server, which then retrieves the match situation and past results from the database. The RAG module generates text such as "Player X scores a goal, giving team Y the lead," and the TTS module converts that text into voice data. The server provides the generated voice data to the user's device in real time, allowing the user to enjoy live commentary along with the match footage.

[1201] This invention allows users to easily request real-time commentary on specific scenes or players, making it possible to maximize the appeal of the match.

[1202] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1203] Step 1:

[1204] A user inputs and submits a request for live commentary on a match.

[1205] Specific behavior:

[1206] Users launch the application on their smartphone and enter the match ID, team name, player names, etc. of a specific match.

[1207] Check the request details and tap the send button.

[1208] Input: Match ID, team name, player name

[1209] Output: Request data

[1210] Step 2:

[1211] The terminal sends the user's request to the server.

[1212] Specific behavior:

[1213] The terminal receives the request data entered by the user and transmits it to the server as an HTTP request.

[1214] Input: Request data

[1215] Output: HTTP request to the server

[1216] Step 3:

[1217] The server analyzes the received request and retrieves the relevant match data from the knowledge base.

[1218] Specific behavior:

[1219] The server analyzes the request and searches and retrieves match-related data from the database based on the match ID, team name, and player name.

[1220] The necessary data to be obtained includes players' past performance, team statistics, current match situation, and past match results.

[1221] Input: Request data

[1222] Output: Match-related data

[1223] Step 4:

[1224] The server generates commentary text based on the acquired data using a generative AI model.

[1225] Specific behavior:

[1226] The server formats the match-related data it receives and inputs it as prompts to the generative AI model.

[1227] Prompt: "Generate text to provide the user with real-time sports commentary based on the following data: Player X scores a goal and Team Y takes the lead; Player X's past goal success rate; Team Y's past win rate when leading; Current game situation. For the customized commentary, please provide detailed commentary in an excited tone."

[1228] A generative AI model dynamically generates appropriate commentary text.

[1229] Input: Match-related data, prompt text

[1230] Output: Commentary text

[1231] Step 5:

[1232] The server sends the generated commentary text to the TTS module, which converts it into voice data.

[1233] Specific behavior:

[1234] The server sends the generated commentary text to the TTS module.

[1235] The TTS module generates speech data based on the text.

[1236] Input: Commentary text

[1237] Output: Audio data

[1238] Step 6:

[1239] The server provides the generated voice data to the user terminal in real time in cooperation with the distribution system.

[1240] Specific behavior:

[1241] The server transfers the audio data to the distribution system and encodes it in real time.

[1242] A distribution system encodes the audio data and distributes it to the user terminal.

[1243] Input: Audio data

[1244] Output: Encoded audio data

[1245] Step 7:

[1246] The terminal plays back the received audio data in synchronization with the game video.

[1247] Specific behavior:

[1248] The user terminal analyzes the audio data received from the distribution system and plays it back in synchronization with the game video.

[1249] Users can enjoy realistic, real-time commentary.

[1250] Input: Encoded audio data

[1251] Output: Synchronized audio and video playback

[1252] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1253] MODE FOR CARRYING OUT THE INVENTION

[1254] Program processing flow explanation

[1255] This invention is a system that allows users to send requests about sports games and provides real-time commentary based on those requests, and by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized commentary. The program processing of this system is explained in detail below.

[1256] Request received

[1257] User

[1258] Using the match broadcasting application, a user inputs and submits a request for live commentary on a particular match, for example, "I would like live commentary on this match."

[1259] Terminal

[1260] The terminal receives the user's request, analyzes the request details (match ID, team name, player name, etc.), and sends the analyzed request data to the server.

[1261] Emotion Recognition and Data Acquisition

[1262] server

[1263] The server analyzes the user's emotion data sent together with the request received from the terminal. The emotion data is generated by an emotion engine installed in the terminal.

[1264] The server searches the knowledge base for relevant game data based on the emotion data. For example, if the user is excited, data on important plays and scoring scenes will be prioritized.

[1265] The knowledge base includes players' past performances, team statistics, and current match situations.

[1266] Text generation

[1267] server

[1268] The server uses a RAG (Search Augmentation and Generation) module to organize the acquired match data and generate commentary text appropriate for a particular scene.

[1269] For example, "Player A hit a home run and Team B added three points."

[1270] The generated text is tailored to the specific commentator's personality and the user's emotions, for example, if the user is excited, it might say, "Great hit! Player A's home run gives Team B three runs!"

[1271] Audio conversion

[1272] server

[1273] The server sends the adjusted commentary text to a Text-to-Speech (TTS) module, which converts the text into voice data.

[1274] The TTS module generates voice data based on a pre-defined voice profile, and also includes the ability to change the voice profile depending on the user's emotions.

[1275] For example, voice data with a lively tone such as "Great hit! Player A's home run gave Team B three more points!" is generated.

[1276] Linking with distribution systems and distribution

[1277] server

[1278] The generated audio data is transferred to a distribution system and encoded in real time.

[1279] The server supports the distribution system in integrating audio data into game footage.

[1280] Terminal

[1281] The device plays back the encoded audio data received from the distribution system, synchronizing it with the game footage to provide users with immersive, real-time commentary.

[1282] Specific examples

[1283] Scenario: A live commentary of a decisive scoring moment in a baseball game

[1284] Request received

[1285] The user sends a request via the terminal saying, "I would like a live commentary of this baseball game."

[1286] The device analyzes the user's emotions (for example, heart rate and excitement level based on facial expression recognition) and sends this data to the server.

[1287] Data Acquisition

[1288] The server analyzes the user's emotional data and retrieves relevant match data (scoring scenes and important plays) from the knowledge base, prioritizing them according to the emotion.

[1289] The data obtained includes player performance, team past performances, and the current game situation.

[1290] Text Generation

[1291] The server uses the RAG module to generate the text "Player B hits a home run and Team C adds 3 runs."

[1292] This text can be adjusted to reflect the user's excitement level, such as "Great hit! Player B's home run gives Team C three more runs!"

[1293] Audio conversion

[1294] The server sends the adjusted text to the TTS module, which converts it into voice data in a lively tone that matches the user's emotions.

[1295] Generates audio data such as "Great hit! Player B's home run gave Team C three more points!"

[1296] delivery

[1297] The server transfers the generated audio data to the distribution system in real time and distributes it to the user's terminal together with the game video.

[1298] The device plays audio and video in sync, allowing users to enjoy personalized, exciting commentary in real time.

[1299] This invention provides a realistic commentary that takes into account the user's emotions, making it possible to maximize the appeal of the match.

[1300] The processing flow will be explained below.

[1301] Step 1:

[1302] Using the game broadcasting application, a user inputs and submits a request for live commentary on a particular game, for example, "I would like live commentary on this game."

[1303] Step 2:

[1304] The device receives the user's request and analyzes the request details (match ID, team name, player name, etc.). After analysis, the emotion engine analyzes the request data and the user's emotional data (e.g., heart rate, facial expression data), and sends them to the server.

[1305] Step 3:

[1306] The server analyzes the request data and emotional data received from the device. The emotional data includes information such as whether the user is excited, nervous, or relaxed.

[1307] Step 4:

[1308] The server searches the knowledge base based on the analyzed data to retrieve data related to the match. For example, if the user is excited, data related to important plays and scoring scenes will be retrieved first.

[1309] Step 5:

[1310] The server uses a RAG (Search Augmentation and Generation) module to organize the acquired game data and generate commentary text appropriate for a specific scene, for example, "Player A hit a home run and Team B scored three runs."

[1311] Step 6:

[1312] Tailor the server-generated text to the personality of a particular commentator and the user's emotions. For example, if the user is excited, tailor the text to "Great hit! Player A's home run gives Team B three runs!"

[1313] Step 7:

[1314] The server sends the adjusted commentary text to the TTS (Text-to-Speech) module, which converts the text into voice data. The TTS module generates voice data based on a pre-defined voice profile and changes the voice profile depending on the user's emotions. For example, it generates voice data with a lively tone, such as "Great hit! Player A's home run gave Team B three points!"

[1315] Step 8:

[1316] The server transfers the generated audio data to a real-time streaming system, which encodes the audio data and synchronizes it with the game video.

[1317] Step 9:

[1318] The device plays back the encoded audio data received from the distribution system, synchronizing the audio with the game footage to provide users with immersive, real-time commentary.

[1319] Specific examples

[1320] Scenario: A live commentary of a decisive scoring moment in a baseball game

[1321] Step 1:

[1322] The user sends a request via the terminal saying, "I would like a live commentary of this baseball game."

[1323] Step 2:

[1324] The device receives the user's request and analyzes details such as the match ID, team name, and player names. At the same time, the device's built-in emotion engine analyzes the user's heart rate and facial expression data, adds emotional information (e.g., excitement level), and sends it to the server.

[1325] Step 3:

[1326] The server receives the request and emotion data from the terminal and confirms that the user is in an excited state.

[1327] Step 4:

[1328] The server searches the knowledge base taking into account the emotional data and prioritizes retrieving match-related data (scoring scenes, important plays).

[1329] Step 5:

[1330] Based on the game data obtained by the server, the RAG module generates the text "Player B hit a home run and Team C added three points."

[1331] Step 6:

[1332] The server adjusts the generated text depending on the user's excitement level, such as "Great hit! Player B's home run gives Team C three more runs!"

[1333] Step 7:

[1334] The server sends the adjusted text to the TTS module, which generates voice data saying in a cheerful tone, "Great hit! Player B's home run gave Team C three more runs!"

[1335] Step 8:

[1336] The server transfers the generated audio data to a real-time distribution system and transmits it to the user's terminal together with the game video.

[1337] Step 9:

[1338] The device receives the audio data and plays it back in sync with the game footage, allowing users to enjoy personalized, exciting commentary in real time.

[1339] Example 2

[1340] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1341] Conventional sports commentary systems have been unable to provide personalized commentary that takes into account the user's emotions, resulting in a lack of realism and a lack of user experience. Specifically, it has been difficult to recognize important moments in a game and provide appropriate commentary in real time that reflects the user's emotional state.

[1342] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for receiving a request related to a game from a user; means for analyzing the received request and the user's emotion data; means for acquiring game-related data from a knowledge base based on the analysis result; means for dynamically generating commentary text based on the acquired data; means for converting the generated text into audio data based on a preset audio profile; and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system. This makes it possible to provide realistic commentary in real time that takes into account the user's emotions.

[1343] "User" refers to an individual who wishes to use the system to provide live commentary of a sports game.

[1344] A "game request" refers to a request sent by a user to the system requesting commentary on a particular sports game.

[1345] "Emotional data" refers to data that indicates the user's current emotional state, and refers to information obtained using, for example, a heart rate sensor or facial expression analysis software.

[1346] "Knowledge base" refers to the database in which the server stores match-related data, including information such as the match situation and player performance.

[1347] "Analyzing" refers to the process of deciphering the content and meaning of data based on received requests and emotional data, and extracting the necessary information.

[1348] "Game-Related Data" refers to data related to a particular sports game, such as statistics, current game status, and player or team performance.

[1349] "Dynamic generation of commentary text" refers to the process of automatically generating appropriate commentary text in real time based on acquired match-related data.

[1350] A "voice profile" is a profile that contains settings for a specific voice synthesis, including pre-defined voice tone, accent, etc.

[1351] "Converting to speech data" refers to the process of converting the generated text into speech using techniques that generate natural language speech from text.

[1352] "Delivery system" refers to the network and software infrastructure for transmitting generated audio data to user terminals in real time.

[1353] "User Device" means the electronic device used by a User to view the live commentary of a match, such as a smartphone, tablet, or PC.

[1354] The present invention provides a system that allows users to submit requests about sports matches and provides personalized commentary in real time. An embodiment of the system will be described in detail below.

[1355] User operation

[1356] When a user wants to receive live commentary of a sports game, he or she uses a game streaming application. The application has a search function, and the user selects a specific game, enters a request such as "I would like live commentary for this game," and submits it. For example, the user might enter "I would like live commentary for this baseball game." This operation also collects the user's emotional data (e.g., heart rate and facial expression data obtained through facial recognition) via the device.

[1357] Device Features

[1358] The device analyzes the request received from the user. The analysis targets the text data in the request and the acquired emotion data. As a result of the analysis, detailed information such as the match ID, team name, and player names is extracted. This information is then sent to the server. A natural language processing engine and an emotion recognition engine are used for the analysis. Specific examples include software such as "SpaCy" for natural language processing and "Affectiva" for emotion recognition.

[1359] Server Processing

[1360] The server analyzes the request data and emotion data received from the device. The emotion recognition engine used for the analysis is API-based emotion recognition software or a machine learning model (e.g., OpenAI's GPT-based emotion recognition model). The server determines the user's emotional state and searches a knowledge base for relevant match data based on that. The knowledge base consists of a sports database, including team and player statistics, past match results, and the current match situation.

[1361] The acquired data is then organized and a RAG (Search Augmentation and Generation) module is used to generate commentary text. For example, it generates text such as "Player A hits a home run, and Team B scores three runs." This text is tailored to the personality of the specific commentator and the emotions of the user. If the user is excited, it will be tailored to "Great hit! Player A hits a home run, and Team B scores three runs!"

[1362] The generated text is sent to a Text-to-Speech (TTS) module (e.g., Amazon Polly, Google Cloud Text-to-Speech) and converted into voice data. The TTS module generates voice data based on a pre-configured voice profile. The voice profile is adjusted according to the user's emotions, for example, to set a cheerful or calm tone.

[1363] The generated audio data is transferred to a distribution system and provided to user terminals in real time. The distribution system encodes the audio data in real time and distributes it together with the game video.

[1364] Specific examples

[1365] For example, if a user sends a request saying, "I would like a commentary of a baseball game," the server analyzes the user's emotions (e.g., heart rate and facial expression indicating excitement) and retrieves relevant data about the game. Using data about scoring moments and key plays, the server generates text such as, "Player A hits a home run, and Team B scores three runs." This text is then converted into lively audio data and delivered in real time. Users can enjoy immersive commentary in real time on their devices.

[1366] Prompt Sentence Examples

[1367] A prompt to the generative AI model to explain this system might look like this:

[1368] "If a user requests a live commentary of a sports game and the system recognizes the user's emotions and provides a personalized commentary, what would that be?"

[1369] In this way, the present invention takes into account the user's emotions and can enhance the realism and entertainment value of the match.

[1370] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1371] Step 1: Submitting a request

[1372] User

[1373] A user opens a game broadcasting application and enters a request for live commentary on a particular game.

[1374] Example: "I'd like to provide live commentary for this match."

[1375] Input: Request text for commentary on a specific sports game

[1376] Output: Request sent successfully

[1377] Step 2: Parsing and sending the request

[1378] Terminal

[1379] The terminal receives a request from a user and analyzes the text data and emotional data (e.g., heart rate, facial expression analysis) in the request.

[1380] Through analysis, detailed information such as the match ID, team name, and player name is extracted and sent to the server along with emotion data.

[1381] Example: A natural language processing engine (e.g., "SpaCy") analyzes the request and identifies the match ID and team name.

[1382] Input: Request text, emotion data

[1383] Output: Analysis results including match ID, team name, player name, and user emotion data

[1384] Step 3: Emotion Recognition and Data Acquisition

[1385] server

[1386] The server analyzes the request data and emotion data received from the device. For emotion analysis, it uses API-based emotion recognition software (e.g., "Affectiva").

[1387] The server determines the user's emotional state and retrieves relevant match data from a knowledge base (e.g., a sports database).

[1388] Example: If it is determined that the user is excited, data related to important play scenes and scoring scenes will be prioritized.

[1389] Input: Analyzed request data, emotion data

[1390] Output: Relevant match data (player performance, team past match results, current match status)

[1391] Step 4: Generate commentary text

[1392] server

[1393] The server generates commentary text based on the acquired match data using a RAG (Search Augmentation and Generation) module.

[1394] This text is tailored to the personality of the particular commentator and the emotions of the user.

[1395] Example: Generate "Player A hits a home run, and Team B scores 3 points," then adjust for an excited user to say "Great hit! Player A hits a home run, and Team B scores 3 points!"

[1396] Input: Related match data, user emotion data

[1397] Output: Adjusted commentary text

[1398] Step 5: Convert commentary text to audio

[1399] server

[1400] The server sends the adjusted commentary text to a Text-to-Speech (TTS) module, which converts it into audio data using a TTS service such as Amazon Polly or Google Cloud Text-to-Speech.

[1401] The generated voice data is tone-adjusted to match the user's emotional state.

[1402] Example: For an excited user, generate a voice in an upbeat tone saying, "Great hit! Player A's home run gives Team B three points!"

[1403] Input: Adjusted commentary text

[1404] Output: Voice data that matches the user's emotions

[1405] Step 6: Streaming audio data

[1406] server

[1407] The server transmits the generated audio data to the distribution system in real time.

[1408] The distribution system integrates and encodes the audio data with the game footage and provides it to the user's terminal in real time.

[1409] Example: Integrating audio data and game footage for real-time distribution.

[1410] Input: Audio data, match footage

[1411] Output: Real-time streaming stream

[1412] Step 7: Synchronize audio and video

[1413] Terminal

[1414] The device plays back the encoded audio data sent from the distribution system in real time, synchronizing it with the game footage to provide users with a realistic commentary experience.

[1415] Example: Audio and video are played synchronously on the user's device.

[1416] Input: Real-time streaming stream

[1417] Output: Synchronized audio and video commentary display

[1418] (Application example 2)

[1419] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1420] Conventional sports commentary systems do not take user emotions into account, resulting in a uniform viewing experience and no personalized commentary. Furthermore, they are unable to provide commentary at the appropriate time based on the user's emotions, making it difficult to maximize the sense of realism and excitement. Therefore, there is a demand for a more personalized viewing experience.

[1421] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1422] In this invention, the server includes means for receiving a request about a match from a user, means for acquiring match-related data from a knowledge base based on the received request, means for dynamically generating commentary text based on the acquired data and the user's emotional data, means for adjusting the generated text to match the user's emotional state and converting it into audio data, and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system, thereby enabling the provision of personalized commentary in real time that takes into account the user's emotions.

[1423] The "means for receiving a request from a user regarding a match" is a function that allows a user to input a request for live commentary of a match and for the system to receive the request.

[1424] "Means for obtaining match-related data from a knowledge base" is a function for searching and obtaining information related to a match from a knowledge base (database) based on a request.

[1425] "Means for dynamically generating commentary text based on acquired data and user emotional data" refers to a function for automatically generating appropriate commentary text in real time based on match-related data acquired from the knowledge base and the user's emotional state.

[1426] The "means for adjusting the generated text to match the emotional state and converting it into audio data" is a function for adjusting the content and tone of the live commentary text generated according to the user's emotions and converting the text into audio data.

[1427] "Means for providing converted voice data to a user terminal in real time in cooperation with a distribution system" is a function that cooperates with a distribution system in order to deliver voice data to a user terminal in real time.

[1428] "The personality of a particular commentator" refers to the commentator's unique way of speaking and expressing himself when providing live commentary.

[1429] "Emotion data" is data that represents the user's emotional state, and is information analyzed based on, for example, facial expressions and heart rate.

[1430] A "voice profile" is a set of settings and parameters used to generate voice data to reproduce the voice and tone of a particular speaker.

[1431] The present invention provides a system for providing personalized live sports commentary in real time that takes into account the user's emotions. The system includes the following main components and processing flow:

[1432] The server receives a match request sent from a user's terminal as a means of receiving a match-related request from the user. This request includes detailed information such as the match ID, team name, and player names. Upon receiving the request, the server performs a database search to obtain the relevant match-related data from the knowledge base.

[1433] At the same time, the user's device collects emotional data. This emotional data is analyzed using cameras and sensors built into the smartphone or smart glasses to analyze the user's facial expressions and heart rate. A facial expression recognition engine called "EmotionDetector" is used for the analysis.

[1434] The server generates commentary text in real time based on the acquired match-related data and emotional data. It uses a generative AI model called the "RAG model" to dynamically generate commentary text, which is adjusted to the user's emotional state.

[1435] Next, the generated text is converted into voice data using the "TTS module (pyttsx3)." The TTS module generates voice data based on a pre-set voice profile and adjusts the voice tone according to the user's emotional data. For example, if the user is happy, it creates voice data with a cheerful tone.

[1436] The generated audio data is provided in real time to user devices via a distribution system, where the audio is played back in synchronization with the game footage, providing users with personalized commentary.

[1437] A concrete example of this system is when a user sends a request saying, "I would like live commentary for this game," and the server generates a live commentary based on the user's emotional data, such as, "Great hit! Player A's home run gave Team B three points!" This allows users to enjoy live commentary in real time.

[1438] Example prompt sentence:

[1439] "Prompt for generating commentary when the user is happy while watching a game: 'The user is smiling. Please commentate on the good plays in the game.'"

[1440] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1441] Step 1:

[1442] The user inputs a request for a match and transmits the request from the terminal.

[1443] Input: The user inputs a match request (match ID, team name, player name, etc.).

[1444] Output: The device sends a request to the server.

[1445] Step 2:

[1446] The server receives the user's request and analyzes the request details.

[1447] Input: A match request sent from the device.

[1448] Output: Parsed request data (match ID, team name, player name, etc.).

[1449] Step 3:

[1450] The terminal acquires the user's emotion data using an emotion engine.

[1451] Input: User's facial expressions and heart rate data.

[1452] Output: Parsed user emotion data.

[1453] Specific operation: Using the device's camera and sensors, Emotion Detector analyzes facial expressions and heart rate.

[1454] Step 4:

[1455] The server retrieves relevant match data from a knowledge base based on the parsed request and emotion data.

[1456] Input: Parsed request data and sentiment data.

[1457] Output: Match-related data retrieved from the knowledge base (player performances, team statistics, current match situation, etc.).

[1458] Specific operation: The server performs a database search and extracts relevant match data.

[1459] Step 5:

[1460] The server acquires the match data and emotional data and generates commentary text based on the data.

[1461] Input: Obtained match data and emotion data.

[1462] Output: The generated commentary text.

[1463] Specific operation: The server generates commentary text using the RAG model and adjusts the text content to match the user's emotional state.

[1464] Step 6:

[1465] The server converts the generated commentary text into voice data using a TTS module.

[1466] Input: Generated commentary text.

[1467] Output: The generated audio data.

[1468] Specific operation: The server uses a TTS module (pyttsx3) to generate speech based on pre-configured voice profiles and emotion data.

[1469] Step 7:

[1470] The server provides the converted voice data to the user terminal in real time in cooperation with the distribution system.

[1471] Input: The converted audio data.

[1472] Output: Audio data played on the user's device.

[1473] Specific operation: The server transfers the audio data to the distribution system, and the audio is played on the user's terminal so that it is synchronized with the game video.

[1474] Step 8:

[1475] The user terminal plays the game video and audio commentary in sync.

[1476] Input: Audio data and game footage received from the distribution system.

[1477] Output: Audio commentary synchronized with match footage.

[1478] Specific operation: The user device simultaneously plays the received audio and video, providing personalized commentary in real time.

[1479] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1480] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1481] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1482] [Fourth embodiment]

[1483] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1484] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1485] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1486] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1487] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1488] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1489] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1490] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1491] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1492] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1493] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1494] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1495] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1496] MODE FOR CARRYING OUT THE INVENTION

[1497] Program processing flow explanation

[1498] This invention is a system that allows users to send requests about sports matches and provides real-time commentary based on the requests. The program processing of this system will be specifically described below.

[1499] Request received

[1500] User

[1501] A user uses the game broadcasting application to input and send a request for live commentary of a particular game, for example, "I would like live commentary of this game."

[1502] Terminal

[1503] The terminal receives the user's request and sends data including detailed information (e.g., match ID, team name, player name) to the server.

[1504] Data acquisition and preparation

[1505] server

[1506] The server analyzes the request received from the terminal and searches for relevant match data from a knowledge base (database).

[1507] This match data includes players' past performances, team statistics, current match status, past match results, and more.

[1508] Organize the data you obtain and convert it into a format that can be used in the next step.

[1509] Text generation

[1510] server

[1511] The server uses a RAG (Search Augmentation Generation) module to generate commentary text appropriate for a particular scene in real time based on the acquired data.

[1512] For example, it generates text such as "Player A hit a home run" or "Team B added three points."

[1513] The generated text is tailored to a particular commentator's personality, such as a calm, detailed commentator saying, "Great hit! Player A hit a home run and Team B added three runs."

[1514] Audio conversion

[1515] server

[1516] The server sends the generated commentary text to a TTS (Text-to-Speech) module, which converts it into voice data.

[1517] The TTS module generates voice data based on a pre-defined voice profile (e.g., a deep male voice).

[1518] For example, voice data such as "Great hit! Player A hit a home run, and Team B added three points" is generated.

[1519] Linking with distribution systems and distribution

[1520] server

[1521] The generated audio data is transferred to a distribution system and encoded in real time.

[1522] The server supports the distribution system in integrating audio data into game footage.

[1523] Terminal

[1524] The terminal receives encoded audio data from the distribution system and plays it back in synchronization with the game video.

[1525] Users can watch immersive real-time commentary on their devices.

[1526] Specific examples

[1527] Scenario: A live commentary of a decisive scoring moment in a baseball game

[1528] Request received

[1529] The user sends a request via the terminal saying, "I would like a live commentary of this baseball game."

[1530] Data Acquisition

[1531] The server analyzes the user's request and retrieves the player's performance, the team's past match results, and the current match situation from the match database.

[1532] Text Generation

[1533] The server uses the RAG module to generate the text "Player B hits a home run and Team C adds 3 runs."

[1534] This text can be tailored to fit the personality of a particular commentator, such as "Great hit! Team C scores three runs with a home run by Player B."

[1535] Audio conversion

[1536] The server sends this adjusted text to the TTS module, which converts it into voice data.

[1537] Generates audio data such as, "Great hit! Player B's home run gave Team C three more runs."

[1538] delivery

[1539] The server transfers the generated audio data to a distribution system in real time and distributes it to user terminals together with the game video.

[1540] The device plays audio and video in sync, allowing users to enjoy live commentary in real time.

[1541] This invention allows users to receive live commentary in real time with a sense of presence, making it possible to maximize the appeal of the match.

[1542] The processing flow will be explained below.

[1543] Step 1:

[1544] Using the game broadcasting application, a user inputs and submits a request for live commentary on a particular game, for example, "I would like live commentary on this game."

[1545] Step 2:

[1546] The device receives the user's request, analyzes the request details (match ID, team name, player name, etc.), and sends the analyzed request data to the server.

[1547] Step 3:

[1548] The server analyzes the requests received from the devices and searches for relevant match data in a knowledge base, which includes information such as players' past performances, team statistics, and the current match situation.

[1549] Step 4:

[1550] The server uses a RAG (Search Augmentation and Generation) module to organize the acquired game data and generate commentary text appropriate for a specific scene, for example, "Player A hit a home run and Team B scored three runs."

[1551] Step 5:

[1552] The server-generated text can be tailored to fit a specific commentator's personality, such as "Great hit! Player A hit a home run, and Team B added three runs."

[1553] Step 6:

[1554] The server sends the adjusted commentary text to a Text-to-Speech (TTS) module, which converts the text into speech data, such as "Great hit! Player A hit a home run, and Team B added three points."

[1555] Step 7:

[1556] The server transfers the generated audio data to a real-time streaming system, which encodes the audio data and synchronizes it with the game video.

[1557] Step 8:

[1558] The device plays back the encoded audio data received from the distribution system, synchronizing the audio with the game footage to provide users with immersive, real-time commentary.

[1559] Example 1

[1560] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1561] In modern sports management, real-time commentary is crucial to enhancing the appeal of matches. However, many systems struggle to provide fast and appropriate commentary for specific matches or scenes requested by users. Furthermore, problems arise with adjusting the commentary to suit the individual styles of commentators and converting the commentary into audio data. Therefore, there is a need for a more flexible system that can respond to individual user requests.

[1562] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1563] In this invention, the server includes means for receiving requests about a match from a user, means for retrieving match-related data from a knowledge base based on the received requests, means for using a generative AI model to generate commentary text in real time based on the retrieved data, means for tailoring the generated text to the personality of a specific commentator, means for converting the tailored text into audio data, and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system, thereby enabling flexible and appropriate commentary to be provided in real time based on user requests.

[1564] A "user" is a person who utilizes the system's game broadcast application to request live commentary about a particular game or team.

[1565] A "request for a match" is information that a user uses to inform the system that they would like live commentary on a specific match or scene.

[1566] A "knowledge base" is a database or collection of information containing game-related data, such as player performance, team statistics, current game situation, and past match results.

[1567] "Match-related data" refers to any information related to a match, such as player performance, team statistics, the current match situation, and past match results.

[1568] A "generative AI model" is an artificial intelligence model that generates live commentary text in real time based on acquired data.

[1569] "Live commentary text" is commentary text that follows the progress of the match and is generated based on the content requested by the user.

[1570] "The personality of a particular commentator" refers to the commentator's unique speaking style or manner of delivery, including, for example, calm commentary or enthusiastic commentary styles.

[1571] "Audio data" refers to an audio file that has been converted from the generated commentary text using text-to-speech (TTS) technology.

[1572] The "distribution system" is an infrastructure for synchronizing the generated audio data with the game video and providing it to user terminals in real time.

[1573] A "user terminal" is a device that a user uses to view live commentary of a match, and includes a smartphone, tablet, PC, etc.

[1574] "Flexible and appropriate live commentary" means responding quickly to individual user requests and generating and providing commentary that matches the progress of the game.

[1575] "Real-time" refers to the immediacy of time in which commentary is provided as the game progresses.

[1576] "Tuning" refers to the process of converting and modifying the generated commentary text to fit the personality and voice profile of a particular commentator.

[1577] "Game Footage" means a live video feed of an actual game or recorded game video.

[1578] MODE FOR CARRYING OUT THE INVENTION

[1579] The present invention is a system that allows users to submit requests for sports matches and provides real-time commentary based on the requests. Details of how to implement this system are described below.

[1580] Basic configuration

[1581] User

[1582] A user uses the game broadcasting application to input and send a request for commentary on a particular game or scene, for example, "I would like commentary on this baseball game."

[1583] Terminal

[1584] The user's device (smartphone, tablet, PC, etc.) receives the user's request and sends detailed information (match ID, team name, player names, etc.) in JSON format to the server.

[1585] server

[1586] The server analyzes the JSON data sent from the device, identifies the request, and then accesses the match database to retrieve relevant match data (player performance, team statistics, current match status, past match results, etc.).

[1587] The server generates commentary text in real time based on the acquired data. It uses a generative AI model to generate appropriate commentary based on the data. For example, it generates text such as "Player A hit a home run" or "Team B added three points."

[1588] The generated text is then tailored to fit the personality of a particular commentator, for example, a calm commentary style might say, "Great hit! Player A hit a home run, and Team B added three runs."

[1589] The adjusted text is sent to the Text-to-Speech (TTS) module, which converts it into speech data. The TTS module generates speech data based on a pre-defined voice profile (e.g., a deep male voice). For example, the generated speech data is, "Great hit! Player A hit a home run, and Team B scored three points."

[1590] The server then transfers the generated audio data to the distribution system, where it is encoded in real time. The encoded audio data is generated in a format that is synchronized with the game video.

[1591] Terminal

[1592] The user's device receives the encoded audio data from the distribution system and plays it in sync with the game footage, allowing the user to enjoy immersive, real-time commentary.

[1593] Specific examples

[1594] For example, consider a scenario where a user requests commentary on a baseball game, saying, "I'd like a commentator for this baseball game." The server receives this request and retrieves data from the game database, such as player performance, past team records, and the current game situation. It then uses a generative AI model to generate text like, "Player B hit a home run, and Team C added three runs," and adjusts it to fit the commentator's personality, such as, "Great hit! Player B hit a home run, and Team C added three runs." Finally, this adjusted text is sent to a TTS module, which converts it into voice data. The voice data is then distributed to the user's device in real time along with the game video.

[1595] Prompt Sentence Examples

[1596] "Please show us the process flow for this system, which generates commentary text in real time as the game progresses based on the match data requested by the user, and converts it into voice data using a TTS module."

[1597] In this way, this invention allows users to enjoy flexible and appropriate live commentary in real time, making it possible to maximize the appeal of the match.

[1598] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1599] Step 1:

[1600] User request submission

[1601] User: Using the match broadcasting application, the user inputs and submits a request for live commentary on a particular match or scene. For example, the user inputs the phrase "I would like live commentary on this match" and presses the submit button.

[1602] Input: The request text entered by the user.

[1603] Output: The request information is sent to the terminal.

[1604] Step 2:

[1605] Terminal request reception and server transmission

[1606] Terminal: The terminal receives the user's request, generates JSON data containing the details (match ID, team name, player name, etc.), and sends it to the server.

[1607] Input: User request information.

[1608] Output: The request data in JSON format that is sent to the server.

[1609] Step 3:

[1610] Server request analysis

[1611] Server: The server parses the received JSON data and identifies the request content. Specifically, it extracts information such as the match ID, team name, and player names.

[1612] Input: JSON data sent from the terminal.

[1613] Output: Analysis results such as match ID, team name, player name, etc.

[1614] Step 4:

[1615] Obtaining match data

[1616] Server: Based on the extracted request, the server searches and retrieves relevant match data (e.g., player performance, team statistics, current match situation, past match results, etc.) from its knowledge base.

[1617] Input: Analysis results (match ID, team name, player name, etc.).

[1618] Output: Match-related data obtained.

[1619] Step 5:

[1620] Data organization and format conversion

[1621] Server: Organizes the match data and converts it into JSON format for use in the next processing step.

[1622] Input: Retrieved match-related data.

[1623] Output: Match data converted to JSON format.

[1624] Step 6:

[1625] Generating commentary text

[1626] Server: The server uses a generative AI model to generate commentary text in real time based on the organized data, such as "Player A hit a home run" or "Team B scored three runs."

[1627] Input: Match data in JSON format.

[1628] Output: Text of the commentary.

[1629] Step 7:

[1630] Text adjustment to suit the commentator's personality

[1631] Server: Tailors the generated text to fit the personality of a particular commentator. For a calm commentary style, the text might be tailored to something like, "Great hit! Player A hit a home run and Team B added three runs."

[1632] Input: The generated commentary text.

[1633] Output: Adjusted commentary text.

[1634] Step 8:

[1635] Text-to-speech

[1636] Server: The adjusted text is sent to the TTS module, which converts it into voice data. The TTS module generates voice data based on the configured voice profile (e.g., a deep male voice).

[1637] Input: Adjusted commentary text.

[1638] Output: The generated audio data.

[1639] Step 9:

[1640] Preparing for audio data distribution

[1641] Server: The generated audio data is transferred to the distribution system and encoded in real time. The encoded audio data is generated in a format that is synchronized with the game video.

[1642] Input: The generated audio data.

[1643] Output: The encoded audio data.

[1644] Step 10:

[1645] Synchronized audio and video playback on devices

[1646] Terminal: The terminal receives the encoded audio data and plays it in sync with the game footage, allowing users to enjoy immersive real-time commentary.

[1647] Input: Encoded audio data

[1648] Output: Real-time commentary with synchronized audio and video.

[1649] (Application example 1)

[1650] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1651] Conventional sports commentary systems lack a way for users to request real-time commentary on specific scenes or players. They also struggle to provide realistic commentary in real time, limiting the user experience. Furthermore, commentary that reflects the commentator's individuality and real-time audio delivery are complex and ineffective, preventing the full appeal of the game.

[1652] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1653] In this invention, the server includes: means for receiving a request about a match from a user; means for acquiring match-related data from a knowledge base based on the received request; means for dynamically generating commentary text based on the acquired data; means for using a generative AI model to generate commentary text appropriate for a specific scene based on the user request; means for converting the generated text into audio data; and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system. This allows users to easily request real-time commentary about specific scenes or players, enabling them to enjoy immersive commentary in real time. Furthermore, customized commentary reflecting the commentator's personality can be provided to maximize the appeal of the match.

[1654] A "user" is an individual or group of people who desires and submits a request for live commentary on a match.

[1655] A "request" is information that a user sends to request commentary on a specific match or scene that the user desires.

[1656] A "knowledge base" is a database containing game-related data such as players' past performances, team statistics, the current game situation, and past match results.

[1657] "Match-related data" refers to all data necessary for live commentary, such as the progress of the match, player and team performance, and statistical information.

[1658] "Live commentary text" is textual information that describes the progress of the match or specific scenes.

[1659] A "generative AI model" is an artificial intelligence model used to generate commentary text appropriate for a specific scene or match situation based on a user request.

[1660] "Audio data" refers to data in audio format that has been converted from the generated live commentary text using a TTS module.

[1661] A "real-time delivery system" is a system for instantly encoding generated audio data and delivering it to a user terminal.

[1662] A "customized commentary" is a commentary that reflects the personality and tone of a particular commentator and is tailored to the user's request.

[1663] MODE FOR CARRYING OUT THE INVENTION

[1664] This invention is a system that allows users to request live commentary on a match in real time, generates commentary text based on the request, and provides it as audio data. An embodiment of this system will be described in detail below.

[1665] Hardware and software used

[1666] Server: Backend system for data acquisition, analysis, text generation, and speech generation

[1667] Database: Stores players' past performances, team statistics, match status, etc.

[1668] RAG module: A generative AI model for generating commentary text based on acquired data.

[1669] TTS module: Converts generated text into speech data

[1670] User devices: Smartphones, smart glasses, head-mounted displays, etc., for sending requests and viewing commentary

[1671] Data processing and calculation flow

[1672] Receiving requests and retrieving data

[1673] A user inputs and sends a request for live commentary of a game through a smartphone application. The user device then sends this request (including the game ID, team name, and player names) to the server.

[1674] The server analyzes the received request and retrieves relevant match data from the database, including player past performances, team statistics, the current match situation, past match results, etc. The retrieved data is then analyzed, organized, and converted into a format for generating play-by-play commentary.

[1675] Text Generation

[1676] The server uses the RAG module to generate commentary text based on the acquired data in real time, tailored to the specific scene, such as "Player X scores a goal, giving team Y the lead."

[1677] Using this generative AI model, we can generate commentary text appropriate for a specific scene based on the user's request. For example, we can use the following prompt sentence from the generative AI model to provide an emotional commentary:

[1678] Generate text to provide the user with real-time sports commentary based on the following data, where Player X scores a goal and Team Y takes the lead:

[1679] Past goal success rate of player X

[1680] Team Y's past win rate when leading

[1681] Current match situation

[1682] For customized commentary, please provide a detailed explanation in an excited tone.

[1683] Audio Conversion and Distribution

[1684] The generated text is sent from the server to the TTS module, which converts it into speech data, such as "Player X scores a goal, giving team Y the lead."

[1685] The server transfers the converted audio data to the distribution system in real time, encodes the audio data, and distributes it to the user's device. The device then plays the audio commentary in real time in sync with the game footage, allowing the user to enjoy a realistic live commentary.

[1686] Specific examples

[1687] Live commentary of decisive goals in soccer matches

[1688] A user launches the app and inputs and sends "I would like live commentary for this soccer match." The device sends the match ID and related data to the server, which then retrieves the match situation and past results from the database. The RAG module generates text such as "Player X scores a goal, giving team Y the lead," and the TTS module converts that text into voice data. The server provides the generated voice data to the user's device in real time, allowing the user to enjoy live commentary along with the match footage.

[1689] This invention allows users to easily request real-time commentary on specific scenes or players, making it possible to maximize the appeal of the match.

[1690] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1691] Step 1:

[1692] A user inputs and submits a request for live commentary on a match.

[1693] Specific behavior:

[1694] Users launch the application on their smartphone and enter the match ID, team name, player names, etc. of a specific match.

[1695] Check the request details and tap the send button.

[1696] Input: Match ID, team name, player name

[1697] Output: Request data

[1698] Step 2:

[1699] The terminal sends the user's request to the server.

[1700] Specific behavior:

[1701] The terminal receives the request data entered by the user and transmits it to the server as an HTTP request.

[1702] Input: Request data

[1703] Output: HTTP request to the server

[1704] Step 3:

[1705] The server analyzes the received request and retrieves the relevant match data from the knowledge base.

[1706] Specific behavior:

[1707] The server analyzes the request and searches and retrieves match-related data from the database based on the match ID, team name, and player name.

[1708] The necessary data to be obtained includes players' past performance, team statistics, current match situation, and past match results.

[1709] Input: Request data

[1710] Output: Match-related data

[1711] Step 4:

[1712] The server generates commentary text based on the acquired data using a generative AI model.

[1713] Specific behavior:

[1714] The server formats the match-related data it receives and inputs it as prompts to the generative AI model.

[1715] Prompt: "Generate text to provide the user with real-time sports commentary based on the following data: Player X scores a goal and Team Y takes the lead; Player X's past goal success rate; Team Y's past win rate when leading; Current game situation. For the customized commentary, please provide detailed commentary in an excited tone."

[1716] A generative AI model dynamically generates appropriate commentary text.

[1717] Input: Match-related data, prompt text

[1718] Output: Commentary text

[1719] Step 5:

[1720] The server sends the generated commentary text to the TTS module, which converts it into voice data.

[1721] Specific behavior:

[1722] The server sends the generated commentary text to the TTS module.

[1723] The TTS module generates speech data based on the text.

[1724] Input: Commentary text

[1725] Output: Audio data

[1726] Step 6:

[1727] The server provides the generated voice data to the user terminal in real time in cooperation with the distribution system.

[1728] Specific behavior:

[1729] The server transfers the audio data to the distribution system and encodes it in real time.

[1730] A distribution system encodes the audio data and distributes it to the user terminal.

[1731] Input: Audio data

[1732] Output: Encoded audio data

[1733] Step 7:

[1734] The terminal plays back the received audio data in synchronization with the game video.

[1735] Specific behavior:

[1736] The user terminal analyzes the audio data received from the distribution system and plays it back in synchronization with the game video.

[1737] Users can enjoy realistic, real-time commentary.

[1738] Input: Encoded audio data

[1739] Output: Synchronized audio and video playback

[1740] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1741] MODE FOR CARRYING OUT THE INVENTION

[1742] Program processing flow explanation

[1743] This invention is a system that allows users to send requests about sports games and provides real-time commentary based on those requests, and by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized commentary. The program processing of this system is explained in detail below.

[1744] Request received

[1745] User

[1746] Using the match broadcasting application, a user inputs and submits a request for live commentary on a particular match, for example, "I would like live commentary on this match."

[1747] Terminal

[1748] The terminal receives the user's request, analyzes the request details (match ID, team name, player name, etc.), and sends the analyzed request data to the server.

[1749] Emotion Recognition and Data Acquisition

[1750] server

[1751] The server analyzes the user's emotion data sent together with the request received from the terminal. The emotion data is generated by an emotion engine installed in the terminal.

[1752] The server searches the knowledge base for relevant game data based on the emotion data. For example, if the user is excited, data on important plays and scoring scenes will be prioritized.

[1753] The knowledge base includes players' past performances, team statistics, and current match situations.

[1754] Text generation

[1755] server

[1756] The server uses a RAG (Search Augmentation and Generation) module to organize the acquired match data and generate commentary text appropriate for a particular scene.

[1757] For example, "Player A hit a home run and Team B added three points."

[1758] The generated text is tailored to the specific commentator's personality and the user's emotions, for example, if the user is excited, it might say, "Great hit! Player A's home run gives Team B three runs!"

[1759] Audio conversion

[1760] server

[1761] The server sends the adjusted commentary text to a Text-to-Speech (TTS) module, which converts the text into voice data.

[1762] The TTS module generates voice data based on a pre-defined voice profile, and also includes the ability to change the voice profile depending on the user's emotions.

[1763] For example, voice data with a lively tone such as "Great hit! Player A's home run gave Team B three more points!" is generated.

[1764] Linking with distribution systems and distribution

[1765] server

[1766] The generated audio data is transferred to a distribution system and encoded in real time.

[1767] The server supports the distribution system in integrating audio data into game footage.

[1768] Terminal

[1769] The device plays back the encoded audio data received from the distribution system, synchronizing it with the game footage to provide users with immersive, real-time commentary.

[1770] Specific examples

[1771] Scenario: A live commentary of a decisive scoring moment in a baseball game

[1772] Request received

[1773] The user sends a request via the terminal saying, "I would like a live commentary of this baseball game."

[1774] The device analyzes the user's emotions (for example, heart rate and excitement level based on facial expression recognition) and sends this data to the server.

[1775] Data Acquisition

[1776] The server analyzes the user's emotional data and retrieves relevant match data (scoring scenes and important plays) from the knowledge base, prioritizing them according to the emotion.

[1777] The data obtained includes player performance, team past performances, and the current game situation.

[1778] Text Generation

[1779] The server uses the RAG module to generate the text "Player B hits a home run and Team C adds 3 runs."

[1780] This text can be adjusted to reflect the user's excitement level, such as "Great hit! Player B's home run gives Team C three more runs!"

[1781] Audio conversion

[1782] The server sends the adjusted text to the TTS module, which converts it into voice data in a lively tone that matches the user's emotions.

[1783] Generates audio data such as "Great hit! Player B's home run gave Team C three more points!"

[1784] delivery

[1785] The server transfers the generated audio data to the distribution system in real time and distributes it to the user's terminal together with the game video.

[1786] The device plays audio and video in sync, allowing users to enjoy personalized, exciting commentary in real time.

[1787] This invention provides a realistic commentary that takes into account the user's emotions, making it possible to maximize the appeal of the match.

[1788] The processing flow will be explained below.

[1789] Step 1:

[1790] Using the game broadcasting application, a user inputs and submits a request for live commentary on a particular game, for example, "I would like live commentary on this game."

[1791] Step 2:

[1792] The device receives the user's request and analyzes the request details (match ID, team name, player name, etc.). After analysis, the emotion engine analyzes the request data and the user's emotional data (e.g., heart rate, facial expression data), and sends them to the server.

[1793] Step 3:

[1794] The server analyzes the request data and emotional data received from the device. The emotional data includes information such as whether the user is excited, nervous, or relaxed.

[1795] Step 4:

[1796] The server searches the knowledge base based on the analyzed data to retrieve data related to the match. For example, if the user is excited, data related to important plays and scoring scenes will be retrieved first.

[1797] Step 5:

[1798] The server uses a RAG (Search Augmentation and Generation) module to organize the acquired game data and generate commentary text appropriate for a specific scene, for example, "Player A hit a home run and Team B scored three runs."

[1799] Step 6:

[1800] Tailor the server-generated text to the personality of a particular commentator and the user's emotions. For example, if the user is excited, tailor the text to "Great hit! Player A's home run gives Team B three runs!"

[1801] Step 7:

[1802] The server sends the adjusted commentary text to the TTS (Text-to-Speech) module, which converts the text into voice data. The TTS module generates voice data based on a pre-defined voice profile and changes the voice profile depending on the user's emotions. For example, it generates voice data with a lively tone, such as "Great hit! Player A's home run gave Team B three points!"

[1803] Step 8:

[1804] The server transfers the generated audio data to a real-time streaming system, which encodes the audio data and synchronizes it with the game video.

[1805] Step 9:

[1806] The device plays back the encoded audio data received from the distribution system, synchronizing the audio with the game footage to provide users with immersive, real-time commentary.

[1807] Specific examples

[1808] Scenario: A live commentary of a decisive scoring moment in a baseball game

[1809] Step 1:

[1810] The user sends a request via the terminal saying, "I would like a live commentary of this baseball game."

[1811] Step 2:

[1812] The device receives the user's request and analyzes details such as the match ID, team name, and player names. At the same time, the device's built-in emotion engine analyzes the user's heart rate and facial expression data, adds emotional information (e.g., excitement level), and sends it to the server.

[1813] Step 3:

[1814] The server receives the request and emotion data from the terminal and confirms that the user is in an excited state.

[1815] Step 4:

[1816] The server searches the knowledge base taking into account the emotional data and prioritizes retrieving match-related data (scoring scenes, important plays).

[1817] Step 5:

[1818] Based on the game data obtained by the server, the RAG module generates the text "Player B hit a home run and Team C added three points."

[1819] Step 6:

[1820] The server adjusts the generated text depending on the user's excitement level, such as "Great hit! Player B's home run gives Team C three more runs!"

[1821] Step 7:

[1822] The server sends the adjusted text to the TTS module, which generates voice data saying in a cheerful tone, "Great hit! Player B's home run gave Team C three more runs!"

[1823] Step 8:

[1824] The server transfers the generated audio data to a real-time distribution system and transmits it to the user's terminal together with the game video.

[1825] Step 9:

[1826] The device receives the audio data and plays it back in sync with the game footage, allowing users to enjoy personalized, exciting commentary in real time.

[1827] Example 2

[1828] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1829] Conventional sports commentary systems have been unable to provide personalized commentary that takes into account the user's emotions, resulting in a lack of realism and a lack of user experience. Specifically, it has been difficult to recognize important moments in a game and provide appropriate commentary in real time that reflects the user's emotional state.

[1830] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for receiving a request related to a game from a user; means for analyzing the received request and the user's emotion data; means for acquiring game-related data from a knowledge base based on the analysis result; means for dynamically generating commentary text based on the acquired data; means for converting the generated text into audio data based on a preset audio profile; and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system. This makes it possible to provide realistic commentary in real time that takes into account the user's emotions.

[1831] "User" refers to an individual who wishes to use the system to provide live commentary of a sports game.

[1832] A "game request" refers to a request sent by a user to the system requesting commentary on a particular sports game.

[1833] "Emotional data" refers to data that indicates the user's current emotional state, and refers to information obtained using, for example, a heart rate sensor or facial expression analysis software.

[1834] "Knowledge base" refers to the database in which the server stores match-related data, including information such as the match situation and player performance.

[1835] "Analyzing" refers to the process of deciphering the content and meaning of data based on received requests and emotional data, and extracting the necessary information.

[1836] "Game-Related Data" refers to data related to a particular sports game, such as statistics, current game status, and player or team performance.

[1837] "Dynamic generation of commentary text" refers to the process of automatically generating appropriate commentary text in real time based on acquired match-related data.

[1838] A "voice profile" is a profile that contains settings for a specific voice synthesis, including pre-defined voice tone, accent, etc.

[1839] "Converting to speech data" refers to the process of converting the generated text into speech using techniques that generate natural language speech from text.

[1840] "Delivery system" refers to the network and software infrastructure for transmitting generated audio data to user terminals in real time.

[1841] "User Device" means the electronic device used by a User to view the live commentary of a match, such as a smartphone, tablet, or PC.

[1842] The present invention provides a system that allows users to submit requests about sports matches and provides personalized commentary in real time. An embodiment of the system will be described in detail below.

[1843] User operation

[1844] When a user wants to receive live commentary of a sports game, he or she uses a game streaming application. The application has a search function, and the user selects a specific game, enters a request such as "I would like live commentary for this game," and submits it. For example, the user might enter "I would like live commentary for this baseball game." This operation also collects the user's emotional data (e.g., heart rate and facial expression data obtained through facial recognition) via the device.

[1845] Device Features

[1846] The device analyzes the request received from the user. The analysis targets the text data in the request and the acquired emotion data. As a result of the analysis, detailed information such as the match ID, team name, and player names is extracted. This information is then sent to the server. A natural language processing engine and an emotion recognition engine are used for the analysis. Specific examples include software such as "SpaCy" for natural language processing and "Affectiva" for emotion recognition.

[1847] Server Processing

[1848] The server analyzes the request data and emotion data received from the device. The emotion recognition engine used for the analysis is API-based emotion recognition software or a machine learning model (e.g., OpenAI's GPT-based emotion recognition model). The server determines the user's emotional state and searches a knowledge base for relevant match data based on that. The knowledge base consists of a sports database, including team and player statistics, past match results, and the current match situation.

[1849] The acquired data is then organized and a RAG (Search Augmentation and Generation) module is used to generate commentary text. For example, it generates text such as "Player A hits a home run, and Team B scores three runs." This text is tailored to the personality of the specific commentator and the emotions of the user. If the user is excited, it will be tailored to "Great hit! Player A hits a home run, and Team B scores three runs!"

[1850] The generated text is sent to a Text-to-Speech (TTS) module (e.g., Amazon Polly, Google Cloud Text-to-Speech) and converted into voice data. The TTS module generates voice data based on a pre-configured voice profile. The voice profile is adjusted according to the user's emotions, for example, to set a cheerful or calm tone.

[1851] The generated audio data is transferred to a distribution system and provided to user terminals in real time. The distribution system encodes the audio data in real time and distributes it together with the game video.

[1852] Specific examples

[1853] For example, if a user sends a request saying, "I would like a commentary of a baseball game," the server analyzes the user's emotions (e.g., heart rate and facial expression indicating excitement) and retrieves relevant data about the game. Using data about scoring moments and key plays, the server generates text such as, "Player A hits a home run, and Team B scores three runs." This text is then converted into lively audio data and delivered in real time. Users can enjoy immersive commentary in real time on their devices.

[1854] Prompt Sentence Examples

[1855] A prompt to the generative AI model to explain this system might look like this:

[1856] "If a user requests a live commentary of a sports game and the system recognizes the user's emotions and provides a personalized commentary, what would that be?"

[1857] In this way, the present invention takes into account the user's emotions and can enhance the realism and entertainment value of the match.

[1858] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1859] Step 1: Submitting a request

[1860] User

[1861] A user opens a game broadcasting application and enters a request for live commentary on a particular game.

[1862] Example: "I'd like to provide live commentary for this match."

[1863] Input: Request text for commentary on a specific sports game

[1864] Output: Request sent successfully

[1865] Step 2: Parsing and sending the request

[1866] Terminal

[1867] The terminal receives a request from a user and analyzes the text data and emotional data (e.g., heart rate, facial expression analysis) in the request.

[1868] Through analysis, detailed information such as the match ID, team name, and player name is extracted and sent to the server along with emotion data.

[1869] Example: A natural language processing engine (e.g., "SpaCy") analyzes the request and identifies the match ID and team name.

[1870] Input: Request text, emotion data

[1871] Output: Analysis results including match ID, team name, player name, and user emotion data

[1872] Step 3: Emotion Recognition and Data Acquisition

[1873] server

[1874] The server analyzes the request data and emotion data received from the device. For emotion analysis, it uses API-based emotion recognition software (e.g., "Affectiva").

[1875] The server determines the user's emotional state and retrieves relevant match data from a knowledge base (e.g., a sports database).

[1876] Example: If it is determined that the user is excited, data related to important play scenes and scoring scenes will be prioritized.

[1877] Input: Analyzed request data, emotion data

[1878] Output: Relevant match data (player performance, team past match results, current match status)

[1879] Step 4: Generate commentary text

[1880] server

[1881] The server generates commentary text based on the acquired match data using a RAG (Search Augmentation and Generation) module.

[1882] This text is tailored to the personality of the particular commentator and the emotions of the user.

[1883] Example: Generate "Player A hits a home run, and Team B scores 3 points," then adjust for an excited user to say "Great hit! Player A hits a home run, and Team B scores 3 points!"

[1884] Input: Related match data, user emotion data

[1885] Output: Adjusted commentary text

[1886] Step 5: Convert commentary text to audio

[1887] server

[1888] The server sends the adjusted commentary text to a Text-to-Speech (TTS) module, which converts it into audio data using a TTS service such as Amazon Polly or Google Cloud Text-to-Speech.

[1889] The generated voice data is tone-adjusted to match the user's emotional state.

[1890] Example: For an excited user, generate a voice in an upbeat tone saying, "Great hit! Player A's home run gives Team B three points!"

[1891] Input: Adjusted commentary text

[1892] Output: Voice data that matches the user's emotions

[1893] Step 6: Streaming audio data

[1894] server

[1895] The server transmits the generated audio data to the distribution system in real time.

[1896] The distribution system integrates and encodes the audio data with the game footage and provides it to the user's terminal in real time.

[1897] Example: Integrating audio data and game footage for real-time distribution.

[1898] Input: Audio data, match footage

[1899] Output: Real-time streaming stream

[1900] Step 7: Synchronize audio and video

[1901] Terminal

[1902] The device plays back the encoded audio data sent from the distribution system in real time, synchronizing it with the game footage to provide users with a realistic commentary experience.

[1903] Example: Audio and video are played synchronously on the user's device.

[1904] Input: Real-time streaming stream

[1905] Output: Synchronized audio and video commentary display

[1906] (Application example 2)

[1907] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1908] Conventional sports commentary systems do not take user emotions into account, resulting in a uniform viewing experience and no personalized commentary. Furthermore, they are unable to provide commentary at the appropriate time based on the user's emotions, making it difficult to maximize the sense of realism and excitement. Therefore, there is a demand for a more personalized viewing experience.

[1909] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1910] In this invention, the server includes means for receiving a request about a match from a user, means for acquiring match-related data from a knowledge base based on the received request, means for dynamically generating commentary text based on the acquired data and the user's emotional data, means for adjusting the generated text to match the user's emotional state and converting it into audio data, and means for providing the converted audio data to a user terminal in real time in cooperation with a distribution system, thereby enabling the provision of personalized commentary in real time that takes into account the user's emotions.

[1911] The "means for receiving a request from a user regarding a match" is a function that allows a user to input a request for live commentary of a match and for the system to receive the request.

[1912] "Means for obtaining match-related data from a knowledge base" is a function for searching and obtaining information related to a match from a knowledge base (database) based on a request.

[1913] "Means for dynamically generating commentary text based on acquired data and user emotional data" refers to a function for automatically generating appropriate commentary text in real time based on match-related data acquired from the knowledge base and the user's emotional state.

[1914] The "means for adjusting the generated text to match the emotional state and converting it into audio data" is a function for adjusting the content and tone of the live commentary text generated according to the user's emotions and converting the text into audio data.

[1915] "Means for providing converted voice data to a user terminal in real time in cooperation with a distribution system" is a function that cooperates with a distribution system in order to deliver voice data to a user terminal in real time.

[1916] "The personality of a particular commentator" refers to the commentator's unique way of speaking and expressing himself when providing live commentary.

[1917] "Emotion data" is data that represents the user's emotional state, and is information analyzed based on, for example, facial expressions and heart rate.

[1918] A "voice profile" is a set of settings and parameters used to generate voice data to reproduce the voice and tone of a particular speaker.

[1919] The present invention provides a system for providing personalized live sports commentary in real time that takes into account the user's emotions. The system includes the following main components and processing flow:

[1920] The server receives a match request sent from a user's terminal as a means of receiving a match-related request from the user. This request includes detailed information such as the match ID, team name, and player names. Upon receiving the request, the server performs a database search to obtain the relevant match-related data from the knowledge base.

[1921] At the same time, the user's device collects emotional data. This emotional data is analyzed using cameras and sensors built into the smartphone or smart glasses to analyze the user's facial expressions and heart rate. A facial expression recognition engine called "EmotionDetector" is used for the analysis.

[1922] The server generates commentary text in real time based on the acquired match-related data and emotional data. It uses a generative AI model called the "RAG model" to dynamically generate commentary text, which is adjusted to the user's emotional state.

[1923] Next, the generated text is converted into voice data using the "TTS module (pyttsx3)." The TTS module generates voice data based on a pre-set voice profile and adjusts the voice tone according to the user's emotional data. For example, if the user is happy, it creates voice data with a cheerful tone.

[1924] The generated audio data is provided in real time to user devices via a distribution system, where the audio is played back in synchronization with the game footage, providing users with personalized commentary.

[1925] A concrete example of this system is when a user sends a request saying, "I would like live commentary for this game," and the server generates a live commentary based on the user's emotional data, such as, "Great hit! Player A's home run gave Team B three points!" This allows users to enjoy live commentary in real time.

[1926] Example prompt sentence:

[1927] "Prompt for generating commentary when the user is happy while watching a game: 'The user is smiling. Please commentate on the good plays in the game.'"

[1928] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1929] Step 1:

[1930] The user inputs a request for a match and transmits the request from the terminal.

[1931] Input: The user inputs a match request (match ID, team name, player name, etc.).

[1932] Output: The device sends a request to the server.

[1933] Step 2:

[1934] The server receives the user's request and analyzes the request details.

[1935] Input: A match request sent from the device.

[1936] Output: Parsed request data (match ID, team name, player name, etc.).

[1937] Step 3:

[1938] The terminal acquires the user's emotion data using an emotion engine.

[1939] Input: User's facial expressions and heart rate data.

[1940] Output: Parsed user emotion data.

[1941] Specific operation: Using the device's camera and sensors, Emotion Detector analyzes facial expressions and heart rate.

[1942] Step 4:

[1943] The server retrieves relevant match data from a knowledge base based on the parsed request and emotion data.

[1944] Input: Parsed request data and sentiment data.

[1945] Output: Match-related data retrieved from the knowledge base (player performances, team statistics, current match situation, etc.).

[1946] Specific operation: The server performs a database search and extracts relevant match data.

[1947] Step 5:

[1948] The server acquires the match data and emotional data and generates commentary text based on the data.

[1949] Input: Obtained match data and emotion data.

[1950] Output: The generated commentary text.

[1951] Specific operation: The server generates commentary text using the RAG model and adjusts the text content to match the user's emotional state.

[1952] Step 6:

[1953] The server converts the generated commentary text into voice data using a TTS module.

[1954] Input: Generated commentary text.

[1955] Output: The generated audio data.

[1956] Specific operation: The server uses a TTS module (pyttsx3) to generate speech based on pre-configured voice profiles and emotion data.

[1957] Step 7:

[1958] The server provides the converted voice data to the user terminal in real time in cooperation with the distribution system.

[1959] Input: The converted audio data.

[1960] Output: Audio data played on the user's device.

[1961] Specific operation: The server transfers the audio data to the distribution system, and the audio is played on the user's terminal so that it is synchronized with the game video.

[1962] Step 8:

[1963] The user terminal plays the game video and audio commentary in sync.

[1964] Input: Audio data and game footage received from the distribution system.

[1965] Output: Audio commentary synchronized with match footage.

[1966] Specific operation: The user device simultaneously plays the received audio and video, providing personalized commentary in real time.

[1967] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1968] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1969] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1970] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1971] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1972] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1973] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1974] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1975] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1976] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1977] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1978] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1979] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1980] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1981] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1982] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1983] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1984] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1985] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1986] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1987] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1988] The following is further disclosed regarding the above embodiment.

[1989] (Claim 1)

[1990] means for receiving a match request from a user;

[1991] means for retrieving match-related data from a knowledge base based on the received request;

[1992] means for dynamically generating commentary text based on the acquired data;

[1993] means for converting the generated text into audio data;

[1994] a means for providing the converted voice data to a user terminal in real time in cooperation with a distribution system;

[1995] A system including:

[1996] (Claim 2)

[1997] 2. The system of claim 1, wherein the means for dynamically generating commentary text adjusts the content of the text to suit the personality of a particular commentator based on the acquired data.

[1998] (Claim 3)

[1999] 2. The system of claim 1, wherein the means for converting the voice data converts the generated text into voice data based on a pre-defined voice profile.

[2000] "Example 1"

[2001] (Claim 1)

[2002] means for receiving a match request from a user;

[2003] means for retrieving match-related data from a knowledge base based on the received request;

[2004] using a generative AI model to generate commentary text in real time based on the acquired data; and

[2005] a means of tailoring the generated text to the personality of a particular commentator;

[2006] means for converting the adjusted text into audio data;

[2007] a means for providing the converted voice data to a user terminal in real time in cooperation with a distribution system;

[2008] A system including:

[2009] (Claim 2)

[2010] 2. The system of claim 1, wherein the means for generating commentary text using the generative AI model dynamically adjusts the content of the text according to specific configuration information based on the acquired data.

[2011] (Claim 3)

[2012] 2. The system of claim 1, wherein the means for converting the voice data converts the generated text into voice data based on a pre-defined voice profile.

[2013] "Application Example 1"

[2014] (Claim 1)

[2015] means for receiving a match request from a user;

[2016] means for retrieving match-related data from a knowledge base based on the received request;

[2017] means for dynamically generating commentary text based on the acquired data;

[2018] a means for using the generative AI model to generate commentary text appropriate for a particular scene based on a user request;

[2019] means for converting the generated text into audio data;

[2020] a means for providing the converted voice data to a user terminal in real time in cooperation with a distribution system;

[2021] A system including:

[2022] (Claim 2)

[2023] 2. The system of claim 1, wherein the means for dynamically generating commentary text adjusts the content of the text to suit the personality of a particular commentator based on the acquired data.

[2024] (Claim 3)

[2025] 2. The system of claim 1, wherein the means for converting the voice data converts the generated text into voice data based on a pre-defined voice profile.

[2026] "Example 2: Combining Emotion Engines"

[2027] (Claim 1)

[2028] means for receiving a match request from a user;

[2029] means for analyzing received requests and user emotion data;

[2030] means for retrieving match-related data from the knowledge base based on the analysis results;

[2031] means for dynamically generating commentary text based on the acquired data;

[2032] means for converting the generated text into speech data based on a pre-defined speech profile;

[2033] a means for providing the converted voice data to a user terminal in real time in cooperation with a distribution system;

[2034] A system including:

[2035] (Claim 2)

[2036] 2. The system of claim 1, wherein the means for dynamically generating commentary text adjusts the content of the text to suit the personality of a particular commentator and the emotions of the user based on the acquired data.

[2037] (Claim 3)

[2038] 2. The system of claim 1, wherein the means for converting the voice data converts the generated text into voice data according to a voice profile based on the user's emotional state.

[2039] "Application example 2 when combining emotion engines"

[2040] (Claim 1)

[2041] means for receiving a match request from a user;

[2042] means for retrieving match-related data from a knowledge base based on the received request;

[2043] means for dynamically generating commentary text based on the acquired data and user emotion data;

[2044] a means for adjusting the generated text to an emotional state and converting the text into speech data;

[2045] a means for providing the converted voice data to a user terminal in real time in cooperation with a distribution system;

[2046] A system including:

[2047] (Claim 2)

[2048] 2. The system of claim 1, wherein the means for dynamically generating commentary text adjusts the content of the text to suit the personality of a particular commentator and the emotional state of the user based on the acquired data.

[2049] (Claim 3)

[2050] 2. The system of claim 1, wherein the means for converting the voice data converts the generated text into voice data based on a preset voice profile and user emotion data. [Explanation of symbols]

[2051] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving a match request from a user; means for retrieving match-related data from a knowledge base based on the received request; means for dynamically generating commentary text based on the acquired data; means for converting the generated text into audio data; a means for providing the converted voice data to a user terminal in real time in cooperation with a distribution system; A system including:

2. 2. The system of claim 1, wherein the means for dynamically generating commentary text adjusts the content of the text to suit the personality of a particular commentator based on the acquired data.

3. 2. The system of claim 1, wherein the means for converting voice data converts the generated text into voice data based on a predefined voice profile.

Images