system

Abstract

Provide a system. A method for registering player photos and uniform number data in advance; A method for activating the drone camera before the start of the game and recognizing the faces and uniform numbers of players on the field; A means of collecting and analyzing footage of the game in real time, using a generative AI model to generate real-time commentary and commentary based on the analysis results; means for delivering the generated commentary and commentary to a user's terminal; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] It is difficult to secure high-quality commentators and referees for small-scale sporting events. This can prevent participants and spectators from fully enjoying the event. In particular, there is a lack of professional commentators and referees for elementary school sports and amateur baseball, which can result in a lack of realism and fairness in the games. [Means for solving the problem]

[0005] The proposed system includes a means for registering player facial photos and uniform number data in advance, a means for activating a drone camera before the start of a game to recognize the faces and uniform numbers of players on the field, a means for collecting and analyzing footage of the game in real time, a means for using a generative AI model to generate real-time commentary and commentary based on the analysis results, and a means for delivering the generated commentary and commentary to users' devices. This enables high-quality commentary and umpiring to be provided even for small-scale sporting events, creating an environment that is fully enjoyable for participants and spectators. Furthermore, by having the generative AI model learn the rules of the sport and past commentary data in advance, it is possible to achieve more real-time commentary and accurate umpiring.

[0006] "Athlete" refers to an individual athlete who participates in a sporting event and whose facial photograph and uniform number data are registered in advance.

[0007] A "face photo" is image data of a player's face, and is the object that is recognized by the drone camera.

[0008] "Uniform number data" refers to an identification number assigned to each player, and is information that is displayed on the back of the player's clothing.

[0009] A "drone camera" is an aerial camera that flies over the field before and during the game to recognize players' faces and uniform numbers.

[0010] "Real-time video" refers to live video data captured by a drone camera or the like during a sporting event and instantly transmitted to a server.

[0011] "Analysis" is the process of analyzing real-time video and other data to identify player movements and playing situations.

[0012] A "generative AI model" is an artificial intelligence algorithm or program that learns sports rules and past live commentary data in advance, and automatically generates commentary and commentary based on real-time conditions.

[0013] "Commentary" is commentary in audio or text form that reports on plays and situations during a sporting event in real time.

[0014] "Commentary" is content that provides real-time background and detailed explanations of plays during a sporting event.

[0015] A "terminal" is an electronic device through which a user receives and displays commentary and commentary.

[0016] "Users" are spectators, judges, or other participants who use the system to receive commentary and commentary.

[0017] The "server" is a central computer system that works in conjunction with drone cameras and devices to process data and generate and distribute real-time commentary and commentary using generative AI models. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is a system that provides high-quality commentary and refereeing functions for small-scale sporting events by linking a generative AI model, a drone camera, a cloud server, and a user device. This section describes a specific embodiment of this system.

[0040] The main components of the system include:

[0041] 1. Generative AI Models

[0042] 2. Drone Camera

[0043] 3. Cloud Server

[0044] 4. User Device

[0045] Program and Processing Flow

[0046] Generative AI Models

[0047] The server pre-trains the generative AI model on sports rules and past commentary data. Specifically, the server collects text data on sports event rules and commentary from a large dataset, and inputs it into the generative AI model for training. For example, the server trains the model with commentary information such as "Player A hits a home run" or "This play is an out."

[0048] Registering Player Data

[0049] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. Specifically, the user uses a dedicated web form or app to upload event participant information (such as name, face photo, uniform number, etc.). For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0050] Drone camera launch

[0051] The server activates the drone camera just before the event starts and scans the players on the field. The server sends GPS coordinates and flight patterns to the drone camera, which automatically recognizes the players' faces and uniform numbers and collects data. For example, the drone camera flies over the field, recognizes Player A's face and uniform number, and sends that information to the server.

[0052] Analysis and Synchronization

[0053] The server analyzes the facial recognition data and uniform number data of players collected from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, the server synchronizes "Player A: Face Photo, Uniform Number 7" with the AI ​​model and uses it for real-time commentary.

[0054] Real-time video collection and analysis

[0055] The device (drone camera) transmits images of the game to the server in real time. For example, the drone camera periodically captures images of the game and transmits the data to the server in live streaming format.

[0056] The server analyzes real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the ball's position and player movements, and extracts important events as triggers. For example, it recognizes the moment the ball passes home plate as a trigger and flags it as a "possible strike."

[0057] Generate commentary and commentary

[0058] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time and outputs them as commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0059] delivery

[0060] The terminal (user terminal) receives the commentary and explanation sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by multiple users. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[0061] Data storage and feedback

[0062] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[0063] After an event, users provide feedback to the system. For example, they can send feedback about the quality of the commentary and refereeing through a satisfaction survey. The server uses the collected feedback to improve the generative AI model algorithm.

[0064] Specific examples

[0065] Baseball game commentary

[0066] Before the game, the server trains the generative AI model on the rules of baseball and past gameplay commentaries, and the drone camera recognizes the players' faces and uniform numbers when the game starts. During the game, the generative AI model analyzes the play in real time and generates commentary such as "Player B has hit a double!" The server then transmits this commentary to spectators' devices and provides it as a live stream.

[0067] Elementary school sports day

[0068] The server trains the generative AI model on the rules of the sports day and past live commentary data, and the drone camera recognizes the participants before each event begins. During the sports day, the generative AI model provides live commentary as the events progress, such as "The Class A student came out on top in the relay!" The cloud server also makes judgments, assisting the referees in providing accurate results.

[0069] In this way, this system enables fast and accurate commentary and refereeing at a variety of sporting events, realizing sponsored events that can be enjoyed by participants and spectators alike.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The server pre-trains the generative AI model with the rules of sports and past commentary data. Specifically, it collects text data on the rules of sporting events and commentary from a large dataset, and inputs it into the model for learning. For example, the server trains the AI ​​model with commentary information such as "Player A hits a home run" or "This play is an out."

[0073] Step 2:

[0074] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uploads event participant information (name, face photo, uniform number, etc.) using a dedicated web form or app. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0075] Step 3:

[0076] The server activates the drone camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the faces and uniform numbers of players and collects data. For example, the drone camera flies over the field, recognizes the face and uniform number of player A, and sends that information to the server.

[0077] Step 4:

[0078] The server analyzes the player's facial recognition data and uniform number data sent from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, the server synchronizes "Player A: Face Photo, Uniform Number 7" with the AI ​​model and uses it for real-time commentary.

[0079] Step 5:

[0080] The device (drone camera) transmits video footage of the match to the server in real time. The drone camera periodically captures video footage of the match and transmits the data to the server in live streaming format.

[0081] Step 6:

[0082] The server analyzes real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the ball's position and player movements, and extracts important events as triggers. For example, it recognizes the moment the ball passes home plate as a trigger and flags it as a "possible strike."

[0083] Step 7:

[0084] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time and outputs them as commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0085] Step 8:

[0086] The terminal (user terminal) receives the commentary and explanation sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by multiple users. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[0087] Step 9:

[0088] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[0089] Step 10:

[0090] After an event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing through a satisfaction survey to the server. The server uses the collected feedback to improve the generative AI model algorithm.

[0091] Example 1

[0092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] In traditional sporting events, the lack of systems that provide high-quality real-time commentary and refereeing functions can lead to low spectator satisfaction and problems with the progress of the event. Small-scale sporting events, in particular, have budget and resource constraints that make it difficult to introduce advanced technology. This has led to issues such as a decline in the entertainment value for participants and spectators.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0095] In this invention, the server includes means for registering player facial photographs and uniform number data in advance, means for activating the unmanned aerial camera before the start of the game and recognizing the faces and uniform numbers of players on the field, means for collecting and analyzing video footage of the game in real time, means for using a generative AI model to generate real-time commentary and commentary based on the analysis results, means for distributing the generated commentary and commentary to user devices, means for identifying important plays and movements in real time, means for saving video footage and commentary data of the entire game and making them available for later reference, and means for collecting user feedback and improving the algorithm of the generative AI model. This makes it possible to provide high-quality commentary and accurate refereeing functions even for small-scale sporting events, thereby increasing the satisfaction of spectators and participants.

[0096] "Athlete" means an individual player participating in a Sporting Event.

[0097] "Facial photo" refers to image data showing the player's face.

[0098] "Uniform number data" refers to the number information written on the uniform worn by a player.

[0099] "Unmanned aerial camera" refers to a camera device mounted on a drone capable of remote control or autonomous flight.

[0100] "Server" refers to a computer system that processes and manages data.

[0101] "Real-time video" refers to video data that captures and distributes the current situation in real time.

[0102] "Analysis" refers to the act of analyzing collected data and video and extracting meaningful information.

[0103] A "generative AI model" refers to an artificial intelligence system that uses machine learning and data analysis techniques to generate language expressions and explanations from input data.

[0104] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[0105] "Distribution" refers to the act of transmitting generated data or content to a user terminal via a network.

[0106] "Significant plays and actions" refers to game situations or player actions that deserve special attention in a sporting event.

[0107] "Storage" refers to the act of recording collected data and analysis results so that they can be referenced later.

[0108] "Feedback" refers to the act of collecting opinions and ratings from users.

[0109] An "algorithm" refers to a procedure or computational method for solving a specific problem.

[0110] MODE FOR CARRYING OUT THE INVENTION

[0111] System Overview

[0112] This invention is a system that can provide high-quality commentary and refereeing functions for small-scale sporting events by linking a generative AI model, an unmanned aerial camera (drone camera), a cloud server, and user devices. This system aims to achieve fast and accurate data processing and commentary for a variety of sporting events, thereby improving the satisfaction of participants and spectators.

[0113] System Components

[0114] Generative AI model: An artificial intelligence system that uses machine learning and data analysis techniques to generate appropriate language expressions and explanations from input data.

[0115] Unmanned aerial camera (drone camera): A camera device mounted on a drone that can be remotely controlled or fly autonomously, which recognizes the faces and uniform numbers of players on the field and captures real-time footage.

[0116] Cloud server: A computer system that processes and manages data, and performs tasks such as training generative AI models, data analysis, and distribution.

[0117] User device: A device used by a user, such as a computer, smartphone, or tablet, that receives and displays commentary and commentary.

[0118] Program processing

[0119] 1. Training a generative AI model

[0120] The server pre-trains the generative AI model on the rules of sports and past live commentary data. The server collects rules of sports events and live commentary text data from an existing database, and inputs this data into the generative AI model for learning. For example, the server trains the AI ​​model using example sentences such as "Player A hits a home run" and "This play is an out."

[0121] Example prompt: "You hit a home run. How would you describe it?"

[0122] 2. Registering player data

[0123] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uses a dedicated web form or app to enter event participant information (such as name, face photo, and uniform number). For example, the user might upload data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15." The server then stores this information in a database for use in later processing.

[0124] 3. Start the drone camera

[0125] The server activates the unmanned aerial camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, which automatically recognizes players' faces and uniform numbers and collects data. For example, the drone camera flies over the field, recognizes Player A's face and uniform number, and sends that information to the server. The server monitors this and ensures that accurate data is being collected.

[0126] 4. Analysis and synchronization

[0127] The server analyzes the facial recognition data and uniform number data of players collected from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database. For example, the server recognizes "Player A: face photo, uniform number 7," matches it with the data in the server, and sends the matching player information to the generative AI model.

[0128] 5. Real-time video collection and analysis

[0129] The device (drone camera) transmits video footage of the game in real time to a server. For example, the drone camera continuously captures video footage of the game and sends the data to the server in live streaming format. The server analyzes the real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the position of the ball and the movements of the players. For example, it recognizes the "moment the ball passes home base" as a trigger and flags it as a "possible strike."

[0130] 6. Commentary and Commentary Generation

[0131] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0132] 7. Distribution

[0133] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. The server then distributes the generated commentary data to a web stream or application that can be viewed by many users. For example, users can enjoy real-time commentary and commentary on their smartphones or tablets.

[0134] 8. Data Storage and Feedback

[0135] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage so that it can be used for later event data analysis and playback. Users provide feedback to the system after the event ends. For example, users can send feedback about the quality of the commentary and refereeing to the server through a satisfaction survey. The server uses the collected feedback to help improve the generative AI model algorithm.

[0136] In this way, this system can provide fast and accurate commentary and refereeing at a variety of sporting events, enabling events that can be enjoyed by participants and spectators alike.

[0137] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0138] Step 1: Training a generative AI model

[0139] The server pre-trains the generative AI model on sports rules and past commentary data. Specifically, it collects a large amount of sports event rules and commentary text data from a database connected to the server. This collected data is then input into the generative AI model, and a learning process takes place. The input data includes commentary data in text format and rule explanations. The output is an AI model skilled in generating accurate commentary based on the rules and past commentary. For example, the server trains the AI ​​model to generate appropriate language expressions in response to prompt statements such as, "There's a home run. How would you describe it?"

[0140] Step 2: Registering Player Data

[0141] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. Specifically, the user uses a dedicated web form or app to enter event participant information (such as name, face photo, and uniform number). The input data includes each player's face photo, name, and uniform number. As output, this data is saved in a database on the server. For example, data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15" are registered.

[0142] Step 3: Start the drone camera

[0143] The server activates the unmanned aerial camera just before the event starts. Specifically, the server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the players' faces and uniform numbers and collects data. The input data includes the GPS coordinates and flight patterns sent from the server. The output data, which includes the faces and uniform numbers of the players collected by the drone camera, is sent to the server. For example, the drone camera flies over the field, recognizes Player A, and sends the data to the server as "Player A: face photo, uniform number 7."

[0144] Step 4: Analyze and synchronize

[0145] The server analyzes the facial recognition data and uniform number data of players collected from the drone camera and synchronizes it with the generative AI model. Specifically, the server uses a facial recognition algorithm to match the collected player information with pre-registered data in the database. Input data includes the facial photo and uniform number data sent from the drone camera and existing information in the database. The synchronized data is sent as output to the generative AI model. For example, "Player A: facial photo, uniform number 7" is analyzed and synchronized and provided to the generative AI model.

[0146] Step 5: Real-time video collection and analysis

[0147] The device (drone camera) transmits video footage of the game in real time to a server. Specifically, the drone camera continuously captures video footage of the game and sends the data to the server in live streaming format. The input data includes real-time video footage. As output, this video data is sent to the server. The server analyzes the real-time video footage and identifies important plays and movements. The input data includes the received live streaming video. As output, information about the position of the ball and player movements is extracted. For example, the moment the ball passes home base is recognized as a trigger, and a flag is raised as a "possible strike."

[0148] Step 6: Generate commentary and commentary

[0149] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. Specifically, the server inputs the analyzed data into the generative AI model to create appropriate linguistic expressions. The input data includes information about important events and player movements during the game. The output is real-time commentary and commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0150] Step 7: Delivery

[0151] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. Specifically, the server delivers the generated commentary data to the user terminal in real time. The input data includes the real-time commentary and commentary created by the generative AI model. As output, this data is displayed on the user terminal. For example, the user can enjoy the real-time commentary and commentary on a smartphone or tablet.

[0152] Step 8: Data storage and feedback

[0153] The server stores the video and commentary data of the entire match for later reference. Specifically, the server stores all collected data in cloud storage. Input data includes video data and commentary data of the match. Output data is stored in cloud storage. Users provide feedback to the system after the event ends. Specifically, users submit their opinions and evaluations through a dedicated survey form. Input data includes user feedback. Output data is used to improve the generative AI model's algorithm.

[0154] (Application example 1)

[0155] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0156] In conventional sporting events, it has been difficult to provide high-quality commentary and commentary in real time. Furthermore, for smaller sporting events, it is difficult to assign professional commentators and referees, resulting in a poor viewing experience during the event. There is a need for a solution to this problem and to provide high-quality commentary and commentary for more events.

[0157] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0158] In this invention, the server includes a means for registering player facial photographs and uniform number data in advance, a means for activating a drone camera before the start of a game and recognizing the faces and uniform numbers of players on the field, a means for collecting and analyzing video footage during the game in real time, a means for using a generative AI model to generate real-time commentary and commentary based on the analysis results, a means for delivering the generated commentary and commentary to a user's device, and a smartphone application that enables real-time viewing of sporting events using the generated commentary and commentary, thereby providing high-quality commentary and commentary in real time and improving the viewing experience.

[0159] "Player's face photograph and uniform number data" refers to an image of the face of a player participating in a sporting event and number information for identification.

[0160] A "drone camera" is a camera device attached to an unmanned aerial vehicle that has the ability to capture video from the air.

[0161] A "facial recognition algorithm" is a computational procedure for detecting faces in images captured by a camera and identifying them as specific people.

[0162] A "server" is a central device that collects, analyzes, and distributes data over a network.

[0163] A "generative AI model" is an artificial intelligence model that generates appropriate commentary and commentary in real time by learning the rules of a sport and past live commentary data in advance.

[0164] A "user terminal" is an electronic device used by a user, such as a computer or smartphone.

[0165] A "smartphone application" is a program that runs on a smartphone and provides specific functions or services.

[0166] "Real-time viewing" means viewing an event as it is unfolding with minimal delay.

[0167] This invention is a system for providing high-quality live commentary of sporting events in real time. It connects a server, a drone camera, and user devices to collect and analyze footage from the event, and automatically generates commentary and commentary using a generative AI model, which is then distributed to users.

[0168] Program processing overview

[0169] Before the event begins, the server uses the following methods: First, it receives the players' facial photos and uniform number data from the user's device and registers them on the cloud. Next, it activates the drone camera before the event starts and uses a facial recognition algorithm (e.g., OpenCV) to recognize the faces and uniform numbers of the players on the field in real time. The acquired data is then sent to the server.

[0170] During the game, the drone camera continuously collects real-time video and transmits the video data to a server, which analyzes the received video data and identifies important plays and movements using video analysis algorithms (e.g., motion detection algorithms).

[0171] Next, the server generates real-time commentary and commentary based on the analysis results using a generative AI model (such as Hugging Face's GPT-2 model). This generative AI model has the ability to generate appropriate language expressions because it has previously learned the rules of the sport and past commentary data.

[0172] The generated commentary and commentary are delivered to a user device (e.g., a smartphone application) and displayed to the user in real time, allowing the user to experience the event with a sense of presence.

[0173] Hardware and software used

[0174] The main components of the system include:

[0175] Drone camera: A camera device mounted on an unmanned aerial vehicle that captures footage of the field in real time from the air.

[0176] Server: A central device that collects data, analyzes it, runs generative AI models, and distributes it to user devices.

[0177] User Device: An electronic device, such as a smartphone or computer, that allows a user to view and listen to the commentary and commentary in real time.

[0178] software:

[0179] Facial Recognition Algorithm (OpenCV): A computational procedure for recognizing players' faces from footage captured by a drone camera.

[0180] Generative AI model (Hugging Face's GPT-2): Generates commentary in real time based on pre-learned sports rules and past live commentary data.

[0181] Examples of concrete examples and prompts

[0182] 1. Example:

[0183] Baseball game commentary:

[0184] The drone camera monitors the entire field and captures the moment a specific player hits a home run. The server analyzes this data and uses a generative AI model to generate a commentary such as "Player A hits a home run!" The user device then broadcasts this commentary to the user in real time.

[0185] 2. Example prompt:

[0186] "Player A scores a goal. Commentator:"

[0187] "Player B hits a double. Commentator:"

[0188] In this way, this invention cleverly integrates drone cameras, generative AI models, and user devices to provide high-quality real-time commentary and significantly improve the viewing experience of sporting events.

[0189] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0190] Step 1:

[0191] Before the start of an event, users upload player photos and uniform number data to the server using a dedicated web form or smartphone application. The input is the player's photo file and uniform number information, and the output is the player data stored on the cloud server.

[0192] Step 2:

[0193] The server activates the drone camera just before the event starts and sends GPS coordinates and flight patterns to the drone to fly over the field. The input is the GPS coordinates and flight pattern data sent from the server, and the output is the drone camera flying over the field.

[0194] Step 3:

[0195] The drone camera captures the faces and uniform numbers of players on the field in real time and sends the video data to a server. The input is the video data captured by the camera, and the output is the video data sent to the cloud server. Specifically, as the drone camera flies over the field, it uses a facial recognition algorithm to recognize the faces and uniform numbers of players and sends the data to the server.

[0196] Step 4:

[0197] The server uses a facial recognition algorithm (e.g., OpenCV) to identify the faces of players from the received video data and compare them with a registered database. The input is the video data sent from the drone camera, and the output is the recognized faces and uniform number data of the players. Specifically, the facial recognition algorithm is used to detect the faces of players from the video and compare them with their uniform numbers to identify the players.

[0198] Step 5:

[0199] During the match, the drone camera collects video in real time and continuously transmits the data to the server. The input is the video data during the match, and the output is the real-time video transmitted to the server. Specifically, the drone camera captures important moments of the match and live-streams the video data to the server.

[0200] Step 6:

[0201] The server analyzes real-time video to identify important plays and movements. The input is real-time video data sent from the drone camera, and the output is identified important event data. A video analysis algorithm (e.g., a motion detection algorithm) is used for the analysis. Specifically, the server analyzes the video data, tracks the position of the ball and the movements of players, and detects specific events (e.g., goals, hits).

[0202] Step 7:

[0203] The server uses a generative AI model (for example, the GPT-2 model for Hugging Face) to generate real-time commentary and commentary based on the analysis results. The input is the identified event data and the analysis results, and the output is the generated commentary text. Specifically, the analysis results are input to the generative AI model, which generates appropriate language expressions in real time.

[0204] Step 8:

[0205] The generated commentary and commentary are delivered to the user's device (e.g., a smartphone application). The input is the generated commentary and commentary data, and the output is the commentary and commentary delivered to the user's device. Specifically, the server sends the generated commentary data to the user's device, and the user watches it in real time using the smartphone application.

[0206] Step 9:

[0207] Users can enjoy the progress of the event by viewing commentary and commentary in real time through a smartphone application. The input is commentary and commentary data delivered to the user's device, and the output is video and commentary that can be viewed in real time. Specifically, users open the application and enjoy a high-quality experience of the commentary and commentary provided as the event progresses.

[0208] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0209] This invention is a system that links a generative AI model, a drone camera, a cloud server, a user device, and an emotion engine to provide high-quality commentary and commentary for small sporting events, and real-time adjustments based on user emotions. This section describes a specific embodiment of this system.

[0210] The main components of the system include:

[0211] 1. Generative AI Models

[0212] 2. Drone Camera

[0213] 3. Cloud Server

[0214] 4. User Device

[0215] 5. Emotion Engine

[0216] Program and Processing Flow

[0217] Training generative AI models

[0218] The server pre-trains the generative AI model on sports rules and past commentary data. Specifically, it collects text data on sports event rules and commentary from a large dataset and inputs it into the generative AI model for learning. For example, the server trains the AI ​​model on commentary information such as "Player A hits a home run" or "This play is an out."

[0219] Registering Player Data

[0220] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uploads event participant information (name, face photo, uniform number, etc.) using a dedicated web form or app. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0221] Drone camera launch

[0222] The server activates the drone camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the faces and uniform numbers of players and collects data. For example, the drone camera flies over the field, recognizes the face and uniform number of player A, and sends that information to the server.

[0223] Analysis and Synchronization

[0224] The server analyzes the player's facial recognition data and uniform number data sent from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, the server synchronizes "Player A: Face Photo, Uniform Number 7" with the AI ​​model and uses it for real-time commentary.

[0225] Real-time video collection and analysis

[0226] The device (drone camera) transmits images of the game in real time to the server. The drone camera periodically captures images of the game and transmits the data to the server in live streaming format.

[0227] Generate commentary and commentary

[0228] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time and outputs them as commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0229] delivery

[0230] The terminal (user terminal) receives the commentary and explanation sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by multiple users. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[0231] Emotional engine regulation

[0232] The user terminal includes an emotion engine that recognizes the user's emotions. The emotion engine uses a camera and sensors to analyze the user's facial expressions and tone of voice, and generates emotion data. For example, if the user is happy, the emotion engine recognizes a smile or an excited voice.

[0233] The device (user device) transmits this emotional data to the server, which analyzes the collected emotional data and adjusts the commentary and commentary generated by the generative AI model in real time. For example, if the user is excited, the server instructs the generative AI model to generate a more energetic commentary.

[0234] Data storage and feedback

[0235] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[0236] After an event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing through a satisfaction survey to the server. The server uses the collected feedback to improve the generative AI model algorithm.

[0237] Specific examples

[0238] Baseball game commentary

[0239] Before the game, the server trains the generative AI model on the rules of baseball and past gameplay commentaries, and the drone camera recognizes the players' faces and uniform numbers when the game starts. During the game, the generative AI model analyzes the play in real time and generates commentary such as "Player B has hit a double!" The server then transmits this commentary to spectators' devices and provides it as a live stream.

[0240] Additionally, if the emotion engine recognizes the user's excitement, the server will modify the commentary to be more emotional and generate commentary with increased energy.

[0241] Elementary school sports day

[0242] The server trains the generative AI model on the rules of the sports day and past live commentary data, and the drone camera recognizes the participants before each event begins. During the sports day, the generative AI model provides live commentary as the events progress, such as "The Class A student came out on top in the relay!" The cloud server also makes judgments, assisting the referees in providing accurate results.

[0243] Additionally, the emotion engine analyzes the user's emotions in real time, and the system adjusts to generate more positive commentary when the user is happy.

[0244] In this way, the system provides fast and accurate commentary, commentary and emotional adjustments for a variety of sporting events, enhancing the experience for participants and spectators.

[0245] The processing flow will be explained below.

[0246] Step 1:

[0247] The server pre-trains the generative AI model with sports rules and past commentary data. Specifically, the server collects text data on the rules and commentary of sporting events from a large dataset, and inputs it into the generative AI model for training. For example, the model learns commentary information such as "Player A hits a home run" or "This play is an out."

[0248] Step 2:

[0249] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uploads event participant information (name, face photo, uniform number, etc.) using a dedicated web form or app. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0250] Step 3:

[0251] The server activates the drone camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, allowing it to automatically recognize the faces and uniform numbers of players on the field and collect data. For example, the drone camera flies over the field, recognizes the face and uniform number of player A, and sends that information to the server.

[0252] Step 4:

[0253] The server analyzes the facial recognition data and uniform number data of players sent from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, "Player A: Face Photo, Uniform Number 7" is synchronized with the AI ​​model and used for real-time commentary.

[0254] Step 5:

[0255] The device (drone camera) transmits video footage of the match to the server in real time. The drone camera periodically captures video footage of the match and transmits the data to the server in live streaming format.

[0256] Step 6:

[0257] The server analyzes real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the ball's position and player movements, and extracts important events as triggers. For example, it recognizes the moment the ball passes home plate as a trigger and flags it as a "possible strike."

[0258] Step 7:

[0259] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, generates appropriate linguistic expressions in real time, and outputs them as commentary. For example, it generates commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0260] Step 8:

[0261] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by many users. For example, spectators can watch the commentary and commentary in real time as a video stream.

[0262] Step 9:

[0263] The user terminal includes an emotion engine that recognizes the user's emotions. The emotion engine uses a camera and sensors to analyze the user's facial expressions and tone of voice to generate emotion data. For example, if the user is happy, it recognizes a smile or an excited voice.

[0264] Step 10:

[0265] The device (user device) sends emotional data to the server. The server analyzes the collected emotional data and adjusts the commentary and commentary generated by the generative AI model in real time. For example, if the user is excited, the server instructs the generative AI model to generate a more energetic commentary.

[0266] Step 11:

[0267] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[0268] Step 12:

[0269] After an event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing through a satisfaction survey to the server. The server uses the collected feedback to improve the generative AI model algorithm.

[0270] Example 2

[0271] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0272] Conventional sports commentary systems lack real-timeness and a sense of presence, making it difficult to accurately identify players and grasp their movements during the game. Furthermore, they were unable to adjust commentary and commentary based on the emotions of the spectators, making it difficult to improve the user experience. Furthermore, efficient methods for storing and analyzing post-match data were also required.

[0273] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for registering facial photographs and identification data of players in advance, means for activating an aerial-capable camera before the start of the event and recognizing the faces and identification data of players on the competition area, means for collecting and analyzing video footage of the event in real time, means for using a language generation model to generate real-time commentary and commentary based on the analysis results, means for delivering the generated commentary and commentary to user devices, means for using an emotion recognition engine to analyze user emotions and adjust the generated commentary and commentary, means for saving video footage and commentary data of the entire game in the host device and making it available for later reference, and means for collecting user feedback and improving the language generation model. This enables real-time and realistic commentary and commentary, realizes flexible responses according to spectator emotions, and enables efficient storage and analysis of post-game data.

[0274] "Athlete Facial Image and Identification Data" means the facial image and unique identification information of athletes participating in a sporting event, which will be used as the basic data for identifying each athlete.

[0275] "Aerial-capable filming devices" refers to devices that have flight capabilities and can capture images from the air, such as drones.

[0276] "Playing Area" means the place or field where a sporting event takes place, where athletes compete and for which commentary and commentary are provided.

[0277] "Language generation model" refers to an algorithm or system that uses artificial intelligence to generate natural-looking language expressions based on input data, such as a generative AI model.

[0278] "User Device" means the device used to receive and display commentary and commentary, including, but not limited to, a smartphone, tablet, or computer.

[0279] An "emotion recognition engine" refers to a software or hardware system that uses cameras and sensors to analyze a user's facial expressions and tone of voice to generate emotional data.

[0280] "Host device" refers to a central device for managing and processing data, such as a cloud server or a high-performance computer.

[0281] "Feedback" refers to the ratings and opinions provided by users. This information is used to refine and improve the system.

[0282] This invention is a system that combines a generative AI model, an aerial-capable camera, a host device, a user terminal, and an emotion recognition engine to provide high-quality commentary and commentary for small sporting events, and real-time adjustments based on user emotions. This section describes a specific embodiment of this system.

[0283] The main components of the system include:

[0284] 1. Generative AI Models

[0285] 2. Imaging device with aerial capabilities

[0286] 3. Host Device

[0287] 4. User Terminal

[0288] 5. Emotion Recognition Engine

[0289] First, the server (host device) trains the generative AI model on the rules of sporting events and past commentary data. Specifically, it collects text data on the rules of sporting events and commentary from a large dataset, and inputs it into the generative AI model for learning. For example, the server trains the AI ​​model on commentary information such as "Player A hits a home run" or "This play is an out."

[0290] Next, the user (event organizer) registers the participants' face photos and identification data (such as names and uniform numbers) on the server in advance. The user then uses a dedicated web form or app to upload the event participant information (face photos and identification data) to the server. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0291] Just before the event begins, the server activates a drone camera with aerial capabilities. The server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the faces and identification data of athletes on the competition area and begins collecting data. For example, the server sends coordinates X: 123.456, coordinate Y: 789.012 to the drone camera, and the drone camera flies over the field and recognizes athletes.

[0292] Once the match begins, the drone camera will transmit live footage of the match to the host device. The drone camera will periodically capture footage of the match and transmit the data to the host device in live streaming format. Real-time footage is important data for accurately reflecting the progress of the match.

[0293] The server analyzes the data sent from the drone camera and synchronizes it with the generative AI model. Based on the analysis results, the generative AI model generates real-time commentary and commentary. For example, the generative AI model instantly generates commentary such as "Player A hits a home run!" or "This is a home run for Player A!"

[0294] The user terminal receives the generated commentary and explanation and displays it to the user. The server distributes the generated commentary data to a web stream or application, which is received and displayed on the user terminal. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[0295] Additionally, an emotion recognition engine analyzes the user's emotions and adjusts the generated commentary and commentary. The user device uses cameras and sensors to analyze the user's facial expressions and tone of voice to generate emotion data. This emotion data is sent to the host device, which instructs the generative AI model to generate a more energetic commentary. For example, if the user is excited, a more energetic commentary will be provided.

[0296] After the match ends, the server stores the video and commentary data for the entire match. All match data (video, commentary, decisions, etc.) is stored in cloud storage so that it can be used for later event data analysis and playback. For example, the server stores "match data from October 12, 2023" so that it can be analyzed and played back as needed.

[0297] Finally, after the event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing to the server through a satisfaction survey. This feedback is used to improve the generative AI model algorithm.

[0298] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0299] Program processing flow

[0300] Step 1: Training a generative AI model

[0301] The server trains the generative AI model on sports rules and past play-by-play data. Inputs include the sports rule book and past play-by-play records. The server collects this data and inputs it into the generative AI model, which then learns linguistic expressions specific to sporting events. The output is an AI model that can generate play-by-play and commentary appropriate for sporting events. Specifically, the server provides the AI ​​model with text data such as "Player A hits a home run" and "This play is an out."

[0302] Step 2: Registering Player Data

[0303] The user registers the participant's face photo and identification data (name, uniform number, etc.) on the server. The input includes the player's face photo and identification data. The user uses a dedicated web form or app to enter this data and send it to the server. The server stores the received data in a database. The output is the registered player information. Specifically, the user enters data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0304] Step 3: Start up the aerial imaging device

[0305] The server activates the aerial camera just before the event starts. The input includes GPS coordinates and flight pattern data. The server sends this data to the drone camera and commands it to activate. The output is that the camera is ready to fly over the competition area and collect footage. Specifically, the server sends coordinate data such as "X coordinate: 123.456, Y coordinate: 789.012."

[0306] Step 4: Collecting and analyzing footage during the competition

[0307] The terminal (aerial camera) transmits video of the game in real time to the host device. The input is video data captured in real time. The terminal periodically captures video of the game and transmits the data to the host device in live streaming format. The output is real-time game video data sent to the host device. In concrete terms, the drone camera transmits live video of the scene where "Player B is currently holding the ball."

[0308] Step 5: Analyze and synchronize player data

[0309] The server analyzes the facial recognition and identification data of players sent from the drone camera and synchronizes it with the generative AI model. The input is the facial image data and identification data sent from the drone camera. The server analyzes this data using a facial recognition algorithm and compares it with an existing database. The output is player data to be used for real-time commentary. Specifically, the server analyzes "Player A: face photo, uniform number 7" and synchronizes it with the generative AI model.

[0310] Step 6: Generate commentary and commentary

[0311] The server inputs the analyzed data into the generative AI model, which generates commentary and commentary in real time. The input is the analyzed game situation data. The server provides this data to the generative AI model, which generates appropriate linguistic expressions. The output is commentary and commentary generated in real time. In concrete terms, the generative AI model generates commentary such as "Player A has hit a home run!" or "This is a home run for Player A!"

[0312] Step 7: Broadcast commentary and commentary

[0313] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. The input includes the commentary data and commentary data sent from the server. The terminal receives this data and displays it to the user. The output is ready for the user to view the commentary and commentary in real time. In concrete terms, the user terminal displays a real-time commentary such as "Player B has scored a double!"

[0314] Step 8: Adjustment with Emotion Recognition Engine

[0315] The user device includes an emotion recognition engine that recognizes the user's emotions. Input includes data on the user's facial expressions and tone of voice collected by cameras and sensors. The device analyzes this data and generates emotion data. The output is the user's emotion data. The device sends this emotion data to the server. The server analyzes the collected emotion data and adjusts the commentary and commentary generated by the generative AI model in real time. Specifically, if the user is excited, the server instructs the generative AI model to generate a more energetic commentary.

[0316] Step 9: Data storage and feedback

[0317] The server stores the video and commentary data of the entire match. The input includes all video data and commentary data collected during the match. The server stores this data in cloud storage for future reference. The output is the stored match data. Specifically, the server stores the "match data for October 12, 2023" in cloud storage.

[0318] After an event, users provide feedback to the system. The input includes a user satisfaction survey. Users provide their opinions on the quality of the commentary and refereeing, which are then sent to the server. The output is collected feedback data. The server uses the collected feedback to improve the generative AI model's algorithm. Specifically, the server aggregates user feedback such as "The commentary was very easy to understand" and "The referee's decisions were accurate."

[0319] (Application example 2)

[0320] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0321] In conventional customer experiences at sporting events and in-stores, it has been difficult to provide appropriate commentary and commentary in real time, and it has also been difficult to provide information that reflects customer emotions. In particular, at small-scale events and brick-and-mortar stores, there are insufficient resources, making it difficult to provide a personalized experience for each customer. The present invention aims to solve these problems and provide a system for improving the experience at sporting events and in-stores.

[0322] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0323] In this invention, the server includes means for registering facial photographs and identification data of athletes in advance, means for activating an aerial photography device before the start of the event and recognizing the faces and identification information of athletes within the area, means for collecting and analyzing footage of the event in real time, means for using a generative AI model to generate real-time commentary and commentary based on the analysis results, means for delivering the generated commentary and commentary to customer devices, and means including an emotion analysis engine for analyzing customer emotions and adjusting the output of the generative AI model, thereby enabling the provision of real-time, personalized information to individual customers.

[0324] "Athlete" means an individual who participates in an Event and has an identity.

[0325] "Mouthshot" refers to facial image data of a player or customer, and is an image used in facial recognition algorithms.

[0326] "Identification Data" means data that individually recognizes and identifies a player or customer, and includes, by way of example, a name or number.

[0327] An "aerial photography device" is a device used to capture images from the air, such as a drone.

[0328] "Zone" means the specific location or area monitored by the aerial imaging device.

[0329] "Real-time" refers to a state in which processing and communication are carried out instantaneously without delay.

[0330] "Footage" means visual data captured in the form of still images or moving images.

[0331] "Analysis" is a method of processing collected data and images to understand their content and meaning.

[0332] A "generative AI model" refers to an artificial intelligence model that generates appropriate commentary and commentary based on pre-trained data.

[0333] "Commentary" refers to a verbal explanation of the progress of an event in real time.

[0334] "Explanation" refers to supplementary explanations of detailed information and background about an event.

[0335] "Customer" means a person receiving a service or shopping in a store.

[0336] An "emotion analysis engine" is a system that analyzes emotions from customers' facial expressions and voice and outputs them as data.

[0337] A specific system for implementing this invention aims to provide information in real time that corresponds to customer behavior and emotions by linking a server, an aerial photography device, a cloud server, a user terminal, and an emotion analysis engine.

[0338] The server first maintains a database for pre-registering the facial photographs and identification data of all people (players and customers) involved in the event or store, allowing the aerial photography device to recognize the identification information in real time based on the registered data before the event begins.

[0339] Aerial photography equipment (e.g., drones) will be activated before the start of the event and will recognize the faces and identification data of athletes and spectators in a designated area in real time. The equipment will use facial recognition algorithms to transmit the video data to a cloud server.

[0340] The cloud server analyzes the received video data and uses a generative AI model to generate real-time commentary and commentary based on that data. The generative AI model is pre-trained with the event rules and past data, and generates commentary and commentary in the appropriate language.

[0341] The generated commentary and commentary are delivered to the user's device. The user device may be a smartphone or smart glasses, and is capable of receiving and displaying streaming data from the cloud server in real time. The user device is also equipped with an emotion analysis engine that analyzes the user's facial expressions and voice to generate emotion data.

[0342] The emotional data generated by the emotion analysis engine is then sent back to the cloud server, which analyzes the data and uses the generative AI model to adjust the commentary and commentary in real time. This allows for energetic commentary if the user is excited, and calm commentary if the user is relaxed.

[0343] Examples:

[0344] Introducing cosmetics in stores

[0345] An aerial camera patrols the store and recognizes customers' faces. The recognized customer information is sent to a cloud server, and a generative AI model generates a commentary about the lipstick for that customer. For example, "This lipstick is newly released and contains special ingredients." Furthermore, if the emotion analysis engine recognizes the customer's emotion as "joy," the cloud server generates a commentary that is more in line with the emotion, adjusting the content to something like, "This lipstick also contains ingredients that will make you feel really happy!"

[0346] Example prompt sentence:

[0347] Produce an exciting commentary for the product Lipstick with the user's current emotion being happy.

[0348] In this way, the system can be used to provide a real-time, personalized experience in-store or during an event.

[0349] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0350] Step 1:

[0351] The server pre-registers the facial photographs and identification data of athletes and customers at events and stores. The facial photographs and identification data uploaded by users (event organizers or store managers) are used as input. This saves the relevant data in the database, enabling the system to identify athletes and customers.

[0352] Step 2:

[0353] Before the event starts, the server activates the aerial photography device (drone), using the event start time and flight pattern data as input. Based on this, the aerial photography device patrols the designated area and captures video data.

[0354] Step 3:

[0355] The aerial photography equipment collects images of events and stores in real time and sends them to a cloud server. The camera image data is used as input and is sent to the cloud server in real time as output.

[0356] Step 4:

[0357] The cloud server uses a facial recognition algorithm to recognize the faces of players and customers from the received video data and match them with the identification data. The video data and pre-registered identification data are used as input, and a recognition result is generated as output.

[0358] Step 5:

[0359] The cloud server inputs the recognition results into the generative AI model to generate real-time commentary and commentary. The recognition results and pre-trained data are used as inputs to the generative AI model, and appropriate commentary and commentary text is generated as output.

[0360] Step 6:

[0361] The generated commentary and commentary text is delivered from the cloud server to the user terminal, and is used as input and displayed in real time on the user terminal as output.

[0362] Step 7:

[0363] An emotion analysis engine installed in the user device analyzes the user's facial expressions and voice to generate emotion data. Camera footage and audio data are used as input, and emotion data is generated as output.

[0364] Step 8:

[0365] The user device sends emotional data to a cloud server, which then adjusts the output of the generative AI model based on the emotional data. The emotional data and prompts for the generative AI model are used as input, and adjusted commentary and commentary are generated as output according to the emotions.

[0366] Step 9:

[0367] The cloud server then delivers the adjusted commentary and commentary back to the user terminal, where the adjusted commentary and commentary text is used as input and displayed to the user as output.

[0368] In this way, the system processes and analyzes data in real time at each step, ultimately providing personalized information.

[0369] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0370] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0371] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0372] [Second embodiment]

[0373] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0374] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0375] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0376] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0377] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0378] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0379] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0380] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0381] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0382] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0383] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0384] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0385] This invention is a system that provides high-quality commentary and refereeing functions for small-scale sporting events by linking a generative AI model, a drone camera, a cloud server, and a user device. This section describes a specific embodiment of this system.

[0386] The main components of the system include:

[0387] 1. Generative AI Models

[0388] 2. Drone Camera

[0389] 3. Cloud Server

[0390] 4. User Device

[0391] Program and Processing Flow

[0392] Generative AI Models

[0393] The server pre-trains the generative AI model on sports rules and past commentary data. Specifically, the server collects text data on sports event rules and commentary from a large dataset, and inputs it into the generative AI model for training. For example, the server trains the model with commentary information such as "Player A hits a home run" or "This play is an out."

[0394] Registering Player Data

[0395] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. Specifically, the user uses a dedicated web form or app to upload event participant information (such as name, face photo, uniform number, etc.). For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0396] Drone camera launch

[0397] The server activates the drone camera just before the event starts and scans the players on the field. The server sends GPS coordinates and flight patterns to the drone camera, which automatically recognizes the players' faces and uniform numbers and collects data. For example, the drone camera flies over the field, recognizes Player A's face and uniform number, and sends that information to the server.

[0398] Analysis and Synchronization

[0399] The server analyzes the facial recognition data and uniform number data of players collected from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, the server synchronizes "Player A: Face Photo, Uniform Number 7" with the AI ​​model and uses it for real-time commentary.

[0400] Real-time video collection and analysis

[0401] The device (drone camera) transmits images of the game to the server in real time. For example, the drone camera periodically captures images of the game and transmits the data to the server in live streaming format.

[0402] The server analyzes real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the ball's position and player movements, and extracts important events as triggers. For example, it recognizes the moment the ball passes home plate as a trigger and flags it as a "possible strike."

[0403] Generate commentary and commentary

[0404] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time and outputs them as commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0405] delivery

[0406] The terminal (user terminal) receives the commentary and explanation sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by multiple users. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[0407] Data storage and feedback

[0408] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[0409] After an event, users provide feedback to the system. For example, they can send feedback about the quality of the commentary and refereeing through a satisfaction survey. The server uses the collected feedback to improve the generative AI model algorithm.

[0410] Specific examples

[0411] Baseball game commentary

[0412] Before the game, the server trains the generative AI model on the rules of baseball and past gameplay commentaries, and the drone camera recognizes the players' faces and uniform numbers when the game starts. During the game, the generative AI model analyzes the play in real time and generates commentary such as "Player B has hit a double!" The server then transmits this commentary to spectators' devices and provides it as a live stream.

[0413] Elementary school sports day

[0414] The server trains the generative AI model on the rules of the sports day and past live commentary data, and the drone camera recognizes the participants before each event begins. During the sports day, the generative AI model provides live commentary as the events progress, such as "The Class A student came out on top in the relay!" The cloud server also makes judgments, assisting the referees in providing accurate results.

[0415] In this way, this system enables fast and accurate commentary and refereeing at a variety of sporting events, realizing sponsored events that can be enjoyed by participants and spectators alike.

[0416] The processing flow will be explained below.

[0417] Step 1:

[0418] The server pre-trains the generative AI model with the rules of sports and past commentary data. Specifically, it collects text data on the rules of sporting events and commentary from a large dataset, and inputs it into the model for learning. For example, the server trains the AI ​​model with commentary information such as "Player A hits a home run" or "This play is an out."

[0419] Step 2:

[0420] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uploads event participant information (name, face photo, uniform number, etc.) using a dedicated web form or app. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0421] Step 3:

[0422] The server activates the drone camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the faces and uniform numbers of players and collects data. For example, the drone camera flies over the field, recognizes the face and uniform number of player A, and sends that information to the server.

[0423] Step 4:

[0424] The server analyzes the player's facial recognition data and uniform number data sent from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, the server synchronizes "Player A: Face Photo, Uniform Number 7" with the AI ​​model and uses it for real-time commentary.

[0425] Step 5:

[0426] The device (drone camera) transmits video footage of the match to the server in real time. The drone camera periodically captures video footage of the match and transmits the data to the server in live streaming format.

[0427] Step 6:

[0428] The server analyzes real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the ball's position and player movements, and extracts important events as triggers. For example, it recognizes the moment the ball passes home plate as a trigger and flags it as a "possible strike."

[0429] Step 7:

[0430] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time and outputs them as commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0431] Step 8:

[0432] The terminal (user terminal) receives the commentary and explanation sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by multiple users. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[0433] Step 9:

[0434] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[0435] Step 10:

[0436] After an event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing through a satisfaction survey to the server. The server uses the collected feedback to improve the generative AI model algorithm.

[0437] Example 1

[0438] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0439] In traditional sporting events, the lack of systems that provide high-quality real-time commentary and refereeing functions can lead to low spectator satisfaction and problems with the progress of the event. Small-scale sporting events, in particular, have budget and resource constraints that make it difficult to introduce advanced technology. This has led to issues such as a decline in the entertainment value for participants and spectators.

[0440] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0441] In this invention, the server includes means for registering player facial photographs and uniform number data in advance, means for activating the unmanned aerial camera before the start of the game and recognizing the faces and uniform numbers of players on the field, means for collecting and analyzing video footage of the game in real time, means for using a generative AI model to generate real-time commentary and commentary based on the analysis results, means for distributing the generated commentary and commentary to user devices, means for identifying important plays and movements in real time, means for saving video footage and commentary data of the entire game and making them available for later reference, and means for collecting user feedback and improving the algorithm of the generative AI model. This makes it possible to provide high-quality commentary and accurate refereeing functions even for small-scale sporting events, thereby increasing the satisfaction of spectators and participants.

[0442] "Athlete" means an individual player participating in a Sporting Event.

[0443] "Facial photo" refers to image data showing the player's face.

[0444] "Uniform number data" refers to the number information written on the uniform worn by a player.

[0445] "Unmanned aerial camera" refers to a camera device mounted on a drone capable of remote control or autonomous flight.

[0446] "Server" refers to a computer system that processes and manages data.

[0447] "Real-time video" refers to video data that captures and distributes the current situation in real time.

[0448] "Analysis" refers to the act of analyzing collected data and video and extracting meaningful information.

[0449] A "generative AI model" refers to an artificial intelligence system that uses machine learning and data analysis techniques to generate language expressions and explanations from input data.

[0450] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[0451] "Distribution" refers to the act of transmitting generated data or content to a user terminal via a network.

[0452] "Significant plays and actions" refers to game situations or player actions that deserve special attention in a sporting event.

[0453] "Storage" refers to the act of recording collected data and analysis results so that they can be referenced later.

[0454] "Feedback" refers to the act of collecting opinions and ratings from users.

[0455] An "algorithm" refers to a procedure or computational method for solving a specific problem.

[0456] MODE FOR CARRYING OUT THE INVENTION

[0457] System Overview

[0458] This invention is a system that can provide high-quality commentary and refereeing functions for small-scale sporting events by linking a generative AI model, an unmanned aerial camera (drone camera), a cloud server, and user devices. This system aims to achieve fast and accurate data processing and commentary for a variety of sporting events, thereby improving the satisfaction of participants and spectators.

[0459] System Components

[0460] Generative AI model: An artificial intelligence system that uses machine learning and data analysis techniques to generate appropriate language expressions and explanations from input data.

[0461] Unmanned aerial camera (drone camera): A camera device mounted on a drone that can be remotely controlled or fly autonomously, which recognizes the faces and uniform numbers of players on the field and captures real-time footage.

[0462] Cloud server: A computer system that processes and manages data, and performs tasks such as training generative AI models, data analysis, and distribution.

[0463] User device: A device used by a user, such as a computer, smartphone, or tablet, that receives and displays commentary and commentary.

[0464] Program processing

[0465] 1. Training a generative AI model

[0466] The server pre-trains the generative AI model on the rules of sports and past live commentary data. The server collects rules of sports events and live commentary text data from an existing database, and inputs this data into the generative AI model for learning. For example, the server trains the AI ​​model using example sentences such as "Player A hits a home run" and "This play is an out."

[0467] Example prompt: "You hit a home run. How would you describe it?"

[0468] 2. Registering player data

[0469] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uses a dedicated web form or app to enter event participant information (such as name, face photo, and uniform number). For example, the user might upload data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15." The server then stores this information in a database for use in later processing.

[0470] 3. Start the drone camera

[0471] The server activates the unmanned aerial camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, which automatically recognizes players' faces and uniform numbers and collects data. For example, the drone camera flies over the field, recognizes Player A's face and uniform number, and sends that information to the server. The server monitors this and ensures that accurate data is being collected.

[0472] 4. Analysis and synchronization

[0473] The server analyzes the facial recognition data and uniform number data of players collected from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database. For example, the server recognizes "Player A: face photo, uniform number 7," matches it with the data in the server, and sends the matching player information to the generative AI model.

[0474] 5. Real-time video collection and analysis

[0475] The device (drone camera) transmits video footage of the game in real time to a server. For example, the drone camera continuously captures video footage of the game and sends the data to the server in live streaming format. The server analyzes the real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the position of the ball and the movements of the players. For example, it recognizes the "moment the ball passes home base" as a trigger and flags it as a "possible strike."

[0476] 6. Commentary and Commentary Generation

[0477] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0478] 7. Distribution

[0479] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. The server then distributes the generated commentary data to a web stream or application that can be viewed by many users. For example, users can enjoy real-time commentary and commentary on their smartphones or tablets.

[0480] 8. Data Storage and Feedback

[0481] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage so that it can be used for later event data analysis and playback. Users provide feedback to the system after the event ends. For example, users can send feedback about the quality of the commentary and refereeing to the server through a satisfaction survey. The server uses the collected feedback to help improve the generative AI model algorithm.

[0482] In this way, this system can provide fast and accurate commentary and refereeing at a variety of sporting events, enabling events that can be enjoyed by participants and spectators alike.

[0483] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0484] Step 1: Training a generative AI model

[0485] The server pre-trains the generative AI model on sports rules and past commentary data. Specifically, it collects a large amount of sports event rules and commentary text data from a database connected to the server. This collected data is then input into the generative AI model, and a learning process takes place. The input data includes commentary data in text format and rule explanations. The output is an AI model skilled in generating accurate commentary based on the rules and past commentary. For example, the server trains the AI ​​model to generate appropriate language expressions in response to prompt statements such as, "There's a home run. How would you describe it?"

[0486] Step 2: Registering Player Data

[0487] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. Specifically, the user uses a dedicated web form or app to enter event participant information (such as name, face photo, and uniform number). The input data includes each player's face photo, name, and uniform number. As output, this data is saved in a database on the server. For example, data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15" are registered.

[0488] Step 3: Start the drone camera

[0489] The server activates the unmanned aerial camera just before the event starts. Specifically, the server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the players' faces and uniform numbers and collects data. The input data includes the GPS coordinates and flight patterns sent from the server. The output data, which includes the faces and uniform numbers of the players collected by the drone camera, is sent to the server. For example, the drone camera flies over the field, recognizes Player A, and sends the data to the server as "Player A: face photo, uniform number 7."

[0490] Step 4: Analyze and synchronize

[0491] The server analyzes the facial recognition data and uniform number data of players collected from the drone camera and synchronizes it with the generative AI model. Specifically, the server uses a facial recognition algorithm to match the collected player information with pre-registered data in the database. Input data includes the facial photo and uniform number data sent from the drone camera and existing information in the database. The synchronized data is sent as output to the generative AI model. For example, "Player A: facial photo, uniform number 7" is analyzed and synchronized and provided to the generative AI model.

[0492] Step 5: Real-time video collection and analysis

[0493] The device (drone camera) transmits video footage of the game in real time to a server. Specifically, the drone camera continuously captures video footage of the game and sends the data to the server in live streaming format. The input data includes real-time video footage. As output, this video data is sent to the server. The server analyzes the real-time video footage and identifies important plays and movements. The input data includes the received live streaming video. As output, information about the position of the ball and player movements is extracted. For example, the moment the ball passes home base is recognized as a trigger, and a flag is raised as a "possible strike."

[0494] Step 6: Generate commentary and commentary

[0495] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. Specifically, the server inputs the analyzed data into the generative AI model to create appropriate linguistic expressions. The input data includes information about important events and player movements during the game. The output is real-time commentary and commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0496] Step 7: Delivery

[0497] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. Specifically, the server delivers the generated commentary data to the user terminal in real time. The input data includes the real-time commentary and commentary created by the generative AI model. As output, this data is displayed on the user terminal. For example, the user can enjoy the real-time commentary and commentary on a smartphone or tablet.

[0498] Step 8: Data storage and feedback

[0499] The server stores the video and commentary data of the entire match for later reference. Specifically, the server stores all collected data in cloud storage. Input data includes video data and commentary data of the match. Output data is stored in cloud storage. Users provide feedback to the system after the event ends. Specifically, users submit their opinions and evaluations through a dedicated survey form. Input data includes user feedback. Output data is used to improve the generative AI model's algorithm.

[0500] (Application example 1)

[0501] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0502] In conventional sporting events, it has been difficult to provide high-quality commentary and commentary in real time. Furthermore, for smaller sporting events, it is difficult to assign professional commentators and referees, resulting in a poor viewing experience during the event. There is a need for a solution to this problem and to provide high-quality commentary and commentary for more events.

[0503] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0504] In this invention, the server includes a means for registering player facial photographs and uniform number data in advance, a means for activating a drone camera before the start of a game and recognizing the faces and uniform numbers of players on the field, a means for collecting and analyzing video footage during the game in real time, a means for using a generative AI model to generate real-time commentary and commentary based on the analysis results, a means for delivering the generated commentary and commentary to a user's device, and a smartphone application that enables real-time viewing of sporting events using the generated commentary and commentary, thereby providing high-quality commentary and commentary in real time and improving the viewing experience.

[0505] "Player's face photograph and uniform number data" refers to an image of the face of a player participating in a sporting event and number information for identification.

[0506] A "drone camera" is a camera device attached to an unmanned aerial vehicle that has the ability to capture video from the air.

[0507] A "facial recognition algorithm" is a computational procedure for detecting faces in images captured by a camera and identifying them as specific people.

[0508] A "server" is a central device that collects, analyzes, and distributes data over a network.

[0509] A "generative AI model" is an artificial intelligence model that generates appropriate commentary and commentary in real time by learning the rules of a sport and past live commentary data in advance.

[0510] A "user terminal" is an electronic device used by a user, such as a computer or smartphone.

[0511] A "smartphone application" is a program that runs on a smartphone and provides specific functions or services.

[0512] "Real-time viewing" means viewing an event as it is unfolding with minimal delay.

[0513] This invention is a system for providing high-quality live commentary of sporting events in real time. It connects a server, a drone camera, and user devices to collect and analyze footage from the event, and automatically generates commentary and commentary using a generative AI model, which is then distributed to users.

[0514] Program processing overview

[0515] Before the event begins, the server uses the following methods: First, it receives the players' facial photos and uniform number data from the user's device and registers them on the cloud. Next, it activates the drone camera before the event starts and uses a facial recognition algorithm (e.g., OpenCV) to recognize the faces and uniform numbers of the players on the field in real time. The acquired data is then sent to the server.

[0516] During the game, the drone camera continuously collects real-time video and transmits the video data to a server, which analyzes the received video data and identifies important plays and movements using video analysis algorithms (e.g., motion detection algorithms).

[0517] Next, the server generates real-time commentary and commentary based on the analysis results using a generative AI model (such as Hugging Face's GPT-2 model). This generative AI model has the ability to generate appropriate language expressions because it has previously learned the rules of the sport and past commentary data.

[0518] The generated commentary and commentary are delivered to a user device (e.g., a smartphone application) and displayed to the user in real time, allowing the user to experience the event with a sense of presence.

[0519] Hardware and software used

[0520] The main components of the system include:

[0521] Drone camera: A camera device mounted on an unmanned aerial vehicle that captures footage of the field in real time from the air.

[0522] Server: A central device that collects data, analyzes it, runs generative AI models, and distributes it to user devices.

[0523] User Device: An electronic device, such as a smartphone or computer, that allows a user to view and listen to the commentary and commentary in real time.

[0524] software:

[0525] Facial Recognition Algorithm (OpenCV): A computational procedure for recognizing players' faces from footage captured by a drone camera.

[0526] Generative AI model (Hugging Face's GPT-2): Generates commentary in real time based on pre-learned sports rules and past live commentary data.

[0527] Examples of concrete examples and prompts

[0528] 1. Example:

[0529] Baseball game commentary:

[0530] The drone camera monitors the entire field and captures the moment a specific player hits a home run. The server analyzes this data and uses a generative AI model to generate a commentary such as "Player A hits a home run!" The user device then broadcasts this commentary to the user in real time.

[0531] 2. Example prompt:

[0532] "Player A scores a goal. Commentator:"

[0533] "Player B hits a double. Commentator:"

[0534] In this way, this invention cleverly integrates drone cameras, generative AI models, and user devices to provide high-quality real-time commentary and significantly improve the viewing experience of sporting events.

[0535] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0536] Step 1:

[0537] Before the start of an event, users upload player photos and uniform number data to the server using a dedicated web form or smartphone application. The input is the player's photo file and uniform number information, and the output is the player data stored on the cloud server.

[0538] Step 2:

[0539] The server activates the drone camera just before the event starts and sends GPS coordinates and flight patterns to the drone to fly over the field. The input is the GPS coordinates and flight pattern data sent from the server, and the output is the drone camera flying over the field.

[0540] Step 3:

[0541] The drone camera captures the faces and uniform numbers of players on the field in real time and sends the video data to a server. The input is the video data captured by the camera, and the output is the video data sent to the cloud server. Specifically, as the drone camera flies over the field, it uses a facial recognition algorithm to recognize the faces and uniform numbers of players and sends the data to the server.

[0542] Step 4:

[0543] The server uses a facial recognition algorithm (e.g., OpenCV) to identify the faces of players from the received video data and compare them with a registered database. The input is the video data sent from the drone camera, and the output is the recognized faces and uniform number data of the players. Specifically, the facial recognition algorithm is used to detect the faces of players from the video and compare them with their uniform numbers to identify the players.

[0544] Step 5:

[0545] During the match, the drone camera collects video in real time and continuously transmits the data to the server. The input is the video data during the match, and the output is the real-time video transmitted to the server. Specifically, the drone camera captures important moments of the match and live-streams the video data to the server.

[0546] Step 6:

[0547] The server analyzes real-time video to identify important plays and movements. The input is real-time video data sent from the drone camera, and the output is identified important event data. A video analysis algorithm (e.g., a motion detection algorithm) is used for the analysis. Specifically, the server analyzes the video data, tracks the position of the ball and the movements of players, and detects specific events (e.g., goals, hits).

[0548] Step 7:

[0549] The server uses a generative AI model (for example, the GPT-2 model for Hugging Face) to generate real-time commentary and commentary based on the analysis results. The input is the identified event data and the analysis results, and the output is the generated commentary text. Specifically, the analysis results are input to the generative AI model, which generates appropriate language expressions in real time.

[0550] Step 8:

[0551] The generated commentary and commentary are delivered to the user's device (e.g., a smartphone application). The input is the generated commentary and commentary data, and the output is the commentary and commentary delivered to the user's device. Specifically, the server sends the generated commentary data to the user's device, and the user watches it in real time using the smartphone application.

[0552] Step 9:

[0553] Users can enjoy the progress of the event by viewing commentary and commentary in real time through a smartphone application. The input is commentary and commentary data delivered to the user's device, and the output is video and commentary that can be viewed in real time. Specifically, users open the application and enjoy a high-quality experience of the commentary and commentary provided as the event progresses.

[0554] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0555] This invention is a system that links a generative AI model, a drone camera, a cloud server, a user device, and an emotion engine to provide high-quality commentary and commentary for small sporting events, and real-time adjustments based on user emotions. This section describes a specific embodiment of this system.

[0556] The main components of the system include:

[0557] 1. Generative AI Models

[0558] 2. Drone Camera

[0559] 3. Cloud Server

[0560] 4. User Device

[0561] 5. Emotion Engine

[0562] Program and Processing Flow

[0563] Training generative AI models

[0564] The server pre-trains the generative AI model on sports rules and past commentary data. Specifically, it collects text data on sports event rules and commentary from a large dataset and inputs it into the generative AI model for learning. For example, the server trains the AI ​​model on commentary information such as "Player A hits a home run" or "This play is an out."

[0565] Registering Player Data

[0566] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uploads event participant information (name, face photo, uniform number, etc.) using a dedicated web form or app. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0567] Drone camera launch

[0568] The server activates the drone camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the faces and uniform numbers of players and collects data. For example, the drone camera flies over the field, recognizes the face and uniform number of player A, and sends that information to the server.

[0569] Analysis and Synchronization

[0570] The server analyzes the player's facial recognition data and uniform number data sent from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, the server synchronizes "Player A: Face Photo, Uniform Number 7" with the AI ​​model and uses it for real-time commentary.

[0571] Real-time video collection and analysis

[0572] The device (drone camera) transmits images of the game in real time to the server. The drone camera periodically captures images of the game and transmits the data to the server in live streaming format.

[0573] Generate commentary and commentary

[0574] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time and outputs them as commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0575] delivery

[0576] The terminal (user terminal) receives the commentary and explanation sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by multiple users. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[0577] Emotional engine regulation

[0578] The user terminal includes an emotion engine that recognizes the user's emotions. The emotion engine uses a camera and sensors to analyze the user's facial expressions and tone of voice, and generates emotion data. For example, if the user is happy, the emotion engine recognizes a smile or an excited voice.

[0579] The device (user device) transmits this emotional data to the server, which analyzes the collected emotional data and adjusts the commentary and commentary generated by the generative AI model in real time. For example, if the user is excited, the server instructs the generative AI model to generate a more energetic commentary.

[0580] Data storage and feedback

[0581] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[0582] After an event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing through a satisfaction survey to the server. The server uses the collected feedback to improve the generative AI model algorithm.

[0583] Specific examples

[0584] Baseball game commentary

[0585] Before the game, the server trains the generative AI model on the rules of baseball and past gameplay commentaries, and the drone camera recognizes the players' faces and uniform numbers when the game starts. During the game, the generative AI model analyzes the play in real time and generates commentary such as "Player B has hit a double!" The server then transmits this commentary to spectators' devices and provides it as a live stream.

[0586] Additionally, if the emotion engine recognizes the user's excitement, the server will modify the commentary to be more emotional and generate commentary with increased energy.

[0587] Elementary school sports day

[0588] The server trains the generative AI model on the rules of the sports day and past live commentary data, and the drone camera recognizes the participants before each event begins. During the sports day, the generative AI model provides live commentary as the events progress, such as "The Class A student came out on top in the relay!" The cloud server also makes judgments, assisting the referees in providing accurate results.

[0589] Additionally, the emotion engine analyzes the user's emotions in real time, and the system adjusts to generate more positive commentary when the user is happy.

[0590] In this way, the system provides fast and accurate commentary, commentary and emotional adjustments for a variety of sporting events, enhancing the experience for participants and spectators.

[0591] The processing flow will be explained below.

[0592] Step 1:

[0593] The server pre-trains the generative AI model with sports rules and past commentary data. Specifically, the server collects text data on the rules and commentary of sporting events from a large dataset, and inputs it into the generative AI model for training. For example, the model learns commentary information such as "Player A hits a home run" or "This play is an out."

[0594] Step 2:

[0595] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uploads event participant information (name, face photo, uniform number, etc.) using a dedicated web form or app. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0596] Step 3:

[0597] The server activates the drone camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, allowing it to automatically recognize the faces and uniform numbers of players on the field and collect data. For example, the drone camera flies over the field, recognizes the face and uniform number of player A, and sends that information to the server.

[0598] Step 4:

[0599] The server analyzes the facial recognition data and uniform number data of players sent from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, "Player A: Face Photo, Uniform Number 7" is synchronized with the AI ​​model and used for real-time commentary.

[0600] Step 5:

[0601] The device (drone camera) transmits video footage of the match to the server in real time. The drone camera periodically captures video footage of the match and transmits the data to the server in live streaming format.

[0602] Step 6:

[0603] The server analyzes real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the ball's position and player movements, and extracts important events as triggers. For example, it recognizes the moment the ball passes home plate as a trigger and flags it as a "possible strike."

[0604] Step 7:

[0605] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, generates appropriate linguistic expressions in real time, and outputs them as commentary. For example, it generates commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0606] Step 8:

[0607] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by many users. For example, spectators can watch the commentary and commentary in real time as a video stream.

[0608] Step 9:

[0609] The user terminal includes an emotion engine that recognizes the user's emotions. The emotion engine uses a camera and sensors to analyze the user's facial expressions and tone of voice to generate emotion data. For example, if the user is happy, it recognizes a smile or an excited voice.

[0610] Step 10:

[0611] The device (user device) sends emotional data to the server. The server analyzes the collected emotional data and adjusts the commentary and commentary generated by the generative AI model in real time. For example, if the user is excited, the server instructs the generative AI model to generate a more energetic commentary.

[0612] Step 11:

[0613] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[0614] Step 12:

[0615] After an event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing through a satisfaction survey to the server. The server uses the collected feedback to improve the generative AI model algorithm.

[0616] Example 2

[0617] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0618] Conventional sports commentary systems lack real-timeness and a sense of presence, making it difficult to accurately identify players and grasp their movements during the game. Furthermore, they were unable to adjust commentary and commentary based on the emotions of the spectators, making it difficult to improve the user experience. Furthermore, efficient methods for storing and analyzing post-match data were also required.

[0619] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for registering facial photographs and identification data of players in advance, means for activating an aerial-capable camera before the start of the event and recognizing the faces and identification data of players on the competition area, means for collecting and analyzing video footage of the event in real time, means for using a language generation model to generate real-time commentary and commentary based on the analysis results, means for delivering the generated commentary and commentary to user devices, means for using an emotion recognition engine to analyze user emotions and adjust the generated commentary and commentary, means for saving video footage and commentary data of the entire game in the host device and making it available for later reference, and means for collecting user feedback and improving the language generation model. This enables real-time and realistic commentary and commentary, realizes flexible responses according to spectator emotions, and enables efficient storage and analysis of post-game data.

[0620] "Athlete Facial Image and Identification Data" means the facial image and unique identification information of athletes participating in a sporting event, which will be used as the basic data for identifying each athlete.

[0621] "Aerial-capable filming devices" refers to devices that have flight capabilities and can capture images from the air, such as drones.

[0622] "Playing Area" means the place or field where a sporting event takes place, where athletes compete and for which commentary and commentary are provided.

[0623] "Language generation model" refers to an algorithm or system that uses artificial intelligence to generate natural-looking language expressions based on input data, such as a generative AI model.

[0624] "User Device" means the device used to receive and display commentary and commentary, including, but not limited to, a smartphone, tablet, or computer.

[0625] An "emotion recognition engine" refers to a software or hardware system that uses cameras and sensors to analyze a user's facial expressions and tone of voice to generate emotional data.

[0626] "Host device" refers to a central device for managing and processing data, such as a cloud server or a high-performance computer.

[0627] "Feedback" refers to the ratings and opinions provided by users. This information is used to refine and improve the system.

[0628] This invention is a system that combines a generative AI model, an aerial-capable camera, a host device, a user terminal, and an emotion recognition engine to provide high-quality commentary and commentary for small sporting events, and real-time adjustments based on user emotions. This section describes a specific embodiment of this system.

[0629] The main components of the system include:

[0630] 1. Generative AI Models

[0631] 2. Imaging device with aerial capabilities

[0632] 3. Host Device

[0633] 4. User Terminal

[0634] 5. Emotion Recognition Engine

[0635] First, the server (host device) trains the generative AI model on the rules of sporting events and past commentary data. Specifically, it collects text data on the rules of sporting events and commentary from a large dataset, and inputs it into the generative AI model for learning. For example, the server trains the AI ​​model on commentary information such as "Player A hits a home run" or "This play is an out."

[0636] Next, the user (event organizer) registers the participants' face photos and identification data (such as names and uniform numbers) on the server in advance. The user then uses a dedicated web form or app to upload the event participant information (face photos and identification data) to the server. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0637] Just before the event begins, the server activates a drone camera with aerial capabilities. The server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the faces and identification data of athletes on the competition area and begins collecting data. For example, the server sends coordinates X: 123.456, coordinate Y: 789.012 to the drone camera, and the drone camera flies over the field and recognizes athletes.

[0638] Once the match begins, the drone camera will transmit live footage of the match to the host device. The drone camera will periodically capture footage of the match and transmit the data to the host device in live streaming format. Real-time footage is important data for accurately reflecting the progress of the match.

[0639] The server analyzes the data sent from the drone camera and synchronizes it with the generative AI model. Based on the analysis results, the generative AI model generates real-time commentary and commentary. For example, the generative AI model instantly generates commentary such as "Player A hits a home run!" or "This is a home run for Player A!"

[0640] The user terminal receives the generated commentary and explanation and displays it to the user. The server distributes the generated commentary data to a web stream or application, which is received and displayed on the user terminal. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[0641] Additionally, an emotion recognition engine analyzes the user's emotions and adjusts the generated commentary and commentary. The user device uses cameras and sensors to analyze the user's facial expressions and tone of voice to generate emotion data. This emotion data is sent to the host device, which instructs the generative AI model to generate a more energetic commentary. For example, if the user is excited, a more energetic commentary will be provided.

[0642] After the match ends, the server stores the video and commentary data for the entire match. All match data (video, commentary, decisions, etc.) is stored in cloud storage so that it can be used for later event data analysis and playback. For example, the server stores "match data from October 12, 2023" so that it can be analyzed and played back as needed.

[0643] Finally, after the event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing to the server through a satisfaction survey. This feedback is used to improve the generative AI model algorithm.

[0644] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0645] Program processing flow

[0646] Step 1: Training a generative AI model

[0647] The server trains the generative AI model on sports rules and past play-by-play data. Inputs include the sports rule book and past play-by-play records. The server collects this data and inputs it into the generative AI model, which then learns linguistic expressions specific to sporting events. The output is an AI model that can generate play-by-play and commentary appropriate for sporting events. Specifically, the server provides the AI ​​model with text data such as "Player A hits a home run" and "This play is an out."

[0648] Step 2: Registering Player Data

[0649] The user registers the participant's face photo and identification data (name, uniform number, etc.) on the server. The input includes the player's face photo and identification data. The user uses a dedicated web form or app to enter this data and send it to the server. The server stores the received data in a database. The output is the registered player information. Specifically, the user enters data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0650] Step 3: Start up the aerial imaging device

[0651] The server activates the aerial camera just before the event starts. The input includes GPS coordinates and flight pattern data. The server sends this data to the drone camera and commands it to activate. The output is that the camera is ready to fly over the competition area and collect footage. Specifically, the server sends coordinate data such as "X coordinate: 123.456, Y coordinate: 789.012."

[0652] Step 4: Collecting and analyzing footage during the competition

[0653] The terminal (aerial camera) transmits video of the game in real time to the host device. The input is video data captured in real time. The terminal periodically captures video of the game and transmits the data to the host device in live streaming format. The output is real-time game video data sent to the host device. In concrete terms, the drone camera transmits live video of the scene where "Player B is currently holding the ball."

[0654] Step 5: Analyze and synchronize player data

[0655] The server analyzes the facial recognition and identification data of players sent from the drone camera and synchronizes it with the generative AI model. The input is the facial image data and identification data sent from the drone camera. The server analyzes this data using a facial recognition algorithm and compares it with an existing database. The output is player data to be used for real-time commentary. Specifically, the server analyzes "Player A: face photo, uniform number 7" and synchronizes it with the generative AI model.

[0656] Step 6: Generate commentary and commentary

[0657] The server inputs the analyzed data into the generative AI model, which generates commentary and commentary in real time. The input is the analyzed game situation data. The server provides this data to the generative AI model, which generates appropriate linguistic expressions. The output is commentary and commentary generated in real time. In concrete terms, the generative AI model generates commentary such as "Player A has hit a home run!" or "This is a home run for Player A!"

[0658] Step 7: Broadcast commentary and commentary

[0659] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. The input includes the commentary data and commentary data sent from the server. The terminal receives this data and displays it to the user. The output is ready for the user to view the commentary and commentary in real time. In concrete terms, the user terminal displays a real-time commentary such as "Player B has scored a double!"

[0660] Step 8: Adjustment with Emotion Recognition Engine

[0661] The user device includes an emotion recognition engine that recognizes the user's emotions. Input includes data on the user's facial expressions and tone of voice collected by cameras and sensors. The device analyzes this data and generates emotion data. The output is the user's emotion data. The device sends this emotion data to the server. The server analyzes the collected emotion data and adjusts the commentary and commentary generated by the generative AI model in real time. Specifically, if the user is excited, the server instructs the generative AI model to generate a more energetic commentary.

[0662] Step 9: Data storage and feedback

[0663] The server stores the video and commentary data of the entire match. The input includes all video data and commentary data collected during the match. The server stores this data in cloud storage for future reference. The output is the stored match data. Specifically, the server stores the "match data for October 12, 2023" in cloud storage.

[0664] After an event, users provide feedback to the system. The input includes a user satisfaction survey. Users provide their opinions on the quality of the commentary and refereeing, which are then sent to the server. The output is collected feedback data. The server uses the collected feedback to improve the generative AI model's algorithm. Specifically, the server aggregates user feedback such as "The commentary was very easy to understand" and "The referee's decisions were accurate."

[0665] (Application example 2)

[0666] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0667] In conventional customer experiences at sporting events and in-stores, it has been difficult to provide appropriate commentary and commentary in real time, and it has also been difficult to provide information that reflects customer emotions. In particular, at small-scale events and brick-and-mortar stores, there are insufficient resources, making it difficult to provide a personalized experience for each customer. The present invention aims to solve these problems and provide a system for improving the experience at sporting events and in-stores.

[0668] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0669] In this invention, the server includes means for registering facial photographs and identification data of athletes in advance, means for activating an aerial photography device before the start of the event and recognizing the faces and identification information of athletes within the area, means for collecting and analyzing footage of the event in real time, means for using a generative AI model to generate real-time commentary and commentary based on the analysis results, means for delivering the generated commentary and commentary to customer devices, and means including an emotion analysis engine for analyzing customer emotions and adjusting the output of the generative AI model, thereby enabling the provision of real-time, personalized information to individual customers.

[0670] "Athlete" means an individual who participates in an Event and has an identity.

[0671] "Mouthshot" refers to facial image data of a player or customer, and is an image used in facial recognition algorithms.

[0672] "Identification Data" means data that individually recognizes and identifies a player or customer, and includes, by way of example, a name or number.

[0673] An "aerial photography device" is a device used to capture images from the air, such as a drone.

[0674] "Zone" means the specific location or area monitored by the aerial imaging device.

[0675] "Real-time" refers to a state in which processing and communication are carried out instantaneously without delay.

[0676] "Footage" means visual data captured in the form of still images or moving images.

[0677] "Analysis" is a method of processing collected data and images to understand their content and meaning.

[0678] A "generative AI model" refers to an artificial intelligence model that generates appropriate commentary and commentary based on pre-trained data.

[0679] "Commentary" refers to a verbal explanation of the progress of an event in real time.

[0680] "Explanation" refers to supplementary explanations of detailed information and background about an event.

[0681] "Customer" means a person receiving a service or shopping in a store.

[0682] An "emotion analysis engine" is a system that analyzes emotions from customers' facial expressions and voice and outputs them as data.

[0683] A specific system for implementing this invention aims to provide information in real time that corresponds to customer behavior and emotions by linking a server, an aerial photography device, a cloud server, a user terminal, and an emotion analysis engine.

[0684] The server first maintains a database for pre-registering the facial photographs and identification data of all people (players and customers) involved in the event or store, allowing the aerial photography device to recognize the identification information in real time based on the registered data before the event begins.

[0685] Aerial photography equipment (e.g., drones) will be activated before the start of the event and will recognize the faces and identification data of athletes and spectators in a designated area in real time. The equipment will use facial recognition algorithms to transmit the video data to a cloud server.

[0686] The cloud server analyzes the received video data and uses a generative AI model to generate real-time commentary and commentary based on that data. The generative AI model is pre-trained with the event rules and past data, and generates commentary and commentary in the appropriate language.

[0687] The generated commentary and commentary are delivered to the user's device. The user device may be a smartphone or smart glasses, and is capable of receiving and displaying streaming data from the cloud server in real time. The user device is also equipped with an emotion analysis engine that analyzes the user's facial expressions and voice to generate emotion data.

[0688] The emotional data generated by the emotion analysis engine is then sent back to the cloud server, which analyzes the data and uses the generative AI model to adjust the commentary and commentary in real time. This allows for energetic commentary if the user is excited, and calm commentary if the user is relaxed.

[0689] Examples:

[0690] Introducing cosmetics in stores

[0691] An aerial camera patrols the store and recognizes customers' faces. The recognized customer information is sent to a cloud server, and a generative AI model generates a commentary about the lipstick for that customer. For example, "This lipstick is newly released and contains special ingredients." Furthermore, if the emotion analysis engine recognizes the customer's emotion as "joy," the cloud server generates a commentary that is more in line with the emotion, adjusting the content to something like, "This lipstick also contains ingredients that will make you feel really happy!"

[0692] Example prompt sentence:

[0693] Produce an exciting commentary for the product Lipstick with the user's current emotion being happy.

[0694] In this way, the system can be used to provide a real-time, personalized experience in-store or during an event.

[0695] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0696] Step 1:

[0697] The server pre-registers the facial photographs and identification data of athletes and customers at events and stores. The facial photographs and identification data uploaded by users (event organizers or store managers) are used as input. This saves the relevant data in the database, enabling the system to identify athletes and customers.

[0698] Step 2:

[0699] Before the event starts, the server activates the aerial photography device (drone), using the event start time and flight pattern data as input. Based on this, the aerial photography device patrols the designated area and captures video data.

[0700] Step 3:

[0701] The aerial photography equipment collects images of events and stores in real time and sends them to a cloud server. The camera image data is used as input and is sent to the cloud server in real time as output.

[0702] Step 4:

[0703] The cloud server uses a facial recognition algorithm to recognize the faces of players and customers from the received video data and match them with the identification data. The video data and pre-registered identification data are used as input, and a recognition result is generated as output.

[0704] Step 5:

[0705] The cloud server inputs the recognition results into the generative AI model to generate real-time commentary and commentary. The recognition results and pre-trained data are used as inputs to the generative AI model, and appropriate commentary and commentary text is generated as output.

[0706] Step 6:

[0707] The generated commentary and commentary text is delivered from the cloud server to the user terminal, and is used as input and displayed in real time on the user terminal as output.

[0708] Step 7:

[0709] An emotion analysis engine installed in the user device analyzes the user's facial expressions and voice to generate emotion data. Camera footage and audio data are used as input, and emotion data is generated as output.

[0710] Step 8:

[0711] The user device sends emotional data to a cloud server, which then adjusts the output of the generative AI model based on the emotional data. The emotional data and prompts for the generative AI model are used as input, and adjusted commentary and commentary are generated as output according to the emotions.

[0712] Step 9:

[0713] The cloud server then delivers the adjusted commentary and commentary back to the user terminal, where the adjusted commentary and commentary text is used as input and displayed to the user as output.

[0714] In this way, the system processes and analyzes data in real time at each step, ultimately providing personalized information.

[0715] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0716] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0717] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0718] [Third embodiment]

[0719] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0720] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0721] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0722] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0723] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0724] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0725] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0726] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0727] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0728] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0729] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0730] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0731] This invention is a system that provides high-quality commentary and refereeing functions for small-scale sporting events by linking a generative AI model, a drone camera, a cloud server, and a user device. This section describes a specific embodiment of this system.

[0732] The main components of the system include:

[0733] 1. Generative AI Models

[0734] 2. Drone Camera

[0735] 3. Cloud Server

[0736] 4. User Device

[0737] Program and Processing Flow

[0738] Generative AI Models

[0739] The server pre-trains the generative AI model on sports rules and past commentary data. Specifically, the server collects text data on sports event rules and commentary from a large dataset, and inputs it into the generative AI model for training. For example, the server trains the model with commentary information such as "Player A hits a home run" or "This play is an out."

[0740] Registering Player Data

[0741] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. Specifically, the user uses a dedicated web form or app to upload event participant information (such as name, face photo, uniform number, etc.). For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0742] Drone camera launch

[0743] The server activates the drone camera just before the event starts and scans the players on the field. The server sends GPS coordinates and flight patterns to the drone camera, which automatically recognizes the players' faces and uniform numbers and collects data. For example, the drone camera flies over the field, recognizes Player A's face and uniform number, and sends that information to the server.

[0744] Analysis and Synchronization

[0745] The server analyzes the facial recognition data and uniform number data of players collected from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, the server synchronizes "Player A: Face Photo, Uniform Number 7" with the AI ​​model and uses it for real-time commentary.

[0746] Real-time video collection and analysis

[0747] The device (drone camera) transmits images of the game to the server in real time. For example, the drone camera periodically captures images of the game and transmits the data to the server in live streaming format.

[0748] The server analyzes real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the ball's position and player movements, and extracts important events as triggers. For example, it recognizes the moment the ball passes home plate as a trigger and flags it as a "possible strike."

[0749] Generate commentary and commentary

[0750] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time and outputs them as commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0751] delivery

[0752] The terminal (user terminal) receives the commentary and explanation sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by multiple users. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[0753] Data storage and feedback

[0754] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[0755] After an event, users provide feedback to the system. For example, they can send feedback about the quality of the commentary and refereeing through a satisfaction survey. The server uses the collected feedback to improve the generative AI model algorithm.

[0756] Specific examples

[0757] Baseball game commentary

[0758] Before the game, the server trains the generative AI model on the rules of baseball and past gameplay commentaries, and the drone camera recognizes the players' faces and uniform numbers when the game starts. During the game, the generative AI model analyzes the play in real time and generates commentary such as "Player B has hit a double!" The server then transmits this commentary to spectators' devices and provides it as a live stream.

[0759] Elementary school sports day

[0760] The server trains the generative AI model on the rules of the sports day and past live commentary data, and the drone camera recognizes the participants before each event begins. During the sports day, the generative AI model provides live commentary as the events progress, such as "The Class A student came out on top in the relay!" The cloud server also makes judgments, assisting the referees in providing accurate results.

[0761] In this way, this system enables fast and accurate commentary and refereeing at a variety of sporting events, realizing sponsored events that can be enjoyed by participants and spectators alike.

[0762] The processing flow will be explained below.

[0763] Step 1:

[0764] The server pre-trains the generative AI model with the rules of sports and past commentary data. Specifically, it collects text data on the rules of sporting events and commentary from a large dataset, and inputs it into the model for learning. For example, the server trains the AI ​​model with commentary information such as "Player A hits a home run" or "This play is an out."

[0765] Step 2:

[0766] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uploads event participant information (name, face photo, uniform number, etc.) using a dedicated web form or app. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0767] Step 3:

[0768] The server activates the drone camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the faces and uniform numbers of players and collects data. For example, the drone camera flies over the field, recognizes the face and uniform number of player A, and sends that information to the server.

[0769] Step 4:

[0770] The server analyzes the player's facial recognition data and uniform number data sent from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, the server synchronizes "Player A: Face Photo, Uniform Number 7" with the AI ​​model and uses it for real-time commentary.

[0771] Step 5:

[0772] The device (drone camera) transmits video footage of the match to the server in real time. The drone camera periodically captures video footage of the match and transmits the data to the server in live streaming format.

[0773] Step 6:

[0774] The server analyzes real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the ball's position and player movements, and extracts important events as triggers. For example, it recognizes the moment the ball passes home plate as a trigger and flags it as a "possible strike."

[0775] Step 7:

[0776] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time and outputs them as commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0777] Step 8:

[0778] The terminal (user terminal) receives the commentary and explanation sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by multiple users. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[0779] Step 9:

[0780] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[0781] Step 10:

[0782] After an event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing through a satisfaction survey to the server. The server uses the collected feedback to improve the generative AI model algorithm.

[0783] Example 1

[0784] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0785] In traditional sporting events, the lack of systems that provide high-quality real-time commentary and refereeing functions can lead to low spectator satisfaction and problems with the progress of the event. Small-scale sporting events, in particular, have budget and resource constraints that make it difficult to introduce advanced technology. This has led to issues such as a decline in the entertainment value for participants and spectators.

[0786] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0787] In this invention, the server includes means for registering player facial photographs and uniform number data in advance, means for activating the unmanned aerial camera before the start of the game and recognizing the faces and uniform numbers of players on the field, means for collecting and analyzing video footage of the game in real time, means for using a generative AI model to generate real-time commentary and commentary based on the analysis results, means for distributing the generated commentary and commentary to user devices, means for identifying important plays and movements in real time, means for saving video footage and commentary data of the entire game and making them available for later reference, and means for collecting user feedback and improving the algorithm of the generative AI model. This makes it possible to provide high-quality commentary and accurate refereeing functions even for small-scale sporting events, thereby increasing the satisfaction of spectators and participants.

[0788] "Athlete" means an individual player participating in a Sporting Event.

[0789] "Facial photo" refers to image data showing the player's face.

[0790] "Uniform number data" refers to the number information written on the uniform worn by a player.

[0791] "Unmanned aerial camera" refers to a camera device mounted on a drone capable of remote control or autonomous flight.

[0792] "Server" refers to a computer system that processes and manages data.

[0793] "Real-time video" refers to video data that captures and distributes the current situation in real time.

[0794] "Analysis" refers to the act of analyzing collected data and video and extracting meaningful information.

[0795] A "generative AI model" refers to an artificial intelligence system that uses machine learning and data analysis techniques to generate language expressions and explanations from input data.

[0796] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[0797] "Distribution" refers to the act of transmitting generated data or content to a user terminal via a network.

[0798] "Significant plays and actions" refers to game situations or player actions that deserve special attention in a sporting event.

[0799] "Storage" refers to the act of recording collected data and analysis results so that they can be referenced later.

[0800] "Feedback" refers to the act of collecting opinions and ratings from users.

[0801] An "algorithm" refers to a procedure or computational method for solving a specific problem.

[0802] MODE FOR CARRYING OUT THE INVENTION

[0803] System Overview

[0804] This invention is a system that can provide high-quality commentary and refereeing functions for small-scale sporting events by linking a generative AI model, an unmanned aerial camera (drone camera), a cloud server, and user devices. This system aims to achieve fast and accurate data processing and commentary for a variety of sporting events, thereby improving the satisfaction of participants and spectators.

[0805] System Components

[0806] Generative AI model: An artificial intelligence system that uses machine learning and data analysis techniques to generate appropriate language expressions and explanations from input data.

[0807] Unmanned aerial camera (drone camera): A camera device mounted on a drone that can be remotely controlled or fly autonomously, which recognizes the faces and uniform numbers of players on the field and captures real-time footage.

[0808] Cloud server: A computer system that processes and manages data, and performs tasks such as training generative AI models, data analysis, and distribution.

[0809] User device: A device used by a user, such as a computer, smartphone, or tablet, that receives and displays commentary and commentary.

[0810] Program processing

[0811] 1. Training a generative AI model

[0812] The server pre-trains the generative AI model on the rules of sports and past live commentary data. The server collects rules of sports events and live commentary text data from an existing database, and inputs this data into the generative AI model for learning. For example, the server trains the AI ​​model using example sentences such as "Player A hits a home run" and "This play is an out."

[0813] Example prompt: "You hit a home run. How would you describe it?"

[0814] 2. Registering player data

[0815] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uses a dedicated web form or app to enter event participant information (such as name, face photo, and uniform number). For example, the user might upload data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15." The server then stores this information in a database for use in later processing.

[0816] 3. Start the drone camera

[0817] The server activates the unmanned aerial camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, which automatically recognizes players' faces and uniform numbers and collects data. For example, the drone camera flies over the field, recognizes Player A's face and uniform number, and sends that information to the server. The server monitors this and ensures that accurate data is being collected.

[0818] 4. Analysis and synchronization

[0819] The server analyzes the facial recognition data and uniform number data of players collected from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database. For example, the server recognizes "Player A: face photo, uniform number 7," matches it with the data in the server, and sends the matching player information to the generative AI model.

[0820] 5. Real-time video collection and analysis

[0821] The device (drone camera) transmits video footage of the game in real time to a server. For example, the drone camera continuously captures video footage of the game and sends the data to the server in live streaming format. The server analyzes the real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the position of the ball and the movements of the players. For example, it recognizes the "moment the ball passes home base" as a trigger and flags it as a "possible strike."

[0822] 6. Commentary and Commentary Generation

[0823] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0824] 7. Distribution

[0825] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. The server then distributes the generated commentary data to a web stream or application that can be viewed by many users. For example, users can enjoy real-time commentary and commentary on their smartphones or tablets.

[0826] 8. Data Storage and Feedback

[0827] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage so that it can be used for later event data analysis and playback. Users provide feedback to the system after the event ends. For example, users can send feedback about the quality of the commentary and refereeing to the server through a satisfaction survey. The server uses the collected feedback to help improve the generative AI model algorithm.

[0828] In this way, this system can provide fast and accurate commentary and refereeing at a variety of sporting events, enabling events that can be enjoyed by participants and spectators alike.

[0829] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0830] Step 1: Training a generative AI model

[0831] The server pre-trains the generative AI model on sports rules and past commentary data. Specifically, it collects a large amount of sports event rules and commentary text data from a database connected to the server. This collected data is then input into the generative AI model, and a learning process takes place. The input data includes commentary data in text format and rule explanations. The output is an AI model skilled in generating accurate commentary based on the rules and past commentary. For example, the server trains the AI ​​model to generate appropriate language expressions in response to prompt statements such as, "There's a home run. How would you describe it?"

[0832] Step 2: Registering Player Data

[0833] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. Specifically, the user uses a dedicated web form or app to enter event participant information (such as name, face photo, and uniform number). The input data includes each player's face photo, name, and uniform number. As output, this data is saved in a database on the server. For example, data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15" are registered.

[0834] Step 3: Start the drone camera

[0835] The server activates the unmanned aerial camera just before the event starts. Specifically, the server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the players' faces and uniform numbers and collects data. The input data includes the GPS coordinates and flight patterns sent from the server. The output data, which includes the faces and uniform numbers of the players collected by the drone camera, is sent to the server. For example, the drone camera flies over the field, recognizes Player A, and sends the data to the server as "Player A: face photo, uniform number 7."

[0836] Step 4: Analyze and synchronize

[0837] The server analyzes the facial recognition data and uniform number data of players collected from the drone camera and synchronizes it with the generative AI model. Specifically, the server uses a facial recognition algorithm to match the collected player information with pre-registered data in the database. Input data includes the facial photo and uniform number data sent from the drone camera and existing information in the database. The synchronized data is sent as output to the generative AI model. For example, "Player A: facial photo, uniform number 7" is analyzed and synchronized and provided to the generative AI model.

[0838] Step 5: Real-time video collection and analysis

[0839] The device (drone camera) transmits video footage of the game in real time to a server. Specifically, the drone camera continuously captures video footage of the game and sends the data to the server in live streaming format. The input data includes real-time video footage. As output, this video data is sent to the server. The server analyzes the real-time video footage and identifies important plays and movements. The input data includes the received live streaming video. As output, information about the position of the ball and player movements is extracted. For example, the moment the ball passes home base is recognized as a trigger, and a flag is raised as a "possible strike."

[0840] Step 6: Generate commentary and commentary

[0841] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. Specifically, the server inputs the analyzed data into the generative AI model to create appropriate linguistic expressions. The input data includes information about important events and player movements during the game. The output is real-time commentary and commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0842] Step 7: Delivery

[0843] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. Specifically, the server delivers the generated commentary data to the user terminal in real time. The input data includes the real-time commentary and commentary created by the generative AI model. As output, this data is displayed on the user terminal. For example, the user can enjoy the real-time commentary and commentary on a smartphone or tablet.

[0844] Step 8: Data storage and feedback

[0845] The server stores the video and commentary data of the entire match for later reference. Specifically, the server stores all collected data in cloud storage. Input data includes video data and commentary data of the match. Output data is stored in cloud storage. Users provide feedback to the system after the event ends. Specifically, users submit their opinions and evaluations through a dedicated survey form. Input data includes user feedback. Output data is used to improve the generative AI model's algorithm.

[0846] (Application example 1)

[0847] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0848] In conventional sporting events, it has been difficult to provide high-quality commentary and commentary in real time. Furthermore, for smaller sporting events, it is difficult to assign professional commentators and referees, resulting in a poor viewing experience during the event. There is a need for a solution to this problem and to provide high-quality commentary and commentary for more events.

[0849] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0850] In this invention, the server includes a means for registering player facial photographs and uniform number data in advance, a means for activating a drone camera before the start of a game and recognizing the faces and uniform numbers of players on the field, a means for collecting and analyzing video footage during the game in real time, a means for using a generative AI model to generate real-time commentary and commentary based on the analysis results, a means for delivering the generated commentary and commentary to a user's device, and a smartphone application that enables real-time viewing of sporting events using the generated commentary and commentary, thereby providing high-quality commentary and commentary in real time and improving the viewing experience.

[0851] "Player's face photograph and uniform number data" refers to an image of the face of a player participating in a sporting event and number information for identification.

[0852] A "drone camera" is a camera device attached to an unmanned aerial vehicle that has the ability to capture video from the air.

[0853] A "facial recognition algorithm" is a computational procedure for detecting faces in images captured by a camera and identifying them as specific people.

[0854] A "server" is a central device that collects, analyzes, and distributes data over a network.

[0855] A "generative AI model" is an artificial intelligence model that generates appropriate commentary and commentary in real time by learning the rules of a sport and past live commentary data in advance.

[0856] A "user terminal" is an electronic device used by a user, such as a computer or smartphone.

[0857] A "smartphone application" is a program that runs on a smartphone and provides specific functions or services.

[0858] "Real-time viewing" means viewing an event as it is unfolding with minimal delay.

[0859] This invention is a system for providing high-quality live commentary of sporting events in real time. It connects a server, a drone camera, and user devices to collect and analyze footage from the event, and automatically generates commentary and commentary using a generative AI model, which is then distributed to users.

[0860] Program processing overview

[0861] Before the event begins, the server uses the following methods: First, it receives the players' facial photos and uniform number data from the user's device and registers them on the cloud. Next, it activates the drone camera before the event starts and uses a facial recognition algorithm (e.g., OpenCV) to recognize the faces and uniform numbers of the players on the field in real time. The acquired data is then sent to the server.

[0862] During the game, the drone camera continuously collects real-time video and transmits the video data to a server, which analyzes the received video data and identifies important plays and movements using video analysis algorithms (e.g., motion detection algorithms).

[0863] Next, the server generates real-time commentary and commentary based on the analysis results using a generative AI model (such as Hugging Face's GPT-2 model). This generative AI model has the ability to generate appropriate language expressions because it has previously learned the rules of the sport and past commentary data.

[0864] The generated commentary and commentary are delivered to a user device (e.g., a smartphone application) and displayed to the user in real time, allowing the user to experience the event with a sense of presence.

[0865] Hardware and software used

[0866] The main components of the system include:

[0867] Drone camera: A camera device mounted on an unmanned aerial vehicle that captures footage of the field in real time from the air.

[0868] Server: A central device that collects data, analyzes it, runs generative AI models, and distributes it to user devices.

[0869] User Device: An electronic device, such as a smartphone or computer, that allows a user to view and listen to the commentary and commentary in real time.

[0870] software:

[0871] Facial Recognition Algorithm (OpenCV): A computational procedure for recognizing players' faces from footage captured by a drone camera.

[0872] Generative AI model (Hugging Face's GPT-2): Generates commentary in real time based on pre-learned sports rules and past live commentary data.

[0873] Examples of concrete examples and prompts

[0874] 1. Example:

[0875] Baseball game commentary:

[0876] The drone camera monitors the entire field and captures the moment a specific player hits a home run. The server analyzes this data and uses a generative AI model to generate a commentary such as "Player A hits a home run!" The user device then broadcasts this commentary to the user in real time.

[0877] 2. Example prompt:

[0878] "Player A scores a goal. Commentator:"

[0879] "Player B hits a double. Commentator:"

[0880] In this way, this invention cleverly integrates drone cameras, generative AI models, and user devices to provide high-quality real-time commentary and significantly improve the viewing experience of sporting events.

[0881] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0882] Step 1:

[0883] Before the start of an event, users upload player photos and uniform number data to the server using a dedicated web form or smartphone application. The input is the player's photo file and uniform number information, and the output is the player data stored on the cloud server.

[0884] Step 2:

[0885] The server activates the drone camera just before the event starts and sends GPS coordinates and flight patterns to the drone to fly over the field. The input is the GPS coordinates and flight pattern data sent from the server, and the output is the drone camera flying over the field.

[0886] Step 3:

[0887] The drone camera captures the faces and uniform numbers of players on the field in real time and sends the video data to a server. The input is the video data captured by the camera, and the output is the video data sent to the cloud server. Specifically, as the drone camera flies over the field, it uses a facial recognition algorithm to recognize the faces and uniform numbers of players and sends the data to the server.

[0888] Step 4:

[0889] The server uses a facial recognition algorithm (e.g., OpenCV) to identify the faces of players from the received video data and compare them with a registered database. The input is the video data sent from the drone camera, and the output is the recognized faces and uniform number data of the players. Specifically, the facial recognition algorithm is used to detect the faces of players from the video and compare them with their uniform numbers to identify the players.

[0890] Step 5:

[0891] During the match, the drone camera collects video in real time and continuously transmits the data to the server. The input is the video data during the match, and the output is the real-time video transmitted to the server. Specifically, the drone camera captures important moments of the match and live-streams the video data to the server.

[0892] Step 6:

[0893] The server analyzes real-time video to identify important plays and movements. The input is real-time video data sent from the drone camera, and the output is identified important event data. A video analysis algorithm (e.g., a motion detection algorithm) is used for the analysis. Specifically, the server analyzes the video data, tracks the position of the ball and the movements of players, and detects specific events (e.g., goals, hits).

[0894] Step 7:

[0895] The server uses a generative AI model (for example, the GPT-2 model for Hugging Face) to generate real-time commentary and commentary based on the analysis results. The input is the identified event data and the analysis results, and the output is the generated commentary text. Specifically, the analysis results are input to the generative AI model, which generates appropriate language expressions in real time.

[0896] Step 8:

[0897] The generated commentary and commentary are delivered to the user's device (e.g., a smartphone application). The input is the generated commentary and commentary data, and the output is the commentary and commentary delivered to the user's device. Specifically, the server sends the generated commentary data to the user's device, and the user watches it in real time using the smartphone application.

[0898] Step 9:

[0899] Users can enjoy the progress of the event by viewing commentary and commentary in real time through a smartphone application. The input is commentary and commentary data delivered to the user's device, and the output is video and commentary that can be viewed in real time. Specifically, users open the application and enjoy a high-quality experience of the commentary and commentary provided as the event progresses.

[0900] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0901] This invention is a system that links a generative AI model, a drone camera, a cloud server, a user device, and an emotion engine to provide high-quality commentary and commentary for small sporting events, and real-time adjustments based on user emotions. This section describes a specific embodiment of this system.

[0902] The main components of the system include:

[0903] 1. Generative AI Models

[0904] 2. Drone Camera

[0905] 3. Cloud Server

[0906] 4. User Device

[0907] 5. Emotion Engine

[0908] Program and Processing Flow

[0909] Training generative AI models

[0910] The server pre-trains the generative AI model on sports rules and past commentary data. Specifically, it collects text data on sports event rules and commentary from a large dataset and inputs it into the generative AI model for learning. For example, the server trains the AI ​​model on commentary information such as "Player A hits a home run" or "This play is an out."

[0911] Registering Player Data

[0912] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uploads event participant information (name, face photo, uniform number, etc.) using a dedicated web form or app. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0913] Drone camera launch

[0914] The server activates the drone camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the faces and uniform numbers of players and collects data. For example, the drone camera flies over the field, recognizes the face and uniform number of player A, and sends that information to the server.

[0915] Analysis and Synchronization

[0916] The server analyzes the player's facial recognition data and uniform number data sent from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, the server synchronizes "Player A: Face Photo, Uniform Number 7" with the AI ​​model and uses it for real-time commentary.

[0917] Real-time video collection and analysis

[0918] The device (drone camera) transmits images of the game in real time to the server. The drone camera periodically captures images of the game and transmits the data to the server in live streaming format.

[0919] Generate commentary and commentary

[0920] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time and outputs them as commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0921] delivery

[0922] The terminal (user terminal) receives the commentary and explanation sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by multiple users. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[0923] Emotional engine regulation

[0924] The user terminal includes an emotion engine that recognizes the user's emotions. The emotion engine uses a camera and sensors to analyze the user's facial expressions and tone of voice, and generates emotion data. For example, if the user is happy, the emotion engine recognizes a smile or an excited voice.

[0925] The device (user device) transmits this emotional data to the server, which analyzes the collected emotional data and adjusts the commentary and commentary generated by the generative AI model in real time. For example, if the user is excited, the server instructs the generative AI model to generate a more energetic commentary.

[0926] Data storage and feedback

[0927] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[0928] After an event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing through a satisfaction survey to the server. The server uses the collected feedback to improve the generative AI model algorithm.

[0929] Specific examples

[0930] Baseball game commentary

[0931] Before the game, the server trains the generative AI model on the rules of baseball and past gameplay commentaries, and the drone camera recognizes the players' faces and uniform numbers when the game starts. During the game, the generative AI model analyzes the play in real time and generates commentary such as "Player B has hit a double!" The server then transmits this commentary to spectators' devices and provides it as a live stream.

[0932] Additionally, if the emotion engine recognizes the user's excitement, the server will modify the commentary to be more emotional and generate commentary with increased energy.

[0933] Elementary school sports day

[0934] The server trains the generative AI model on the rules of the sports day and past live commentary data, and the drone camera recognizes the participants before each event begins. During the sports day, the generative AI model provides live commentary as the events progress, such as "The Class A student came out on top in the relay!" The cloud server also makes judgments, assisting the referees in providing accurate results.

[0935] Additionally, the emotion engine analyzes the user's emotions in real time, and the system adjusts to generate more positive commentary when the user is happy.

[0936] In this way, the system provides fast and accurate commentary, commentary and emotional adjustments for a variety of sporting events, enhancing the experience for participants and spectators.

[0937] The processing flow will be explained below.

[0938] Step 1:

[0939] The server pre-trains the generative AI model with sports rules and past commentary data. Specifically, the server collects text data on the rules and commentary of sporting events from a large dataset, and inputs it into the generative AI model for training. For example, the model learns commentary information such as "Player A hits a home run" or "This play is an out."

[0940] Step 2:

[0941] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uploads event participant information (name, face photo, uniform number, etc.) using a dedicated web form or app. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0942] Step 3:

[0943] The server activates the drone camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, allowing it to automatically recognize the faces and uniform numbers of players on the field and collect data. For example, the drone camera flies over the field, recognizes the face and uniform number of player A, and sends that information to the server.

[0944] Step 4:

[0945] The server analyzes the facial recognition data and uniform number data of players sent from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, "Player A: Face Photo, Uniform Number 7" is synchronized with the AI ​​model and used for real-time commentary.

[0946] Step 5:

[0947] The device (drone camera) transmits video footage of the match to the server in real time. The drone camera periodically captures video footage of the match and transmits the data to the server in live streaming format.

[0948] Step 6:

[0949] The server analyzes real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the ball's position and player movements, and extracts important events as triggers. For example, it recognizes the moment the ball passes home plate as a trigger and flags it as a "possible strike."

[0950] Step 7:

[0951] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, generates appropriate linguistic expressions in real time, and outputs them as commentary. For example, it generates commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[0952] Step 8:

[0953] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by many users. For example, spectators can watch the commentary and commentary in real time as a video stream.

[0954] Step 9:

[0955] The user terminal includes an emotion engine that recognizes the user's emotions. The emotion engine uses a camera and sensors to analyze the user's facial expressions and tone of voice to generate emotion data. For example, if the user is happy, it recognizes a smile or an excited voice.

[0956] Step 10:

[0957] The device (user device) sends emotional data to the server. The server analyzes the collected emotional data and adjusts the commentary and commentary generated by the generative AI model in real time. For example, if the user is excited, the server instructs the generative AI model to generate a more energetic commentary.

[0958] Step 11:

[0959] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[0960] Step 12:

[0961] After an event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing through a satisfaction survey to the server. The server uses the collected feedback to improve the generative AI model algorithm.

[0962] Example 2

[0963] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0964] Conventional sports commentary systems lack real-timeness and a sense of presence, making it difficult to accurately identify players and grasp their movements during the game. Furthermore, they were unable to adjust commentary and commentary based on the emotions of the spectators, making it difficult to improve the user experience. Furthermore, efficient methods for storing and analyzing post-match data were also required.

[0965] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for registering facial photographs and identification data of players in advance, means for activating an aerial-capable camera before the start of the event and recognizing the faces and identification data of players on the competition area, means for collecting and analyzing video footage of the event in real time, means for using a language generation model to generate real-time commentary and commentary based on the analysis results, means for delivering the generated commentary and commentary to user devices, means for using an emotion recognition engine to analyze user emotions and adjust the generated commentary and commentary, means for saving video footage and commentary data of the entire game in the host device and making it available for later reference, and means for collecting user feedback and improving the language generation model. This enables real-time and realistic commentary and commentary, realizes flexible responses according to spectator emotions, and enables efficient storage and analysis of post-game data.

[0966] "Athlete Facial Image and Identification Data" means the facial image and unique identification information of athletes participating in a sporting event, which will be used as the basic data for identifying each athlete.

[0967] "Aerial-capable filming devices" refers to devices that have flight capabilities and can capture images from the air, such as drones.

[0968] "Playing Area" means the place or field where a sporting event takes place, where athletes compete and for which commentary and commentary are provided.

[0969] "Language generation model" refers to an algorithm or system that uses artificial intelligence to generate natural-looking language expressions based on input data, such as a generative AI model.

[0970] "User Device" means the device used to receive and display commentary and commentary, including, but not limited to, a smartphone, tablet, or computer.

[0971] An "emotion recognition engine" refers to a software or hardware system that uses cameras and sensors to analyze a user's facial expressions and tone of voice to generate emotional data.

[0972] "Host device" refers to a central device for managing and processing data, such as a cloud server or a high-performance computer.

[0973] "Feedback" refers to the ratings and opinions provided by users. This information is used to refine and improve the system.

[0974] This invention is a system that combines a generative AI model, an aerial-capable camera, a host device, a user terminal, and an emotion recognition engine to provide high-quality commentary and commentary for small sporting events, and real-time adjustments based on user emotions. This section describes a specific embodiment of this system.

[0975] The main components of the system include:

[0976] 1. Generative AI Models

[0977] 2. Imaging device with aerial capabilities

[0978] 3. Host Device

[0979] 4. User Terminal

[0980] 5. Emotion Recognition Engine

[0981] First, the server (host device) trains the generative AI model on the rules of sporting events and past commentary data. Specifically, it collects text data on the rules of sporting events and commentary from a large dataset, and inputs it into the generative AI model for learning. For example, the server trains the AI ​​model on commentary information such as "Player A hits a home run" or "This play is an out."

[0982] Next, the user (event organizer) registers the participants' face photos and identification data (such as names and uniform numbers) on the server in advance. The user then uses a dedicated web form or app to upload the event participant information (face photos and identification data) to the server. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0983] Just before the event begins, the server activates a drone camera with aerial capabilities. The server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the faces and identification data of athletes on the competition area and begins collecting data. For example, the server sends coordinates X: 123.456, coordinate Y: 789.012 to the drone camera, and the drone camera flies over the field and recognizes athletes.

[0984] Once the match begins, the drone camera will transmit live footage of the match to the host device. The drone camera will periodically capture footage of the match and transmit the data to the host device in live streaming format. Real-time footage is important data for accurately reflecting the progress of the match.

[0985] The server analyzes the data sent from the drone camera and synchronizes it with the generative AI model. Based on the analysis results, the generative AI model generates real-time commentary and commentary. For example, the generative AI model instantly generates commentary such as "Player A hits a home run!" or "This is a home run for Player A!"

[0986] The user terminal receives the generated commentary and explanation and displays it to the user. The server distributes the generated commentary data to a web stream or application, which is received and displayed on the user terminal. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[0987] Additionally, an emotion recognition engine analyzes the user's emotions and adjusts the generated commentary and commentary. The user device uses cameras and sensors to analyze the user's facial expressions and tone of voice to generate emotion data. This emotion data is sent to the host device, which instructs the generative AI model to generate a more energetic commentary. For example, if the user is excited, a more energetic commentary will be provided.

[0988] After the match ends, the server stores the video and commentary data for the entire match. All match data (video, commentary, decisions, etc.) is stored in cloud storage so that it can be used for later event data analysis and playback. For example, the server stores "match data from October 12, 2023" so that it can be analyzed and played back as needed.

[0989] Finally, after the event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing to the server through a satisfaction survey. This feedback is used to improve the generative AI model algorithm.

[0990] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0991] Program processing flow

[0992] Step 1: Training a generative AI model

[0993] The server trains the generative AI model on sports rules and past play-by-play data. Inputs include the sports rule book and past play-by-play records. The server collects this data and inputs it into the generative AI model, which then learns linguistic expressions specific to sporting events. The output is an AI model that can generate play-by-play and commentary appropriate for sporting events. Specifically, the server provides the AI ​​model with text data such as "Player A hits a home run" and "This play is an out."

[0994] Step 2: Registering Player Data

[0995] The user registers the participant's face photo and identification data (name, uniform number, etc.) on the server. The input includes the player's face photo and identification data. The user uses a dedicated web form or app to enter this data and send it to the server. The server stores the received data in a database. The output is the registered player information. Specifically, the user enters data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[0996] Step 3: Start up the aerial imaging device

[0997] The server activates the aerial camera just before the event starts. The input includes GPS coordinates and flight pattern data. The server sends this data to the drone camera and commands it to activate. The output is that the camera is ready to fly over the competition area and collect footage. Specifically, the server sends coordinate data such as "X coordinate: 123.456, Y coordinate: 789.012."

[0998] Step 4: Collecting and analyzing footage during the competition

[0999] The terminal (aerial camera) transmits video of the game in real time to the host device. The input is video data captured in real time. The terminal periodically captures video of the game and transmits the data to the host device in live streaming format. The output is real-time game video data sent to the host device. In concrete terms, the drone camera transmits live video of the scene where "Player B is currently holding the ball."

[1000] Step 5: Analyze and synchronize player data

[1001] The server analyzes the facial recognition and identification data of players sent from the drone camera and synchronizes it with the generative AI model. The input is the facial image data and identification data sent from the drone camera. The server analyzes this data using a facial recognition algorithm and compares it with an existing database. The output is player data to be used for real-time commentary. Specifically, the server analyzes "Player A: face photo, uniform number 7" and synchronizes it with the generative AI model.

[1002] Step 6: Generate commentary and commentary

[1003] The server inputs the analyzed data into the generative AI model, which generates commentary and commentary in real time. The input is the analyzed game situation data. The server provides this data to the generative AI model, which generates appropriate linguistic expressions. The output is commentary and commentary generated in real time. In concrete terms, the generative AI model generates commentary such as "Player A has hit a home run!" or "This is a home run for Player A!"

[1004] Step 7: Broadcast commentary and commentary

[1005] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. The input includes the commentary data and commentary data sent from the server. The terminal receives this data and displays it to the user. The output is ready for the user to view the commentary and commentary in real time. In concrete terms, the user terminal displays a real-time commentary such as "Player B has scored a double!"

[1006] Step 8: Adjustment with Emotion Recognition Engine

[1007] The user device includes an emotion recognition engine that recognizes the user's emotions. Input includes data on the user's facial expressions and tone of voice collected by cameras and sensors. The device analyzes this data and generates emotion data. The output is the user's emotion data. The device sends this emotion data to the server. The server analyzes the collected emotion data and adjusts the commentary and commentary generated by the generative AI model in real time. Specifically, if the user is excited, the server instructs the generative AI model to generate a more energetic commentary.

[1008] Step 9: Data storage and feedback

[1009] The server stores the video and commentary data of the entire match. The input includes all video data and commentary data collected during the match. The server stores this data in cloud storage for future reference. The output is the stored match data. Specifically, the server stores the "match data for October 12, 2023" in cloud storage.

[1010] After an event, users provide feedback to the system. The input includes a user satisfaction survey. Users provide their opinions on the quality of the commentary and refereeing, which are then sent to the server. The output is collected feedback data. The server uses the collected feedback to improve the generative AI model's algorithm. Specifically, the server aggregates user feedback such as "The commentary was very easy to understand" and "The referee's decisions were accurate."

[1011] (Application example 2)

[1012] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1013] In conventional customer experiences at sporting events and in-stores, it has been difficult to provide appropriate commentary and commentary in real time, and it has also been difficult to provide information that reflects customer emotions. In particular, at small-scale events and brick-and-mortar stores, there are insufficient resources, making it difficult to provide a personalized experience for each customer. The present invention aims to solve these problems and provide a system for improving the experience at sporting events and in-stores.

[1014] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1015] In this invention, the server includes means for registering facial photographs and identification data of athletes in advance, means for activating an aerial photography device before the start of the event and recognizing the faces and identification information of athletes within the area, means for collecting and analyzing footage of the event in real time, means for using a generative AI model to generate real-time commentary and commentary based on the analysis results, means for delivering the generated commentary and commentary to customer devices, and means including an emotion analysis engine for analyzing customer emotions and adjusting the output of the generative AI model, thereby enabling the provision of real-time, personalized information to individual customers.

[1016] "Athlete" means an individual who participates in an Event and has an identity.

[1017] "Mouthshot" refers to facial image data of a player or customer, and is an image used in facial recognition algorithms.

[1018] "Identification Data" means data that individually recognizes and identifies a player or customer, and includes, by way of example, a name or number.

[1019] An "aerial photography device" is a device used to capture images from the air, such as a drone.

[1020] "Zone" means the specific location or area monitored by the aerial imaging device.

[1021] "Real-time" refers to a state in which processing and communication are carried out instantaneously without delay.

[1022] "Footage" means visual data captured in the form of still images or moving images.

[1023] "Analysis" is a method of processing collected data and images to understand their content and meaning.

[1024] A "generative AI model" refers to an artificial intelligence model that generates appropriate commentary and commentary based on pre-trained data.

[1025] "Commentary" refers to a verbal explanation of the progress of an event in real time.

[1026] "Explanation" refers to supplementary explanations of detailed information and background about an event.

[1027] "Customer" means a person receiving a service or shopping in a store.

[1028] An "emotion analysis engine" is a system that analyzes emotions from customers' facial expressions and voice and outputs them as data.

[1029] A specific system for implementing this invention aims to provide information in real time that corresponds to customer behavior and emotions by linking a server, an aerial photography device, a cloud server, a user terminal, and an emotion analysis engine.

[1030] The server first maintains a database for pre-registering the facial photographs and identification data of all people (players and customers) involved in the event or store, allowing the aerial photography device to recognize the identification information in real time based on the registered data before the event begins.

[1031] Aerial photography equipment (e.g., drones) will be activated before the start of the event and will recognize the faces and identification data of athletes and spectators in a designated area in real time. The equipment will use facial recognition algorithms to transmit the video data to a cloud server.

[1032] The cloud server analyzes the received video data and uses a generative AI model to generate real-time commentary and commentary based on that data. The generative AI model is pre-trained with the event rules and past data, and generates commentary and commentary in the appropriate language.

[1033] The generated commentary and commentary are delivered to the user's device. The user device may be a smartphone or smart glasses, and is capable of receiving and displaying streaming data from the cloud server in real time. The user device is also equipped with an emotion analysis engine that analyzes the user's facial expressions and voice to generate emotion data.

[1034] The emotional data generated by the emotion analysis engine is then sent back to the cloud server, which analyzes the data and uses the generative AI model to adjust the commentary and commentary in real time. This allows for energetic commentary if the user is excited, and calm commentary if the user is relaxed.

[1035] Examples:

[1036] Introducing cosmetics in stores

[1037] An aerial camera patrols the store and recognizes customers' faces. The recognized customer information is sent to a cloud server, and a generative AI model generates a commentary about the lipstick for that customer. For example, "This lipstick is newly released and contains special ingredients." Furthermore, if the emotion analysis engine recognizes the customer's emotion as "joy," the cloud server generates a commentary that is more in line with the emotion, adjusting the content to something like, "This lipstick also contains ingredients that will make you feel really happy!"

[1038] Example prompt sentence:

[1039] Produce an exciting commentary for the product Lipstick with the user's current emotion being happy.

[1040] In this way, the system can be used to provide a real-time, personalized experience in-store or during an event.

[1041] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1042] Step 1:

[1043] The server pre-registers the facial photographs and identification data of athletes and customers at events and stores. The facial photographs and identification data uploaded by users (event organizers or store managers) are used as input. This saves the relevant data in the database, enabling the system to identify athletes and customers.

[1044] Step 2:

[1045] Before the event starts, the server activates the aerial photography device (drone), using the event start time and flight pattern data as input. Based on this, the aerial photography device patrols the designated area and captures video data.

[1046] Step 3:

[1047] The aerial photography equipment collects images of events and stores in real time and sends them to a cloud server. The camera image data is used as input and is sent to the cloud server in real time as output.

[1048] Step 4:

[1049] The cloud server uses a facial recognition algorithm to recognize the faces of players and customers from the received video data and match them with the identification data. The video data and pre-registered identification data are used as input, and a recognition result is generated as output.

[1050] Step 5:

[1051] The cloud server inputs the recognition results into the generative AI model to generate real-time commentary and commentary. The recognition results and pre-trained data are used as inputs to the generative AI model, and appropriate commentary and commentary text is generated as output.

[1052] Step 6:

[1053] The generated commentary and commentary text is delivered from the cloud server to the user terminal, and is used as input and displayed in real time on the user terminal as output.

[1054] Step 7:

[1055] An emotion analysis engine installed in the user device analyzes the user's facial expressions and voice to generate emotion data. Camera footage and audio data are used as input, and emotion data is generated as output.

[1056] Step 8:

[1057] The user device sends emotional data to a cloud server, which then adjusts the output of the generative AI model based on the emotional data. The emotional data and prompts for the generative AI model are used as input, and adjusted commentary and commentary are generated as output according to the emotions.

[1058] Step 9:

[1059] The cloud server then delivers the adjusted commentary and commentary back to the user terminal, where the adjusted commentary and commentary text is used as input and displayed to the user as output.

[1060] In this way, the system processes and analyzes data in real time at each step, ultimately providing personalized information.

[1061] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1062] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1063] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1064] [Fourth embodiment]

[1065] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1066] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1067] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1068] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1069] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1070] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1071] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1072] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1073] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1074] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1075] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1076] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1077] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1078] This invention is a system that provides high-quality commentary and refereeing functions for small-scale sporting events by linking a generative AI model, a drone camera, a cloud server, and a user device. This section describes a specific embodiment of this system.

[1079] The main components of the system include:

[1080] 1. Generative AI Models

[1081] 2. Drone Camera

[1082] 3. Cloud Server

[1083] 4. User Device

[1084] Program and Processing Flow

[1085] Generative AI Models

[1086] The server pre-trains the generative AI model on sports rules and past commentary data. Specifically, the server collects text data on sports event rules and commentary from a large dataset, and inputs it into the generative AI model for training. For example, the server trains the model with commentary information such as "Player A hits a home run" or "This play is an out."

[1087] Registering Player Data

[1088] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. Specifically, the user uses a dedicated web form or app to upload event participant information (such as name, face photo, uniform number, etc.). For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[1089] Drone camera launch

[1090] The server activates the drone camera just before the event starts and scans the players on the field. The server sends GPS coordinates and flight patterns to the drone camera, which automatically recognizes the players' faces and uniform numbers and collects data. For example, the drone camera flies over the field, recognizes Player A's face and uniform number, and sends that information to the server.

[1091] Analysis and Synchronization

[1092] The server analyzes the facial recognition data and uniform number data of players collected from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, the server synchronizes "Player A: Face Photo, Uniform Number 7" with the AI ​​model and uses it for real-time commentary.

[1093] Real-time video collection and analysis

[1094] The device (drone camera) transmits images of the game to the server in real time. For example, the drone camera periodically captures images of the game and transmits the data to the server in live streaming format.

[1095] The server analyzes real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the ball's position and player movements, and extracts important events as triggers. For example, it recognizes the moment the ball passes home plate as a trigger and flags it as a "possible strike."

[1096] Generate commentary and commentary

[1097] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time and outputs them as commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[1098] delivery

[1099] The terminal (user terminal) receives the commentary and explanation sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by multiple users. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[1100] Data storage and feedback

[1101] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[1102] After an event, users provide feedback to the system. For example, they can send feedback about the quality of the commentary and refereeing through a satisfaction survey. The server uses the collected feedback to improve the generative AI model algorithm.

[1103] Specific examples

[1104] Baseball game commentary

[1105] Before the game, the server trains the generative AI model on the rules of baseball and past gameplay commentaries, and the drone camera recognizes the players' faces and uniform numbers when the game starts. During the game, the generative AI model analyzes the play in real time and generates commentary such as "Player B has hit a double!" The server then transmits this commentary to spectators' devices and provides it as a live stream.

[1106] Elementary school sports day

[1107] The server trains the generative AI model on the rules of the sports day and past live commentary data, and the drone camera recognizes the participants before each event begins. During the sports day, the generative AI model provides live commentary as the events progress, such as "The Class A student came out on top in the relay!" The cloud server also makes judgments, assisting the referees in providing accurate results.

[1108] In this way, this system enables fast and accurate commentary and refereeing at a variety of sporting events, realizing sponsored events that can be enjoyed by participants and spectators alike.

[1109] The processing flow will be explained below.

[1110] Step 1:

[1111] The server pre-trains the generative AI model with the rules of sports and past commentary data. Specifically, it collects text data on the rules of sporting events and commentary from a large dataset, and inputs it into the model for learning. For example, the server trains the AI ​​model with commentary information such as "Player A hits a home run" or "This play is an out."

[1112] Step 2:

[1113] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uploads event participant information (name, face photo, uniform number, etc.) using a dedicated web form or app. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[1114] Step 3:

[1115] The server activates the drone camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the faces and uniform numbers of players and collects data. For example, the drone camera flies over the field, recognizes the face and uniform number of player A, and sends that information to the server.

[1116] Step 4:

[1117] The server analyzes the player's facial recognition data and uniform number data sent from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, the server synchronizes "Player A: Face Photo, Uniform Number 7" with the AI ​​model and uses it for real-time commentary.

[1118] Step 5:

[1119] The device (drone camera) transmits video footage of the match to the server in real time. The drone camera periodically captures video footage of the match and transmits the data to the server in live streaming format.

[1120] Step 6:

[1121] The server analyzes real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the ball's position and player movements, and extracts important events as triggers. For example, it recognizes the moment the ball passes home plate as a trigger and flags it as a "possible strike."

[1122] Step 7:

[1123] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time and outputs them as commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[1124] Step 8:

[1125] The terminal (user terminal) receives the commentary and explanation sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by multiple users. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[1126] Step 9:

[1127] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[1128] Step 10:

[1129] After an event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing through a satisfaction survey to the server. The server uses the collected feedback to improve the generative AI model algorithm.

[1130] Example 1

[1131] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1132] In traditional sporting events, the lack of systems that provide high-quality real-time commentary and refereeing functions can lead to low spectator satisfaction and problems with the progress of the event. Small-scale sporting events, in particular, have budget and resource constraints that make it difficult to introduce advanced technology. This has led to issues such as a decline in the entertainment value for participants and spectators.

[1133] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1134] In this invention, the server includes means for registering player facial photographs and uniform number data in advance, means for activating the unmanned aerial camera before the start of the game and recognizing the faces and uniform numbers of players on the field, means for collecting and analyzing video footage of the game in real time, means for using a generative AI model to generate real-time commentary and commentary based on the analysis results, means for distributing the generated commentary and commentary to user devices, means for identifying important plays and movements in real time, means for saving video footage and commentary data of the entire game and making them available for later reference, and means for collecting user feedback and improving the algorithm of the generative AI model. This makes it possible to provide high-quality commentary and accurate refereeing functions even for small-scale sporting events, thereby increasing the satisfaction of spectators and participants.

[1135] "Athlete" means an individual player participating in a Sporting Event.

[1136] "Facial photo" refers to image data showing the player's face.

[1137] "Uniform number data" refers to the number information written on the uniform worn by a player.

[1138] "Unmanned aerial camera" refers to a camera device mounted on a drone capable of remote control or autonomous flight.

[1139] "Server" refers to a computer system that processes and manages data.

[1140] "Real-time video" refers to video data that captures and distributes the current situation in real time.

[1141] "Analysis" refers to the act of analyzing collected data and video and extracting meaningful information.

[1142] A "generative AI model" refers to an artificial intelligence system that uses machine learning and data analysis techniques to generate language expressions and explanations from input data.

[1143] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[1144] "Distribution" refers to the act of transmitting generated data or content to a user terminal via a network.

[1145] "Significant plays and actions" refers to game situations or player actions that deserve special attention in a sporting event.

[1146] "Storage" refers to the act of recording collected data and analysis results so that they can be referenced later.

[1147] "Feedback" refers to the act of collecting opinions and ratings from users.

[1148] An "algorithm" refers to a procedure or computational method for solving a specific problem.

[1149] MODE FOR CARRYING OUT THE INVENTION

[1150] System Overview

[1151] This invention is a system that can provide high-quality commentary and refereeing functions for small-scale sporting events by linking a generative AI model, an unmanned aerial camera (drone camera), a cloud server, and user devices. This system aims to achieve fast and accurate data processing and commentary for a variety of sporting events, thereby improving the satisfaction of participants and spectators.

[1152] System Components

[1153] Generative AI model: An artificial intelligence system that uses machine learning and data analysis techniques to generate appropriate language expressions and explanations from input data.

[1154] Unmanned aerial camera (drone camera): A camera device mounted on a drone that can be remotely controlled or fly autonomously, which recognizes the faces and uniform numbers of players on the field and captures real-time footage.

[1155] Cloud server: A computer system that processes and manages data, and performs tasks such as training generative AI models, data analysis, and distribution.

[1156] User device: A device used by a user, such as a computer, smartphone, or tablet, that receives and displays commentary and commentary.

[1157] Program processing

[1158] 1. Training a generative AI model

[1159] The server pre-trains the generative AI model on the rules of sports and past live commentary data. The server collects rules of sports events and live commentary text data from an existing database, and inputs this data into the generative AI model for learning. For example, the server trains the AI ​​model using example sentences such as "Player A hits a home run" and "This play is an out."

[1160] Example prompt: "You hit a home run. How would you describe it?"

[1161] 2. Registering player data

[1162] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uses a dedicated web form or app to enter event participant information (such as name, face photo, and uniform number). For example, the user might upload data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15." The server then stores this information in a database for use in later processing.

[1163] 3. Start the drone camera

[1164] The server activates the unmanned aerial camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, which automatically recognizes players' faces and uniform numbers and collects data. For example, the drone camera flies over the field, recognizes Player A's face and uniform number, and sends that information to the server. The server monitors this and ensures that accurate data is being collected.

[1165] 4. Analysis and synchronization

[1166] The server analyzes the facial recognition data and uniform number data of players collected from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database. For example, the server recognizes "Player A: face photo, uniform number 7," matches it with the data in the server, and sends the matching player information to the generative AI model.

[1167] 5. Real-time video collection and analysis

[1168] The device (drone camera) transmits video footage of the game in real time to a server. For example, the drone camera continuously captures video footage of the game and sends the data to the server in live streaming format. The server analyzes the real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the position of the ball and the movements of the players. For example, it recognizes the "moment the ball passes home base" as a trigger and flags it as a "possible strike."

[1169] 6. Commentary and Commentary Generation

[1170] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[1171] 7. Distribution

[1172] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. The server then distributes the generated commentary data to a web stream or application that can be viewed by many users. For example, users can enjoy real-time commentary and commentary on their smartphones or tablets.

[1173] 8. Data Storage and Feedback

[1174] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage so that it can be used for later event data analysis and playback. Users provide feedback to the system after the event ends. For example, users can send feedback about the quality of the commentary and refereeing to the server through a satisfaction survey. The server uses the collected feedback to help improve the generative AI model algorithm.

[1175] In this way, this system can provide fast and accurate commentary and refereeing at a variety of sporting events, enabling events that can be enjoyed by participants and spectators alike.

[1176] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1177] Step 1: Training a generative AI model

[1178] The server pre-trains the generative AI model on sports rules and past commentary data. Specifically, it collects a large amount of sports event rules and commentary text data from a database connected to the server. This collected data is then input into the generative AI model, and a learning process takes place. The input data includes commentary data in text format and rule explanations. The output is an AI model skilled in generating accurate commentary based on the rules and past commentary. For example, the server trains the AI ​​model to generate appropriate language expressions in response to prompt statements such as, "There's a home run. How would you describe it?"

[1179] Step 2: Registering Player Data

[1180] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. Specifically, the user uses a dedicated web form or app to enter event participant information (such as name, face photo, and uniform number). The input data includes each player's face photo, name, and uniform number. As output, this data is saved in a database on the server. For example, data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15" are registered.

[1181] Step 3: Start the drone camera

[1182] The server activates the unmanned aerial camera just before the event starts. Specifically, the server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the players' faces and uniform numbers and collects data. The input data includes the GPS coordinates and flight patterns sent from the server. The output data, which includes the faces and uniform numbers of the players collected by the drone camera, is sent to the server. For example, the drone camera flies over the field, recognizes Player A, and sends the data to the server as "Player A: face photo, uniform number 7."

[1183] Step 4: Analyze and synchronize

[1184] The server analyzes the facial recognition data and uniform number data of players collected from the drone camera and synchronizes it with the generative AI model. Specifically, the server uses a facial recognition algorithm to match the collected player information with pre-registered data in the database. Input data includes the facial photo and uniform number data sent from the drone camera and existing information in the database. The synchronized data is sent as output to the generative AI model. For example, "Player A: facial photo, uniform number 7" is analyzed and synchronized and provided to the generative AI model.

[1185] Step 5: Real-time video collection and analysis

[1186] The device (drone camera) transmits video footage of the game in real time to a server. Specifically, the drone camera continuously captures video footage of the game and sends the data to the server in live streaming format. The input data includes real-time video footage. As output, this video data is sent to the server. The server analyzes the real-time video footage and identifies important plays and movements. The input data includes the received live streaming video. As output, information about the position of the ball and player movements is extracted. For example, the moment the ball passes home base is recognized as a trigger, and a flag is raised as a "possible strike."

[1187] Step 6: Generate commentary and commentary

[1188] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. Specifically, the server inputs the analyzed data into the generative AI model to create appropriate linguistic expressions. The input data includes information about important events and player movements during the game. The output is real-time commentary and commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[1189] Step 7: Delivery

[1190] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. Specifically, the server delivers the generated commentary data to the user terminal in real time. The input data includes the real-time commentary and commentary created by the generative AI model. As output, this data is displayed on the user terminal. For example, the user can enjoy the real-time commentary and commentary on a smartphone or tablet.

[1191] Step 8: Data storage and feedback

[1192] The server stores the video and commentary data of the entire match for later reference. Specifically, the server stores all collected data in cloud storage. Input data includes video data and commentary data of the match. Output data is stored in cloud storage. Users provide feedback to the system after the event ends. Specifically, users submit their opinions and evaluations through a dedicated survey form. Input data includes user feedback. Output data is used to improve the generative AI model's algorithm.

[1193] (Application example 1)

[1194] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1195] In conventional sporting events, it has been difficult to provide high-quality commentary and commentary in real time. Furthermore, for smaller sporting events, it is difficult to assign professional commentators and referees, resulting in a poor viewing experience during the event. There is a need for a solution to this problem and to provide high-quality commentary and commentary for more events.

[1196] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1197] In this invention, the server includes a means for registering player facial photographs and uniform number data in advance, a means for activating a drone camera before the start of a game and recognizing the faces and uniform numbers of players on the field, a means for collecting and analyzing video footage during the game in real time, a means for using a generative AI model to generate real-time commentary and commentary based on the analysis results, a means for delivering the generated commentary and commentary to a user's device, and a smartphone application that enables real-time viewing of sporting events using the generated commentary and commentary, thereby providing high-quality commentary and commentary in real time and improving the viewing experience.

[1198] "Player's face photograph and uniform number data" refers to an image of the face of a player participating in a sporting event and number information for identification.

[1199] A "drone camera" is a camera device attached to an unmanned aerial vehicle that has the ability to capture video from the air.

[1200] A "facial recognition algorithm" is a computational procedure for detecting faces in images captured by a camera and identifying them as specific people.

[1201] A "server" is a central device that collects, analyzes, and distributes data over a network.

[1202] A "generative AI model" is an artificial intelligence model that generates appropriate commentary and commentary in real time by learning the rules of a sport and past live commentary data in advance.

[1203] A "user terminal" is an electronic device used by a user, such as a computer or smartphone.

[1204] A "smartphone application" is a program that runs on a smartphone and provides specific functions or services.

[1205] "Real-time viewing" means viewing an event as it is unfolding with minimal delay.

[1206] This invention is a system for providing high-quality live commentary of sporting events in real time. It connects a server, a drone camera, and user devices to collect and analyze footage from the event, and automatically generates commentary and commentary using a generative AI model, which is then distributed to users.

[1207] Program processing overview

[1208] Before the event begins, the server uses the following methods: First, it receives the players' facial photos and uniform number data from the user's device and registers them on the cloud. Next, it activates the drone camera before the event starts and uses a facial recognition algorithm (e.g., OpenCV) to recognize the faces and uniform numbers of the players on the field in real time. The acquired data is then sent to the server.

[1209] During the game, the drone camera continuously collects real-time video and transmits the video data to a server, which analyzes the received video data and identifies important plays and movements using video analysis algorithms (e.g., motion detection algorithms).

[1210] Next, the server generates real-time commentary and commentary based on the analysis results using a generative AI model (such as Hugging Face's GPT-2 model). This generative AI model has the ability to generate appropriate language expressions because it has previously learned the rules of the sport and past commentary data.

[1211] The generated commentary and commentary are delivered to a user device (e.g., a smartphone application) and displayed to the user in real time, allowing the user to experience the event with a sense of presence.

[1212] Hardware and software used

[1213] The main components of the system include:

[1214] Drone camera: A camera device mounted on an unmanned aerial vehicle that captures footage of the field in real time from the air.

[1215] Server: A central device that collects data, analyzes it, runs generative AI models, and distributes it to user devices.

[1216] User Device: An electronic device, such as a smartphone or computer, that allows a user to view and listen to the commentary and commentary in real time.

[1217] software:

[1218] Facial Recognition Algorithm (OpenCV): A computational procedure for recognizing players' faces from footage captured by a drone camera.

[1219] Generative AI model (Hugging Face's GPT-2): Generates commentary in real time based on pre-learned sports rules and past live commentary data.

[1220] Examples of concrete examples and prompts

[1221] 1. Example:

[1222] Baseball game commentary:

[1223] The drone camera monitors the entire field and captures the moment a specific player hits a home run. The server analyzes this data and uses a generative AI model to generate a commentary such as "Player A hits a home run!" The user device then broadcasts this commentary to the user in real time.

[1224] 2. Example prompt:

[1225] "Player A scores a goal. Commentator:"

[1226] "Player B hits a double. Commentator:"

[1227] In this way, this invention cleverly integrates drone cameras, generative AI models, and user devices to provide high-quality real-time commentary and significantly improve the viewing experience of sporting events.

[1228] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1229] Step 1:

[1230] Before the start of an event, users upload player photos and uniform number data to the server using a dedicated web form or smartphone application. The input is the player's photo file and uniform number information, and the output is the player data stored on the cloud server.

[1231] Step 2:

[1232] The server activates the drone camera just before the event starts and sends GPS coordinates and flight patterns to the drone to fly over the field. The input is the GPS coordinates and flight pattern data sent from the server, and the output is the drone camera flying over the field.

[1233] Step 3:

[1234] The drone camera captures the faces and uniform numbers of players on the field in real time and sends the video data to a server. The input is the video data captured by the camera, and the output is the video data sent to the cloud server. Specifically, as the drone camera flies over the field, it uses a facial recognition algorithm to recognize the faces and uniform numbers of players and sends the data to the server.

[1235] Step 4:

[1236] The server uses a facial recognition algorithm (e.g., OpenCV) to identify the faces of players from the received video data and compare them with a registered database. The input is the video data sent from the drone camera, and the output is the recognized faces and uniform number data of the players. Specifically, the facial recognition algorithm is used to detect the faces of players from the video and compare them with their uniform numbers to identify the players.

[1237] Step 5:

[1238] During the match, the drone camera collects video in real time and continuously transmits the data to the server. The input is the video data during the match, and the output is the real-time video transmitted to the server. Specifically, the drone camera captures important moments of the match and live-streams the video data to the server.

[1239] Step 6:

[1240] The server analyzes real-time video to identify important plays and movements. The input is real-time video data sent from the drone camera, and the output is identified important event data. A video analysis algorithm (e.g., a motion detection algorithm) is used for the analysis. Specifically, the server analyzes the video data, tracks the position of the ball and the movements of players, and detects specific events (e.g., goals, hits).

[1241] Step 7:

[1242] The server uses a generative AI model (for example, the GPT-2 model for Hugging Face) to generate real-time commentary and commentary based on the analysis results. The input is the identified event data and the analysis results, and the output is the generated commentary text. Specifically, the analysis results are input to the generative AI model, which generates appropriate language expressions in real time.

[1243] Step 8:

[1244] The generated commentary and commentary are delivered to the user's device (e.g., a smartphone application). The input is the generated commentary and commentary data, and the output is the commentary and commentary delivered to the user's device. Specifically, the server sends the generated commentary data to the user's device, and the user watches it in real time using the smartphone application.

[1245] Step 9:

[1246] Users can enjoy the progress of the event by viewing commentary and commentary in real time through a smartphone application. The input is commentary and commentary data delivered to the user's device, and the output is video and commentary that can be viewed in real time. Specifically, users open the application and enjoy a high-quality experience of the commentary and commentary provided as the event progresses.

[1247] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1248] This invention is a system that links a generative AI model, a drone camera, a cloud server, a user device, and an emotion engine to provide high-quality commentary and commentary for small sporting events, and real-time adjustments based on user emotions. This section describes a specific embodiment of this system.

[1249] The main components of the system include:

[1250] 1. Generative AI Models

[1251] 2. Drone Camera

[1252] 3. Cloud Server

[1253] 4. User Device

[1254] 5. Emotion Engine

[1255] Program and Processing Flow

[1256] Training generative AI models

[1257] The server pre-trains the generative AI model on sports rules and past commentary data. Specifically, it collects text data on sports event rules and commentary from a large dataset and inputs it into the generative AI model for learning. For example, the server trains the AI ​​model on commentary information such as "Player A hits a home run" or "This play is an out."

[1258] Registering Player Data

[1259] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uploads event participant information (name, face photo, uniform number, etc.) using a dedicated web form or app. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[1260] Drone camera launch

[1261] The server activates the drone camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the faces and uniform numbers of players and collects data. For example, the drone camera flies over the field, recognizes the face and uniform number of player A, and sends that information to the server.

[1262] Analysis and Synchronization

[1263] The server analyzes the player's facial recognition data and uniform number data sent from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, the server synchronizes "Player A: Face Photo, Uniform Number 7" with the AI ​​model and uses it for real-time commentary.

[1264] Real-time video collection and analysis

[1265] The device (drone camera) transmits images of the game in real time to the server. The drone camera periodically captures images of the game and transmits the data to the server in live streaming format.

[1266] Generate commentary and commentary

[1267] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, which then generates appropriate language expressions in real time and outputs them as commentary. For example, the generative AI model might generate commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[1268] delivery

[1269] The terminal (user terminal) receives the commentary and explanation sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by multiple users. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[1270] Emotional engine regulation

[1271] The user terminal includes an emotion engine that recognizes the user's emotions. The emotion engine uses a camera and sensors to analyze the user's facial expressions and tone of voice, and generates emotion data. For example, if the user is happy, the emotion engine recognizes a smile or an excited voice.

[1272] The device (user device) transmits this emotional data to the server, which analyzes the collected emotional data and adjusts the commentary and commentary generated by the generative AI model in real time. For example, if the user is excited, the server instructs the generative AI model to generate a more energetic commentary.

[1273] Data storage and feedback

[1274] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[1275] After an event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing through a satisfaction survey to the server. The server uses the collected feedback to improve the generative AI model algorithm.

[1276] Specific examples

[1277] Baseball game commentary

[1278] Before the game, the server trains the generative AI model on the rules of baseball and past gameplay commentaries, and the drone camera recognizes the players' faces and uniform numbers when the game starts. During the game, the generative AI model analyzes the play in real time and generates commentary such as "Player B has hit a double!" The server then transmits this commentary to spectators' devices and provides it as a live stream.

[1279] Additionally, if the emotion engine recognizes the user's excitement, the server will modify the commentary to be more emotional and generate commentary with increased energy.

[1280] Elementary school sports day

[1281] The server trains the generative AI model on the rules of the sports day and past live commentary data, and the drone camera recognizes the participants before each event begins. During the sports day, the generative AI model provides live commentary as the events progress, such as "The Class A student came out on top in the relay!" The cloud server also makes judgments, assisting the referees in providing accurate results.

[1282] Additionally, the emotion engine analyzes the user's emotions in real time, and the system adjusts to generate more positive commentary when the user is happy.

[1283] In this way, the system provides fast and accurate commentary, commentary and emotional adjustments for a variety of sporting events, enhancing the experience for participants and spectators.

[1284] The processing flow will be explained below.

[1285] Step 1:

[1286] The server pre-trains the generative AI model with sports rules and past commentary data. Specifically, the server collects text data on the rules and commentary of sporting events from a large dataset, and inputs it into the generative AI model for training. For example, the model learns commentary information such as "Player A hits a home run" or "This play is an out."

[1287] Step 2:

[1288] The user (event organizer) registers participants' face photos and uniform number data on the server in advance. The user uploads event participant information (name, face photo, uniform number, etc.) using a dedicated web form or app. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[1289] Step 3:

[1290] The server activates the drone camera just before the event starts. The server sends GPS coordinates and flight patterns to the drone camera, allowing it to automatically recognize the faces and uniform numbers of players on the field and collect data. For example, the drone camera flies over the field, recognizes the face and uniform number of player A, and sends that information to the server.

[1291] Step 4:

[1292] The server analyzes the facial recognition data and uniform number data of players sent from the drone camera and synchronizes it with the generative AI model. The server uses a facial recognition algorithm to match the collected player information with an existing database and sends matching player information to the generative AI model. For example, "Player A: Face Photo, Uniform Number 7" is synchronized with the AI ​​model and used for real-time commentary.

[1293] Step 5:

[1294] The device (drone camera) transmits video footage of the match to the server in real time. The drone camera periodically captures video footage of the match and transmits the data to the server in live streaming format.

[1295] Step 6:

[1296] The server analyzes real-time video and identifies important plays and movements. The server uses a video analysis algorithm to recognize the ball's position and player movements, and extracts important events as triggers. For example, it recognizes the moment the ball passes home plate as a trigger and flags it as a "possible strike."

[1297] Step 7:

[1298] The server uses the generative AI model to generate real-time commentary and commentary based on trigger events. The server inputs the analyzed data into the generative AI model, generates appropriate linguistic expressions in real time, and outputs them as commentary. For example, it generates commentary such as, "Player A hits a home run!" or "This is a home run for Player A!"

[1299] Step 8:

[1300] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. The server distributes the generated commentary data to a web stream or application that can be viewed by many users. For example, spectators can watch the commentary and commentary in real time as a video stream.

[1301] Step 9:

[1302] The user terminal includes an emotion engine that recognizes the user's emotions. The emotion engine uses a camera and sensors to analyze the user's facial expressions and tone of voice to generate emotion data. For example, if the user is happy, it recognizes a smile or an excited voice.

[1303] Step 10:

[1304] The device (user device) sends emotional data to the server. The server analyzes the collected emotional data and adjusts the commentary and commentary generated by the generative AI model in real time. For example, if the user is excited, the server instructs the generative AI model to generate a more energetic commentary.

[1305] Step 11:

[1306] The server stores the entire match footage and commentary data for later reference. For example, the server stores all match data (footage, commentary, and decisions) in cloud storage for later event data analysis and playback.

[1307] Step 12:

[1308] After an event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing through a satisfaction survey to the server. The server uses the collected feedback to improve the generative AI model algorithm.

[1309] Example 2

[1310] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1311] Conventional sports commentary systems lack real-timeness and a sense of presence, making it difficult to accurately identify players and grasp their movements during the game. Furthermore, they were unable to adjust commentary and commentary based on the emotions of the spectators, making it difficult to improve the user experience. Furthermore, efficient methods for storing and analyzing post-match data were also required.

[1312] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for registering facial photographs and identification data of players in advance, means for activating an aerial-capable camera before the start of the event and recognizing the faces and identification data of players on the competition area, means for collecting and analyzing video footage of the event in real time, means for using a language generation model to generate real-time commentary and commentary based on the analysis results, means for delivering the generated commentary and commentary to user devices, means for using an emotion recognition engine to analyze user emotions and adjust the generated commentary and commentary, means for saving video footage and commentary data of the entire game in the host device and making it available for later reference, and means for collecting user feedback and improving the language generation model. This enables real-time and realistic commentary and commentary, realizes flexible responses according to spectator emotions, and enables efficient storage and analysis of post-game data.

[1313] "Athlete Facial Image and Identification Data" means the facial image and unique identification information of athletes participating in a sporting event, which will be used as the basic data for identifying each athlete.

[1314] "Aerial-capable filming devices" refers to devices that have flight capabilities and can capture images from the air, such as drones.

[1315] "Playing Area" means the place or field where a sporting event takes place, where athletes compete and for which commentary and commentary are provided.

[1316] "Language generation model" refers to an algorithm or system that uses artificial intelligence to generate natural-looking language expressions based on input data, such as a generative AI model.

[1317] "User Device" means the device used to receive and display commentary and commentary, including, but not limited to, a smartphone, tablet, or computer.

[1318] An "emotion recognition engine" refers to a software or hardware system that uses cameras and sensors to analyze a user's facial expressions and tone of voice to generate emotional data.

[1319] "Host device" refers to a central device for managing and processing data, such as a cloud server or a high-performance computer.

[1320] "Feedback" refers to the ratings and opinions provided by users. This information is used to refine and improve the system.

[1321] This invention is a system that combines a generative AI model, an aerial-capable camera, a host device, a user terminal, and an emotion recognition engine to provide high-quality commentary and commentary for small sporting events, and real-time adjustments based on user emotions. This section describes a specific embodiment of this system.

[1322] The main components of the system include:

[1323] 1. Generative AI Models

[1324] 2. Imaging device with aerial capabilities

[1325] 3. Host Device

[1326] 4. User Terminal

[1327] 5. Emotion Recognition Engine

[1328] First, the server (host device) trains the generative AI model on the rules of sporting events and past commentary data. Specifically, it collects text data on the rules of sporting events and commentary from a large dataset, and inputs it into the generative AI model for learning. For example, the server trains the AI ​​model on commentary information such as "Player A hits a home run" or "This play is an out."

[1329] Next, the user (event organizer) registers the participants' face photos and identification data (such as names and uniform numbers) on the server in advance. The user then uses a dedicated web form or app to upload the event participant information (face photos and identification data) to the server. For example, the user might enter data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[1330] Just before the event begins, the server activates a drone camera with aerial capabilities. The server sends GPS coordinates and flight patterns to the drone camera, which then automatically recognizes the faces and identification data of athletes on the competition area and begins collecting data. For example, the server sends coordinates X: 123.456, coordinate Y: 789.012 to the drone camera, and the drone camera flies over the field and recognizes athletes.

[1331] Once the match begins, the drone camera will transmit live footage of the match to the host device. The drone camera will periodically capture footage of the match and transmit the data to the host device in live streaming format. Real-time footage is important data for accurately reflecting the progress of the match.

[1332] The server analyzes the data sent from the drone camera and synchronizes it with the generative AI model. Based on the analysis results, the generative AI model generates real-time commentary and commentary. For example, the generative AI model instantly generates commentary such as "Player A hits a home run!" or "This is a home run for Player A!"

[1333] The user terminal receives the generated commentary and explanation and displays it to the user. The server distributes the generated commentary data to a web stream or application, which is received and displayed on the user terminal. For example, multiple users can watch the commentary and explanation in real time as video streaming.

[1334] Additionally, an emotion recognition engine analyzes the user's emotions and adjusts the generated commentary and commentary. The user device uses cameras and sensors to analyze the user's facial expressions and tone of voice to generate emotion data. This emotion data is sent to the host device, which instructs the generative AI model to generate a more energetic commentary. For example, if the user is excited, a more energetic commentary will be provided.

[1335] After the match ends, the server stores the video and commentary data for the entire match. All match data (video, commentary, decisions, etc.) is stored in cloud storage so that it can be used for later event data analysis and playback. For example, the server stores "match data from October 12, 2023" so that it can be analyzed and played back as needed.

[1336] Finally, after the event, users provide feedback to the system. Users send feedback about the quality of the commentary and refereeing to the server through a satisfaction survey. This feedback is used to improve the generative AI model algorithm.

[1337] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1338] Program processing flow

[1339] Step 1: Training a generative AI model

[1340] The server trains the generative AI model on sports rules and past play-by-play data. Inputs include the sports rule book and past play-by-play records. The server collects this data and inputs it into the generative AI model, which then learns linguistic expressions specific to sporting events. The output is an AI model that can generate play-by-play and commentary appropriate for sporting events. Specifically, the server provides the AI ​​model with text data such as "Player A hits a home run" and "This play is an out."

[1341] Step 2: Registering Player Data

[1342] The user registers the participant's face photo and identification data (name, uniform number, etc.) on the server. The input includes the player's face photo and identification data. The user uses a dedicated web form or app to enter this data and send it to the server. The server stores the received data in a database. The output is the registered player information. Specifically, the user enters data such as "Player A: face photo, uniform number 7" and "Player B: face photo, uniform number 15."

[1343] Step 3: Start up the aerial imaging device

[1344] The server activates the aerial camera just before the event starts. The input includes GPS coordinates and flight pattern data. The server sends this data to the drone camera and commands it to activate. The output is that the camera is ready to fly over the competition area and collect footage. Specifically, the server sends coordinate data such as "X coordinate: 123.456, Y coordinate: 789.012."

[1345] Step 4: Collecting and analyzing footage during the competition

[1346] The terminal (aerial camera) transmits video of the game in real time to the host device. The input is video data captured in real time. The terminal periodically captures video of the game and transmits the data to the host device in live streaming format. The output is real-time game video data sent to the host device. In concrete terms, the drone camera transmits live video of the scene where "Player B is currently holding the ball."

[1347] Step 5: Analyze and synchronize player data

[1348] The server analyzes the facial recognition and identification data of players sent from the drone camera and synchronizes it with the generative AI model. The input is the facial image data and identification data sent from the drone camera. The server analyzes this data using a facial recognition algorithm and compares it with an existing database. The output is player data to be used for real-time commentary. Specifically, the server analyzes "Player A: face photo, uniform number 7" and synchronizes it with the generative AI model.

[1349] Step 6: Generate commentary and commentary

[1350] The server inputs the analyzed data into the generative AI model, which generates commentary and commentary in real time. The input is the analyzed game situation data. The server provides this data to the generative AI model, which generates appropriate linguistic expressions. The output is commentary and commentary generated in real time. In concrete terms, the generative AI model generates commentary such as "Player A has hit a home run!" or "This is a home run for Player A!"

[1351] Step 7: Broadcast commentary and commentary

[1352] The terminal (user terminal) receives the commentary and commentary sent from the server and displays it to the user. The input includes the commentary data and commentary data sent from the server. The terminal receives this data and displays it to the user. The output is ready for the user to view the commentary and commentary in real time. In concrete terms, the user terminal displays a real-time commentary such as "Player B has scored a double!"

[1353] Step 8: Adjustment with Emotion Recognition Engine

[1354] The user device includes an emotion recognition engine that recognizes the user's emotions. Input includes data on the user's facial expressions and tone of voice collected by cameras and sensors. The device analyzes this data and generates emotion data. The output is the user's emotion data. The device sends this emotion data to the server. The server analyzes the collected emotion data and adjusts the commentary and commentary generated by the generative AI model in real time. Specifically, if the user is excited, the server instructs the generative AI model to generate a more energetic commentary.

[1355] Step 9: Data storage and feedback

[1356] The server stores the video and commentary data of the entire match. The input includes all video data and commentary data collected during the match. The server stores this data in cloud storage for future reference. The output is the stored match data. Specifically, the server stores the "match data for October 12, 2023" in cloud storage.

[1357] After an event, users provide feedback to the system. The input includes a user satisfaction survey. Users provide their opinions on the quality of the commentary and refereeing, which are then sent to the server. The output is collected feedback data. The server uses the collected feedback to improve the generative AI model's algorithm. Specifically, the server aggregates user feedback such as "The commentary was very easy to understand" and "The referee's decisions were accurate."

[1358] (Application example 2)

[1359] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1360] In conventional customer experiences at sporting events and in-stores, it has been difficult to provide appropriate commentary and commentary in real time, and it has also been difficult to provide information that reflects customer emotions. In particular, at small-scale events and brick-and-mortar stores, there are insufficient resources, making it difficult to provide a personalized experience for each customer. The present invention aims to solve these problems and provide a system for improving the experience at sporting events and in-stores.

[1361] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1362] In this invention, the server includes means for registering facial photographs and identification data of athletes in advance, means for activating an aerial photography device before the start of the event and recognizing the faces and identification information of athletes within the area, means for collecting and analyzing footage of the event in real time, means for using a generative AI model to generate real-time commentary and commentary based on the analysis results, means for delivering the generated commentary and commentary to customer devices, and means including an emotion analysis engine for analyzing customer emotions and adjusting the output of the generative AI model, thereby enabling the provision of real-time, personalized information to individual customers.

[1363] "Athlete" means an individual who participates in an Event and has an identity.

[1364] "Mouthshot" refers to facial image data of a player or customer, and is an image used in facial recognition algorithms.

[1365] "Identification Data" means data that individually recognizes and identifies a player or customer, and includes, by way of example, a name or number.

[1366] An "aerial photography device" is a device used to capture images from the air, such as a drone.

[1367] "Zone" means the specific location or area monitored by the aerial imaging device.

[1368] "Real-time" refers to a state in which processing and communication are carried out instantaneously without delay.

[1369] "Footage" means visual data captured in the form of still images or moving images.

[1370] "Analysis" is a method of processing collected data and images to understand their content and meaning.

[1371] A "generative AI model" refers to an artificial intelligence model that generates appropriate commentary and commentary based on pre-trained data.

[1372] "Commentary" refers to a verbal explanation of the progress of an event in real time.

[1373] "Explanation" refers to supplementary explanations of detailed information and background about an event.

[1374] "Customer" means a person receiving a service or shopping in a store.

[1375] An "emotion analysis engine" is a system that analyzes emotions from customers' facial expressions and voice and outputs them as data.

[1376] A specific system for implementing this invention aims to provide information in real time that corresponds to customer behavior and emotions by linking a server, an aerial photography device, a cloud server, a user terminal, and an emotion analysis engine.

[1377] The server first maintains a database for pre-registering the facial photographs and identification data of all people (players and customers) involved in the event or store, allowing the aerial photography device to recognize the identification information in real time based on the registered data before the event begins.

[1378] Aerial photography equipment (e.g., drones) will be activated before the start of the event and will recognize the faces and identification data of athletes and spectators in a designated area in real time. The equipment will use facial recognition algorithms to transmit the video data to a cloud server.

[1379] The cloud server analyzes the received video data and uses a generative AI model to generate real-time commentary and commentary based on that data. The generative AI model is pre-trained with the event rules and past data, and generates commentary and commentary in the appropriate language.

[1380] The generated commentary and commentary are delivered to the user's device. The user device may be a smartphone or smart glasses, and is capable of receiving and displaying streaming data from the cloud server in real time. The user device is also equipped with an emotion analysis engine that analyzes the user's facial expressions and voice to generate emotion data.

[1381] The emotional data generated by the emotion analysis engine is then sent back to the cloud server, which analyzes the data and uses the generative AI model to adjust the commentary and commentary in real time. This allows for energetic commentary if the user is excited, and calm commentary if the user is relaxed.

[1382] Examples:

[1383] Introducing cosmetics in stores

[1384] An aerial camera patrols the store and recognizes customers' faces. The recognized customer information is sent to a cloud server, and a generative AI model generates a commentary about the lipstick for that customer. For example, "This lipstick is newly released and contains special ingredients." Furthermore, if the emotion analysis engine recognizes the customer's emotion as "joy," the cloud server generates a commentary that is more in line with the emotion, adjusting the content to something like, "This lipstick also contains ingredients that will make you feel really happy!"

[1385] Example prompt sentence:

[1386] Produce an exciting commentary for the product Lipstick with the user's current emotion being happy.

[1387] In this way, the system can be used to provide a real-time, personalized experience in-store or during an event.

[1388] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1389] Step 1:

[1390] The server pre-registers the facial photographs and identification data of athletes and customers at events and stores. The facial photographs and identification data uploaded by users (event organizers or store managers) are used as input. This saves the relevant data in the database, enabling the system to identify athletes and customers.

[1391] Step 2:

[1392] Before the event starts, the server activates the aerial photography device (drone), using the event start time and flight pattern data as input. Based on this, the aerial photography device patrols the designated area and captures video data.

[1393] Step 3:

[1394] The aerial photography equipment collects images of events and stores in real time and sends them to a cloud server. The camera image data is used as input and is sent to the cloud server in real time as output.

[1395] Step 4:

[1396] The cloud server uses a facial recognition algorithm to recognize the faces of players and customers from the received video data and match them with the identification data. The video data and pre-registered identification data are used as input, and a recognition result is generated as output.

[1397] Step 5:

[1398] The cloud server inputs the recognition results into the generative AI model to generate real-time commentary and commen...

Claims

1. A means to register player photos and uniform number data in advance, A method for activating the drone camera before the start of the game and recognizing the faces and uniform numbers of players on the field; A means of collecting and analyzing footage of the game in real time, using a generative AI model to generate real-time commentary and commentary based on the analysis results; means for delivering the generated commentary and commentary to a user's terminal; A system including:

2. The system of claim 1 , further comprising means for the drone camera to recognize faces of players using a facial recognition algorithm and transmit the faces to a server.

3. The system of claim 1 , wherein the generative AI model comprises means for learning sports rules and past live data in advance.

Images