System

Abstract

A system is provided.SOLUTION: Means for acquiring moving image data of a game from a plurality of cameras in real time, means for segmenting the acquired moving image data for each frame and identifying positions of a player and a ball using an object detection model, means for tracking movements of the player and the ball using a tracking algorithm between consecutive frames, means for identifying a specific event from the tracking data and updating a score in real time, means for creating play commentary using a generated AI based on the identified event data, and means for converting the updated score and the generated play commentary into a data format for display; This system is provided with a means for real-time distribution and a means for receiving the distributed data and displaying them as a score board and play explanation on a screen.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditionally, the work of table officials (TOs) in basketball games requires monitoring the movements of players and referees and manually recording scores and events, which requires a lot of manpower and time. As a result, the workload of TOs is heavy in amateur basketball at the elementary and junior high school level, as well as among working adults, and game management can be disrupted. Furthermore, spectators have limited access to detailed information about the game's progress in real time, which can diminish their enjoyment of the game. Therefore, there is a need for a system that reduces the workload of TOs and allows all parties involved, including spectators, to enjoy the game more. [Means for solving the problem]

[0005] The present invention provides a system that includes: means for acquiring video data of a match from multiple cameras in real time; means for breaking down the acquired video data into frames and identifying the positions of players and the ball using an object detection model; means for tracking the movements of players and the ball between consecutive frames using a tracking algorithm; means for identifying specific events from the tracking data and updating the score in real time; means for creating play commentary using a generation AI based on the identified event data; means for converting the updated score and generated play commentary into a data format for display and distributing them in real time; and means for receiving the distributed data and displaying it on a screen as a scoreboard and play commentary. This reduces the burden on TOs and realizes a system that allows match participants, including spectators, to understand the details of the match in real time and enjoy the game more.

[0006] "Camera data" refers to video signals acquired from multiple cameras capturing a basketball game.

[0007] An "object detection model" is a machine learning and deep learning algorithm for identifying and locating specific objects in image data.

[0008] "Player" refers to a player who participates in a basketball game and belongs to a particular team.

[0009] A "ball" is a spherical object used in the game of basketball.

[0010] A "tracking algorithm" is a computer algorithm for tracking the movement of an object between successive frames and analyzing its position and movement.

[0011] An "event" refers to a specific action or situation that occurs during a basketball game, such as a made shot, a foul, or a timeout.

[0012] A "score" is the number of points a team scores during a basketball game.

[0013] "Generative AI" is artificial intelligence that uses natural language processing technology to automatically generate text.

[0014] "Play Commentary" means text and / or audio commentary that explains specific plays or events during a match to the viewer.

[0015] A "data format for display" is a format suitable for displaying the distributed information on a screen, such as HTML or JSON.

[0016] "Real-time" is a time constraint that means information is obtained and processed immediately, with little delay.

[0017] "Distribution" is the act of sending information from a server to a terminal and having it received in real time.

[0018] "Terminal" refers to a device for receiving and displaying information, such as a smartphone or screen. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention is a system that digitizes and automates the work of table officials (TOs) in basketball games by capturing video data of the game in real time from multiple cameras and analyzing the movements of the players and the ball. This system allows each component to interact with each other, reflecting the game situation in the score in real time and automatically generating commentary on plays.

[0041] System configuration

[0042] The system of the present invention consists of the following main components:

[0043] 1. Video Data Acquisition Unit: The server acquires video data in real time from multiple cameras installed in the venue. This data is used to analyze the movements of players and the ball.

[0044] 2. Image analysis unit: The server breaks down the acquired video data into frames and identifies the positions of players and the ball using an object detection model.

[0045] 3. Tracking: The server uses a tracking algorithm to track the movements of players and the ball between successive frames and records these movements as time-series data.

[0046] 4. Event Identification: The server analyzes the tracking data and identifies specific events such as successful shots, fouls, and timeouts.

[0047] 5. Score Updater: The server updates the score in real time based on the identified events.

[0048] 6. Play commentary generation unit: The server uses a generation AI to create a play commentary in natural language based on the identified event data.

[0049] 7. Data distribution unit: The server converts the updated scores and generated play commentary into a data format for display and distributes them in real time.

[0050] 8. Display terminal: The terminal receives the distributed data and displays it on the screen as a scoreboard and play commentary.

[0051] Program processing flow (explained in natural language)

[0052] Server: Acquire video data

[0053] The server collects real-time video data from multiple cameras installed in the venue, each capturing footage from a different angle or position, and streams it in chronological order.

[0054] Server: Image analysis

[0055] The server breaks down the acquired video data into frames and applies an object detection model to identify the positions of players and the ball, thereby obtaining positional information for players and the ball within each frame.

[0056] Server:Tracking

[0057] The server uses a tracking algorithm to track the movement of players and the ball between successive frames, recording this data as time series of player and ball positions for use in motion analysis.

[0058] Server:Event Identification

[0059] The server identifies specific events (e.g., successful shots, fouls, timeouts) from the tracking data. For example, if it detects that the ball has passed through the hoop, it identifies that as a score event.

[0060] Server:Score Update

[0061] The server updates the team's score in real time based on the identified events, for example, adding a point to the team's score when a made shot event is identified.

[0062] Server: Play commentary generation

[0063] The server uses the generated AI to generate natural language commentary based on the identified event data, providing viewers with easy-to-understand details about the match.

[0064] Server: Data distribution

[0065] The server converts the updated scores and generated commentary into a displayable data format and distributes them in real time, making the latest match information instantly available.

[0066] Terminal: Receiving and displaying data

[0067] The device receives data distributed from the server and displays it on the screen as a scoreboard and commentary on the game, allowing spectators to understand the situation in real time.

[0068] Specific examples

[0069] For example, if a player makes a three-point shot during a game, the process is as follows:

[0070] 1. Acquisition of video data (server): The server acquires video data in real time from multiple cameras installed in the spectator seats.

[0071] 2. Image analysis (server): The server breaks down the captured video data into frames and applies object detection models to identify the positions of players and the ball.

[0072] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[0073] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[0074] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[0075] 6. Play commentary generation (server): The server uses a generation AI to generate a play commentary such as, "Player X made a great three-point shot!"

[0076] 7. Data distribution (server): The server converts the updated scores and play commentary into a data format for display and distributes them in real time.

[0077] 8. Receiving and displaying data (terminal): The terminal receives the distributed data and displays the latest score and play commentary on the screen in real time.

[0078] This will significantly reduce the burden on TOs, and allow game officials and spectators to understand game details in real time, allowing them to enjoy basketball even more.

[0079] The processing flow will be explained below.

[0080] Program processing steps

[0081] Step 1: Obtaining video data

[0082] server:

[0083] The server receives real-time video data of the match from multiple cameras in the venue, each positioned to capture the match from a different angle.

[0084] Step 2: Frame Decomposition

[0085] server:

[0086] The server breaks down the acquired video data into frames and processes each frame in parallel for the video data from each camera.

[0087] Step 3: Object detection

[0088] server:

[0089] The server uses object detection models such as YOLO (You Only Look Once) to identify players and the ball in each frame.

[0090] The object detection model outputs the position coordinates and labels of the players and ball (Player 1, Player 2, Ball, etc.).

[0091] Step 4: Tracking

[0092] server:

[0093] The server tracks the movement of players and the ball between successive frames using tracking algorithms such as Kalman filters or optical flow.

[0094] The tracking algorithm analyzes the position and movement vector of each object and records them as time series data.

[0095] Step 5: Event Identification

[0096] server:

[0097] The server identifies specific events (e.g., successful shots, fouls, timeouts) based on the tracking data.

[0098] For example, if the ball passes over the hoop and then changes position, this is identified as a "scoring event."

[0099] Step 6: Update your score

[0100] server:

[0101] The server updates the team scores in real time based on the identified events.

[0102] For example, if the shot is successful, the team will receive points corresponding to their score (e.g., 2 points, 3 points).

[0103] Step 7: Generate play instructions

[0104] server:

[0105] The server generates a play commentary in natural language based on the event data identified using the generation AI.

[0106] For example, it generates a sentence such as, "Player X broke through the defense with a sharp dribble and scored a stunning layup!"

[0107] Step 8: Prepare for data distribution

[0108] server:

[0109] The server converts the updated scores and generated commentary into a data format for display, such as JSON or HTML.

[0110] Step 9: Data Distribution

[0111] server:

[0112] The server distributes the converted data in real time to the screens at the venue and to the audience's devices.

[0113] Data is delivered using real-time communication protocols such as WebSocket.

[0114] Step 10: Receive and display data

[0115] Device:

[0116] The terminal receives the data distributed from the server and displays it on the screen as a scoreboard and commentary on the play.

[0117] The display is updated in real time, allowing spectators to instantly understand the situation in the game.

[0118] These are the specific processing steps of the "Digital Tournament for Basketball" system program, which reduces the burden on Tournament Officers and allows spectators to enjoy the details of the game in real time.

[0119] Example 1

[0120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0121] In the current match management system, a large amount of manpower is required for the duties of table officials (TOs), which means that real-time updates of scores and prompt commentary on plays cannot always be provided. As a result, it is difficult for spectators and officials to grasp the progress of the match in real time, which can diminish the appeal of the match. Furthermore, incorrect score entries and time lags due to human error are also issues. It is necessary to provide a system that can solve these problems and improve the efficiency and sophistication of match management.

[0122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0123] In this invention, the server includes means for acquiring video data of a game in real time from multiple camera devices, means for breaking down the acquired video data into frames and identifying the positions of players and the ball using a machine learning model, means for tracking the movements of players and the ball between consecutive frames using a tracking algorithm, means for identifying specific events from the tracking data and updating the score in real time, means for creating play commentary based on the identified event data using artificial intelligence, means for converting the updated score and the generated play commentary into a data format for display and distributing them in real time, and means for receiving the distributed data and displaying it on a screen as a score display device and play commentary. This enables the automation of TO operations and realizes the provision of accurate score updates and play commentary in real time.

[0124] "Multiple camera devices" refers to multiple camera devices that capture video data of the game in real time from different angles and positions.

[0125] "Video data" refers to digital video data showing the progress of a match captured by multiple camera devices.

[0126] "Machine learning model" refers to an artificial intelligence model used to analyze captured video data and identify the positions of players and the ball.

[0127] A "tracking algorithm" refers to a mathematical technique or calculation method for tracking the movement of players and the ball between successive frames.

[0128] "Tracking Data" refers to data containing player and ball position information obtained between successive frames by a tracking algorithm.

[0129] "Specific event" refers to a significant occurrence during a game (e.g., a successful shot, a foul, a timeout, etc.).

[0130] "Generative Artificial Intelligence" refers to a generative AI model used to generate natural language play commentary based on data.

[0131] "Points" refers to the number of points a team scores during a match.

[0132] "Score display device" refers to a device (e.g., a digital scoreboard or terminal display) that visually displays updated scores and commentary to spectators and other involved parties.

[0133] "Real-time" refers to processing or updating information occurring almost immediately or with very little delay.

[0134] This invention is a system that digitizes and automates the work of table officials (TOs) in sports matches by acquiring game video data in real time from multiple camera devices and analyzing the movements of players and the ball. Each component of this system works together, reflecting the game situation in real time in the score and automatically generating commentary on plays.

[0135] Key Components of the System

[0136] 1. Video data acquisition unit:

[0137] The server acquires real-time video data of the match from multiple cameras (e.g., general-purpose digital cameras) installed in the venue. Each camera captures the match from a different angle or position, and transmits the video data to the server via a network.

[0138] 2. Image analysis unit:

[0139] The server breaks down the captured video data into frames using libraries such as OpenCV. It then applies an object detection model (e.g., YOLOv3 or SSD model) trained using machine learning libraries such as TensorFlow or PyTorch to identify the positions of players and the ball within each frame. The detected position information is recorded as coordinate data.

[0140] 3. Tracking section:

[0141] The server uses tracking algorithms such as Kalman filters and SORT to track the movement of players and the ball between successive frames, recording this data as time series data and analyzing patterns of player and ball movement.

[0142] 4. Event Identification:

[0143] The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts), specifically when the ball passes through the hoop, which is identified as a scoring event. Identified events are recorded in a database.

[0144] 5. Score Update Section:

[0145] The server updates the team's score in real time based on the identified events. For example, if a shot is made, a point is added to the corresponding team's score. This information is managed as scoreboard data.

[0146] 6. Play commentary generation unit:

[0147] The server generates play commentary in natural language based on the identified event data using generative AI (e.g., GPT-3). For example, it generates commentary such as, "Player A made a great three-point shot!"

[0148] 7. Data Distribution Department:

[0149] The server converts the updated scores and generated commentary into a displayable data format such as JSON and distributes it in real time via WebSocket or REST API, making the latest information instantly available.

[0150] 8. Display terminal:

[0151] Devices (e.g., tablets and smartphones) and digital scoreboards receive the data distributed from the server. The devices display the received data on the screen in real time as a scoreboard and play commentary. Appropriate UI / UX design is used to improve visibility.

[0152] Specific examples

[0153] For example, if Player A makes a three-point shot during a game, the system will behave as follows:

[0154] 1. Acquiring video data:

[0155] The server collects video data in real time from multiple cameras in the venue, each capturing footage of the match from a different angle, and transmits the data to the server via a network.

[0156] 2. Image Analysis:

[0157] The server breaks down the video data into frames using OpenCV or similar tools, and applies an object detection model (such as YOLOv3) to identify the positions of players and the ball.

[0158] 3. Tracking:

[0159] The server uses a tracking algorithm (such as a Kalman filter) to track the ball's movement between successive frames and records the trajectory of its movement.

[0160] 4. Event Identification:

[0161] The server analyzes the tracking data and detects when the ball passes through the hoop, identifying this as a successful shot event.

[0162] 5. Score Update:

[0163] The server will award the successful shot three points to Team A's score and record this information in the database.

[0164] 6. Play commentary generation:

[0165] Using generative AI (GPT-3), we generate commentary on the play, such as "Player A made a great three-point shot!"

[0166] 7. Data Distribution:

[0167] The server converts the updated scores and generated play commentary into JSON format and delivers them to the device in real time via WebSocket.

[0168] 8. Receiving and Displaying Data:

[0169] The device receives the data and displays the latest score and commentary on the game in real time on the screen.

[0170] Prompt Sentence Examples

[0171] An example of an input prompt for the generation AI is, "Please generate commentary if player A makes a successful three-point shot."

[0172] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0173] Step 1: Obtaining video data

[0174] The server acquires real-time video data of the match from multiple camera devices. Specifically, multiple camera devices capture the match from different angles and transmit the video data to the server via a network. The input is the video data captured by each camera device, and the output is multiple real-time video streams stored on the server.

[0175] Step 2: Image analysis

[0176] The server breaks down the captured video data into frames. To do this, it uses libraries such as OpenCV to convert the video data into still images. Next, it applies an object detection model (e.g., YOLOv3 or SSD) trained using machine learning libraries such as TensorFlow or PyTorch to identify the positions of players and the ball within each frame. The input is the captured video data (frames), and the output is the position information (coordinate data) of players and the ball for each frame.

[0177] Step 3: Tracking

[0178] The server uses tracking algorithms such as Kalman filter and SORT to track the movements of players and the ball between consecutive frames. This allows the position information of players and the ball to be recorded as time-series data. The input is consecutive frames of data containing position information, and the output is time-series position information as tracking data.

[0179] Step 4: Event Identification

[0180] The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts). In particular, if the ball's trajectory passes through the hoop, it is identified as a successful shot. The input is the tracking data, and the output is the identified event information.

[0181] Step 5: Update your score

[0182] The server updates the team's score in real time based on the identified event information. For example, when a shot is made, a point is added to the corresponding team's score. This information is managed as scoreboard data. The input is the identified event information, and the output is updated score information.

[0183] Step 6: Generate play instructions

[0184] The server generates commentary based on the identified event data using a generation AI (e.g., GPT-3). To do this, a prompt is prepared for the generation AI and given as input. For example, the prompt might be, "Please generate commentary if player A makes a successful three-point shot." The input is the identified event data and the prompt, and the output is the generated commentary text.

[0185] Step 7: Data Distribution

[0186] The server converts the updated score and generated commentary into a displayable data format such as JSON and distributes it in real time via WebSocket or REST API. The input is the updated score information and the generated commentary text, and the output is the distributed JSON data.

[0187] Step 8: Receive and display data

[0188] The terminal receives data delivered from the server and displays it on the screen as a scoreboard and commentary in real time. For this purpose, a web application or a native application is used. The input is the delivered JSON data, and the output is the latest score and commentary displayed on the screen.

[0189] (Application example 1)

[0190] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0191] It is difficult for table officials (TOs) to perform their duties quickly and accurately during a basketball game. Furthermore, there is a lack of means to accurately communicate the real-time status of the game to spectators. Therefore, there is a need for a system that can reduce the burden on TOs and provide spectators with real-time information on the game's status.

[0192] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0193] In this invention, the server includes means for acquiring video data in real time from multiple camera devices, means for breaking down the acquired video data into frames and identifying the positions of the target and the sphere using an object detection model, means for tracking the movement of the target and the sphere between consecutive frames using a tracking algorithm, means for identifying specific events from the tracking data and updating scores in real time, means for creating action descriptions using a generation AI based on the identified event data, means for converting the updated scores and generated action descriptions into a data format for display and distributing them in real time, and means for receiving the distributed data and displaying them as a score board and action descriptions on a virtual space display device. This reduces the burden on TOs and makes it possible to accurately convey the situation of the game to spectators in real time.

[0194] A "camera" is a device for acquiring video data, and is responsible for capturing video of the game in real time from multiple angles and positions.

[0195] "Video data" refers to visual information of the match obtained from a camera, and is data that shows the progress of the match in real time.

[0196] An "object detection model" is an analytical algorithm or technology that analyzes acquired video data to identify the position of a specific object (such as a player or a ball).

[0197] "Target" refers to a person or object that is being tracked and identified, such as a player in a match.

[0198] The term "sphere" refers to a sports ball used in a game, and in the present invention refers to a basketball.

[0199] A "frame" is an individual still image that makes up video data, and a moving image is formed by successive frames.

[0200] A "tracking algorithm" is a technique used to track the movement of an object or sphere between successive frames and record its position information as time-series data.

[0201] "Specific events" refer to important events that occur during a game (such as a successful shot or a foul) that the system automatically identifies.

[0202] "Generative AI" is an artificial intelligence technology that automatically generates natural language descriptions of behavior based on identified event data.

[0203] An "action explanation" is an explanatory text about a specific event during a match, created by the generation AI.

[0204] The "score board" is an interface that is displayed on the virtual space display device to visually provide the latest score information.

[0205] A "virtual space display device" is a device that uses virtual reality technology to allow users to visually experience a match in a virtual space, and generally includes a VR headset.

[0206] A system for implementing this invention comprises the following major components:

[0207] 1. Server

[0208] The server has a means to acquire video data in real time from multiple camera devices, which are installed at different angles and positions within the venue to capture visual information of the match.

[0209] The server has a means to break down the acquired video data into frames and identify the positions of the objects and spheres using an object detection model, which is implemented using a library such as OpenCV.

[0210] The server has a means to track the movement of the object and the ball using a tracking algorithm between successive frames, allowing the position data of the object (player) and the ball (basketball) to be recorded over time.

[0211] The server has the means to identify specific events from the tracking data and update the score in real time. For example, when a ball passes through a hoop, it is identified as a scoring event and the score is automatically updated.

[0212] The server has a means to generate behavioral explanations based on the identified event data using a generative AI model, such as GPT-3.

[0213] The server has a means for converting the updated scores and generated action descriptions into a data format for display and distributing them in real time, thereby enabling the virtual space display device to quickly provide the latest match information.

[0214] 2. Terminal (Virtual Space Display Device)

[0215] The device receives data from the server and displays it in the virtual space as a scoreboard and action explanations, allowing users to visually grasp the progress of the game in real time. A VR headset or similar device is used as the virtual space display device.

[0216] 3. Users

[0217] By wearing the virtual space display device, users can watch the game in a virtual space. By referring to the score and action explanations updated in real time, users can intuitively understand the situation of the game.

[0218] Specific examples

[0219] For example, if a ball (basketball) passes through the hoop during a game and a three-point shot is successful, the process is as follows:

[0220] 1. Server

[0221] Video data acquisition: Video data is acquired in real time from the imaging device.

[0222] Image analysis: Video data is broken down into frames and object detection models are applied to identify the positions of objects (players) and balls (basketballs).

[0223] Tracking: The movement of the object and the sphere is tracked using a tracking algorithm between successive frames.

[0224] Event Identification: The tracking data is analyzed and when the sphere passes through the ring it is identified as a scoring event.

[0225] Score Update: Based on the identified incident, add 3 points to the team's score.

[0226] Play commentary generation: Using a generation AI, we generate an action explanation such as "Player A made a great three-point shot!"

[0227] Data distribution: Updated scores and commentary are converted into a displayable data format and distributed in real time.

[0228] 2. Terminal (Virtual Space Display Device)

[0229] Display of received data: Receive the distributed data and display it in the virtual space as a score board and action explanation.

[0230] Prompt Sentence Examples

[0231] "When a three-point shot is successful, generate a score and commentary of the play and display them in the virtual space."

[0232] The above steps reduce the burden on TOs and enable them to provide users with accurate information on the progress of the match in real time.

[0233] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0234] Step 1:

[0235] The server receives video data from the camera in real time. Each camera captures the game from a different angle or position and sends the video data to the server. The server receives the data and starts processing. The input is real-time video data, and the output is the video data.

[0236] Step 2:

[0237] The server uses the OpenCV library to decompose the video data into frames, which converts it into a sequence of still images (frames). The input is the video data acquired in step 1, and the output is the individual frames.

[0238] Step 3:

[0239] The server applies an object detection model to each decomposed frame to identify the positions of the objects (players) and the ball (basketball). This uses a machine learning object detection algorithm. The input is the image data for each frame, and the output is the position information of the objects and the ball within each frame.

[0240] Step 4:

[0241] The server uses a tracking algorithm to track the movement of the object and the sphere between successive frames, generating position information as time-series data. The input is the position information within each frame, and the output is time-series position data of the object and the sphere.

[0242] Step 5:

[0243] The server analyzes the tracking data and identifies specific events, such as a ball passing through a ring, as a scoring event. This is achieved using an event identification algorithm, whose input is the time-series position data and whose output is the identified specific event.

[0244] Step 6:

[0245] The server updates the scores in real time based on the identified events. Each scoring event automatically increments the team's score. The input is the identified event, and the output is the updated score.

[0246] Step 7:

[0247] The server uses a generative AI to create a behavioral explanation based on the identified event data. For example, it generates an explanation in the form of "Player A made a great three-point shot!" The input is the identified event data, and the output is the generated behavioral explanation.

[0248] Step 8:

[0249] The server converts the updated scores and generated behavior descriptions into a data format for display and distributes them in real time using data streaming technology. The input is the updated scores and generated behavior descriptions, and the output is data that can be distributed.

[0250] Step 9:

[0251] The terminal (virtual space display device) receives the data distributed from the server and displays it in the virtual space as a score board and action explanations. This allows the user to visually grasp the progress of the game in real time. The input is the distributed data, and the output is the display in the virtual space.

[0252] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0253] This invention is a system that digitizes and automates the work of table officials (TOs) in basketball games, and combines it with an emotion engine that recognizes user emotions to provide a more interactive and immersive game-watching experience.

[0254] System configuration

[0255] The system of the present invention consists of the following main components:

[0256] 1. Video Data Acquisition Unit: The server acquires video data of the match in real time from multiple cameras installed in the venue. This data is used to analyze the movements of players and the ball.

[0257] 2. Image analysis unit: The server breaks down the acquired video data into frames and identifies the positions of players and the ball using an object detection model.

[0258] 3. Tracking: The server uses a tracking algorithm to track the movements of players and the ball between successive frames and records these movements as time-series data.

[0259] 4. Event Identification: The server analyzes the tracking data and identifies specific events such as successful shots, fouls, and timeouts.

[0260] 5. Score Updater: The server updates the score in real time based on the identified events.

[0261] 6. Play commentary generation unit: The server uses a generation AI to create a play commentary in natural language based on the identified event data.

[0262] 7. Emotion engine: The server includes an emotion engine that acquires the user's face and voice data and recognizes their emotions.

[0263] 8. Data distribution unit: The server converts the updated scores and generated play commentary into a data format for display and distributes them in real time.

[0264] 9. Display terminal: The terminal receives the distributed data and displays it on the screen as a scoreboard and play commentary.

[0265] Program processing flow (explained in natural language)

[0266] Server: Acquire video data

[0267] The server collects real-time video data from multiple cameras installed in the venue, each capturing footage of the match from a different angle or position, and streams the footage in chronological order.

[0268] Server: Image analysis

[0269] The server breaks down the acquired video data into frames and applies an object detection model to identify the positions of players and the ball, thereby obtaining positional information for players and the ball within each frame.

[0270] Server:Tracking

[0271] The server uses a tracking algorithm to track the movement of players and the ball between successive frames, recording this data as time series of player and ball positions for use in motion analysis.

[0272] Server:Event Identification

[0273] The server identifies specific events (e.g., successful shots, fouls, timeouts) from the tracking data. For example, if it detects that the ball has passed through the hoop, it identifies that as a score event.

[0274] Server:Score Update

[0275] The server updates the team's score in real time based on the identified events, for example, adding a point to the team's score when a made shot event is identified.

[0276] Server: Play commentary generation

[0277] The server generates play commentary in natural language based on the event data identified using the AI ​​generation technology, which provides viewers with easy-to-understand details about the match.

[0278] Server: Emotion recognition

[0279] The server acquires the user's facial and voice data and uses an emotion engine to recognize their emotions. For example, by analyzing facial expression data and tone of voice acquired through a camera or microphone, it can determine whether the user is in an emotional state such as "joy," "excitement," or "dissatisfaction."

[0280] Server: Adjustment of gameplay commentary

[0281] The server dynamically adjusts the content and display of the commentary based on the user's emotions as recognized by the emotion engine. For example, if the user is excited, the server can display more detailed and enthusiastic commentary.

[0282] Server: Real-time analysis of audience sentiment

[0283] The server aggregates the emotional data of multiple users and analyzes the emotional state of the entire audience in real time. This data may be reflected in the progress and direction of the match.

[0284] Server: Prepare data distribution

[0285] The server converts the updated scores and generated commentary into a data format for display, such as JSON or HTML.

[0286] Server: Data distribution

[0287] The server then transmits the converted data to the screens at the venue and to the audience's devices in real time using a real-time communication protocol such as WebSocket.

[0288] Terminal: Receiving and displaying data

[0289] The device receives data from the server and displays it on the screen as a scoreboard and commentary on the game, which is updated in real time, allowing spectators to instantly understand the situation in the game.

[0290] Specific examples

[0291] For example, if a player makes a three-point shot during a game, the process is as follows:

[0292] 1. Acquisition of video data (server): The server acquires video data in real time from multiple cameras installed within the venue.

[0293] 2. Image analysis (server): The server breaks down the captured video data into frames and applies object detection models to identify the positions of players and the ball.

[0294] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[0295] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[0296] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[0297] 6. Play commentary generation (server): The server uses a generation AI to generate a play commentary such as, "Player X made a great three-point shot!"

[0298] 7. Emotion recognition (server): The server acquires the user's facial and voice data and uses the emotion engine to recognize when the user is excited.

[0299] 8. Adjusting play commentary (server): The server will reflect the user's excitement and adjust the commentary to be more detailed and energetic.

[0300] 9. Data distribution preparation (server): The server converts the updated scores and adjusted play commentary into a data format for display.

[0301] 10. Data distribution (server): The server distributes data in real time to the screen and audience devices.

[0302] 11. Receiving and displaying data (terminal): The terminal receives the distributed data and displays the scoreboard and play commentary in real time.

[0303] This will significantly reduce the burden on TOs and allow audiences to have a more immersive experience by receiving individually tailored commentary through emotion recognition.

[0304] The processing flow will be explained below.

[0305] Program processing steps

[0306] Step 1: Obtaining video data

[0307] server:

[0308] The server collects real-time video data from multiple cameras installed in the venue, each capturing footage from a different angle, and streams it in chronological order.

[0309] Step 2: Frame Decomposition

[0310] server:

[0311] The server breaks down the acquired video data into frames and processes each frame in parallel for the video data from each camera.

[0312] Step 3: Object detection

[0313] server:

[0314] The server uses object detection models such as YOLO (You Only Look Once) to identify players and the ball in each frame.

[0315] The detection model outputs the position coordinates and labels of players and the ball (Player 1, Player 2, Ball, etc.).

[0316] Step 4: Tracking

[0317] server:

[0318] The server uses a tracking algorithm to track the movement of players and the ball between successive frames.

[0319] The algorithm analyzes the position and movement vectors of each object and records them as time series data.

[0320] Step 5: Event Identification

[0321] server:

[0322] The server identifies specific events (e.g., successful shots, fouls, timeouts) based on the tracking data.

[0323] For example, if the ball passes over the hoop, this is identified as a "scoring event."

[0324] Step 6: Update your score

[0325] server:

[0326] The server updates the team scores in real time based on the identified events.

[0327] For example, if the shot is successful, the team will receive points corresponding to their score (e.g., 2 points, 3 points).

[0328] Step 7: Generate play instructions

[0329] server:

[0330] The server generates a play commentary in natural language based on the event data identified using the generation AI.

[0331] For example, it generates a sentence such as, "Player X broke through the defense with a sharp dribble and scored a stunning layup!"

[0332] Step 8: Emotion Recognition

[0333] server:

[0334] The server acquires the user's facial and voice data and recognizes their emotions using an emotion engine.

[0335] For example, facial expression data and tone of voice obtained from a camera or microphone are analyzed to determine whether the user is in an emotional state such as "joy," "excitement," or "dissatisfaction."

[0336] Step 9: Adjust the commentary

[0337] server:

[0338] The server dynamically adjusts the content and display method of the play commentary based on the user's emotions recognized by the emotion engine.

[0339] For example, if the user is excited, a more detailed and energetic commentary will be displayed.

[0340] Step 10: Real-time analysis of audience sentiment

[0341] server:

[0342] The server aggregates the emotional data of multiple users and analyzes the emotional state of the entire audience in real time.

[0343] This data may be reflected in the progress and presentation of the match.

[0344] Step 11: Prepare for data distribution

[0345] server:

[0346] The server converts the updated scores and generated commentary into a data format for display, such as JSON or HTML.

[0347] Step 12: Data Distribution

[0348] server:

[0349] The server distributes the converted data in real time to the screens at the venue and to the audience's devices.

[0350] Data is delivered using real-time communication protocols such as WebSocket.

[0351] Step 13: Receiving and displaying data

[0352] Device:

[0353] The terminal receives the data distributed from the server and displays it on the screen as a scoreboard and commentary on the game.

[0354] The display is updated in real time, allowing spectators to instantly understand the situation in the game.

[0355] Specific examples

[0356] For example, if a player makes a three-point shot during a game, the process is as follows:

[0357] 1. Acquisition of video data (server): The server acquires video data in real time from multiple cameras installed within the venue.

[0358] 2. Image analysis (server): The server breaks down the captured video data into frames and applies object detection models to identify the positions of players and the ball.

[0359] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[0360] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[0361] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[0362] 6. Play commentary generation (server): The server uses a generation AI to generate a play commentary such as, "Player X made a great three-point shot!"

[0363] 7. Emotion recognition (server): The server acquires the user's facial and voice data and uses the emotion engine to recognize when the user is excited.

[0364] 8. Adjusting play commentary (server): The server will reflect the user's excitement and adjust the commentary to be more detailed and energetic.

[0365] 9. Data distribution preparation (server): The server converts the updated scores and adjusted play commentary into a data format for display.

[0366] 10. Data distribution (server): The server distributes data in real time to the screen and audience devices.

[0367] 11. Receiving and displaying data (terminal): The terminal receives the distributed data and displays the scoreboard and play commentary in real time.

[0368] This will significantly reduce the burden on TOs and allow audiences to have a more immersive experience by receiving individually tailored commentary through emotion recognition.

[0369] Example 2

[0370] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0371] At sports games such as basketball, the goal is to improve the efficiency and accuracy of table officials (TOs) by digitizing and automating their work. There is also a need for a system that can provide real-time commentary and information on the game to spectators, creating a more immersive and realistic experience. However, existing systems face challenges in accurately grasping the game situation and updating and distributing necessary information in real time. Furthermore, there are currently no systems that can recognize and reflect spectator emotions and provide personalized commentary.

[0372] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0373] In this invention, the server includes a means for acquiring image data of the game in real time from multiple camera devices, a means for dividing the acquired image data into successive images and using an object detection algorithm to identify the positions of players and the ball, and a means for tracking the movement of players and the ball between successive images using a tracking algorithm. This enables accurate understanding of the game situation, real-time score updates, and the creation and distribution of game commentary using generative AI. Furthermore, the server can acquire user facial and voice data, analyze their emotional state using an emotion engine, and dynamically adjust the content and display method of the game commentary based on that data, providing spectators with a more realistic and immersive game viewing experience.

[0374] "Filming equipment" refers to video cameras installed within the venue to capture the state of the match.

[0375] "Image data" refers to video data of a match captured by a camera.

[0376] "Continuous images" refers to a set of still images obtained by breaking down video data into frames.

[0377] An "object detection algorithm" refers to a machine learning model for identifying specific objects (players or balls) within image data.

[0378] "Player" means a player taking part in a match.

[0379] "Sphere" refers to the ball used during a match.

[0380] A "tracking algorithm" refers to a computational method for tracking the position of a particular object (a player or a ball) between successive image frames.

[0381] "Specific occurrences" refer to significant events that occur during a game, such as a successful shot, a foul, or a timeout.

[0382] "Score" refers to points awarded to a team based on a specific event.

[0383] "Generative algorithm" refers to AI technology that generates natural language commentary on sports events based on given data.

[0384] "Sports commentary" refers to natural language text that describes the progress of a match or an event.

[0385] "Display data format" refers to a data format (e.g., JSON, HTML) suitable for displaying scores and commentary on a display device.

[0386] An "emotion engine" refers to software or algorithms that analyze a user's facial expressions and voice data to recognize their emotional state.

[0387] "Users" refers to spectators who attend matches.

[0388] "Emotional state" refers to the user's psychological state, such as joy, excitement, or dissatisfaction.

[0389] "Dynamic adjustment" refers to changing the content and display format in real time depending on the situation.

[0390] This invention aims to digitize and automate the work of table officials (TOs) at sports games, and is a system that combines it with an emotion engine that recognizes user emotions to provide a more interactive and immersive game-watching experience.

[0391] System configuration

[0392] Server: Acquire video data

[0393] The server acquires image data of the match in real time from multiple camera devices installed in the venue. Each camera captures the match from a different angle or position and streams the data. Specific examples of such cameras include high-resolution digital cameras and IP cameras.

[0394] Server: Image analysis

[0395] The server divides the acquired image data into successive images and applies an object detection algorithm to identify the positions of players and the ball. The object detection algorithm uses machine learning models such as YOLO (You Only Look Once) and Mask R-CNN. This allows the server to obtain the positional information of players and the ball within each frame.

[0396] Server:Tracking

[0397] The server tracks the movements of the players and the ball between successive image frames using a tracking algorithm, such as a Kalman filter and the tracking algorithm SORT (Simple Online and Realtime Tracking). This process stores the positional information of the players and the ball as time-series data.

[0398] Server:Event Identification

[0399] The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts). For example, a ball passing through the hoop is captured as a "successful shot." Event identification is achieved using statistical analysis and rule-based techniques.

[0400] Server:Score Update

[0401] The server updates the team's score in real time based on the identified incidents, and points are automatically added to the team's score for successful shots, for example, three points for a successful three-point shot.

[0402] Server: Play commentary generation

[0403] The server uses a generative algorithm to generate a natural language commentary of the game based on the identified event data. Generative AI models such as GPT-4 are used. For example, a commentary such as "Player X made a successful three-point shot!" is generated.

[0404] Server: Emotion recognition

[0405] The server collects the user's facial and voice data and analyzes their emotional state using an emotion engine, which uses tools such as Affectiva and the Microsoft Azure Emotion API. This allows the server to determine in real time whether the user is in a state of joy, excitement, dissatisfaction, or other emotional state.

[0406] Server: Adjustment of gameplay commentary

[0407] The server dynamically adjusts the content and display method of the generated commentary based on the user's emotions determined by the emotion engine. For example, if the user is excited, the generated commentary will be changed to be more detailed and passionate.

[0408] Server: Real-time analysis of audience sentiment

[0409] The server aggregates emotional data collected from multiple users and analyzes the emotional state of the entire venue in real time. This data is reflected in the progress and direction of the match.

[0410] Server: Prepare data distribution

[0411] The server converts the updated scores and generated commentary into a displayable data format (e.g., JSON or HTML), which facilitates distribution.

[0412] Server: Data distribution

[0413] The server then distributes the converted data in real time to the screens at the venue and to the audience's devices, using a real-time communication protocol such as WebSocket.

[0414] Terminal: Receiving and displaying data

[0415] The terminals receive the data distributed from the server and display it on the screen as scores and commentary on the match. The terminals can be smartphones, tablets, digital signage, etc. The display is updated in real time, allowing spectators to instantly understand the situation of the match.

[0416] Specific examples

[0417] For example, if a player makes a three-point shot during a game, the process is as follows:

[0418] 1. Acquisition of video data (server): The server acquires video data in real time from multiple camera devices installed within the venue.

[0419] 2. Image analysis (server): The server decomposes the captured video data and applies object detection algorithms to identify the positions of players and the ball.

[0420] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[0421] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[0422] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[0423] 6. Play commentary generation (server): The server uses a generation AI to generate commentary on the game, such as "Player X made a successful three-point shot!"

[0424] 7. Emotion recognition (server): The server acquires data on the user's facial expressions and voice, and uses the emotion engine to recognize when the user is excited.

[0425] 8. Adjusting play commentary (server): The server will reflect the user's excitement and adjust the commentary to be more detailed and energetic.

[0426] 9. Data distribution preparation (server): The server converts the updated scores and adjusted game commentary into a data format for display.

[0427] 10. Data distribution (server): The server distributes data in real time to the screen and audience devices.

[0428] 11. Data reception and display (terminal): The terminal receives the distributed data and displays the score and commentary in real time, allowing spectators to instantly grasp the situation of the match, significantly reducing the burden on the TO. Spectators can also receive individually tailored commentary through emotion recognition, providing a more immersive experience.

[0429] Prompt Sentence Examples

[0430] The following prompt sentence is used for the generative AI model to automatically generate an explanation such as "Player X made a successful three-point shot!"

[0431] Prompt statement:

[0432] "With the crowd going wild, player X made a brilliant three-point shot!"

[0433] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0434] Step 1:

[0435] Input: Real-time video data from multiple imaging modalities

[0436] Specific operation: The server acquires video data of the match in real time from multiple camera devices installed in the venue. Each camera captures the match from a different angle or position and streams the video data. Specifically, the data is transmitted in real time using the RTSP protocol.

[0437] Output: Real-time video data of the match

[0438] Step 2:

[0439] Input: Real-time video data of the match

[0440] Specific operation: The server divides the acquired video data into successive image frames. Specifically, it uses a tool such as ffmpeg to break the video down into 30 frames per second.

[0441] Output: Sequential image frames

[0442] Step 3:

[0443] Input: Sequential image frames

[0444] How it works: The server applies an object detection algorithm to identify the positions of players and spheres in each frame. Specifically, it uses models such as YOLO (You Only Look Once) and Mask R-CNN to output the coordinates of players and spheres in each frame.

[0445] Output: Position information of the player and the ball in each frame

[0446] Step 4:

[0447] Input: Position information of the player and the ball in each frame

[0448] How it works: The server tracks the movement of players and the ball using a tracking algorithm between consecutive frames, specifically using a Kalman filter and the SORT (Simple Online and Realtime Tracking) algorithm to record position information as time-series data.

[0449] Output: Time series position data of players and balls

[0450] Step 5:

[0451] Input: Time series position data of players and balls

[0452] Specific operation: The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts). Specifically, it checks whether the ball has passed through the hoop, and if it has, identifies it as a "successful shot."

[0453] Output: Identified event data

[0454] Step 6:

[0455] Input: Identified event data

[0456] Specific behavior: The server updates the team's score in real time based on the identified events. Specifically, for successful shots, the server automatically adds 2 or 3 points to the team's score.

[0457] Output: Updated score data

[0458] Step 7:

[0459] Input: Identified event data, updated score data

[0460] Specific behavior: The server uses a generative AI model (e.g., GPT-4) to generate natural language commentary on the game based on the identified event data. An example of a specific prompt sentence is, "Player X made a three-point shot!"

[0461] Output: Generated commentary

[0462] Step 8:

[0463] Input: User's facial expression data, voice data

[0464] Specific operation: The server acquires the user's facial expression and voice data and analyzes their emotional state using an emotion engine (e.g., Affectiva, Microsoft Azure Emotion API). This determines whether the user is in an emotional state such as "joy," "excitement," or "dissatisfaction."

[0465] Output: Analyzed sentiment data

[0466] Step 9:

[0467] Input: Analyzed emotion data, generated commentary

[0468] Specific behavior: The server dynamically adjusts the content and presentation of the generated commentary based on the user's emotional state. For example, if the user is excited, the server displays more detailed and energetic commentary.

[0469] Output: Coordinated competition commentary

[0470] Step 10:

[0471] Input: Adjusted game commentary, updated scoring data

[0472] Specific operation: The server converts the adjusted competition commentary and score data into a data format for display (e.g., JSON, HTML).

[0473] Output: Competition commentary and score data converted into a displayable data format

[0474] Step 11:

[0475] Input: Competition commentary and score data converted into a display format

[0476] How it works: The server distributes the converted data in real time to the screens in the venue and to the audience's devices, using a real-time communication protocol such as WebSocket.

[0477] Output: Real-time broadcast of competition commentary and score data

[0478] Step 12:

[0479] Input: Real-time game commentary and score data

[0480] Specific operation: The terminal receives data distributed from the server and displays it on the screen as score and commentary. The display is updated in real time, allowing spectators to instantly understand the situation of the game. Specifically, smartphones, tablets, digital signage, etc. are used.

[0481] Output: On-screen commentary and score information

[0482] (Application example 2)

[0483] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0484] Traditionally, watching basketball games required manual work by table officials (TOs), which placed a heavy burden on spectators and sometimes caused delays in real-time game progress. Furthermore, there was no way to provide an interactive experience that responded to spectators' emotions, limiting the quality of the viewing experience. These issues can be resolved using new technology.

[0485] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0486] In this invention, the server includes means for acquiring game video data from multiple cameras in real time, means for breaking down the acquired video data into frames and identifying the positions of objects and the ball using an object detection model, means for tracking the movement of the object and the ball between consecutive frames using a tracking algorithm, means for identifying specific events from the tracking data and updating the score in real time, means for creating play commentary based on the identified event data using a generation AI, means for converting the updated score and the generated play commentary into a display data format and distributing them in real time, means for receiving the distributed data and displaying it as a scoreboard and play commentary on a screen, means for acquiring user facial and voice data and recognizing the user's emotions using an emotion engine, and means for dynamically adjusting the content and display method of the play commentary based on the recognized emotion data. This improves the quality of game viewing and makes it possible to provide an interactive experience that corresponds to the user's emotions.

[0487] "Multiple cameras" means two or more video recording devices used to capture the action of a match in real time from different angles and positions.

[0488] "Video data" refers to digital data that is captured in real time by a camera and used for analysis and storage, either as is or after processing.

[0489] "Frame decomposition" refers to the process of dividing continuous video data into individual still images (frames).

[0490] An "object detection model" is an artificial intelligence model trained to identify specific objects (e.g., players or balls) in video or images.

[0491] A "tracking algorithm" is a computational method or process for continuously tracking the movement of an identified object between frames of video.

[0492] A "specific occurrence" is a significant action or event that occurs during a match (e.g., a goal, a foul).

[0493] "Generative AI" is a technology that uses artificial intelligence to automatically generate natural language explanations and sentences from input data.

[0494] "Play commentary" refers to written or audio data that explains specific events during a match in a way that is easy for viewers to understand.

[0495] A "data format" is a recording format or protocol for storing and communicating data (e.g., JSON, HTML).

[0496] An "emotion engine" is a technology or model that analyzes a user's facial expressions and voice to recognize their emotional state (e.g., joy, excitement, dissatisfaction).

[0497] The system of the present invention includes multiple cameras, a server, a user terminal, an emotion engine, etc. to provide an interactive and immersive experience in watching a basketball game.

[0498] The server collects video data of the match in real time from multiple cameras installed in the venue. This allows for time-series streaming of footage shot from different angles and positions. The video data is broken down into frames, and the positions of players and the ball are identified using an object detection model powered by TensorFlow. At this stage, positional information within each frame is obtained.

[0499] The server uses a SORT algorithm to track the movements of detected players and the ball between consecutive frames, recording the movements of each object during the match as time-series data for later analysis.

[0500] The server then analyzes the tracking data and identifies specific occurrences (e.g., goals, fouls, timeouts). For example, when the ball passes through the hoop, it detects that as a scoring event and updates the score in real time.

[0501] Based on the identified event data, the server uses a generative AI model to generate commentary in natural language, providing viewers with a clear understanding of the game situation.

[0502] The emotion engine recognizes the user's emotional state by capturing and analyzing their facial and voice data. Based on the captured emotion data, the server dynamically adjusts the content and display of the gameplay commentary. For example, if the user is excited, the server will provide more detailed and energetic commentary.

[0503] The updated scores and generated commentary are converted into a displayable data format and delivered to the user's terminal in real time. The user's terminal receives the delivered data and displays the scoreboard and commentary on the screen in real time.

[0504] As a concrete example, the system behavior when a player makes a successful three-point shot during a game is shown below.

[0505] 1. Server: Acquire video data

[0506] The server collects video data in real time from multiple cameras within the venue.

[0507] 2. Server: Image analysis

[0508] The server breaks down the video data into frames and uses TensorFlow to identify the positions of players and the ball.

[0509] 3. Server:Tracking

[0510] The server tracks the ball's movement between successive frames using the SORT algorithm.

[0511] 4. Server: Event Identification

[0512] The server detects when the ball passes through the hoop and identifies it as a scoring event.

[0513] 5. Server: Update score

[0514] Add 3 points to the team's score.

[0515] 6. Server: Play commentary generation

[0516] The server uses a generative AI model to generate commentary such as, "Player X made a great three-point shot!"

[0517] 7. Server: Emotion Recognition

[0518] The server analyzes the user's face and voice and determines whether the user is excited.

[0519] 8. Server: Game commentary adjustment

[0520] Tailor your commentary to be more energetic based on sentiment data.

[0521] 9. Server: Data Distribution

[0522] Updated scores and adjusted play commentary are delivered to user devices in real time.

[0523] 10. Terminal: Receiving and Displaying

[0524] The user's device receives the distributed data and displays the scoreboard and play commentary in real time.

[0525] The following are examples of prompt sentences that can be used:

[0526] "Users get very excited when a three-point shot is made, so we want more detailed and exciting commentary. Please generate appropriate commentary based on the current game situation."

[0527] This provides a more interactive and interesting viewing experience that is tailored to the user's emotions.

[0528] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0529] Step 1:

[0530] Acquiring video data

[0531] The server collects real-time video data of the match from multiple cameras installed in the venue. This data consists of footage from different angles and positions. The data collected from each camera is streamed to the server and used as input data. This aggregates real-time video data of the entire match. The output is frame-by-frame video data.

[0532] Step 2:

[0533] Image analysis

[0534] The server breaks down the acquired video data into frames and runs an object detection model using TensorFlow on each frame. This extracts the positional information of players and the ball. The input is the video data frame by frame, and the output is the positional data of players and the ball for each frame. Specifically, the object detection model is applied to identify the coordinates of objects in each frame.

[0535] Step 3:

[0536] tracking

[0537] The server uses the SORT algorithm to track the movements of detected players and the ball between successive frames. This generates tracking data, recording the movement path of each object in chronological order. The input is position data for each frame, and the output is tracking data in chronological order. Specifically, the server maintains the continuity of the object's position between successive frames and analyzes its movement pattern.

[0538] Step 4:

[0539] Event Identification

[0540] The server analyzes the tracking data and identifies specific events (e.g., a goal, a foul, or a timeout). For example, if the ball passes through a hoop, it is recognized as a score event. The input is tracking data, and the output is event data. Specifically, it detects an event when a specific condition is met (e.g., the ball's coordinates pass through the hoop).

[0541] Step 5:

[0542] Score Update

[0543] The server updates the team's score in real time based on the recognized events. The input is the event data, and the output is the updated score. Specifically, it adds the score for the corresponding team based on the scoring event and modifies the scoreboard in real time.

[0544] Step 6:

[0545] Play commentary generation

[0546] The server uses a generative AI model to generate play commentary based on the identified event data. The input is the event data, and the output is the generated commentary. Specifically, the server inputs a prompt sentence into the generative AI model, which then generates a natural language commentary based on the prompt sentence.

[0547] Step 7:

[0548] emotion recognition

[0549] The server acquires the user's facial and voice data and uses an emotion engine to recognize their emotions. The input is the user's facial and voice data, and the output is emotional state data. Specifically, the server analyzes the data acquired through the camera and microphone and classifies the user's emotions into categories such as "joy," "excitement," and "dissatisfaction."

[0550] Step 8:

[0551] Game commentary adjustments

[0552] The server dynamically adjusts the content and display method of the play commentary based on the user's emotions recognized by the emotion engine. The input is emotional state data and the generated commentary, and the output is the adjusted commentary. Specifically, the commentary is changed according to the user's emotions, and if the user is excited, the commentary is adjusted to be more energetic, for example.

[0553] Step 9:

[0554] Data Distribution

[0555] The server converts the updated score and adjusted commentary into a data format for display and delivers it to the user's device in real time. The input is the adjusted commentary and updated score, and the output is the delivered data. Specifically, the server converts the data into JSON or HTML format and sends it using a communication protocol such as WebSocket.

[0556] Step 10:

[0557] Receiving and Displaying

[0558] The user device receives the data distributed from the server and displays it on the screen as a scoreboard and commentary on the play. The input is the distributed data, and the output is the displayed score and commentary. Specifically, the data is analyzed and displayed to the user in real time.

[0559] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0560] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0561] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0562] [Second embodiment]

[0563] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0564] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0565] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0566] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0567] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0568] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0569] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0570] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0571] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0572] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0573] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0574] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0575] This invention is a system that digitizes and automates the work of table officials (TOs) in basketball games by capturing video data of the game in real time from multiple cameras and analyzing the movements of the players and the ball. This system allows each component to interact with each other, reflecting the game situation in the score in real time and automatically generating commentary on plays.

[0576] System configuration

[0577] The system of the present invention consists of the following main components:

[0578] 1. Video Data Acquisition Unit: The server acquires video data in real time from multiple cameras installed in the venue. This data is used to analyze the movements of players and the ball.

[0579] 2. Image analysis unit: The server breaks down the acquired video data into frames and identifies the positions of players and the ball using an object detection model.

[0580] 3. Tracking: The server uses a tracking algorithm to track the movements of players and the ball between successive frames and records these movements as time-series data.

[0581] 4. Event Identification: The server analyzes the tracking data and identifies specific events such as successful shots, fouls, and timeouts.

[0582] 5. Score Updater: The server updates the score in real time based on the identified events.

[0583] 6. Play commentary generation unit: The server uses a generation AI to create a play commentary in natural language based on the identified event data.

[0584] 7. Data distribution unit: The server converts the updated scores and generated play commentary into a data format for display and distributes them in real time.

[0585] 8. Display terminal: The terminal receives the distributed data and displays it on the screen as a scoreboard and play commentary.

[0586] Program processing flow (explained in natural language)

[0587] Server: Acquire video data

[0588] The server collects real-time video data from multiple cameras installed in the venue, each capturing footage from a different angle or position, and streams it in chronological order.

[0589] Server: Image analysis

[0590] The server breaks down the acquired video data into frames and applies an object detection model to identify the positions of players and the ball, thereby obtaining positional information for players and the ball within each frame.

[0591] Server:Tracking

[0592] The server uses a tracking algorithm to track the movement of players and the ball between successive frames, recording this data as time series of player and ball positions for use in motion analysis.

[0593] Server:Event Identification

[0594] The server identifies specific events (e.g., successful shots, fouls, timeouts) from the tracking data. For example, if it detects that the ball has passed through the hoop, it identifies that as a score event.

[0595] Server:Score Update

[0596] The server updates the team's score in real time based on the identified events, for example, adding a point to the team's score when a made shot event is identified.

[0597] Server: Play commentary generation

[0598] The server uses the generated AI to generate natural language commentary based on the identified event data, providing viewers with easy-to-understand details about the match.

[0599] Server: Data distribution

[0600] The server converts the updated scores and generated commentary into a displayable data format and distributes them in real time, making the latest match information instantly available.

[0601] Terminal: Receiving and displaying data

[0602] The device receives data distributed from the server and displays it on the screen as a scoreboard and commentary on the game, allowing spectators to understand the situation in real time.

[0603] Specific examples

[0604] For example, if a player makes a three-point shot during a game, the process is as follows:

[0605] 1. Acquisition of video data (server): The server acquires video data in real time from multiple cameras installed in the spectator seats.

[0606] 2. Image analysis (server): The server breaks down the captured video data into frames and applies object detection models to identify the positions of players and the ball.

[0607] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[0608] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[0609] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[0610] 6. Play commentary generation (server): The server uses a generation AI to generate a play commentary such as, "Player X made a great three-point shot!"

[0611] 7. Data distribution (server): The server converts the updated scores and play commentary into a data format for display and distributes them in real time.

[0612] 8. Receiving and displaying data (terminal): The terminal receives the distributed data and displays the latest score and play commentary on the screen in real time.

[0613] This will significantly reduce the burden on TOs, and allow game officials and spectators to understand game details in real time, allowing them to enjoy basketball even more.

[0614] The processing flow will be explained below.

[0615] Program processing steps

[0616] Step 1: Obtaining video data

[0617] server:

[0618] The server receives real-time video data of the match from multiple cameras in the venue, each positioned to capture the match from a different angle.

[0619] Step 2: Frame Decomposition

[0620] server:

[0621] The server breaks down the acquired video data into frames and processes each frame in parallel for the video data from each camera.

[0622] Step 3: Object detection

[0623] server:

[0624] The server uses object detection models such as YOLO (You Only Look Once) to identify players and the ball in each frame.

[0625] The object detection model outputs the position coordinates and labels of the players and ball (Player 1, Player 2, Ball, etc.).

[0626] Step 4: Tracking

[0627] server:

[0628] The server tracks the movement of players and the ball between successive frames using tracking algorithms such as Kalman filters or optical flow.

[0629] The tracking algorithm analyzes the position and movement vector of each object and records them as time series data.

[0630] Step 5: Event Identification

[0631] server:

[0632] The server identifies specific events (e.g., successful shots, fouls, timeouts) based on the tracking data.

[0633] For example, if the ball passes over the hoop and then changes position, this is identified as a "scoring event."

[0634] Step 6: Update your score

[0635] server:

[0636] The server updates the team scores in real time based on the identified events.

[0637] For example, if the shot is successful, the team will receive points corresponding to their score (e.g., 2 points, 3 points).

[0638] Step 7: Generate play instructions

[0639] server:

[0640] The server generates a play commentary in natural language based on the event data identified using the generation AI.

[0641] For example, it generates a sentence such as, "Player X broke through the defense with a sharp dribble and scored a stunning layup!"

[0642] Step 8: Prepare for data distribution

[0643] server:

[0644] The server converts the updated scores and generated commentary into a data format for display, such as JSON or HTML.

[0645] Step 9: Data Distribution

[0646] server:

[0647] The server distributes the converted data in real time to the screens at the venue and to the audience's devices.

[0648] Data is delivered using real-time communication protocols such as WebSocket.

[0649] Step 10: Receive and display data

[0650] Device:

[0651] The terminal receives the data distributed from the server and displays it on the screen as a scoreboard and commentary on the play.

[0652] The display is updated in real time, allowing spectators to instantly understand the situation in the game.

[0653] These are the specific processing steps of the "Digital Tournament for Basketball" system program, which reduces the burden on Tournament Officers and allows spectators to enjoy the details of the game in real time.

[0654] Example 1

[0655] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0656] In the current match management system, a large amount of manpower is required for the duties of table officials (TOs), which means that real-time updates of scores and prompt commentary on plays cannot always be provided. As a result, it is difficult for spectators and officials to grasp the progress of the match in real time, which can diminish the appeal of the match. Furthermore, incorrect score entries and time lags due to human error are also issues. It is necessary to provide a system that can solve these problems and improve the efficiency and sophistication of match management.

[0657] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0658] In this invention, the server includes means for acquiring video data of a game in real time from multiple camera devices, means for breaking down the acquired video data into frames and identifying the positions of players and the ball using a machine learning model, means for tracking the movements of players and the ball between consecutive frames using a tracking algorithm, means for identifying specific events from the tracking data and updating the score in real time, means for creating play commentary based on the identified event data using artificial intelligence, means for converting the updated score and the generated play commentary into a data format for display and distributing them in real time, and means for receiving the distributed data and displaying it on a screen as a score display device and play commentary. This enables the automation of TO operations and realizes the provision of accurate score updates and play commentary in real time.

[0659] "Multiple camera devices" refers to multiple camera devices that capture video data of the game in real time from different angles and positions.

[0660] "Video data" refers to digital video data showing the progress of a match captured by multiple camera devices.

[0661] "Machine learning model" refers to an artificial intelligence model used to analyze captured video data and identify the positions of players and the ball.

[0662] A "tracking algorithm" refers to a mathematical technique or calculation method for tracking the movement of players and the ball between successive frames.

[0663] "Tracking Data" refers to data containing player and ball position information obtained between successive frames by a tracking algorithm.

[0664] "Specific event" refers to a significant occurrence during a game (e.g., a successful shot, a foul, a timeout, etc.).

[0665] "Generative Artificial Intelligence" refers to a generative AI model used to generate natural language play commentary based on data.

[0666] "Points" refers to the number of points a team scores during a match.

[0667] "Score display device" refers to a device (e.g., a digital scoreboard or terminal display) that visually displays updated scores and commentary to spectators and other involved parties.

[0668] "Real-time" refers to processing or updating information occurring almost immediately or with very little delay.

[0669] This invention is a system that digitizes and automates the work of table officials (TOs) in sports matches by acquiring game video data in real time from multiple camera devices and analyzing the movements of players and the ball. Each component of this system works together, reflecting the game situation in real time in the score and automatically generating commentary on plays.

[0670] Key Components of the System

[0671] 1. Video data acquisition unit:

[0672] The server acquires real-time video data of the match from multiple cameras (e.g., general-purpose digital cameras) installed in the venue. Each camera captures the match from a different angle or position, and transmits the video data to the server via a network.

[0673] 2. Image analysis unit:

[0674] The server breaks down the captured video data into frames using libraries such as OpenCV. It then applies an object detection model (e.g., YOLOv3 or SSD model) trained using machine learning libraries such as TensorFlow or PyTorch to identify the positions of players and the ball within each frame. The detected position information is recorded as coordinate data.

[0675] 3. Tracking section:

[0676] The server uses tracking algorithms such as Kalman filters and SORT to track the movement of players and the ball between successive frames, recording this data as time series data and analyzing patterns of player and ball movement.

[0677] 4. Event Identification:

[0678] The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts), specifically when the ball passes through the hoop, which is identified as a scoring event. Identified events are recorded in a database.

[0679] 5. Score Update Section:

[0680] The server updates the team's score in real time based on the identified events. For example, if a shot is made, a point is added to the corresponding team's score. This information is managed as scoreboard data.

[0681] 6. Play commentary generation unit:

[0682] The server generates play commentary in natural language based on the identified event data using generative AI (e.g., GPT-3). For example, it generates commentary such as, "Player A made a great three-point shot!"

[0683] 7. Data Distribution Department:

[0684] The server converts the updated scores and generated commentary into a displayable data format such as JSON and distributes it in real time via WebSocket or REST API, making the latest information instantly available.

[0685] 8. Display terminal:

[0686] Devices (e.g., tablets and smartphones) and digital scoreboards receive the data distributed from the server. The devices display the received data on the screen in real time as a scoreboard and play commentary. Appropriate UI / UX design is used to improve visibility.

[0687] Specific examples

[0688] For example, if Player A makes a three-point shot during a game, the system will behave as follows:

[0689] 1. Acquiring video data:

[0690] The server collects video data in real time from multiple cameras in the venue, each capturing footage of the match from a different angle, and transmits the data to the server via a network.

[0691] 2. Image Analysis:

[0692] The server breaks down the video data into frames using OpenCV or similar tools, and applies an object detection model (such as YOLOv3) to identify the positions of players and the ball.

[0693] 3. Tracking:

[0694] The server uses a tracking algorithm (such as a Kalman filter) to track the ball's movement between successive frames and records the trajectory of its movement.

[0695] 4. Event Identification:

[0696] The server analyzes the tracking data and detects when the ball passes through the hoop, identifying this as a successful shot event.

[0697] 5. Score Update:

[0698] The server will award the successful shot three points to Team A's score and record this information in the database.

[0699] 6. Play commentary generation:

[0700] Using generative AI (GPT-3), we generate commentary on the play, such as "Player A made a great three-point shot!"

[0701] 7. Data Distribution:

[0702] The server converts the updated scores and generated play commentary into JSON format and delivers them to the device in real time via WebSocket.

[0703] 8. Receiving and Displaying Data:

[0704] The device receives the data and displays the latest score and commentary on the game in real time on the screen.

[0705] Prompt Sentence Examples

[0706] An example of an input prompt for the generation AI is, "Please generate commentary if player A makes a successful three-point shot."

[0707] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0708] Step 1: Obtaining video data

[0709] The server acquires real-time video data of the match from multiple camera devices. Specifically, multiple camera devices capture the match from different angles and transmit the video data to the server via a network. The input is the video data captured by each camera device, and the output is multiple real-time video streams stored on the server.

[0710] Step 2: Image analysis

[0711] The server breaks down the captured video data into frames. To do this, it uses libraries such as OpenCV to convert the video data into still images. Next, it applies an object detection model (e.g., YOLOv3 or SSD) trained using machine learning libraries such as TensorFlow or PyTorch to identify the positions of players and the ball within each frame. The input is the captured video data (frames), and the output is the position information (coordinate data) of players and the ball for each frame.

[0712] Step 3: Tracking

[0713] The server uses tracking algorithms such as Kalman filter and SORT to track the movements of players and the ball between consecutive frames. This allows the position information of players and the ball to be recorded as time-series data. The input is consecutive frames of data containing position information, and the output is time-series position information as tracking data.

[0714] Step 4: Event Identification

[0715] The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts). In particular, if the ball's trajectory passes through the hoop, it is identified as a successful shot. The input is the tracking data, and the output is the identified event information.

[0716] Step 5: Update your score

[0717] The server updates the team's score in real time based on the identified event information. For example, when a shot is made, a point is added to the corresponding team's score. This information is managed as scoreboard data. The input is the identified event information, and the output is updated score information.

[0718] Step 6: Generate play instructions

[0719] The server generates commentary based on the identified event data using a generation AI (e.g., GPT-3). To do this, a prompt is prepared for the generation AI and given as input. For example, the prompt might be, "Please generate commentary if player A makes a successful three-point shot." The input is the identified event data and the prompt, and the output is the generated commentary text.

[0720] Step 7: Data Distribution

[0721] The server converts the updated score and generated commentary into a displayable data format such as JSON and distributes it in real time via WebSocket or REST API. The input is the updated score information and the generated commentary text, and the output is the distributed JSON data.

[0722] Step 8: Receive and display data

[0723] The terminal receives data delivered from the server and displays it on the screen as a scoreboard and commentary in real time. For this purpose, a web application or a native application is used. The input is the delivered JSON data, and the output is the latest score and commentary displayed on the screen.

[0724] (Application example 1)

[0725] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0726] It is difficult for table officials (TOs) to perform their duties quickly and accurately during a basketball game. Furthermore, there is a lack of means to accurately communicate the real-time status of the game to spectators. Therefore, there is a need for a system that can reduce the burden on TOs and provide spectators with real-time information on the game's status.

[0727] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0728] In this invention, the server includes means for acquiring video data in real time from multiple camera devices, means for breaking down the acquired video data into frames and identifying the positions of the target and the sphere using an object detection model, means for tracking the movement of the target and the sphere between consecutive frames using a tracking algorithm, means for identifying specific events from the tracking data and updating scores in real time, means for creating action descriptions using a generation AI based on the identified event data, means for converting the updated scores and generated action descriptions into a data format for display and distributing them in real time, and means for receiving the distributed data and displaying them as a score board and action descriptions on a virtual space display device. This reduces the burden on TOs and makes it possible to accurately convey the situation of the game to spectators in real time.

[0729] A "camera" is a device for acquiring video data, and is responsible for capturing video of the game in real time from multiple angles and positions.

[0730] "Video data" refers to visual information of the match obtained from a camera, and is data that shows the progress of the match in real time.

[0731] An "object detection model" is an analytical algorithm or technology that analyzes acquired video data to identify the position of a specific object (such as a player or a ball).

[0732] "Target" refers to a person or object that is being tracked and identified, such as a player in a match.

[0733] The term "sphere" refers to a sports ball used in a game, and in the present invention refers to a basketball.

[0734] A "frame" is an individual still image that makes up video data, and a moving image is formed by successive frames.

[0735] A "tracking algorithm" is a technique used to track the movement of an object or sphere between successive frames and record its position information as time-series data.

[0736] "Specific events" refer to important events that occur during a game (such as a successful shot or a foul) that the system automatically identifies.

[0737] "Generative AI" is an artificial intelligence technology that automatically generates natural language descriptions of behavior based on identified event data.

[0738] An "action explanation" is an explanatory text about a specific event during a match, created by the generation AI.

[0739] The "score board" is an interface that is displayed on the virtual space display device to visually provide the latest score information.

[0740] A "virtual space display device" is a device that uses virtual reality technology to allow users to visually experience a match in a virtual space, and generally includes a VR headset.

[0741] A system for implementing this invention comprises the following major components:

[0742] 1. Server

[0743] The server has a means to acquire video data in real time from multiple camera devices, which are installed at different angles and positions within the venue to capture visual information of the match.

[0744] The server has a means to break down the acquired video data into frames and identify the positions of the objects and spheres using an object detection model, which is implemented using a library such as OpenCV.

[0745] The server has a means to track the movement of the object and the ball using a tracking algorithm between successive frames, allowing the position data of the object (player) and the ball (basketball) to be recorded over time.

[0746] The server has the means to identify specific events from the tracking data and update the score in real time. For example, when a ball passes through a hoop, it is identified as a scoring event and the score is automatically updated.

[0747] The server has a means to generate behavioral explanations based on the identified event data using a generative AI model, such as GPT-3.

[0748] The server has a means for converting the updated scores and generated action descriptions into a data format for display and distributing them in real time, thereby enabling the virtual space display device to quickly provide the latest match information.

[0749] 2. Terminal (Virtual Space Display Device)

[0750] The device receives data from the server and displays it in the virtual space as a scoreboard and action explanations, allowing users to visually grasp the progress of the game in real time. A VR headset or similar device is used as the virtual space display device.

[0751] 3. Users

[0752] By wearing the virtual space display device, users can watch the game in a virtual space. By referring to the score and action explanations updated in real time, users can intuitively understand the situation of the game.

[0753] Specific examples

[0754] For example, if a ball (basketball) passes through the hoop during a game and a three-point shot is successful, the process is as follows:

[0755] 1. Server

[0756] Video data acquisition: Video data is acquired in real time from the imaging device.

[0757] Image analysis: Video data is broken down into frames and object detection models are applied to identify the positions of objects (players) and balls (basketballs).

[0758] Tracking: The movement of the object and the sphere is tracked using a tracking algorithm between successive frames.

[0759] Event Identification: The tracking data is analyzed and when the sphere passes through the ring it is identified as a scoring event.

[0760] Score Update: Based on the identified incident, add 3 points to the team's score.

[0761] Play commentary generation: Using a generation AI, we generate an action explanation such as "Player A made a great three-point shot!"

[0762] Data distribution: Updated scores and commentary are converted into a displayable data format and distributed in real time.

[0763] 2. Terminal (Virtual Space Display Device)

[0764] Display of received data: Receive the distributed data and display it in the virtual space as a score board and action explanation.

[0765] Prompt Sentence Examples

[0766] "When a three-point shot is successful, generate a score and commentary of the play and display them in the virtual space."

[0767] The above steps reduce the burden on TOs and enable them to provide users with accurate information on the progress of the match in real time.

[0768] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0769] Step 1:

[0770] The server receives video data from the camera in real time. Each camera captures the game from a different angle or position and sends the video data to the server. The server receives the data and starts processing. The input is real-time video data, and the output is the video data.

[0771] Step 2:

[0772] The server uses the OpenCV library to decompose the video data into frames, which converts it into a sequence of still images (frames). The input is the video data acquired in step 1, and the output is the individual frames.

[0773] Step 3:

[0774] The server applies an object detection model to each decomposed frame to identify the positions of the objects (players) and the ball (basketball). This uses a machine learning object detection algorithm. The input is the image data for each frame, and the output is the position information of the objects and the ball within each frame.

[0775] Step 4:

[0776] The server uses a tracking algorithm to track the movement of the object and the sphere between successive frames, generating position information as time-series data. The input is the position information within each frame, and the output is time-series position data of the object and the sphere.

[0777] Step 5:

[0778] The server analyzes the tracking data and identifies specific events, such as a ball passing through a ring, as a scoring event. This is achieved using an event identification algorithm, whose input is the time-series position data and whose output is the identified specific event.

[0779] Step 6:

[0780] The server updates the scores in real time based on the identified events. Each scoring event automatically increments the team's score. The input is the identified event, and the output is the updated score.

[0781] Step 7:

[0782] The server uses a generative AI to create a behavioral explanation based on the identified event data. For example, it generates an explanation in the form of "Player A made a great three-point shot!" The input is the identified event data, and the output is the generated behavioral explanation.

[0783] Step 8:

[0784] The server converts the updated scores and generated behavior descriptions into a data format for display and distributes them in real time using data streaming technology. The input is the updated scores and generated behavior descriptions, and the output is data that can be distributed.

[0785] Step 9:

[0786] The terminal (virtual space display device) receives the data distributed from the server and displays it in the virtual space as a score board and action explanations. This allows the user to visually grasp the progress of the game in real time. The input is the distributed data, and the output is the display in the virtual space.

[0787] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0788] This invention is a system that digitizes and automates the work of table officials (TOs) in basketball games, and combines it with an emotion engine that recognizes user emotions to provide a more interactive and immersive game-watching experience.

[0789] System configuration

[0790] The system of the present invention consists of the following main components:

[0791] 1. Video Data Acquisition Unit: The server acquires video data of the match in real time from multiple cameras installed in the venue. This data is used to analyze the movements of players and the ball.

[0792] 2. Image analysis unit: The server breaks down the acquired video data into frames and identifies the positions of players and the ball using an object detection model.

[0793] 3. Tracking: The server uses a tracking algorithm to track the movements of players and the ball between successive frames and records these movements as time-series data.

[0794] 4. Event Identification: The server analyzes the tracking data and identifies specific events such as successful shots, fouls, and timeouts.

[0795] 5. Score Updater: The server updates the score in real time based on the identified events.

[0796] 6. Play commentary generation unit: The server uses a generation AI to create a play commentary in natural language based on the identified event data.

[0797] 7. Emotion engine: The server includes an emotion engine that acquires the user's face and voice data and recognizes their emotions.

[0798] 8. Data distribution unit: The server converts the updated scores and generated play commentary into a data format for display and distributes them in real time.

[0799] 9. Display terminal: The terminal receives the distributed data and displays it on the screen as a scoreboard and play commentary.

[0800] Program processing flow (explained in natural language)

[0801] Server: Acquire video data

[0802] The server collects real-time video data from multiple cameras installed in the venue, each capturing footage of the match from a different angle or position, and streams the footage in chronological order.

[0803] Server: Image analysis

[0804] The server breaks down the acquired video data into frames and applies an object detection model to identify the positions of players and the ball, thereby obtaining positional information for players and the ball within each frame.

[0805] Server:Tracking

[0806] The server uses a tracking algorithm to track the movement of players and the ball between successive frames, recording this data as time series of player and ball positions for use in motion analysis.

[0807] Server:Event Identification

[0808] The server identifies specific events (e.g., successful shots, fouls, timeouts) from the tracking data. For example, if it detects that the ball has passed through the hoop, it identifies that as a score event.

[0809] Server:Score Update

[0810] The server updates the team's score in real time based on the identified events, for example, adding a point to the team's score when a made shot event is identified.

[0811] Server: Play commentary generation

[0812] The server generates play commentary in natural language based on the event data identified using the AI ​​generation technology, which provides viewers with easy-to-understand details about the match.

[0813] Server: Emotion recognition

[0814] The server acquires the user's facial and voice data and uses an emotion engine to recognize their emotions. For example, by analyzing facial expression data and tone of voice acquired through a camera or microphone, it can determine whether the user is in an emotional state such as "joy," "excitement," or "dissatisfaction."

[0815] Server: Adjustment of gameplay commentary

[0816] The server dynamically adjusts the content and display of the commentary based on the user's emotions as recognized by the emotion engine. For example, if the user is excited, the server can display more detailed and enthusiastic commentary.

[0817] Server: Real-time analysis of audience sentiment

[0818] The server aggregates the emotional data of multiple users and analyzes the emotional state of the entire audience in real time. This data may be reflected in the progress and direction of the match.

[0819] Server: Prepare data distribution

[0820] The server converts the updated scores and generated commentary into a data format for display, such as JSON or HTML.

[0821] Server: Data distribution

[0822] The server then transmits the converted data to the screens at the venue and to the audience's devices in real time using a real-time communication protocol such as WebSocket.

[0823] Terminal: Receiving and displaying data

[0824] The device receives data from the server and displays it on the screen as a scoreboard and commentary on the game, which is updated in real time, allowing spectators to instantly understand the situation in the game.

[0825] Specific examples

[0826] For example, if a player makes a three-point shot during a game, the process is as follows:

[0827] 1. Acquisition of video data (server): The server acquires video data in real time from multiple cameras installed within the venue.

[0828] 2. Image analysis (server): The server breaks down the captured video data into frames and applies object detection models to identify the positions of players and the ball.

[0829] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[0830] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[0831] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[0832] 6. Play commentary generation (server): The server uses a generation AI to generate a play commentary such as, "Player X made a great three-point shot!"

[0833] 7. Emotion recognition (server): The server acquires the user's facial and voice data and uses the emotion engine to recognize when the user is excited.

[0834] 8. Adjusting play commentary (server): The server will reflect the user's excitement and adjust the commentary to be more detailed and energetic.

[0835] 9. Data distribution preparation (server): The server converts the updated scores and adjusted play commentary into a data format for display.

[0836] 10. Data distribution (server): The server distributes data in real time to the screen and audience devices.

[0837] 11. Receiving and displaying data (terminal): The terminal receives the distributed data and displays the scoreboard and play commentary in real time.

[0838] This will significantly reduce the burden on TOs and allow audiences to have a more immersive experience by receiving individually tailored commentary through emotion recognition.

[0839] The processing flow will be explained below.

[0840] Program processing steps

[0841] Step 1: Obtaining video data

[0842] server:

[0843] The server collects real-time video data from multiple cameras installed in the venue, each capturing footage from a different angle, and streams it in chronological order.

[0844] Step 2: Frame Decomposition

[0845] server:

[0846] The server breaks down the acquired video data into frames and processes each frame in parallel for the video data from each camera.

[0847] Step 3: Object detection

[0848] server:

[0849] The server uses object detection models such as YOLO (You Only Look Once) to identify players and the ball in each frame.

[0850] The detection model outputs the position coordinates and labels of players and the ball (Player 1, Player 2, Ball, etc.).

[0851] Step 4: Tracking

[0852] server:

[0853] The server uses a tracking algorithm to track the movement of players and the ball between successive frames.

[0854] The algorithm analyzes the position and movement vectors of each object and records them as time series data.

[0855] Step 5: Event Identification

[0856] server:

[0857] The server identifies specific events (e.g., successful shots, fouls, timeouts) based on the tracking data.

[0858] For example, if the ball passes over the hoop, this is identified as a "scoring event."

[0859] Step 6: Update your score

[0860] server:

[0861] The server updates the team scores in real time based on the identified events.

[0862] For example, if the shot is successful, the team will receive points corresponding to their score (e.g., 2 points, 3 points).

[0863] Step 7: Generate play instructions

[0864] server:

[0865] The server generates a play commentary in natural language based on the event data identified using the generation AI.

[0866] For example, it generates a sentence such as, "Player X broke through the defense with a sharp dribble and scored a stunning layup!"

[0867] Step 8: Emotion Recognition

[0868] server:

[0869] The server acquires the user's facial and voice data and recognizes their emotions using an emotion engine.

[0870] For example, facial expression data and tone of voice obtained from a camera or microphone are analyzed to determine whether the user is in an emotional state such as "joy," "excitement," or "dissatisfaction."

[0871] Step 9: Adjust the commentary

[0872] server:

[0873] The server dynamically adjusts the content and display method of the play commentary based on the user's emotions recognized by the emotion engine.

[0874] For example, if the user is excited, a more detailed and energetic commentary will be displayed.

[0875] Step 10: Real-time analysis of audience sentiment

[0876] server:

[0877] The server aggregates the emotional data of multiple users and analyzes the emotional state of the entire audience in real time.

[0878] This data may be reflected in the progress and presentation of the match.

[0879] Step 11: Prepare for data distribution

[0880] server:

[0881] The server converts the updated scores and generated commentary into a data format for display, such as JSON or HTML.

[0882] Step 12: Data Distribution

[0883] server:

[0884] The server distributes the converted data in real time to the screens at the venue and to the audience's devices.

[0885] Data is delivered using real-time communication protocols such as WebSocket.

[0886] Step 13: Receiving and displaying data

[0887] Device:

[0888] The terminal receives the data distributed from the server and displays it on the screen as a scoreboard and commentary on the game.

[0889] The display is updated in real time, allowing spectators to instantly understand the situation in the game.

[0890] Specific examples

[0891] For example, if a player makes a three-point shot during a game, the process is as follows:

[0892] 1. Acquisition of video data (server): The server acquires video data in real time from multiple cameras installed within the venue.

[0893] 2. Image analysis (server): The server breaks down the captured video data into frames and applies object detection models to identify the positions of players and the ball.

[0894] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[0895] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[0896] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[0897] 6. Play commentary generation (server): The server uses a generation AI to generate a play commentary such as, "Player X made a great three-point shot!"

[0898] 7. Emotion recognition (server): The server acquires the user's facial and voice data and uses the emotion engine to recognize when the user is excited.

[0899] 8. Adjusting play commentary (server): The server will reflect the user's excitement and adjust the commentary to be more detailed and energetic.

[0900] 9. Data distribution preparation (server): The server converts the updated scores and adjusted play commentary into a data format for display.

[0901] 10. Data distribution (server): The server distributes data in real time to the screen and audience devices.

[0902] 11. Receiving and displaying data (terminal): The terminal receives the distributed data and displays the scoreboard and play commentary in real time.

[0903] This will significantly reduce the burden on TOs and allow audiences to have a more immersive experience by receiving individually tailored commentary through emotion recognition.

[0904] Example 2

[0905] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0906] At sports games such as basketball, the goal is to improve the efficiency and accuracy of table officials (TOs) by digitizing and automating their work. There is also a need for a system that can provide real-time commentary and information on the game to spectators, creating a more immersive and realistic experience. However, existing systems face challenges in accurately grasping the game situation and updating and distributing necessary information in real time. Furthermore, there are currently no systems that can recognize and reflect spectator emotions and provide personalized commentary.

[0907] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0908] In this invention, the server includes a means for acquiring image data of the game in real time from multiple camera devices, a means for dividing the acquired image data into successive images and using an object detection algorithm to identify the positions of players and the ball, and a means for tracking the movement of players and the ball between successive images using a tracking algorithm. This enables accurate understanding of the game situation, real-time score updates, and the creation and distribution of game commentary using generative AI. Furthermore, the server can acquire user facial and voice data, analyze their emotional state using an emotion engine, and dynamically adjust the content and display method of the game commentary based on that data, providing spectators with a more realistic and immersive game viewing experience.

[0909] "Filming equipment" refers to video cameras installed within the venue to capture the state of the match.

[0910] "Image data" refers to video data of a match captured by a camera.

[0911] "Continuous images" refers to a set of still images obtained by breaking down video data into frames.

[0912] An "object detection algorithm" refers to a machine learning model for identifying specific objects (players or balls) within image data.

[0913] "Player" means a player taking part in a match.

[0914] "Sphere" refers to the ball used during a match.

[0915] A "tracking algorithm" refers to a computational method for tracking the position of a particular object (a player or a ball) between successive image frames.

[0916] "Specific occurrences" refer to significant events that occur during a game, such as a successful shot, a foul, or a timeout.

[0917] "Score" refers to points awarded to a team based on a specific event.

[0918] "Generative algorithm" refers to AI technology that generates natural language commentary on sports events based on given data.

[0919] "Sports commentary" refers to natural language text that describes the progress of a match or an event.

[0920] "Display data format" refers to a data format (e.g., JSON, HTML) suitable for displaying scores and commentary on a display device.

[0921] An "emotion engine" refers to software or algorithms that analyze a user's facial expressions and voice data to recognize their emotional state.

[0922] "Users" refers to spectators who attend matches.

[0923] "Emotional state" refers to the user's psychological state, such as joy, excitement, or dissatisfaction.

[0924] "Dynamic adjustment" refers to changing the content and display format in real time depending on the situation.

[0925] This invention aims to digitize and automate the work of table officials (TOs) at sports games, and is a system that combines it with an emotion engine that recognizes user emotions to provide a more interactive and immersive game-watching experience.

[0926] System configuration

[0927] Server: Acquire video data

[0928] The server acquires image data of the match in real time from multiple camera devices installed in the venue. Each camera captures the match from a different angle or position and streams the data. Specific examples of such cameras include high-resolution digital cameras and IP cameras.

[0929] Server: Image analysis

[0930] The server divides the acquired image data into successive images and applies an object detection algorithm to identify the positions of players and the ball. The object detection algorithm uses machine learning models such as YOLO (You Only Look Once) and Mask R-CNN. This allows the server to obtain the positional information of players and the ball within each frame.

[0931] Server:Tracking

[0932] The server tracks the movements of the players and the ball between successive image frames using a tracking algorithm, such as a Kalman filter and the tracking algorithm SORT (Simple Online and Realtime Tracking). This process stores the positional information of the players and the ball as time-series data.

[0933] Server:Event Identification

[0934] The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts). For example, a ball passing through the hoop is captured as a "successful shot." Event identification is achieved using statistical analysis and rule-based techniques.

[0935] Server:Score Update

[0936] The server updates the team's score in real time based on the identified incidents, and points are automatically added to the team's score for successful shots, for example, three points for a successful three-point shot.

[0937] Server: Play commentary generation

[0938] The server uses a generative algorithm to generate a natural language commentary of the game based on the identified event data. Generative AI models such as GPT-4 are used. For example, a commentary such as "Player X made a successful three-point shot!" is generated.

[0939] Server: Emotion recognition

[0940] The server collects the user's facial and voice data and analyzes their emotional state using an emotion engine, which uses tools such as Affectiva and the Microsoft Azure Emotion API. This allows the server to determine in real time whether the user is in a state of joy, excitement, dissatisfaction, or other emotional state.

[0941] Server: Adjustment of gameplay commentary

[0942] The server dynamically adjusts the content and display method of the generated commentary based on the user's emotions determined by the emotion engine. For example, if the user is excited, the generated commentary will be changed to be more detailed and passionate.

[0943] Server: Real-time analysis of audience sentiment

[0944] The server aggregates emotional data collected from multiple users and analyzes the emotional state of the entire venue in real time. This data is reflected in the progress and direction of the match.

[0945] Server: Prepare data distribution

[0946] The server converts the updated scores and generated commentary into a displayable data format (e.g., JSON or HTML), which facilitates distribution.

[0947] Server: Data distribution

[0948] The server then distributes the converted data in real time to the screens at the venue and to the audience's devices, using a real-time communication protocol such as WebSocket.

[0949] Terminal: Receiving and displaying data

[0950] The terminals receive the data distributed from the server and display it on the screen as scores and commentary on the match. The terminals can be smartphones, tablets, digital signage, etc. The display is updated in real time, allowing spectators to instantly understand the situation of the match.

[0951] Specific examples

[0952] For example, if a player makes a three-point shot during a game, the process is as follows:

[0953] 1. Acquisition of video data (server): The server acquires video data in real time from multiple camera devices installed within the venue.

[0954] 2. Image analysis (server): The server decomposes the captured video data and applies object detection algorithms to identify the positions of players and the ball.

[0955] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[0956] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[0957] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[0958] 6. Play commentary generation (server): The server uses a generation AI to generate commentary on the game, such as "Player X made a successful three-point shot!"

[0959] 7. Emotion recognition (server): The server acquires data on the user's facial expressions and voice, and uses the emotion engine to recognize when the user is excited.

[0960] 8. Adjusting play commentary (server): The server will reflect the user's excitement and adjust the commentary to be more detailed and energetic.

[0961] 9. Data distribution preparation (server): The server converts the updated scores and adjusted game commentary into a data format for display.

[0962] 10. Data distribution (server): The server distributes data in real time to the screen and audience devices.

[0963] 11. Data reception and display (terminal): The terminal receives the distributed data and displays the score and commentary in real time, allowing spectators to instantly grasp the situation of the match, significantly reducing the burden on the TO. Spectators can also receive individually tailored commentary through emotion recognition, providing a more immersive experience.

[0964] Prompt Sentence Examples

[0965] The following prompt sentence is used for the generative AI model to automatically generate an explanation such as "Player X made a successful three-point shot!"

[0966] Prompt statement:

[0967] "With the crowd going wild, player X made a brilliant three-point shot!"

[0968] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0969] Step 1:

[0970] Input: Real-time video data from multiple imaging modalities

[0971] Specific operation: The server acquires video data of the match in real time from multiple camera devices installed in the venue. Each camera captures the match from a different angle or position and streams the video data. Specifically, the data is transmitted in real time using the RTSP protocol.

[0972] Output: Real-time video data of the match

[0973] Step 2:

[0974] Input: Real-time video data of the match

[0975] Specific operation: The server divides the acquired video data into successive image frames. Specifically, it uses a tool such as ffmpeg to break the video down into 30 frames per second.

[0976] Output: Sequential image frames

[0977] Step 3:

[0978] Input: Sequential image frames

[0979] How it works: The server applies an object detection algorithm to identify the positions of players and spheres in each frame. Specifically, it uses models such as YOLO (You Only Look Once) and Mask R-CNN to output the coordinates of players and spheres in each frame.

[0980] Output: Position information of the player and the ball in each frame

[0981] Step 4:

[0982] Input: Position information of the player and the ball in each frame

[0983] How it works: The server tracks the movement of players and the ball using a tracking algorithm between consecutive frames, specifically using a Kalman filter and the SORT (Simple Online and Realtime Tracking) algorithm to record position information as time-series data.

[0984] Output: Time series position data of players and balls

[0985] Step 5:

[0986] Input: Time series position data of players and balls

[0987] Specific operation: The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts). Specifically, it checks whether the ball has passed through the hoop, and if it has, identifies it as a "successful shot."

[0988] Output: Identified event data

[0989] Step 6:

[0990] Input: Identified event data

[0991] Specific behavior: The server updates the team's score in real time based on the identified events. Specifically, for successful shots, the server automatically adds 2 or 3 points to the team's score.

[0992] Output: Updated score data

[0993] Step 7:

[0994] Input: Identified event data, updated score data

[0995] Specific behavior: The server uses a generative AI model (e.g., GPT-4) to generate natural language commentary on the game based on the identified event data. An example of a specific prompt sentence is, "Player X made a three-point shot!"

[0996] Output: Generated commentary

[0997] Step 8:

[0998] Input: User's facial expression data, voice data

[0999] Specific operation: The server acquires the user's facial expression and voice data and analyzes their emotional state using an emotion engine (e.g., Affectiva, Microsoft Azure Emotion API). This determines whether the user is in an emotional state such as "joy," "excitement," or "dissatisfaction."

[1000] Output: Analyzed sentiment data

[1001] Step 9:

[1002] Input: Analyzed emotion data, generated commentary

[1003] Specific behavior: The server dynamically adjusts the content and presentation of the generated commentary based on the user's emotional state. For example, if the user is excited, the server displays more detailed and energetic commentary.

[1004] Output: Coordinated competition commentary

[1005] Step 10:

[1006] Input: Adjusted game commentary, updated scoring data

[1007] Specific operation: The server converts the adjusted competition commentary and score data into a data format for display (e.g., JSON, HTML).

[1008] Output: Competition commentary and score data converted into a displayable data format

[1009] Step 11:

[1010] Input: Competition commentary and score data converted into a display format

[1011] How it works: The server distributes the converted data in real time to the screens in the venue and to the audience's devices, using a real-time communication protocol such as WebSocket.

[1012] Output: Real-time broadcast of competition commentary and score data

[1013] Step 12:

[1014] Input: Real-time game commentary and score data

[1015] Specific operation: The terminal receives data distributed from the server and displays it on the screen as score and commentary. The display is updated in real time, allowing spectators to instantly understand the situation of the game. Specifically, smartphones, tablets, digital signage, etc. are used.

[1016] Output: On-screen commentary and score information

[1017] (Application example 2)

[1018] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1019] Traditionally, watching basketball games required manual work by table officials (TOs), which placed a heavy burden on spectators and sometimes caused delays in real-time game progress. Furthermore, there was no way to provide an interactive experience that responded to spectators' emotions, limiting the quality of the viewing experience. These issues can be resolved using new technology.

[1020] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1021] In this invention, the server includes means for acquiring game video data from multiple cameras in real time, means for breaking down the acquired video data into frames and identifying the positions of objects and the ball using an object detection model, means for tracking the movement of the object and the ball between consecutive frames using a tracking algorithm, means for identifying specific events from the tracking data and updating the score in real time, means for creating play commentary based on the identified event data using a generation AI, means for converting the updated score and the generated play commentary into a display data format and distributing them in real time, means for receiving the distributed data and displaying it as a scoreboard and play commentary on a screen, means for acquiring user facial and voice data and recognizing the user's emotions using an emotion engine, and means for dynamically adjusting the content and display method of the play commentary based on the recognized emotion data. This improves the quality of game viewing and makes it possible to provide an interactive experience that corresponds to the user's emotions.

[1022] "Multiple cameras" means two or more video recording devices used to capture the action of a match in real time from different angles and positions.

[1023] "Video data" refers to digital data that is captured in real time by a camera and used for analysis and storage, either as is or after processing.

[1024] "Frame decomposition" refers to the process of dividing continuous video data into individual still images (frames).

[1025] An "object detection model" is an artificial intelligence model trained to identify specific objects (e.g., players or balls) in video or images.

[1026] A "tracking algorithm" is a computational method or process for continuously tracking the movement of an identified object between frames of video.

[1027] A "specific occurrence" is a significant action or event that occurs during a match (e.g., a goal, a foul).

[1028] "Generative AI" is a technology that uses artificial intelligence to automatically generate natural language explanations and sentences from input data.

[1029] "Play commentary" refers to written or audio data that explains specific events during a match in a way that is easy for viewers to understand.

[1030] A "data format" is a recording format or protocol for storing and communicating data (e.g., JSON, HTML).

[1031] An "emotion engine" is a technology or model that analyzes a user's facial expressions and voice to recognize their emotional state (e.g., joy, excitement, dissatisfaction).

[1032] The system of the present invention includes multiple cameras, a server, a user terminal, an emotion engine, etc. to provide an interactive and immersive experience in watching a basketball game.

[1033] The server collects video data of the match in real time from multiple cameras installed in the venue. This allows for time-series streaming of footage shot from different angles and positions. The video data is broken down into frames, and the positions of players and the ball are identified using an object detection model powered by TensorFlow. At this stage, positional information within each frame is obtained.

[1034] The server uses a SORT algorithm to track the movements of detected players and the ball between consecutive frames, recording the movements of each object during the match as time-series data for later analysis.

[1035] The server then analyzes the tracking data and identifies specific occurrences (e.g., goals, fouls, timeouts). For example, when the ball passes through the hoop, it detects that as a scoring event and updates the score in real time.

[1036] Based on the identified event data, the server uses a generative AI model to generate commentary in natural language, providing viewers with a clear understanding of the game situation.

[1037] The emotion engine recognizes the user's emotional state by capturing and analyzing their facial and voice data. Based on the captured emotion data, the server dynamically adjusts the content and display of the gameplay commentary. For example, if the user is excited, the server will provide more detailed and energetic commentary.

[1038] The updated scores and generated commentary are converted into a displayable data format and delivered to the user's terminal in real time. The user's terminal receives the delivered data and displays the scoreboard and commentary on the screen in real time.

[1039] As a concrete example, the system behavior when a player makes a successful three-point shot during a game is shown below.

[1040] 1. Server: Acquire video data

[1041] The server collects video data in real time from multiple cameras within the venue.

[1042] 2. Server: Image analysis

[1043] The server breaks down the video data into frames and uses TensorFlow to identify the positions of players and the ball.

[1044] 3. Server:Tracking

[1045] The server tracks the ball's movement between successive frames using the SORT algorithm.

[1046] 4. Server: Event Identification

[1047] The server detects when the ball passes through the hoop and identifies it as a scoring event.

[1048] 5. Server: Update score

[1049] Add 3 points to the team's score.

[1050] 6. Server: Play commentary generation

[1051] The server uses a generative AI model to generate commentary such as, "Player X made a great three-point shot!"

[1052] 7. Server: Emotion Recognition

[1053] The server analyzes the user's face and voice and determines whether the user is excited.

[1054] 8. Server: Game commentary adjustment

[1055] Tailor your commentary to be more energetic based on sentiment data.

[1056] 9. Server: Data Distribution

[1057] Updated scores and adjusted play commentary are delivered to user devices in real time.

[1058] 10. Terminal: Receiving and Displaying

[1059] The user's device receives the distributed data and displays the scoreboard and play commentary in real time.

[1060] The following are examples of prompt sentences that can be used:

[1061] "Users get very excited when a three-point shot is made, so we want more detailed and exciting commentary. Please generate appropriate commentary based on the current game situation."

[1062] This provides a more interactive and interesting viewing experience that is tailored to the user's emotions.

[1063] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1064] Step 1:

[1065] Acquiring video data

[1066] The server collects real-time video data of the match from multiple cameras installed in the venue. This data consists of footage from different angles and positions. The data collected from each camera is streamed to the server and used as input data. This aggregates real-time video data of the entire match. The output is frame-by-frame video data.

[1067] Step 2:

[1068] Image analysis

[1069] The server breaks down the acquired video data into frames and runs an object detection model using TensorFlow on each frame. This extracts the positional information of players and the ball. The input is the video data frame by frame, and the output is the positional data of players and the ball for each frame. Specifically, the object detection model is applied to identify the coordinates of objects in each frame.

[1070] Step 3:

[1071] tracking

[1072] The server uses the SORT algorithm to track the movements of detected players and the ball between successive frames. This generates tracking data, recording the movement path of each object in chronological order. The input is position data for each frame, and the output is tracking data in chronological order. Specifically, the server maintains the continuity of the object's position between successive frames and analyzes its movement pattern.

[1073] Step 4:

[1074] Event Identification

[1075] The server analyzes the tracking data and identifies specific events (e.g., a goal, a foul, or a timeout). For example, if the ball passes through a hoop, it is recognized as a score event. The input is tracking data, and the output is event data. Specifically, it detects an event when a specific condition is met (e.g., the ball's coordinates pass through the hoop).

[1076] Step 5:

[1077] Score Update

[1078] The server updates the team's score in real time based on the recognized events. The input is the event data, and the output is the updated score. Specifically, it adds the score for the corresponding team based on the scoring event and modifies the scoreboard in real time.

[1079] Step 6:

[1080] Play commentary generation

[1081] The server uses a generative AI model to generate play commentary based on the identified event data. The input is the event data, and the output is the generated commentary. Specifically, the server inputs a prompt sentence into the generative AI model, which then generates a natural language commentary based on the prompt sentence.

[1082] Step 7:

[1083] emotion recognition

[1084] The server acquires the user's facial and voice data and uses an emotion engine to recognize their emotions. The input is the user's facial and voice data, and the output is emotional state data. Specifically, the server analyzes the data acquired through the camera and microphone and classifies the user's emotions into categories such as "joy," "excitement," and "dissatisfaction."

[1085] Step 8:

[1086] Game commentary adjustments

[1087] The server dynamically adjusts the content and display method of the play commentary based on the user's emotions recognized by the emotion engine. The input is emotional state data and the generated commentary, and the output is the adjusted commentary. Specifically, the commentary is changed according to the user's emotions, and if the user is excited, the commentary is adjusted to be more energetic, for example.

[1088] Step 9:

[1089] Data Distribution

[1090] The server converts the updated score and adjusted commentary into a data format for display and delivers it to the user's device in real time. The input is the adjusted commentary and updated score, and the output is the delivered data. Specifically, the server converts the data into JSON or HTML format and sends it using a communication protocol such as WebSocket.

[1091] Step 10:

[1092] Receiving and Displaying

[1093] The user device receives the data distributed from the server and displays it on the screen as a scoreboard and commentary on the play. The input is the distributed data, and the output is the displayed score and commentary. Specifically, the data is analyzed and displayed to the user in real time.

[1094] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1095] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1096] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1097] [Third embodiment]

[1098] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1099] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1100] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1101] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1102] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1103] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1104] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1105] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1106] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1107] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1108] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1109] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1110] This invention is a system that digitizes and automates the work of table officials (TOs) in basketball games by capturing video data of the game in real time from multiple cameras and analyzing the movements of the players and the ball. This system allows each component to interact with each other, reflecting the game situation in the score in real time and automatically generating commentary on plays.

[1111] System configuration

[1112] The system of the present invention consists of the following main components:

[1113] 1. Video Data Acquisition Unit: The server acquires video data in real time from multiple cameras installed in the venue. This data is used to analyze the movements of players and the ball.

[1114] 2. Image analysis unit: The server breaks down the acquired video data into frames and identifies the positions of players and the ball using an object detection model.

[1115] 3. Tracking: The server uses a tracking algorithm to track the movements of players and the ball between successive frames and records these movements as time-series data.

[1116] 4. Event Identification: The server analyzes the tracking data and identifies specific events such as successful shots, fouls, and timeouts.

[1117] 5. Score Updater: The server updates the score in real time based on the identified events.

[1118] 6. Play commentary generation unit: The server uses a generation AI to create a play commentary in natural language based on the identified event data.

[1119] 7. Data distribution unit: The server converts the updated scores and generated play commentary into a data format for display and distributes them in real time.

[1120] 8. Display terminal: The terminal receives the distributed data and displays it on the screen as a scoreboard and play commentary.

[1121] Program processing flow (explained in natural language)

[1122] Server: Acquire video data

[1123] The server collects real-time video data from multiple cameras installed in the venue, each capturing footage from a different angle or position, and streams it in chronological order.

[1124] Server: Image analysis

[1125] The server breaks down the acquired video data into frames and applies an object detection model to identify the positions of players and the ball, thereby obtaining positional information for players and the ball within each frame.

[1126] Server:Tracking

[1127] The server uses a tracking algorithm to track the movement of players and the ball between successive frames, recording this data as time series of player and ball positions for use in motion analysis.

[1128] Server:Event Identification

[1129] The server identifies specific events (e.g., successful shots, fouls, timeouts) from the tracking data. For example, if it detects that the ball has passed through the hoop, it identifies that as a score event.

[1130] Server:Score Update

[1131] The server updates the team's score in real time based on the identified events, for example, adding a point to the team's score when a made shot event is identified.

[1132] Server: Play commentary generation

[1133] The server uses the generated AI to generate natural language commentary based on the identified event data, providing viewers with easy-to-understand details about the match.

[1134] Server: Data distribution

[1135] The server converts the updated scores and generated commentary into a displayable data format and distributes them in real time, making the latest match information instantly available.

[1136] Terminal: Receiving and displaying data

[1137] The device receives data distributed from the server and displays it on the screen as a scoreboard and commentary on the game, allowing spectators to understand the situation in real time.

[1138] Specific examples

[1139] For example, if a player makes a three-point shot during a game, the process is as follows:

[1140] 1. Acquisition of video data (server): The server acquires video data in real time from multiple cameras installed in the spectator seats.

[1141] 2. Image analysis (server): The server breaks down the captured video data into frames and applies object detection models to identify the positions of players and the ball.

[1142] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[1143] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[1144] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[1145] 6. Play commentary generation (server): The server uses a generation AI to generate a play commentary such as, "Player X made a great three-point shot!"

[1146] 7. Data distribution (server): The server converts the updated scores and play commentary into a data format for display and distributes them in real time.

[1147] 8. Receiving and displaying data (terminal): The terminal receives the distributed data and displays the latest score and play commentary on the screen in real time.

[1148] This will significantly reduce the burden on TOs, and allow game officials and spectators to understand game details in real time, allowing them to enjoy basketball even more.

[1149] The processing flow will be explained below.

[1150] Program processing steps

[1151] Step 1: Obtaining video data

[1152] server:

[1153] The server receives real-time video data of the match from multiple cameras in the venue, each positioned to capture the match from a different angle.

[1154] Step 2: Frame Decomposition

[1155] server:

[1156] The server breaks down the acquired video data into frames and processes each frame in parallel for the video data from each camera.

[1157] Step 3: Object detection

[1158] server:

[1159] The server uses object detection models such as YOLO (You Only Look Once) to identify players and the ball in each frame.

[1160] The object detection model outputs the position coordinates and labels of the players and ball (Player 1, Player 2, Ball, etc.).

[1161] Step 4: Tracking

[1162] server:

[1163] The server tracks the movement of players and the ball between successive frames using tracking algorithms such as Kalman filters or optical flow.

[1164] The tracking algorithm analyzes the position and movement vector of each object and records them as time series data.

[1165] Step 5: Event Identification

[1166] server:

[1167] The server identifies specific events (e.g., successful shots, fouls, timeouts) based on the tracking data.

[1168] For example, if the ball passes over the hoop and then changes position, this is identified as a "scoring event."

[1169] Step 6: Update your score

[1170] server:

[1171] The server updates the team scores in real time based on the identified events.

[1172] For example, if the shot is successful, the team will receive points corresponding to their score (e.g., 2 points, 3 points).

[1173] Step 7: Generate play instructions

[1174] server:

[1175] The server generates a play commentary in natural language based on the event data identified using the generation AI.

[1176] For example, it generates a sentence such as, "Player X broke through the defense with a sharp dribble and scored a stunning layup!"

[1177] Step 8: Prepare for data distribution

[1178] server:

[1179] The server converts the updated scores and generated commentary into a data format for display, such as JSON or HTML.

[1180] Step 9: Data Distribution

[1181] server:

[1182] The server distributes the converted data in real time to the screens at the venue and to the audience's devices.

[1183] Data is delivered using real-time communication protocols such as WebSocket.

[1184] Step 10: Receive and display data

[1185] Device:

[1186] The terminal receives the data distributed from the server and displays it on the screen as a scoreboard and commentary on the play.

[1187] The display is updated in real time, allowing spectators to instantly understand the situation in the game.

[1188] These are the specific processing steps of the "Digital Tournament for Basketball" system program, which reduces the burden on Tournament Officers and allows spectators to enjoy the details of the game in real time.

[1189] Example 1

[1190] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1191] In the current match management system, a large amount of manpower is required for the duties of table officials (TOs), which means that real-time updates of scores and prompt commentary on plays cannot always be provided. As a result, it is difficult for spectators and officials to grasp the progress of the match in real time, which can diminish the appeal of the match. Furthermore, incorrect score entries and time lags due to human error are also issues. It is necessary to provide a system that can solve these problems and improve the efficiency and sophistication of match management.

[1192] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1193] In this invention, the server includes means for acquiring video data of a game in real time from multiple camera devices, means for breaking down the acquired video data into frames and identifying the positions of players and the ball using a machine learning model, means for tracking the movements of players and the ball between consecutive frames using a tracking algorithm, means for identifying specific events from the tracking data and updating the score in real time, means for creating play commentary based on the identified event data using artificial intelligence, means for converting the updated score and the generated play commentary into a data format for display and distributing them in real time, and means for receiving the distributed data and displaying it on a screen as a score display device and play commentary. This enables the automation of TO operations and realizes the provision of accurate score updates and play commentary in real time.

[1194] "Multiple camera devices" refers to multiple camera devices that capture video data of the game in real time from different angles and positions.

[1195] "Video data" refers to digital video data showing the progress of a match captured by multiple camera devices.

[1196] "Machine learning model" refers to an artificial intelligence model used to analyze captured video data and identify the positions of players and the ball.

[1197] A "tracking algorithm" refers to a mathematical technique or calculation method for tracking the movement of players and the ball between successive frames.

[1198] "Tracking Data" refers to data containing player and ball position information obtained between successive frames by a tracking algorithm.

[1199] "Specific event" refers to a significant occurrence during a game (e.g., a successful shot, a foul, a timeout, etc.).

[1200] "Generative Artificial Intelligence" refers to a generative AI model used to generate natural language play commentary based on data.

[1201] "Points" refers to the number of points a team scores during a match.

[1202] "Score display device" refers to a device (e.g., a digital scoreboard or terminal display) that visually displays updated scores and commentary to spectators and other involved parties.

[1203] "Real-time" refers to processing or updating information occurring almost immediately or with very little delay.

[1204] This invention is a system that digitizes and automates the work of table officials (TOs) in sports matches by acquiring game video data in real time from multiple camera devices and analyzing the movements of players and the ball. Each component of this system works together, reflecting the game situation in real time in the score and automatically generating commentary on plays.

[1205] Key Components of the System

[1206] 1. Video data acquisition unit:

[1207] The server acquires real-time video data of the match from multiple cameras (e.g., general-purpose digital cameras) installed in the venue. Each camera captures the match from a different angle or position, and transmits the video data to the server via a network.

[1208] 2. Image analysis unit:

[1209] The server breaks down the captured video data into frames using libraries such as OpenCV. It then applies an object detection model (e.g., YOLOv3 or SSD model) trained using machine learning libraries such as TensorFlow or PyTorch to identify the positions of players and the ball within each frame. The detected position information is recorded as coordinate data.

[1210] 3. Tracking section:

[1211] The server uses tracking algorithms such as Kalman filters and SORT to track the movement of players and the ball between successive frames, recording this data as time series data and analyzing patterns of player and ball movement.

[1212] 4. Event Identification:

[1213] The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts), specifically when the ball passes through the hoop, which is identified as a scoring event. Identified events are recorded in a database.

[1214] 5. Score Update Section:

[1215] The server updates the team's score in real time based on the identified events. For example, if a shot is made, a point is added to the corresponding team's score. This information is managed as scoreboard data.

[1216] 6. Play commentary generation unit:

[1217] The server generates play commentary in natural language based on the identified event data using generative AI (e.g., GPT-3). For example, it generates commentary such as, "Player A made a great three-point shot!"

[1218] 7. Data Distribution Department:

[1219] The server converts the updated scores and generated commentary into a displayable data format such as JSON and distributes it in real time via WebSocket or REST API, making the latest information instantly available.

[1220] 8. Display terminal:

[1221] Devices (e.g., tablets and smartphones) and digital scoreboards receive the data distributed from the server. The devices display the received data on the screen in real time as a scoreboard and play commentary. Appropriate UI / UX design is used to improve visibility.

[1222] Specific examples

[1223] For example, if Player A makes a three-point shot during a game, the system will behave as follows:

[1224] 1. Acquiring video data:

[1225] The server collects video data in real time from multiple cameras in the venue, each capturing footage of the match from a different angle, and transmits the data to the server via a network.

[1226] 2. Image Analysis:

[1227] The server breaks down the video data into frames using OpenCV or similar tools, and applies an object detection model (such as YOLOv3) to identify the positions of players and the ball.

[1228] 3. Tracking:

[1229] The server uses a tracking algorithm (such as a Kalman filter) to track the ball's movement between successive frames and records the trajectory of its movement.

[1230] 4. Event Identification:

[1231] The server analyzes the tracking data and detects when the ball passes through the hoop, identifying this as a successful shot event.

[1232] 5. Score Update:

[1233] The server will award the successful shot three points to Team A's score and record this information in the database.

[1234] 6. Play commentary generation:

[1235] Using generative AI (GPT-3), we generate commentary on the play, such as "Player A made a great three-point shot!"

[1236] 7. Data Distribution:

[1237] The server converts the updated scores and generated play commentary into JSON format and delivers them to the device in real time via WebSocket.

[1238] 8. Receiving and Displaying Data:

[1239] The device receives the data and displays the latest score and commentary on the game in real time on the screen.

[1240] Prompt Sentence Examples

[1241] An example of an input prompt for the generation AI is, "Please generate commentary if player A makes a successful three-point shot."

[1242] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1243] Step 1: Obtaining video data

[1244] The server acquires real-time video data of the match from multiple camera devices. Specifically, multiple camera devices capture the match from different angles and transmit the video data to the server via a network. The input is the video data captured by each camera device, and the output is multiple real-time video streams stored on the server.

[1245] Step 2: Image analysis

[1246] The server breaks down the captured video data into frames. To do this, it uses libraries such as OpenCV to convert the video data into still images. Next, it applies an object detection model (e.g., YOLOv3 or SSD) trained using machine learning libraries such as TensorFlow or PyTorch to identify the positions of players and the ball within each frame. The input is the captured video data (frames), and the output is the position information (coordinate data) of players and the ball for each frame.

[1247] Step 3: Tracking

[1248] The server uses tracking algorithms such as Kalman filter and SORT to track the movements of players and the ball between consecutive frames. This allows the position information of players and the ball to be recorded as time-series data. The input is consecutive frames of data containing position information, and the output is time-series position information as tracking data.

[1249] Step 4: Event Identification

[1250] The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts). In particular, if the ball's trajectory passes through the hoop, it is identified as a successful shot. The input is the tracking data, and the output is the identified event information.

[1251] Step 5: Update your score

[1252] The server updates the team's score in real time based on the identified event information. For example, when a shot is made, a point is added to the corresponding team's score. This information is managed as scoreboard data. The input is the identified event information, and the output is updated score information.

[1253] Step 6: Generate play instructions

[1254] The server generates commentary based on the identified event data using a generation AI (e.g., GPT-3). To do this, a prompt is prepared for the generation AI and given as input. For example, the prompt might be, "Please generate commentary if player A makes a successful three-point shot." The input is the identified event data and the prompt, and the output is the generated commentary text.

[1255] Step 7: Data Distribution

[1256] The server converts the updated score and generated commentary into a displayable data format such as JSON and distributes it in real time via WebSocket or REST API. The input is the updated score information and the generated commentary text, and the output is the distributed JSON data.

[1257] Step 8: Receive and display data

[1258] The terminal receives data delivered from the server and displays it on the screen as a scoreboard and commentary in real time. For this purpose, a web application or a native application is used. The input is the delivered JSON data, and the output is the latest score and commentary displayed on the screen.

[1259] (Application example 1)

[1260] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1261] It is difficult for table officials (TOs) to perform their duties quickly and accurately during a basketball game. Furthermore, there is a lack of means to accurately communicate the real-time status of the game to spectators. Therefore, there is a need for a system that can reduce the burden on TOs and provide spectators with real-time information on the game's status.

[1262] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1263] In this invention, the server includes means for acquiring video data in real time from multiple camera devices, means for breaking down the acquired video data into frames and identifying the positions of the target and the sphere using an object detection model, means for tracking the movement of the target and the sphere between consecutive frames using a tracking algorithm, means for identifying specific events from the tracking data and updating scores in real time, means for creating action descriptions using a generation AI based on the identified event data, means for converting the updated scores and generated action descriptions into a data format for display and distributing them in real time, and means for receiving the distributed data and displaying them as a score board and action descriptions on a virtual space display device. This reduces the burden on TOs and makes it possible to accurately convey the situation of the game to spectators in real time.

[1264] A "camera" is a device for acquiring video data, and is responsible for capturing video of the game in real time from multiple angles and positions.

[1265] "Video data" refers to visual information of the match obtained from a camera, and is data that shows the progress of the match in real time.

[1266] An "object detection model" is an analytical algorithm or technology that analyzes acquired video data to identify the position of a specific object (such as a player or a ball).

[1267] "Target" refers to a person or object that is being tracked and identified, such as a player in a match.

[1268] The term "sphere" refers to a sports ball used in a game, and in the present invention refers to a basketball.

[1269] A "frame" is an individual still image that makes up video data, and a moving image is formed by successive frames.

[1270] A "tracking algorithm" is a technique used to track the movement of an object or sphere between successive frames and record its position information as time-series data.

[1271] "Specific events" refer to important events that occur during a game (such as a successful shot or a foul) that the system automatically identifies.

[1272] "Generative AI" is an artificial intelligence technology that automatically generates natural language descriptions of behavior based on identified event data.

[1273] An "action explanation" is an explanatory text about a specific event during a match, created by the generation AI.

[1274] The "score board" is an interface that is displayed on the virtual space display device to visually provide the latest score information.

[1275] A "virtual space display device" is a device that uses virtual reality technology to allow users to visually experience a match in a virtual space, and generally includes a VR headset.

[1276] A system for implementing this invention comprises the following major components:

[1277] 1. Server

[1278] The server has a means to acquire video data in real time from multiple camera devices, which are installed at different angles and positions within the venue to capture visual information of the match.

[1279] The server has a means to break down the acquired video data into frames and identify the positions of the objects and spheres using an object detection model, which is implemented using a library such as OpenCV.

[1280] The server has a means to track the movement of the object and the ball using a tracking algorithm between successive frames, allowing the position data of the object (player) and the ball (basketball) to be recorded over time.

[1281] The server has the means to identify specific events from the tracking data and update the score in real time. For example, when a ball passes through a hoop, it is identified as a scoring event and the score is automatically updated.

[1282] The server has a means to generate behavioral explanations based on the identified event data using a generative AI model, such as GPT-3.

[1283] The server has a means for converting the updated scores and generated action descriptions into a data format for display and distributing them in real time, thereby enabling the virtual space display device to quickly provide the latest match information.

[1284] 2. Terminal (Virtual Space Display Device)

[1285] The device receives data from the server and displays it in the virtual space as a scoreboard and action explanations, allowing users to visually grasp the progress of the game in real time. A VR headset or similar device is used as the virtual space display device.

[1286] 3. Users

[1287] By wearing the virtual space display device, users can watch the game in a virtual space. By referring to the score and action explanations updated in real time, users can intuitively understand the situation of the game.

[1288] Specific examples

[1289] For example, if a ball (basketball) passes through the hoop during a game and a three-point shot is successful, the process is as follows:

[1290] 1. Server

[1291] Video data acquisition: Video data is acquired in real time from the imaging device.

[1292] Image analysis: Video data is broken down into frames and object detection models are applied to identify the positions of objects (players) and balls (basketballs).

[1293] Tracking: The movement of the object and the sphere is tracked using a tracking algorithm between successive frames.

[1294] Event Identification: The tracking data is analyzed and when the sphere passes through the ring it is identified as a scoring event.

[1295] Score Update: Based on the identified incident, add 3 points to the team's score.

[1296] Play commentary generation: Using a generation AI, we generate an action explanation such as "Player A made a great three-point shot!"

[1297] Data distribution: Updated scores and commentary are converted into a displayable data format and distributed in real time.

[1298] 2. Terminal (Virtual Space Display Device)

[1299] Display of received data: Receive the distributed data and display it in the virtual space as a score board and action explanation.

[1300] Prompt Sentence Examples

[1301] "When a three-point shot is successful, generate a score and commentary of the play and display them in the virtual space."

[1302] The above steps reduce the burden on TOs and enable them to provide users with accurate information on the progress of the match in real time.

[1303] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1304] Step 1:

[1305] The server receives video data from the camera in real time. Each camera captures the game from a different angle or position and sends the video data to the server. The server receives the data and starts processing. The input is real-time video data, and the output is the video data.

[1306] Step 2:

[1307] The server uses the OpenCV library to decompose the video data into frames, which converts it into a sequence of still images (frames). The input is the video data acquired in step 1, and the output is the individual frames.

[1308] Step 3:

[1309] The server applies an object detection model to each decomposed frame to identify the positions of the objects (players) and the ball (basketball). This uses a machine learning object detection algorithm. The input is the image data for each frame, and the output is the position information of the objects and the ball within each frame.

[1310] Step 4:

[1311] The server uses a tracking algorithm to track the movement of the object and the sphere between successive frames, generating position information as time-series data. The input is the position information within each frame, and the output is time-series position data of the object and the sphere.

[1312] Step 5:

[1313] The server analyzes the tracking data and identifies specific events, such as a ball passing through a ring, as a scoring event. This is achieved using an event identification algorithm, whose input is the time-series position data and whose output is the identified specific event.

[1314] Step 6:

[1315] The server updates the scores in real time based on the identified events. Each scoring event automatically increments the team's score. The input is the identified event, and the output is the updated score.

[1316] Step 7:

[1317] The server uses a generative AI to create a behavioral explanation based on the identified event data. For example, it generates an explanation in the form of "Player A made a great three-point shot!" The input is the identified event data, and the output is the generated behavioral explanation.

[1318] Step 8:

[1319] The server converts the updated scores and generated behavior descriptions into a data format for display and distributes them in real time using data streaming technology. The input is the updated scores and generated behavior descriptions, and the output is data that can be distributed.

[1320] Step 9:

[1321] The terminal (virtual space display device) receives the data distributed from the server and displays it in the virtual space as a score board and action explanations. This allows the user to visually grasp the progress of the game in real time. The input is the distributed data, and the output is the display in the virtual space.

[1322] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1323] This invention is a system that digitizes and automates the work of table officials (TOs) in basketball games, and combines it with an emotion engine that recognizes user emotions to provide a more interactive and immersive game-watching experience.

[1324] System configuration

[1325] The system of the present invention consists of the following main components:

[1326] 1. Video Data Acquisition Unit: The server acquires video data of the match in real time from multiple cameras installed in the venue. This data is used to analyze the movements of players and the ball.

[1327] 2. Image analysis unit: The server breaks down the acquired video data into frames and identifies the positions of players and the ball using an object detection model.

[1328] 3. Tracking: The server uses a tracking algorithm to track the movements of players and the ball between successive frames and records these movements as time-series data.

[1329] 4. Event Identification: The server analyzes the tracking data and identifies specific events such as successful shots, fouls, and timeouts.

[1330] 5. Score Updater: The server updates the score in real time based on the identified events.

[1331] 6. Play commentary generation unit: The server uses a generation AI to create a play commentary in natural language based on the identified event data.

[1332] 7. Emotion engine: The server includes an emotion engine that acquires the user's face and voice data and recognizes their emotions.

[1333] 8. Data distribution unit: The server converts the updated scores and generated play commentary into a data format for display and distributes them in real time.

[1334] 9. Display terminal: The terminal receives the distributed data and displays it on the screen as a scoreboard and play commentary.

[1335] Program processing flow (explained in natural language)

[1336] Server: Acquire video data

[1337] The server collects real-time video data from multiple cameras installed in the venue, each capturing footage of the match from a different angle or position, and streams the footage in chronological order.

[1338] Server: Image analysis

[1339] The server breaks down the acquired video data into frames and applies an object detection model to identify the positions of players and the ball, thereby obtaining positional information for players and the ball within each frame.

[1340] Server:Tracking

[1341] The server uses a tracking algorithm to track the movement of players and the ball between successive frames, recording this data as time series of player and ball positions for use in motion analysis.

[1342] Server:Event Identification

[1343] The server identifies specific events (e.g., successful shots, fouls, timeouts) from the tracking data. For example, if it detects that the ball has passed through the hoop, it identifies that as a score event.

[1344] Server:Score Update

[1345] The server updates the team's score in real time based on the identified events, for example, adding a point to the team's score when a made shot event is identified.

[1346] Server: Play commentary generation

[1347] The server generates play commentary in natural language based on the event data identified using the AI ​​generation technology, which provides viewers with easy-to-understand details about the match.

[1348] Server: Emotion recognition

[1349] The server acquires the user's facial and voice data and uses an emotion engine to recognize their emotions. For example, by analyzing facial expression data and tone of voice acquired through a camera or microphone, it can determine whether the user is in an emotional state such as "joy," "excitement," or "dissatisfaction."

[1350] Server: Adjustment of gameplay commentary

[1351] The server dynamically adjusts the content and display of the commentary based on the user's emotions as recognized by the emotion engine. For example, if the user is excited, the server can display more detailed and enthusiastic commentary.

[1352] Server: Real-time analysis of audience sentiment

[1353] The server aggregates the emotional data of multiple users and analyzes the emotional state of the entire audience in real time. This data may be reflected in the progress and direction of the match.

[1354] Server: Prepare data distribution

[1355] The server converts the updated scores and generated commentary into a data format for display, such as JSON or HTML.

[1356] Server: Data distribution

[1357] The server then transmits the converted data to the screens at the venue and to the audience's devices in real time using a real-time communication protocol such as WebSocket.

[1358] Terminal: Receiving and displaying data

[1359] The device receives data from the server and displays it on the screen as a scoreboard and commentary on the game, which is updated in real time, allowing spectators to instantly understand the situation in the game.

[1360] Specific examples

[1361] For example, if a player makes a three-point shot during a game, the process is as follows:

[1362] 1. Acquisition of video data (server): The server acquires video data in real time from multiple cameras installed within the venue.

[1363] 2. Image analysis (server): The server breaks down the captured video data into frames and applies object detection models to identify the positions of players and the ball.

[1364] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[1365] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[1366] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[1367] 6. Play commentary generation (server): The server uses a generation AI to generate a play commentary such as, "Player X made a great three-point shot!"

[1368] 7. Emotion recognition (server): The server acquires the user's facial and voice data and uses the emotion engine to recognize when the user is excited.

[1369] 8. Adjusting play commentary (server): The server will reflect the user's excitement and adjust the commentary to be more detailed and energetic.

[1370] 9. Data distribution preparation (server): The server converts the updated scores and adjusted play commentary into a data format for display.

[1371] 10. Data distribution (server): The server distributes data in real time to the screen and audience devices.

[1372] 11. Receiving and displaying data (terminal): The terminal receives the distributed data and displays the scoreboard and play commentary in real time.

[1373] This will significantly reduce the burden on TOs and allow audiences to have a more immersive experience by receiving individually tailored commentary through emotion recognition.

[1374] The processing flow will be explained below.

[1375] Program processing steps

[1376] Step 1: Obtaining video data

[1377] server:

[1378] The server collects real-time video data from multiple cameras installed in the venue, each capturing footage from a different angle, and streams it in chronological order.

[1379] Step 2: Frame Decomposition

[1380] server:

[1381] The server breaks down the acquired video data into frames and processes each frame in parallel for the video data from each camera.

[1382] Step 3: Object detection

[1383] server:

[1384] The server uses object detection models such as YOLO (You Only Look Once) to identify players and the ball in each frame.

[1385] The detection model outputs the position coordinates and labels of players and the ball (Player 1, Player 2, Ball, etc.).

[1386] Step 4: Tracking

[1387] server:

[1388] The server uses a tracking algorithm to track the movement of players and the ball between successive frames.

[1389] The algorithm analyzes the position and movement vectors of each object and records them as time series data.

[1390] Step 5: Event Identification

[1391] server:

[1392] The server identifies specific events (e.g., successful shots, fouls, timeouts) based on the tracking data.

[1393] For example, if the ball passes over the hoop, this is identified as a "scoring event."

[1394] Step 6: Update your score

[1395] server:

[1396] The server updates the team scores in real time based on the identified events.

[1397] For example, if the shot is successful, the team will receive points corresponding to their score (e.g., 2 points, 3 points).

[1398] Step 7: Generate play instructions

[1399] server:

[1400] The server generates a play commentary in natural language based on the event data identified using the generation AI.

[1401] For example, it generates a sentence such as, "Player X broke through the defense with a sharp dribble and scored a stunning layup!"

[1402] Step 8: Emotion Recognition

[1403] server:

[1404] The server acquires the user's facial and voice data and recognizes their emotions using an emotion engine.

[1405] For example, facial expression data and tone of voice obtained from a camera or microphone are analyzed to determine whether the user is in an emotional state such as "joy," "excitement," or "dissatisfaction."

[1406] Step 9: Adjust the commentary

[1407] server:

[1408] The server dynamically adjusts the content and display method of the play commentary based on the user's emotions recognized by the emotion engine.

[1409] For example, if the user is excited, a more detailed and energetic commentary will be displayed.

[1410] Step 10: Real-time analysis of audience sentiment

[1411] server:

[1412] The server aggregates the emotional data of multiple users and analyzes the emotional state of the entire audience in real time.

[1413] This data may be reflected in the progress and presentation of the match.

[1414] Step 11: Prepare for data distribution

[1415] server:

[1416] The server converts the updated scores and generated commentary into a data format for display, such as JSON or HTML.

[1417] Step 12: Data Distribution

[1418] server:

[1419] The server distributes the converted data in real time to the screens at the venue and to the audience's devices.

[1420] Data is delivered using real-time communication protocols such as WebSocket.

[1421] Step 13: Receiving and displaying data

[1422] Device:

[1423] The terminal receives the data distributed from the server and displays it on the screen as a scoreboard and commentary on the game.

[1424] The display is updated in real time, allowing spectators to instantly understand the situation in the game.

[1425] Specific examples

[1426] For example, if a player makes a three-point shot during a game, the process is as follows:

[1427] 1. Acquisition of video data (server): The server acquires video data in real time from multiple cameras installed within the venue.

[1428] 2. Image analysis (server): The server breaks down the captured video data into frames and applies object detection models to identify the positions of players and the ball.

[1429] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[1430] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[1431] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[1432] 6. Play commentary generation (server): The server uses a generation AI to generate a play commentary such as, "Player X made a great three-point shot!"

[1433] 7. Emotion recognition (server): The server acquires the user's facial and voice data and uses the emotion engine to recognize when the user is excited.

[1434] 8. Adjusting play commentary (server): The server will reflect the user's excitement and adjust the commentary to be more detailed and energetic.

[1435] 9. Data distribution preparation (server): The server converts the updated scores and adjusted play commentary into a data format for display.

[1436] 10. Data distribution (server): The server distributes data in real time to the screen and audience devices.

[1437] 11. Receiving and displaying data (terminal): The terminal receives the distributed data and displays the scoreboard and play commentary in real time.

[1438] This will significantly reduce the burden on TOs and allow audiences to have a more immersive experience by receiving individually tailored commentary through emotion recognition.

[1439] Example 2

[1440] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1441] At sports games such as basketball, the goal is to improve the efficiency and accuracy of table officials (TOs) by digitizing and automating their work. There is also a need for a system that can provide real-time commentary and information on the game to spectators, creating a more immersive and realistic experience. However, existing systems face challenges in accurately grasping the game situation and updating and distributing necessary information in real time. Furthermore, there are currently no systems that can recognize and reflect spectator emotions and provide personalized commentary.

[1442] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1443] In this invention, the server includes a means for acquiring image data of the game in real time from multiple camera devices, a means for dividing the acquired image data into successive images and using an object detection algorithm to identify the positions of players and the ball, and a means for tracking the movement of players and the ball between successive images using a tracking algorithm. This enables accurate understanding of the game situation, real-time score updates, and the creation and distribution of game commentary using generative AI. Furthermore, the server can acquire user facial and voice data, analyze their emotional state using an emotion engine, and dynamically adjust the content and display method of the game commentary based on that data, providing spectators with a more realistic and immersive game viewing experience.

[1444] "Filming equipment" refers to video cameras installed within the venue to capture the state of the match.

[1445] "Image data" refers to video data of a match captured by a camera.

[1446] "Continuous images" refers to a set of still images obtained by breaking down video data into frames.

[1447] An "object detection algorithm" refers to a machine learning model for identifying specific objects (players or balls) within image data.

[1448] "Player" means a player taking part in a match.

[1449] "Sphere" refers to the ball used during a match.

[1450] A "tracking algorithm" refers to a computational method for tracking the position of a particular object (a player or a ball) between successive image frames.

[1451] "Specific occurrences" refer to significant events that occur during a game, such as a successful shot, a foul, or a timeout.

[1452] "Score" refers to points awarded to a team based on a specific event.

[1453] "Generative algorithm" refers to AI technology that generates natural language commentary on sports events based on given data.

[1454] "Sports commentary" refers to natural language text that describes the progress of a match or an event.

[1455] "Display data format" refers to a data format (e.g., JSON, HTML) suitable for displaying scores and commentary on a display device.

[1456] An "emotion engine" refers to software or algorithms that analyze a user's facial expressions and voice data to recognize their emotional state.

[1457] "Users" refers to spectators who attend matches.

[1458] "Emotional state" refers to the user's psychological state, such as joy, excitement, or dissatisfaction.

[1459] "Dynamic adjustment" refers to changing the content and display format in real time depending on the situation.

[1460] This invention aims to digitize and automate the work of table officials (TOs) at sports games, and is a system that combines it with an emotion engine that recognizes user emotions to provide a more interactive and immersive game-watching experience.

[1461] System configuration

[1462] Server: Acquire video data

[1463] The server acquires image data of the match in real time from multiple camera devices installed in the venue. Each camera captures the match from a different angle or position and streams the data. Specific examples of such cameras include high-resolution digital cameras and IP cameras.

[1464] Server: Image analysis

[1465] The server divides the acquired image data into successive images and applies an object detection algorithm to identify the positions of players and the ball. The object detection algorithm uses machine learning models such as YOLO (You Only Look Once) and Mask R-CNN. This allows the server to obtain the positional information of players and the ball within each frame.

[1466] Server:Tracking

[1467] The server tracks the movements of the players and the ball between successive image frames using a tracking algorithm, such as a Kalman filter and the tracking algorithm SORT (Simple Online and Realtime Tracking). This process stores the positional information of the players and the ball as time-series data.

[1468] Server:Event Identification

[1469] The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts). For example, a ball passing through the hoop is captured as a "successful shot." Event identification is achieved using statistical analysis and rule-based techniques.

[1470] Server:Score Update

[1471] The server updates the team's score in real time based on the identified incidents, and points are automatically added to the team's score for successful shots, for example, three points for a successful three-point shot.

[1472] Server: Play commentary generation

[1473] The server uses a generative algorithm to generate a natural language commentary of the game based on the identified event data. Generative AI models such as GPT-4 are used. For example, a commentary such as "Player X made a successful three-point shot!" is generated.

[1474] Server: Emotion recognition

[1475] The server collects the user's facial and voice data and analyzes their emotional state using an emotion engine, which uses tools such as Affectiva and the Microsoft Azure Emotion API. This allows the server to determine in real time whether the user is in a state of joy, excitement, dissatisfaction, or other emotional state.

[1476] Server: Adjustment of gameplay commentary

[1477] The server dynamically adjusts the content and display method of the generated commentary based on the user's emotions determined by the emotion engine. For example, if the user is excited, the generated commentary will be changed to be more detailed and passionate.

[1478] Server: Real-time analysis of audience sentiment

[1479] The server aggregates emotional data collected from multiple users and analyzes the emotional state of the entire venue in real time. This data is reflected in the progress and direction of the match.

[1480] Server: Prepare data distribution

[1481] The server converts the updated scores and generated commentary into a displayable data format (e.g., JSON or HTML), which facilitates distribution.

[1482] Server: Data distribution

[1483] The server then distributes the converted data in real time to the screens at the venue and to the audience's devices, using a real-time communication protocol such as WebSocket.

[1484] Terminal: Receiving and displaying data

[1485] The terminals receive the data distributed from the server and display it on the screen as scores and commentary on the match. The terminals can be smartphones, tablets, digital signage, etc. The display is updated in real time, allowing spectators to instantly understand the situation of the match.

[1486] Specific examples

[1487] For example, if a player makes a three-point shot during a game, the process is as follows:

[1488] 1. Acquisition of video data (server): The server acquires video data in real time from multiple camera devices installed within the venue.

[1489] 2. Image analysis (server): The server decomposes the captured video data and applies object detection algorithms to identify the positions of players and the ball.

[1490] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[1491] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[1492] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[1493] 6. Play commentary generation (server): The server uses a generation AI to generate commentary on the game, such as "Player X made a successful three-point shot!"

[1494] 7. Emotion recognition (server): The server acquires data on the user's facial expressions and voice, and uses the emotion engine to recognize when the user is excited.

[1495] 8. Adjusting play commentary (server): The server will reflect the user's excitement and adjust the commentary to be more detailed and energetic.

[1496] 9. Data distribution preparation (server): The server converts the updated scores and adjusted game commentary into a data format for display.

[1497] 10. Data distribution (server): The server distributes data in real time to the screen and audience devices.

[1498] 11. Data reception and display (terminal): The terminal receives the distributed data and displays the score and commentary in real time, allowing spectators to instantly grasp the situation of the match, significantly reducing the burden on the TO. Spectators can also receive individually tailored commentary through emotion recognition, providing a more immersive experience.

[1499] Prompt Sentence Examples

[1500] The following prompt sentence is used for the generative AI model to automatically generate an explanation such as "Player X made a successful three-point shot!"

[1501] Prompt statement:

[1502] "With the crowd going wild, player X made a brilliant three-point shot!"

[1503] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1504] Step 1:

[1505] Input: Real-time video data from multiple imaging modalities

[1506] Specific operation: The server acquires video data of the match in real time from multiple camera devices installed in the venue. Each camera captures the match from a different angle or position and streams the video data. Specifically, the data is transmitted in real time using the RTSP protocol.

[1507] Output: Real-time video data of the match

[1508] Step 2:

[1509] Input: Real-time video data of the match

[1510] Specific operation: The server divides the acquired video data into successive image frames. Specifically, it uses a tool such as ffmpeg to break the video down into 30 frames per second.

[1511] Output: Sequential image frames

[1512] Step 3:

[1513] Input: Sequential image frames

[1514] How it works: The server applies an object detection algorithm to identify the positions of players and spheres in each frame. Specifically, it uses models such as YOLO (You Only Look Once) and Mask R-CNN to output the coordinates of players and spheres in each frame.

[1515] Output: Position information of the player and the ball in each frame

[1516] Step 4:

[1517] Input: Position information of the player and the ball in each frame

[1518] How it works: The server tracks the movement of players and the ball using a tracking algorithm between consecutive frames, specifically using a Kalman filter and the SORT (Simple Online and Realtime Tracking) algorithm to record position information as time-series data.

[1519] Output: Time series position data of players and balls

[1520] Step 5:

[1521] Input: Time series position data of players and balls

[1522] Specific operation: The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts). Specifically, it checks whether the ball has passed through the hoop, and if it has, identifies it as a "successful shot."

[1523] Output: Identified event data

[1524] Step 6:

[1525] Input: Identified event data

[1526] Specific behavior: The server updates the team's score in real time based on the identified events. Specifically, for successful shots, the server automatically adds 2 or 3 points to the team's score.

[1527] Output: Updated score data

[1528] Step 7:

[1529] Input: Identified event data, updated score data

[1530] Specific behavior: The server uses a generative AI model (e.g., GPT-4) to generate natural language commentary on the game based on the identified event data. An example of a specific prompt sentence is, "Player X made a three-point shot!"

[1531] Output: Generated commentary

[1532] Step 8:

[1533] Input: User's facial expression data, voice data

[1534] Specific operation: The server acquires the user's facial expression and voice data and analyzes their emotional state using an emotion engine (e.g., Affectiva, Microsoft Azure Emotion API). This determines whether the user is in an emotional state such as "joy," "excitement," or "dissatisfaction."

[1535] Output: Analyzed sentiment data

[1536] Step 9:

[1537] Input: Analyzed emotion data, generated commentary

[1538] Specific behavior: The server dynamically adjusts the content and presentation of the generated commentary based on the user's emotional state. For example, if the user is excited, the server displays more detailed and energetic commentary.

[1539] Output: Coordinated competition commentary

[1540] Step 10:

[1541] Input: Adjusted game commentary, updated scoring data

[1542] Specific operation: The server converts the adjusted competition commentary and score data into a data format for display (e.g., JSON, HTML).

[1543] Output: Competition commentary and score data converted into a displayable data format

[1544] Step 11:

[1545] Input: Competition commentary and score data converted into a display format

[1546] How it works: The server distributes the converted data in real time to the screens in the venue and to the audience's devices, using a real-time communication protocol such as WebSocket.

[1547] Output: Real-time broadcast of competition commentary and score data

[1548] Step 12:

[1549] Input: Real-time game commentary and score data

[1550] Specific operation: The terminal receives data distributed from the server and displays it on the screen as score and commentary. The display is updated in real time, allowing spectators to instantly understand the situation of the game. Specifically, smartphones, tablets, digital signage, etc. are used.

[1551] Output: On-screen commentary and score information

[1552] (Application example 2)

[1553] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1554] Traditionally, watching basketball games required manual work by table officials (TOs), which placed a heavy burden on spectators and sometimes caused delays in real-time game progress. Furthermore, there was no way to provide an interactive experience that responded to spectators' emotions, limiting the quality of the viewing experience. These issues can be resolved using new technology.

[1555] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1556] In this invention, the server includes means for acquiring game video data from multiple cameras in real time, means for breaking down the acquired video data into frames and identifying the positions of objects and the ball using an object detection model, means for tracking the movement of the object and the ball between consecutive frames using a tracking algorithm, means for identifying specific events from the tracking data and updating the score in real time, means for creating play commentary based on the identified event data using a generation AI, means for converting the updated score and the generated play commentary into a display data format and distributing them in real time, means for receiving the distributed data and displaying it as a scoreboard and play commentary on a screen, means for acquiring user facial and voice data and recognizing the user's emotions using an emotion engine, and means for dynamically adjusting the content and display method of the play commentary based on the recognized emotion data. This improves the quality of game viewing and makes it possible to provide an interactive experience that corresponds to the user's emotions.

[1557] "Multiple cameras" means two or more video recording devices used to capture the action of a match in real time from different angles and positions.

[1558] "Video data" refers to digital data that is captured in real time by a camera and used for analysis and storage, either as is or after processing.

[1559] "Frame decomposition" refers to the process of dividing continuous video data into individual still images (frames).

[1560] An "object detection model" is an artificial intelligence model trained to identify specific objects (e.g., players or balls) in video or images.

[1561] A "tracking algorithm" is a computational method or process for continuously tracking the movement of an identified object between frames of video.

[1562] A "specific occurrence" is a significant action or event that occurs during a match (e.g., a goal, a foul).

[1563] "Generative AI" is a technology that uses artificial intelligence to automatically generate natural language explanations and sentences from input data.

[1564] "Play commentary" refers to written or audio data that explains specific events during a match in a way that is easy for viewers to understand.

[1565] A "data format" is a recording format or protocol for storing and communicating data (e.g., JSON, HTML).

[1566] An "emotion engine" is a technology or model that analyzes a user's facial expressions and voice to recognize their emotional state (e.g., joy, excitement, dissatisfaction).

[1567] The system of the present invention includes multiple cameras, a server, a user terminal, an emotion engine, etc. to provide an interactive and immersive experience in watching a basketball game.

[1568] The server collects video data of the match in real time from multiple cameras installed in the venue. This allows for time-series streaming of footage shot from different angles and positions. The video data is broken down into frames, and the positions of players and the ball are identified using an object detection model powered by TensorFlow. At this stage, positional information within each frame is obtained.

[1569] The server uses a SORT algorithm to track the movements of detected players and the ball between consecutive frames, recording the movements of each object during the match as time-series data for later analysis.

[1570] The server then analyzes the tracking data and identifies specific occurrences (e.g., goals, fouls, timeouts). For example, when the ball passes through the hoop, it detects that as a scoring event and updates the score in real time.

[1571] Based on the identified event data, the server uses a generative AI model to generate commentary in natural language, providing viewers with a clear understanding of the game situation.

[1572] The emotion engine recognizes the user's emotional state by capturing and analyzing their facial and voice data. Based on the captured emotion data, the server dynamically adjusts the content and display of the gameplay commentary. For example, if the user is excited, the server will provide more detailed and energetic commentary.

[1573] The updated scores and generated commentary are converted into a displayable data format and delivered to the user's terminal in real time. The user's terminal receives the delivered data and displays the scoreboard and commentary on the screen in real time.

[1574] As a concrete example, the system behavior when a player makes a successful three-point shot during a game is shown below.

[1575] 1. Server: Acquire video data

[1576] The server collects video data in real time from multiple cameras within the venue.

[1577] 2. Server: Image analysis

[1578] The server breaks down the video data into frames and uses TensorFlow to identify the positions of players and the ball.

[1579] 3. Server:Tracking

[1580] The server tracks the ball's movement between successive frames using the SORT algorithm.

[1581] 4. Server: Event Identification

[1582] The server detects when the ball passes through the hoop and identifies it as a scoring event.

[1583] 5. Server: Update score

[1584] Add 3 points to the team's score.

[1585] 6. Server: Play commentary generation

[1586] The server uses a generative AI model to generate commentary such as, "Player X made a great three-point shot!"

[1587] 7. Server: Emotion Recognition

[1588] The server analyzes the user's face and voice and determines whether the user is excited.

[1589] 8. Server: Game commentary adjustment

[1590] Tailor your commentary to be more energetic based on sentiment data.

[1591] 9. Server: Data Distribution

[1592] Updated scores and adjusted play commentary are delivered to user devices in real time.

[1593] 10. Terminal: Receiving and Displaying

[1594] The user's device receives the distributed data and displays the scoreboard and play commentary in real time.

[1595] The following are examples of prompt sentences that can be used:

[1596] "Users get very excited when a three-point shot is made, so we want more detailed and exciting commentary. Please generate appropriate commentary based on the current game situation."

[1597] This provides a more interactive and interesting viewing experience that is tailored to the user's emotions.

[1598] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1599] Step 1:

[1600] Acquiring video data

[1601] The server collects real-time video data of the match from multiple cameras installed in the venue. This data consists of footage from different angles and positions. The data collected from each camera is streamed to the server and used as input data. This aggregates real-time video data of the entire match. The output is frame-by-frame video data.

[1602] Step 2:

[1603] Image analysis

[1604] The server breaks down the acquired video data into frames and runs an object detection model using TensorFlow on each frame. This extracts the positional information of players and the ball. The input is the video data frame by frame, and the output is the positional data of players and the ball for each frame. Specifically, the object detection model is applied to identify the coordinates of objects in each frame.

[1605] Step 3:

[1606] tracking

[1607] The server uses the SORT algorithm to track the movements of detected players and the ball between successive frames. This generates tracking data, recording the movement path of each object in chronological order. The input is position data for each frame, and the output is tracking data in chronological order. Specifically, the server maintains the continuity of the object's position between successive frames and analyzes its movement pattern.

[1608] Step 4:

[1609] Event Identification

[1610] The server analyzes the tracking data and identifies specific events (e.g., a goal, a foul, or a timeout). For example, if the ball passes through a hoop, it is recognized as a score event. The input is tracking data, and the output is event data. Specifically, it detects an event when a specific condition is met (e.g., the ball's coordinates pass through the hoop).

[1611] Step 5:

[1612] Score Update

[1613] The server updates the team's score in real time based on the recognized events. The input is the event data, and the output is the updated score. Specifically, it adds the score for the corresponding team based on the scoring event and modifies the scoreboard in real time.

[1614] Step 6:

[1615] Play commentary generation

[1616] The server uses a generative AI model to generate play commentary based on the identified event data. The input is the event data, and the output is the generated commentary. Specifically, the server inputs a prompt sentence into the generative AI model, which then generates a natural language commentary based on the prompt sentence.

[1617] Step 7:

[1618] emotion recognition

[1619] The server acquires the user's facial and voice data and uses an emotion engine to recognize their emotions. The input is the user's facial and voice data, and the output is emotional state data. Specifically, the server analyzes the data acquired through the camera and microphone and classifies the user's emotions into categories such as "joy," "excitement," and "dissatisfaction."

[1620] Step 8:

[1621] Game commentary adjustments

[1622] The server dynamically adjusts the content and display method of the play commentary based on the user's emotions recognized by the emotion engine. The input is emotional state data and the generated commentary, and the output is the adjusted commentary. Specifically, the commentary is changed according to the user's emotions, and if the user is excited, the commentary is adjusted to be more energetic, for example.

[1623] Step 9:

[1624] Data Distribution

[1625] The server converts the updated score and adjusted commentary into a data format for display and delivers it to the user's device in real time. The input is the adjusted commentary and updated score, and the output is the delivered data. Specifically, the server converts the data into JSON or HTML format and sends it using a communication protocol such as WebSocket.

[1626] Step 10:

[1627] Receiving and Displaying

[1628] The user device receives the data distributed from the server and displays it on the screen as a scoreboard and commentary on the play. The input is the distributed data, and the output is the displayed score and commentary. Specifically, the data is analyzed and displayed to the user in real time.

[1629] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1630] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1631] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1632] [Fourth embodiment]

[1633] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1634] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1635] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1636] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1637] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1638] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1639] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1640] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1641] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1642] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1643] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1644] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1645] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1646] This invention is a system that digitizes and automates the work of table officials (TOs) in basketball games by capturing video data of the game in real time from multiple cameras and analyzing the movements of the players and the ball. This system allows each component to interact with each other, reflecting the game situation in the score in real time and automatically generating commentary on plays.

[1647] System configuration

[1648] The system of the present invention consists of the following main components:

[1649] 1. Video Data Acquisition Unit: The server acquires video data in real time from multiple cameras installed in the venue. This data is used to analyze the movements of players and the ball.

[1650] 2. Image analysis unit: The server breaks down the acquired video data into frames and identifies the positions of players and the ball using an object detection model.

[1651] 3. Tracking: The server uses a tracking algorithm to track the movements of players and the ball between successive frames and records these movements as time-series data.

[1652] 4. Event Identification: The server analyzes the tracking data and identifies specific events such as successful shots, fouls, and timeouts.

[1653] 5. Score Updater: The server updates the score in real time based on the identified events.

[1654] 6. Play commentary generation unit: The server uses a generation AI to create a play commentary in natural language based on the identified event data.

[1655] 7. Data distribution unit: The server converts the updated scores and generated play commentary into a data format for display and distributes them in real time.

[1656] 8. Display terminal: The terminal receives the distributed data and displays it on the screen as a scoreboard and play commentary.

[1657] Program processing flow (explained in natural language)

[1658] Server: Acquire video data

[1659] The server collects real-time video data from multiple cameras installed in the venue, each capturing footage from a different angle or position, and streams it in chronological order.

[1660] Server: Image analysis

[1661] The server breaks down the acquired video data into frames and applies an object detection model to identify the positions of players and the ball, thereby obtaining positional information for players and the ball within each frame.

[1662] Server:Tracking

[1663] The server uses a tracking algorithm to track the movement of players and the ball between successive frames, recording this data as time series of player and ball positions for use in motion analysis.

[1664] Server:Event Identification

[1665] The server identifies specific events (e.g., successful shots, fouls, timeouts) from the tracking data. For example, if it detects that the ball has passed through the hoop, it identifies that as a score event.

[1666] Server:Score Update

[1667] The server updates the team's score in real time based on the identified events, for example, adding a point to the team's score when a made shot event is identified.

[1668] Server: Play commentary generation

[1669] The server uses the generated AI to generate natural language commentary based on the identified event data, providing viewers with easy-to-understand details about the match.

[1670] Server: Data distribution

[1671] The server converts the updated scores and generated commentary into a displayable data format and distributes them in real time, making the latest match information instantly available.

[1672] Terminal: Receiving and displaying data

[1673] The device receives data distributed from the server and displays it on the screen as a scoreboard and commentary on the game, allowing spectators to understand the situation in real time.

[1674] Specific examples

[1675] For example, if a player makes a three-point shot during a game, the process is as follows:

[1676] 1. Acquisition of video data (server): The server acquires video data in real time from multiple cameras installed in the spectator seats.

[1677] 2. Image analysis (server): The server breaks down the captured video data into frames and applies object detection models to identify the positions of players and the ball.

[1678] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[1679] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[1680] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[1681] 6. Play commentary generation (server): The server uses a generation AI to generate a play commentary such as, "Player X made a great three-point shot!"

[1682] 7. Data distribution (server): The server converts the updated scores and play commentary into a data format for display and distributes them in real time.

[1683] 8. Receiving and displaying data (terminal): The terminal receives the distributed data and displays the latest score and play commentary on the screen in real time.

[1684] This will significantly reduce the burden on TOs, and allow game officials and spectators to understand game details in real time, allowing them to enjoy basketball even more.

[1685] The processing flow will be explained below.

[1686] Program processing steps

[1687] Step 1: Obtaining video data

[1688] server:

[1689] The server receives real-time video data of the match from multiple cameras in the venue, each positioned to capture the match from a different angle.

[1690] Step 2: Frame Decomposition

[1691] server:

[1692] The server breaks down the acquired video data into frames and processes each frame in parallel for the video data from each camera.

[1693] Step 3: Object detection

[1694] server:

[1695] The server uses object detection models such as YOLO (You Only Look Once) to identify players and the ball in each frame.

[1696] The object detection model outputs the position coordinates and labels of the players and ball (Player 1, Player 2, Ball, etc.).

[1697] Step 4: Tracking

[1698] server:

[1699] The server tracks the movement of players and the ball between successive frames using tracking algorithms such as Kalman filters or optical flow.

[1700] The tracking algorithm analyzes the position and movement vector of each object and records them as time series data.

[1701] Step 5: Event Identification

[1702] server:

[1703] The server identifies specific events (e.g., successful shots, fouls, timeouts) based on the tracking data.

[1704] For example, if the ball passes over the hoop and then changes position, this is identified as a "scoring event."

[1705] Step 6: Update your score

[1706] server:

[1707] The server updates the team scores in real time based on the identified events.

[1708] For example, if the shot is successful, the team will receive points corresponding to their score (e.g., 2 points, 3 points).

[1709] Step 7: Generate play instructions

[1710] server:

[1711] The server generates a play commentary in natural language based on the event data identified using the generation AI.

[1712] For example, it generates a sentence such as, "Player X broke through the defense with a sharp dribble and scored a stunning layup!"

[1713] Step 8: Prepare for data distribution

[1714] server:

[1715] The server converts the updated scores and generated commentary into a data format for display, such as JSON or HTML.

[1716] Step 9: Data Distribution

[1717] server:

[1718] The server distributes the converted data in real time to the screens at the venue and to the audience's devices.

[1719] Data is delivered using real-time communication protocols such as WebSocket.

[1720] Step 10: Receive and display data

[1721] Device:

[1722] The terminal receives the data distributed from the server and displays it on the screen as a scoreboard and commentary on the play.

[1723] The display is updated in real time, allowing spectators to instantly understand the situation in the game.

[1724] These are the specific processing steps of the "Digital Tournament for Basketball" system program, which reduces the burden on Tournament Officers and allows spectators to enjoy the details of the game in real time.

[1725] Example 1

[1726] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1727] In the current match management system, a large amount of manpower is required for the duties of table officials (TOs), which means that real-time updates of scores and prompt commentary on plays cannot always be provided. As a result, it is difficult for spectators and officials to grasp the progress of the match in real time, which can diminish the appeal of the match. Furthermore, incorrect score entries and time lags due to human error are also issues. It is necessary to provide a system that can solve these problems and improve the efficiency and sophistication of match management.

[1728] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1729] In this invention, the server includes means for acquiring video data of a game in real time from multiple camera devices, means for breaking down the acquired video data into frames and identifying the positions of players and the ball using a machine learning model, means for tracking the movements of players and the ball between consecutive frames using a tracking algorithm, means for identifying specific events from the tracking data and updating the score in real time, means for creating play commentary based on the identified event data using artificial intelligence, means for converting the updated score and the generated play commentary into a data format for display and distributing them in real time, and means for receiving the distributed data and displaying it on a screen as a score display device and play commentary. This enables the automation of TO operations and realizes the provision of accurate score updates and play commentary in real time.

[1730] "Multiple camera devices" refers to multiple camera devices that capture video data of the game in real time from different angles and positions.

[1731] "Video data" refers to digital video data showing the progress of a match captured by multiple camera devices.

[1732] "Machine learning model" refers to an artificial intelligence model used to analyze captured video data and identify the positions of players and the ball.

[1733] A "tracking algorithm" refers to a mathematical technique or calculation method for tracking the movement of players and the ball between successive frames.

[1734] "Tracking Data" refers to data containing player and ball position information obtained between successive frames by a tracking algorithm.

[1735] "Specific event" refers to a significant occurrence during a game (e.g., a successful shot, a foul, a timeout, etc.).

[1736] "Generative Artificial Intelligence" refers to a generative AI model used to generate natural language play commentary based on data.

[1737] "Points" refers to the number of points a team scores during a match.

[1738] "Score display device" refers to a device (e.g., a digital scoreboard or terminal display) that visually displays updated scores and commentary to spectators and other involved parties.

[1739] "Real-time" refers to processing or updating information occurring almost immediately or with very little delay.

[1740] This invention is a system that digitizes and automates the work of table officials (TOs) in sports matches by acquiring game video data in real time from multiple camera devices and analyzing the movements of players and the ball. Each component of this system works together, reflecting the game situation in real time in the score and automatically generating commentary on plays.

[1741] Key Components of the System

[1742] 1. Video data acquisition unit:

[1743] The server acquires real-time video data of the match from multiple cameras (e.g., general-purpose digital cameras) installed in the venue. Each camera captures the match from a different angle or position, and transmits the video data to the server via a network.

[1744] 2. Image analysis unit:

[1745] The server breaks down the captured video data into frames using libraries such as OpenCV. It then applies an object detection model (e.g., YOLOv3 or SSD model) trained using machine learning libraries such as TensorFlow or PyTorch to identify the positions of players and the ball within each frame. The detected position information is recorded as coordinate data.

[1746] 3. Tracking section:

[1747] The server uses tracking algorithms such as Kalman filters and SORT to track the movement of players and the ball between successive frames, recording this data as time series data and analyzing patterns of player and ball movement.

[1748] 4. Event Identification:

[1749] The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts), specifically when the ball passes through the hoop, which is identified as a scoring event. Identified events are recorded in a database.

[1750] 5. Score Update Section:

[1751] The server updates the team's score in real time based on the identified events. For example, if a shot is made, a point is added to the corresponding team's score. This information is managed as scoreboard data.

[1752] 6. Play commentary generation unit:

[1753] The server generates play commentary in natural language based on the identified event data using generative AI (e.g., GPT-3). For example, it generates commentary such as, "Player A made a great three-point shot!"

[1754] 7. Data Distribution Department:

[1755] The server converts the updated scores and generated commentary into a displayable data format such as JSON and distributes it in real time via WebSocket or REST API, making the latest information instantly available.

[1756] 8. Display terminal:

[1757] Devices (e.g., tablets and smartphones) and digital scoreboards receive the data distributed from the server. The devices display the received data on the screen in real time as a scoreboard and play commentary. Appropriate UI / UX design is used to improve visibility.

[1758] Specific examples

[1759] For example, if Player A makes a three-point shot during a game, the system will behave as follows:

[1760] 1. Acquiring video data:

[1761] The server collects video data in real time from multiple cameras in the venue, each capturing footage of the match from a different angle, and transmits the data to the server via a network.

[1762] 2. Image Analysis:

[1763] The server breaks down the video data into frames using OpenCV or similar tools, and applies an object detection model (such as YOLOv3) to identify the positions of players and the ball.

[1764] 3. Tracking:

[1765] The server uses a tracking algorithm (such as a Kalman filter) to track the ball's movement between successive frames and records the trajectory of its movement.

[1766] 4. Event Identification:

[1767] The server analyzes the tracking data and detects when the ball passes through the hoop, identifying this as a successful shot event.

[1768] 5. Score Update:

[1769] The server will award the successful shot three points to Team A's score and record this information in the database.

[1770] 6. Play commentary generation:

[1771] Using generative AI (GPT-3), we generate commentary on the play, such as "Player A made a great three-point shot!"

[1772] 7. Data Distribution:

[1773] The server converts the updated scores and generated play commentary into JSON format and delivers them to the device in real time via WebSocket.

[1774] 8. Receiving and Displaying Data:

[1775] The device receives the data and displays the latest score and commentary on the game in real time on the screen.

[1776] Prompt Sentence Examples

[1777] An example of an input prompt for the generation AI is, "Please generate commentary if player A makes a successful three-point shot."

[1778] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1779] Step 1: Obtaining video data

[1780] The server acquires real-time video data of the match from multiple camera devices. Specifically, multiple camera devices capture the match from different angles and transmit the video data to the server via a network. The input is the video data captured by each camera device, and the output is multiple real-time video streams stored on the server.

[1781] Step 2: Image analysis

[1782] The server breaks down the captured video data into frames. To do this, it uses libraries such as OpenCV to convert the video data into still images. Next, it applies an object detection model (e.g., YOLOv3 or SSD) trained using machine learning libraries such as TensorFlow or PyTorch to identify the positions of players and the ball within each frame. The input is the captured video data (frames), and the output is the position information (coordinate data) of players and the ball for each frame.

[1783] Step 3: Tracking

[1784] The server uses tracking algorithms such as Kalman filter and SORT to track the movements of players and the ball between consecutive frames. This allows the position information of players and the ball to be recorded as time-series data. The input is consecutive frames of data containing position information, and the output is time-series position information as tracking data.

[1785] Step 4: Event Identification

[1786] The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts). In particular, if the ball's trajectory passes through the hoop, it is identified as a successful shot. The input is the tracking data, and the output is the identified event information.

[1787] Step 5: Update your score

[1788] The server updates the team's score in real time based on the identified event information. For example, when a shot is made, a point is added to the corresponding team's score. This information is managed as scoreboard data. The input is the identified event information, and the output is updated score information.

[1789] Step 6: Generate play instructions

[1790] The server generates commentary based on the identified event data using a generation AI (e.g., GPT-3). To do this, a prompt is prepared for the generation AI and given as input. For example, the prompt might be, "Please generate commentary if player A makes a successful three-point shot." The input is the identified event data and the prompt, and the output is the generated commentary text.

[1791] Step 7: Data Distribution

[1792] The server converts the updated score and generated commentary into a displayable data format such as JSON and distributes it in real time via WebSocket or REST API. The input is the updated score information and the generated commentary text, and the output is the distributed JSON data.

[1793] Step 8: Receive and display data

[1794] The terminal receives data delivered from the server and displays it on the screen as a scoreboard and commentary in real time. For this purpose, a web application or a native application is used. The input is the delivered JSON data, and the output is the latest score and commentary displayed on the screen.

[1795] (Application example 1)

[1796] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1797] It is difficult for table officials (TOs) to perform their duties quickly and accurately during a basketball game. Furthermore, there is a lack of means to accurately communicate the real-time status of the game to spectators. Therefore, there is a need for a system that can reduce the burden on TOs and provide spectators with real-time information on the game's status.

[1798] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1799] In this invention, the server includes means for acquiring video data in real time from multiple camera devices, means for breaking down the acquired video data into frames and identifying the positions of the target and the sphere using an object detection model, means for tracking the movement of the target and the sphere between consecutive frames using a tracking algorithm, means for identifying specific events from the tracking data and updating scores in real time, means for creating action descriptions using a generation AI based on the identified event data, means for converting the updated scores and generated action descriptions into a data format for display and distributing them in real time, and means for receiving the distributed data and displaying them as a score board and action descriptions on a virtual space display device. This reduces the burden on TOs and makes it possible to accurately convey the situation of the game to spectators in real time.

[1800] A "camera" is a device for acquiring video data, and is responsible for capturing video of the game in real time from multiple angles and positions.

[1801] "Video data" refers to visual information of the match obtained from a camera, and is data that shows the progress of the match in real time.

[1802] An "object detection model" is an analytical algorithm or technology that analyzes acquired video data to identify the position of a specific object (such as a player or a ball).

[1803] "Target" refers to a person or object that is being tracked and identified, such as a player in a match.

[1804] The term "sphere" refers to a sports ball used in a game, and in the present invention refers to a basketball.

[1805] A "frame" is an individual still image that makes up video data, and a moving image is formed by successive frames.

[1806] A "tracking algorithm" is a technique used to track the movement of an object or sphere between successive frames and record its position information as time-series data.

[1807] "Specific events" refer to important events that occur during a game (such as a successful shot or a foul) that the system automatically identifies.

[1808] "Generative AI" is an artificial intelligence technology that automatically generates natural language descriptions of behavior based on identified event data.

[1809] An "action explanation" is an explanatory text about a specific event during a match, created by the generation AI.

[1810] The "score board" is an interface that is displayed on the virtual space display device to visually provide the latest score information.

[1811] A "virtual space display device" is a device that uses virtual reality technology to allow users to visually experience a match in a virtual space, and generally includes a VR headset.

[1812] A system for implementing this invention comprises the following major components:

[1813] 1. Server

[1814] The server has a means to acquire video data in real time from multiple camera devices, which are installed at different angles and positions within the venue to capture visual information of the match.

[1815] The server has a means to break down the acquired video data into frames and identify the positions of the objects and spheres using an object detection model, which is implemented using a library such as OpenCV.

[1816] The server has a means to track the movement of the object and the ball using a tracking algorithm between successive frames, allowing the position data of the object (player) and the ball (basketball) to be recorded over time.

[1817] The server has the means to identify specific events from the tracking data and update the score in real time. For example, when a ball passes through a hoop, it is identified as a scoring event and the score is automatically updated.

[1818] The server has a means to generate behavioral explanations based on the identified event data using a generative AI model, such as GPT-3.

[1819] The server has a means for converting the updated scores and generated action descriptions into a data format for display and distributing them in real time, thereby enabling the virtual space display device to quickly provide the latest match information.

[1820] 2. Terminal (Virtual Space Display Device)

[1821] The device receives data from the server and displays it in the virtual space as a scoreboard and action explanations, allowing users to visually grasp the progress of the game in real time. A VR headset or similar device is used as the virtual space display device.

[1822] 3. Users

[1823] By wearing the virtual space display device, users can watch the game in a virtual space. By referring to the score and action explanations updated in real time, users can intuitively understand the situation of the game.

[1824] Specific examples

[1825] For example, if a ball (basketball) passes through the hoop during a game and a three-point shot is successful, the process is as follows:

[1826] 1. Server

[1827] Video data acquisition: Video data is acquired in real time from the imaging device.

[1828] Image analysis: Video data is broken down into frames and object detection models are applied to identify the positions of objects (players) and balls (basketballs).

[1829] Tracking: The movement of the object and the sphere is tracked using a tracking algorithm between successive frames.

[1830] Event Identification: The tracking data is analyzed and when the sphere passes through the ring it is identified as a scoring event.

[1831] Score Update: Based on the identified incident, add 3 points to the team's score.

[1832] Play commentary generation: Using a generation AI, we generate an action explanation such as "Player A made a great three-point shot!"

[1833] Data distribution: Updated scores and commentary are converted into a displayable data format and distributed in real time.

[1834] 2. Terminal (Virtual Space Display Device)

[1835] Display of received data: Receive the distributed data and display it in the virtual space as a score board and action explanation.

[1836] Prompt Sentence Examples

[1837] "When a three-point shot is successful, generate a score and commentary of the play and display them in the virtual space."

[1838] The above steps reduce the burden on TOs and enable them to provide users with accurate information on the progress of the match in real time.

[1839] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1840] Step 1:

[1841] The server receives video data from the camera in real time. Each camera captures the game from a different angle or position and sends the video data to the server. The server receives the data and starts processing. The input is real-time video data, and the output is the video data.

[1842] Step 2:

[1843] The server uses the OpenCV library to decompose the video data into frames, which converts it into a sequence of still images (frames). The input is the video data acquired in step 1, and the output is the individual frames.

[1844] Step 3:

[1845] The server applies an object detection model to each decomposed frame to identify the positions of the objects (players) and the ball (basketball). This uses a machine learning object detection algorithm. The input is the image data for each frame, and the output is the position information of the objects and the ball within each frame.

[1846] Step 4:

[1847] The server uses a tracking algorithm to track the movement of the object and the sphere between successive frames, generating position information as time-series data. The input is the position information within each frame, and the output is time-series position data of the object and the sphere.

[1848] Step 5:

[1849] The server analyzes the tracking data and identifies specific events, such as a ball passing through a ring, as a scoring event. This is achieved using an event identification algorithm, whose input is the time-series position data and whose output is the identified specific event.

[1850] Step 6:

[1851] The server updates the scores in real time based on the identified events. Each scoring event automatically increments the team's score. The input is the identified event, and the output is the updated score.

[1852] Step 7:

[1853] The server uses a generative AI to create a behavioral explanation based on the identified event data. For example, it generates an explanation in the form of "Player A made a great three-point shot!" The input is the identified event data, and the output is the generated behavioral explanation.

[1854] Step 8:

[1855] The server converts the updated scores and generated behavior descriptions into a data format for display and distributes them in real time using data streaming technology. The input is the updated scores and generated behavior descriptions, and the output is data that can be distributed.

[1856] Step 9:

[1857] The terminal (virtual space display device) receives the data distributed from the server and displays it in the virtual space as a score board and action explanations. This allows the user to visually grasp the progress of the game in real time. The input is the distributed data, and the output is the display in the virtual space.

[1858] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1859] This invention is a system that digitizes and automates the work of table officials (TOs) in basketball games, and combines it with an emotion engine that recognizes user emotions to provide a more interactive and immersive game-watching experience.

[1860] System configuration

[1861] The system of the present invention consists of the following main components:

[1862] 1. Video Data Acquisition Unit: The server acquires video data of the match in real time from multiple cameras installed in the venue. This data is used to analyze the movements of players and the ball.

[1863] 2. Image analysis unit: The server breaks down the acquired video data into frames and identifies the positions of players and the ball using an object detection model.

[1864] 3. Tracking: The server uses a tracking algorithm to track the movements of players and the ball between successive frames and records these movements as time-series data.

[1865] 4. Event Identification: The server analyzes the tracking data and identifies specific events such as successful shots, fouls, and timeouts.

[1866] 5. Score Updater: The server updates the score in real time based on the identified events.

[1867] 6. Play commentary generation unit: The server uses a generation AI to create a play commentary in natural language based on the identified event data.

[1868] 7. Emotion engine: The server includes an emotion engine that acquires the user's face and voice data and recognizes their emotions.

[1869] 8. Data distribution unit: The server converts the updated scores and generated play commentary into a data format for display and distributes them in real time.

[1870] 9. Display terminal: The terminal receives the distributed data and displays it on the screen as a scoreboard and play commentary.

[1871] Program processing flow (explained in natural language)

[1872] Server: Acquire video data

[1873] The server collects real-time video data from multiple cameras installed in the venue, each capturing footage of the match from a different angle or position, and streams the footage in chronological order.

[1874] Server: Image analysis

[1875] The server breaks down the acquired video data into frames and applies an object detection model to identify the positions of players and the ball, thereby obtaining positional information for players and the ball within each frame.

[1876] Server:Tracking

[1877] The server uses a tracking algorithm to track the movement of players and the ball between successive frames, recording this data as time series of player and ball positions for use in motion analysis.

[1878] Server:Event Identification

[1879] The server identifies specific events (e.g., successful shots, fouls, timeouts) from the tracking data. For example, if it detects that the ball has passed through the hoop, it identifies that as a score event.

[1880] Server:Score Update

[1881] The server updates the team's score in real time based on the identified events, for example, adding a point to the team's score when a made shot event is identified.

[1882] Server: Play commentary generation

[1883] The server generates play commentary in natural language based on the event data identified using the AI ​​generation technology, which provides viewers with easy-to-understand details about the match.

[1884] Server: Emotion recognition

[1885] The server acquires the user's facial and voice data and uses an emotion engine to recognize their emotions. For example, by analyzing facial expression data and tone of voice acquired through a camera or microphone, it can determine whether the user is in an emotional state such as "joy," "excitement," or "dissatisfaction."

[1886] Server: Adjustment of gameplay commentary

[1887] The server dynamically adjusts the content and display of the commentary based on the user's emotions as recognized by the emotion engine. For example, if the user is excited, the server can display more detailed and enthusiastic commentary.

[1888] Server: Real-time analysis of audience sentiment

[1889] The server aggregates the emotional data of multiple users and analyzes the emotional state of the entire audience in real time. This data may be reflected in the progress and direction of the match.

[1890] Server: Prepare data distribution

[1891] The server converts the updated scores and generated commentary into a data format for display, such as JSON or HTML.

[1892] Server: Data distribution

[1893] The server then transmits the converted data to the screens at the venue and to the audience's devices in real time using a real-time communication protocol such as WebSocket.

[1894] Terminal: Receiving and displaying data

[1895] The device receives data from the server and displays it on the screen as a scoreboard and commentary on the game, which is updated in real time, allowing spectators to instantly understand the situation in the game.

[1896] Specific examples

[1897] For example, if a player makes a three-point shot during a game, the process is as follows:

[1898] 1. Acquisition of video data (server): The server acquires video data in real time from multiple cameras installed within the venue.

[1899] 2. Image analysis (server): The server breaks down the captured video data into frames and applies object detection models to identify the positions of players and the ball.

[1900] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[1901] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[1902] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[1903] 6. Play commentary generation (server): The server uses a generation AI to generate a play commentary such as, "Player X made a great three-point shot!"

[1904] 7. Emotion recognition (server): The server acquires the user's facial and voice data and uses the emotion engine to recognize when the user is excited.

[1905] 8. Adjusting play commentary (server): The server will reflect the user's excitement and adjust the commentary to be more detailed and energetic.

[1906] 9. Data distribution preparation (server): The server converts the updated scores and adjusted play commentary into a data format for display.

[1907] 10. Data distribution (server): The server distributes data in real time to the screen and audience devices.

[1908] 11. Receiving and displaying data (terminal): The terminal receives the distributed data and displays the scoreboard and play commentary in real time.

[1909] This will significantly reduce the burden on TOs and allow audiences to have a more immersive experience by receiving individually tailored commentary through emotion recognition.

[1910] The processing flow will be explained below.

[1911] Program processing steps

[1912] Step 1: Obtaining video data

[1913] server:

[1914] The server collects real-time video data from multiple cameras installed in the venue, each capturing footage from a different angle, and streams it in chronological order.

[1915] Step 2: Frame Decomposition

[1916] server:

[1917] The server breaks down the acquired video data into frames and processes each frame in parallel for the video data from each camera.

[1918] Step 3: Object detection

[1919] server:

[1920] The server uses object detection models such as YOLO (You Only Look Once) to identify players and the ball in each frame.

[1921] The detection model outputs the position coordinates and labels of players and the ball (Player 1, Player 2, Ball, etc.).

[1922] Step 4: Tracking

[1923] server:

[1924] The server uses a tracking algorithm to track the movement of players and the ball between successive frames.

[1925] The algorithm analyzes the position and movement vectors of each object and records them as time series data.

[1926] Step 5: Event Identification

[1927] server:

[1928] The server identifies specific events (e.g., successful shots, fouls, timeouts) based on the tracking data.

[1929] For example, if the ball passes over the hoop, this is identified as a "scoring event."

[1930] Step 6: Update your score

[1931] server:

[1932] The server updates the team scores in real time based on the identified events.

[1933] For example, if the shot is successful, the team will receive points corresponding to their score (e.g., 2 points, 3 points).

[1934] Step 7: Generate play instructions

[1935] server:

[1936] The server generates a play commentary in natural language based on the event data identified using the generation AI.

[1937] For example, it generates a sentence such as, "Player X broke through the defense with a sharp dribble and scored a stunning layup!"

[1938] Step 8: Emotion Recognition

[1939] server:

[1940] The server acquires the user's facial and voice data and recognizes their emotions using an emotion engine.

[1941] For example, facial expression data and tone of voice obtained from a camera or microphone are analyzed to determine whether the user is in an emotional state such as "joy," "excitement," or "dissatisfaction."

[1942] Step 9: Adjust the commentary

[1943] server:

[1944] The server dynamically adjusts the content and display method of the play commentary based on the user's emotions recognized by the emotion engine.

[1945] For example, if the user is excited, a more detailed and energetic commentary will be displayed.

[1946] Step 10: Real-time analysis of audience sentiment

[1947] server:

[1948] The server aggregates the emotional data of multiple users and analyzes the emotional state of the entire audience in real time.

[1949] This data may be reflected in the progress and presentation of the match.

[1950] Step 11: Prepare for data distribution

[1951] server:

[1952] The server converts the updated scores and generated commentary into a data format for display, such as JSON or HTML.

[1953] Step 12: Data Distribution

[1954] server:

[1955] The server distributes the converted data in real time to the screens at the venue and to the audience's devices.

[1956] Data is delivered using real-time communication protocols such as WebSocket.

[1957] Step 13: Receiving and displaying data

[1958] Device:

[1959] The terminal receives the data distributed from the server and displays it on the screen as a scoreboard and commentary on the game.

[1960] The display is updated in real time, allowing spectators to instantly understand the situation in the game.

[1961] Specific examples

[1962] For example, if a player makes a three-point shot during a game, the process is as follows:

[1963] 1. Acquisition of video data (server): The server acquires video data in real time from multiple cameras installed within the venue.

[1964] 2. Image analysis (server): The server breaks down the captured video data into frames and applies object detection models to identify the positions of players and the ball.

[1965] 3. Tracking (Server): The server tracks the ball's movement between successive frames using a tracking algorithm.

[1966] 4. Event Identification (Server): The server analyzes the tracking data and identifies the ball passing through the hoop as a scoring event.

[1967] 5. Score Update (Server): The server detects that the three-point shot was successful and adds three points to the team's score.

[1968] 6. Play commentary generation (server): The server uses a generation AI to generate a play commentary such as, "Player X made a great three-point shot!"

[1969] 7. Emotion recognition (server): The server acquires the user's facial and voice data and uses the emotion engine to recognize when the user is excited.

[1970] 8. Adjusting play commentary (server): The server will reflect the user's excitement and adjust the commentary to be more detailed and energetic.

[1971] 9. Data distribution preparation (server): The server converts the updated scores and adjusted play commentary into a data format for display.

[1972] 10. Data distribution (server): The server distributes data in real time to the screen and audience devices.

[1973] 11. Receiving and displaying data (terminal): The terminal receives the distributed data and displays the scoreboard and play commentary in real time.

[1974] This will significantly reduce the burden on TOs and allow audiences to have a more immersive experience by receiving individually tailored commentary through emotion recognition.

[1975] Example 2

[1976] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1977] At sports games such as basketball, the goal is to improve the efficiency and accuracy of table officials (TOs) by digitizing and automating their work. There is also a need for a system that can provide real-time commentary and information on the game to spectators, creating a more immersive and realistic experience. However, existing systems face challenges in accurately grasping the game situation and updating and distributing necessary information in real time. Furthermore, there are currently no systems that can recognize and reflect spectator emotions and provide personalized commentary.

[1978] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1979] In this invention, the server includes a means for acquiring image data of the game in real time from multiple camera devices, a means for dividing the acquired image data into successive images and using an object detection algorithm to identify the positions of players and the ball, and a means for tracking the movement of players and the ball between successive images using a tracking algorithm. This enables accurate understanding of the game situation, real-time score updates, and the creation and distribution of game commentary using generative AI. Furthermore, the server can acquire user facial and voice data, analyze their emotional state using an emotion engine, and dynamically adjust the content and display method of the game commentary based on that data, providing spectators with a more realistic and immersive game viewing experience.

[1980] "Filming equipment" refers to video cameras installed within the venue to capture the state of the match.

[1981] "Image data" refers to video data of a match captured by a camera.

[1982] "Continuous images" refers to a set of still images obtained by breaking down video data into frames.

[1983] An "object detection algorithm" refers to a machine learning model for identifying specific objects (players or balls) within image data.

[1984] "Player" means a player taking part in a match.

[1985] "Sphere" refers to the ball used during a match.

[1986] A "tracking algorithm" refers to a computational method for tracking the position of a particular object (a player or a ball) between successive image frames.

[1987] "Specific occurrences" refer to significant events that occur during a game, such as a successful shot, a foul, or a timeout.

[1988] "Score" refers to points awarded to a team based on a specific event.

[1989] "Generative algorithm" refers to AI technology that generates natural language commentary on sports events based on given data.

[1990] "Sports commentary" refers to natural language text that describes the progress of a match or an event.

[1991] "Display data format" refers to a data format (e.g., JSON, HTML) suitable for displaying scores and commentary on a display device.

[1992] An "emotion engine" refers to software or algorithms that analyze a user's facial expressions and voice data to recognize their emotional state.

[1993] "Users" refers to spectators who attend matches.

[1994] "Emotional state" refers to the user's psychological state, such as joy, excitement, or dissatisfaction.

[1995] "Dynamic adjustment" refers to changing the content and display format in real time depending on the situation.

[1996] This invention aims to digitize and automate the work of table officials (TOs) at sports games, and is a system that combines it with an emotion engine that recognizes user emotions to provide a more interactive and immersive game-watching experience.

[1997] System configuration

[1998] Server: Acquire video data

[1999] The server acquires image data of the match in real time from multiple camera devices installed in the venue. Each camera captures the match from a different angle or position and streams the data. Specific examples of such cameras include high-resolution digital cameras and IP cameras.

[2000] Server: Image analysis

[2001] The server divides the acquired image data into successive images and applies an object detection algorithm to identify the positions of players and the ball. The object detection algorithm uses machine learning models such as YOLO (You Only Look Once) and Mask R-CNN. This allows the server to obtain the positional information of players and the ball within each frame.

[2002] Server:Tracking

[2003] The server tracks the movements of the players and the ball between successive image frames using a tracking algorithm, such as a Kalman filter and the tracking algorithm SORT (Simple Online and Realtime Tracking). This process stores the positional information of the players and the ball as time-series data.

[2004] Server:Event Identification

[2005] The server analyzes the tracking data and identifies specific events (e.g., successful shots, fouls, timeouts). For example, a ball passing through the hoop is captured as a "successful shot." Event identification is achieved using statistical analysis and rule-based techniques.

[2006] Server:Score Update

[2007] The server updates the team's score in real time based on the identified incidents, and points are automatically added to the team's score for successful shots, for example, three points for a successful three-point shot.

[2008] Server: Play commentary generation

[2009] The server uses a generative algorithm to generate a natural language commentary of the game based on the identified event data. Generative AI models such as GPT-4 are used. For example, a commentary such as "Player X made a successful three-point shot!" is generated.

[2010] Server: Emotion recognition

[2011] The server collects the user's facial and voice data and analyzes their emotional state using an emotion engine, which uses tools such as Affectiva and the Microsoft Azure Emotion API. This allows the server to determine in real time whether the user is in a state of joy, excitement, dissatisfaction, or other emotional state.

[2012] Server: Adjustment of gameplay commentary

[2013] The server dynamically adjusts the content and display method of the generated commentary based on the user's emotions determined by the emotion engine. For example, if the user is excited, the generated commentary will be changed to be more detailed and passionate.

[2014] Server: Real-time analysis of audience sentiment

[2015] The server aggregates emotional data collected from multiple users and analyzes the emotional state of the entire venue in real time. This data is reflected in the progress and direction of the match.

[2016] Server: Prepare data distribution

[2017] The server converts the updated scores and generated commentary into a displayable data format (e.g., JSON or HTML), which facilitates distribution.

[2018] Server: Data distribution

[2019] The server then distributes the converted data in real time to the screens at the venue and to the audience's devices, using a real-time communication protocol such as WebSocket.

[2020] Terminal: Receiving and displaying data

[2021] The terminals receive the data distributed from the server and display it on the screen as scores and commentary on the match. The terminals can be smartphones, tablets, digital signage, etc. The display is updated in real time, allowing spectators to instantly understand the situation of the match.

[2022] Specific examples

[2023] For example, if a player makes a three-point shot during a game, the process is as follows:

[2024] 1. Acquisition of video data (server): The server acquir...

Claims

1. A means of acquiring video data of the match in real time from multiple cameras; A means for breaking down the acquired video data into frames and identifying the positions of players and the ball using an object detection model; means for tracking player and ball movement between successive frames using a tracking algorithm; A means of identifying specific events from the tracking data and updating the scores in real time; A means for generating play commentary using a generation AI based on the identified event data; A means for converting the updated scores and generated commentary into a displayable data format and distributing them in real time; The system includes means for receiving the distributed data and displaying it on a screen as a scoreboard and commentary on the play.

2. 10. The system of claim 1, further comprising means for analyzing the tracking data and identifying a scoring event when the ball passes through a hoop.

3. 10. The system of claim 1, further comprising means for automatically incrementing a team's score based on the identified events.

Images