System

Abstract

A system is provided.SOLUTION: The system includes a means for acquiring video, a means for analyzing the acquired video data and detecting the motion, position and position of a ball of a player, a means for generating a live commentary sentence in a natural language based on the detected data, and a means for integrating the generated live commentary sentence and the video and distributing them in real time.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional sports broadcasting systems require specialized teams and expensive equipment, making it difficult to easily broadcast sports with small audiences or children's games. Even in such environments, there is a demand for a system that can easily broadcast live sports and provide information to viewers in real time. [Means for solving the problem]

[0005] The present invention relates to a system for easily broadcasting live sports coverage, and includes the following means: First, a means for acquiring video is provided; Second, a means for analyzing the acquired video data and detecting the movements and positions of players and the position of the ball is used; Third, a means for generating commentary in natural language based on the detected data is provided; Finally, by configuring a system including a means for integrating the generated commentary with video and distributing it in real time, it is possible to easily broadcast sports coverage at low cost.

[0006] "Means for acquiring video" refers to the ability to capture live video using a smartphone camera or other video capture device and collect that data.

[0007] "Captured video data" refers to video information captured by a video capture device and imported into the system.

[0008] The "means of analysis" refers to a function that uses computer vision technology and image analysis algorithms to detect the movements and positions of players and the position of the ball from the acquired video data.

[0009] "Player movements and positions" refers to the movements and current position information of athletes within the filmed footage.

[0010] "Ball position" refers to information about where a sports ball is currently located within the captured video.

[0011] "Means for detection" refers to the ability to identify specific objects (players and balls) in the video and determine their positions and movements.

[0012] "Natural language commentary" is a text message generated to inform viewers about the content of a match based on the progress and events of the match.

[0013] The "means of generation" is a function that uses AI technology to automatically create commentary based on the detected data.

[0014] The "means for integrating" is a function that combines the generated commentary with the acquired video data and compiles them into a single distribution stream.

[0015] "Real-time delivery means" refers to the ability to deliver an integrated video and commentary stream live to viewers without delay. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This paper describes a system for automatically broadcasting live sports events, and explains its method in detail. This system performs all steps from video capture and analysis to natural language commentary generation and real-time distribution.

[0038] First, a user uses a smartphone to capture video of a sports game. The device transmits the captured video data to a server in real time. The video data is then encoded and sent to the server using a low-latency protocol.

[0039] The server decodes the received video data and stores the video stream in a buffer. The server then uses computer vision technology to analyze the movement and position of players and the ball in the video. This includes player tracking and ball location detection, and identifies significant events such as goals and points scored.

[0040] Based on the event data generated by the analysis, the server automatically generates commentary using a natural language generation algorithm (e.g., GPT-4 (registered trademark)). This commentary corresponds to a specific event and is managed along with a timestamp.

[0041] The server then integrates the generated commentary with the video, adding the commentary as a text overlay to the video stream, and then encodes the integrated video and commentary for delivery in real time using a streaming protocol.

[0042] As a concrete example, let's take a children's soccer match. A user uses a smartphone to film the match, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." If player number 8 then scores a goal, the server generates a commentary such as "Goal! Player number 8 scores a great shot!" These commentaries are integrated with the video and distributed to viewers in real time.

[0043] This system allows anyone to easily broadcast live sports without the need for expensive dedicated equipment or a professional broadcasting team. It is particularly useful for sports with small audiences and children's games.

[0044] The processing flow will be explained below.

[0045] Step 1:

[0046] Users use their smartphone cameras to capture footage of sports matches, and the device acquires the footage in real time.

[0047] Step 2:

[0048] The device compresses and encodes the acquired video data and sends it to the server using a low-latency protocol (e.g., RTSP, WebRTC).

[0049] Step 3:

[0050] The server receives the video data sent from the terminal, decodes it, and temporarily stores it in a buffer.

[0051] Step 4:

[0052] The server analyzes the stored video data using computer vision technology (e.g., OpenCV, TENSORFLOW (registered trademark)), which detects the movements and positions of players and the position of the ball.

[0053] Step 5:

[0054] Based on the analysis results, the server generates data about player and ball movements and identifies important events (e.g. goals, shots, etc.).

[0055] Step 6:

[0056] The server uses the generated event data to apply a natural language generation algorithm (e.g., GPT-4) to automatically generate commentary.

[0057] Step 7:

[0058] The server timestamps the generated commentary and synchronizes it with the actual game time.

[0059] Step 8:

[0060] The server adds the commentary as an overlay to the video stream, integrating the video and commentary.

[0061] Step 9:

[0062] The server encodes the aggregated video stream and converts it into a streaming format (e.g., HLS, DASH).

[0063] Step 10:

[0064] The server delivers the encoded video stream in real time, allowing users to watch the game by accessing the live stream on their smartphones or computers.

[0065] These processing steps allow anyone to easily enjoy live coverage of sports games in real time.

[0066] Example 1

[0067] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0068] Conventional live broadcasts of sporting events require expensive equipment and a specialized broadcasting team, and preparation and operation are costly and labor-intensive. Furthermore, it is difficult to broadcast professionally when the audience is small, such as for small-scale sporting events or children's games. For this reason, there is a demand for a system that allows anyone to easily broadcast live sports at low cost.

[0069] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0070] In this invention, the server includes means for acquiring video, means for encoding the acquired video data and transmitting it to the server using a low-latency protocol, means for decoding the received video data and storing it in a buffer, means for analyzing the video data and detecting the movements and positions of players and objects in the sporting event, means for generating a natural language commentary using a generative AI model that creates generated text based on the detected data, means for integrating the generated commentary with the video and adding it as a text overlay, and means for encoding the integrated video and commentary and distributing it in real time using a streaming protocol, thereby enabling anyone to easily broadcast a sporting event in real time without the need for expensive equipment.

[0071] "Means for acquiring video" refers to means for capturing video in real time using a photographic device such as a smartphone or camera.

[0072] "Means for encoding acquired video data and transmitting it to a server using a low-latency protocol" refers to means for encoding video data based on a compression method and transmitting it to a server using a low-latency data transmission protocol such as WebRTC or RTMP.

[0073] "Means for decoding received video data and storing it in a buffer" refers to means for the server to decode encoded video data received from a client terminal and store it in a temporary memory space (buffer).

[0074] "Means for analyzing the video data and detecting the movements and positions of players and objects in the sporting event" refers to means for identifying the positions and movements of players and the ball in the video using computer vision technology and machine learning algorithms.

[0075] A "means for generating natural language commentary using a generative AI model that creates generated text based on detected data" means a means for using the analyzed data as input and a generative AI model (e.g., a deep learning model) to create a natural language commentary that includes details of the event.

[0076] The "means for integrating the generated commentary with the video and adding it as a text overlay" refers to a means for visually displaying the generated commentary as text overlaid at a specific position on the video stream.

[0077] "Means for encoding the integrated video and commentary and delivering it in real time using a streaming protocol" refers to a means for re-encoding the video stream with the added text overlay and delivering it to viewers in real time using a streaming protocol such as HLS or DASH.

[0078] This invention is a system for automatically broadcasting live sports events, and is realized using specific devices and algorithms. This system performs all processes from video capture and analysis to natural language commentary generation and real-time distribution.

[0079] First, a user uses a smartphone to capture video of a sports game. The smartphone used here must be equipped with a camera function and a streaming application. The captured video data is sent from the device to a server in real time. At this time, the device encodes the video data using a low-latency codec such as H.264 or H.265, and sends it to the server using a low-latency protocol such as WebRTC or RTMP.

[0080] The server receives the encoded video data, decodes it using a decoding library such as FFmpeg, and temporarily stores the video stream in a buffer. The server then uses computer vision techniques to analyze the movement and position of players and the ball in the video. Specific algorithms used include OpenCV and YOLO. This analysis process includes player tracking and ball position detection, and identifies important events such as goals and points.

[0081] Based on the analysis data, the server uses a generative AI model (such as GPT-4) to generate commentary. The generated commentary is managed along with a timestamp. For example, if the analysis detects that player number 8 is dribbling the ball, the prompt "Player number 8 is dribbling the ball" is generated. If player number 8 scores a goal, the commentary will read "Goal! Player number 8 made a great shot!"

[0082] The server then uses a library such as FFmpeg to add the generated commentary as a text overlay to the video stream. The commentary is integrated into the video at the appropriate time based on the timestamp. After integration, the server distributes the encoded video stream in real time using a streaming protocol such as HLS or DASH. Viewers can then watch the distributed video in real time using devices such as smartphones and PCs.

[0083] As a concrete example, let's take a children's soccer match. A user uses a smartphone to film the match, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." If player number 8 then scores a goal, the server generates a commentary such as "Goal! Player number 8 scores a great shot!" These commentaries are integrated with the video and distributed to viewers in real time.

[0084] As a result, this system allows anyone to easily broadcast live sports without the need for expensive dedicated equipment or a specialized broadcasting team. This is particularly useful for sports with small audiences or for children's games.

[0085] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0086] Step 1:

[0087] Video shooting by users

[0088] Input: Sports event footage, smartphone

[0089] Processing: A user uses a smartphone to capture video of a sporting event in real time, where the smartphone has a camera function and an application installed to capture the video.

[0090] Output: Real-time video data captured by a smartphone camera

[0091] Step 2:

[0092] Video encoding and transmission by the device

[0093] Input: Real-time video data

[0094] Processing: The device encodes the captured video using a codec such as H.264 or H.265 to compress the video data. It then transmits the data to the server using a low-latency protocol such as WebRTC or RTMP. It also monitors the network connection status to minimize latency.

[0095] Output: The encoded video data is sent to the server.

[0096] Step 3:

[0097] Video reception and decoding by the server

[0098] Input: Encoded video data sent from the device

[0099] Processing: The server receives the encoded video data sent from the device, decodes the video data using a decoding library such as FFmpeg, and stores it in a buffer memory in real time. This information is used for later analysis and integration processing.

[0100] Output: Decoded video data stored in a buffer

[0101] Step 4:

[0102] Server-based video analysis

[0103] Input: Decoded video data stored in a buffer

[0104] Processing: The server uses computer vision techniques (e.g., OpenCV or YOLO) to analyze the video data. Specifically, it detects the tracking, movement, and position of players and the ball within the video. This analysis identifies important events, such as goals scored.

[0105] Output: Analysis data (player position, ball position, event information, etc.)

[0106] Step 5:

[0107] Server-based natural language generation

[0108] Input: Analysis data (player position, ball position, event information, etc.)

[0109] Processing: The server uses a generative AI model (e.g., GPT-4) to generate commentary in natural language based on the analyzed data. For each specific event, a prompt is provided as input, and a natural-sounding commentary is generated based on that. For example, the commentary generated might be, "Player No. 8 is dribbling the ball."

[0110] Output: Generated commentary (with timestamp)

[0111] Step 6:

[0112] Server-based integration of video and commentary

[0113] Input: Decoded video data, generated commentary

[0114] Processing: The server adds the generated commentary to the video stream as a text overlay, using FFmpeg filters to insert the text at the right time and in the right position, based on the timestamps.

[0115] Output: Video stream with text overlay added

[0116] Step 7:

[0117] Server-based video content distribution

[0118] Input: Video stream with text overlay added

[0119] Processing: The server re-encodes the combined video stream and delivers it in real time using a streaming protocol such as HLS or DASH, allowing viewers to enjoy the video and commentary smoothly without delay.

[0120] Output: Real-time integrated video stream

[0121] (Application example 1)

[0122] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0123] Conventional monitoring systems require manual confirmation, making it difficult to respond quickly. It is also difficult to generate warning messages in real time or automatically recognize suspicious activity, making it difficult to implement effective security measures. The goal of this project is to solve these issues and provide a more effective and efficient security system.

[0124] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0125] In this invention, the server includes a means for acquiring video, a means for analyzing the acquired video data to detect the movement and position of a target or the position of an object, a means for generating a warning message in natural language based on the detected data, and a means for integrating the generated warning message with the video and distributing it in real time, thereby enabling the rapid detection of suspicious movements and the immediate issuance of a warning.

[0126] "Means for acquiring video" refers to devices that capture video data in real time using cameras, sensors, etc.

[0127] "Means for analyzing acquired video data and detecting the movement and position of a target or object" refers to a device or software that uses computer vision technology and algorithms to detect the movement and position of a specific target or object in a video and analyze its behavior.

[0128] A "means for generating natural language warning text based on detected data" is a device or program that uses generative AI models or natural language processing techniques to automatically generate appropriate warning messages based on detected events or situations.

[0129] "Means for integrating generated warning text with video and delivering it in real time" refers to a technology or system that overlays generated warning text onto video data and delivers it to users in real time.

[0130] The "means for encoding video data and transmitting it to the server" refers to a device or software that converts (encodes) the captured video data into an appropriate format and transmits it to the server using a low-latency protocol.

[0131] "Means for detecting the position of a target or object using computer vision technology" refers to technologies or programs that use image processing algorithms or machine learning models to accurately detect the position of a specific target or object within a video.

[0132] The purpose of this invention is to automate and improve the efficiency of a monitoring system. We will now explain in detail how the server, terminals, and users cooperate to realize this system.

[0133] First, a surveillance camera, smartphone, or smart glasses camera captures video in real time. This is the "means of video capture." The device captures the video data and sends it to a streaming server using a low-latency protocol (e.g., RTMP).

[0134] The server then analyzes the received video data. Specifically, it uses computer vision technology such as OpenCV to detect specific objects in the video, such as people, vehicles, or suspicious movements. This is "a means of analyzing acquired video data and detecting the movement and location of objects."

[0135] Based on the detected data, a generative AI model (e.g., GPT-4) is used to generate a natural language warning. This is a "means of generating a natural language warning based on the detected data." The following prompts can be used:

[0136] "We've detected suspicious activity. What should we do?"

[0137] "I detected suspicious activity at night. What is the appropriate response?"

[0138] The generated warning text is integrated with the video data. Specifically, the warning text is added as a text overlay to the video stream. This is the "means of integrating the generated warning text with the video and delivering it in real time." This integrated data is delivered using a real-time streaming protocol (such as HLS or DASH), allowing users to check the monitoring results in real time.

[0139] Additionally, when important events occur, emergency response teams can be alerted using push notification services such as Firebase Cloud Messaging.

[0140] For example, if a surveillance camera detects motion at night, the server analyzes the video data to detect suspicious activity. Using this information, a generative AI model generates a warning message: "Suspicious activity detected. Please check immediately." This message is overlaid on the video stream and distributed in real time, simultaneously notifying emergency response teams.

[0141] In this way, the present invention realizes an efficient monitoring system and effective security measures.

[0142] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0143] Step 1:

[0144] The device (smartphone, smart glasses) captures video in real time using a camera. The input is the camera image, and the output is the captured video data. This video data is sent to a streaming server using a low-latency protocol (e.g., RTMP).

[0145] Step 2:

[0146] The server decodes the received video data. The input is the encoded video data, and the output is the decoded video frames. This is handled efficiently by the streaming protocol, for example, using the FFmpeg library for decoding.

[0147] Step 3:

[0148] The server analyzes the decoded video frames using computer vision technology. The input is the video frame, and the output is detection data such as the movement and position of the target and the location of the object. For example, OpenCV is used to detect suspicious movements and abnormal behavior. Specifically, it uses background subtraction to extract moving objects, and performs facial recognition and motion detection.

[0149] Step 4:

[0150] The server generates a warning in natural language based on the detection data. The input is the detection data and the output is the warning. The warning is generated using a generative AI model (e.g., GPT-4). As a specific example, an appropriate warning is generated based on the prompt, "Suspicious activity has been detected. What should be done?"

[0151] Step 5:

[0152] The server overlays the generated warning text onto the video frame. The input is the warning text and the video frame, and the output is the video with the warning text overlaid. Specifically, we add text to the video using OpenCV.

[0153] Step 6:

[0154] The server re-encodes the overlaid video and delivers it in real time. The input is the overlaid video, and the output is the streamed video. It is delivered to users using a streaming protocol (e.g., HLS or DASH).

[0155] Step 7:

[0156] The server sends a push notification when an important event occurs. The input is the event data, and the output is the notification message. A push notification service such as Firebase Cloud Messaging is used to send an alert to an emergency response team. Specifically, the alert text is set as the message and a Firebase API is called.

[0157] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0158] This invention is a system for automatically broadcasting live sports commentary, and combines it with an emotion engine that recognizes user emotions. This system enables video capture and analysis, commentary generation in natural language, real-time distribution, and even more realistic commentary that incorporates user emotions.

[0159] First, a user uses a smartphone to capture video of a sports game. The device transmits the captured video data to a server in real time, where it is encoded and transmitted to the server using a low-latency protocol.

[0160] The server decodes the received video data and temporarily stores it in a buffer. The server then uses computer vision technology to analyze the movement and position of players and the ball in the video. This analysis allows for player tracking, ball location detection, and identification of significant events such as goals and points scored.

[0161] The server then uses a natural language generation algorithm to automatically generate commentary based on the analysis results, which corresponds to a specific event and is maintained with a timestamp.

[0162] The server then uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial expressions and movements based on the camera footage to recognize their emotions. Based on the recognized emotion data, the content of the commentary can be adjusted in real time. For example, if the user is excited, the tone and content of the commentary can be made more exciting.

[0163] The server merges the generated commentary with the video, adds a text overlay of the commentary to the video stream, and then encodes the merged video and commentary for delivery in real time using a streaming protocol.

[0164] As a concrete example, let's take a children's soccer game. The user uses a smartphone to film the game, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." Furthermore, if the server recognizes that the user is excited, it generates an exciting commentary such as "Great dribbling! Player number 8 is moving forward!"

[0165] In this way, by using the emotion engine, this system can provide more realistic live broadcasts that reflect the user's emotions. This makes it possible for anyone to easily broadcast live sports without the need for expensive dedicated equipment or a specialized broadcasting team. This is particularly useful for sports with small audiences and children's games.

[0166] The processing flow will be explained below.

[0167] Step 1:

[0168] Users use their smartphone cameras to capture footage of sports matches, and the device captures the footage in real time and stores it in an internal buffer.

[0169] Step 2:

[0170] The device compresses and encodes the captured video data and sequentially transmits it to the server using a low-latency protocol (e.g., RTSP, WebRTC).

[0171] Step 3:

[0172] The server receives the video data sent from the terminal, decodes it, and stores the decoded video data in the server's buffer.

[0173] Step 4:

[0174] The server uses computer vision techniques (e.g., OpenCV, TensorFlow) to analyze the video data, determining the movements and positions of players and the position of the ball.

[0175] Step 5:

[0176] Based on the analysis results, the server generates event data about player and ball movements, e.g., "Player No. 8 is dribbling the ball" or "Player No. 8 has scored a goal."

[0177] Step 6:

[0178] The server uses a natural language generation algorithm (e.g., GPT-4) to generate commentary based on the generated event data. Examples include, "Player 8 is dribbling the ball," and "Goal! Player 8 made a great shot!"

[0179] Step 7:

[0180] The server recognizes the user's emotions using an emotion engine within the server, which analyzes the video data of the user's face and voice to identify his emotional state (e.g., excitement, joy, surprise).

[0181] Step 8:

[0182] The server adjusts the content and tone of the commentary based on the user's emotional data. For example, if the user is excited, the server will generate an exciting commentary such as, "Player No. 8 is showing off some great dribbling skills!"

[0183] Step 9:

[0184] The server combines the generated commentary with the decoded video data, and the commentary is overlaid onto the video stream with a timestamp.

[0185] Step 10:

[0186] The server compresses and encodes the aggregated video stream and delivers it in real time using a streaming protocol (e.g., HLS, DASH).

[0187] Step 11:

[0188] Users can watch the video stream in real time using their smartphones or PCs, and can enjoy a realistic commentary that reflects their own emotions.

[0189] By using these specific processing steps, the present invention can realize a live sports broadcast that is full of realism and incorporates the user's emotions.

[0190] Example 2

[0191] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0192] Conventional sports commentary systems have been unable to reflect the user's emotions in the acquisition and analysis of video footage, and in the generation and distribution of commentary, resulting in a lack of realism and a sense of unity. In particular, for sports with small audiences or children's games, expensive dedicated equipment and a specialized broadcasting team are required, making it difficult to easily provide live commentary.

[0193] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0194] In this invention, the server includes means for acquiring video, means for analyzing the acquired video data and detecting the movements and positions of players and the position of the ball, means for generating commentary in natural language based on the detected data, means for integrating the generated commentary with the video and distributing it in real time, and means for recognizing the user's emotions and adjusting the commentary based on the recognized emotion data. This makes it possible to easily provide a realistic live broadcast that reflects the user's emotions.

[0195] "Means for acquiring video footage" refers to the technology and methods for capturing video footage of sports games using devices such as cameras and smartphones.

[0196] "Means for analyzing acquired video data and detecting the movement and position of players and the position of the ball" refers to techniques and methods that use computer vision technology to analyze and identify the movement and position of players and the ball in the video.

[0197] "Means for generating commentary in natural language based on detected data" refers to technologies and methods that utilize artificial intelligence and natural language generation algorithms to automatically generate commentary on sports based on analyzed data.

[0198] "Means for integrating generated commentary with video and distributing it in real time" refers to a technology or method for overlaying generated commentary on video data, encoding it, and streaming it in real time using a distribution protocol.

[0199] "Means for recognizing the user's emotions and adjusting the commentary text based on the recognized emotion data" refers to a technology or method that uses an emotion recognition engine to analyze the user's facial expressions and movements, and changes the content and tone of the commentary text in real time according to the emotions.

[0200] The present invention is a system for automatically broadcasting live sports games, providing a realistic commentary by recognizing the user's emotions. This system involves capturing and analyzing video footage, generating commentary in natural language, delivering it in real time, and adjusting the commentary to incorporate the user's emotions. Specific embodiments are described below.

[0201] First, a user uses a smartphone to capture video of a sports game. The smartphone has a camera function and can capture high-quality images. The captured video data is encoded in H.264 format and sent to a server in real time using a low-latency protocol (e.g., RTMP).

[0202] The device encodes the video and sends the encoded video data to the server. The data is compressed and sent with low latency, maintaining real-time performance.

[0203] The server decodes the received video data and temporarily stores it in a buffer. The decoding is performed using the FFmpeg library for efficient data processing. The server then uses computer vision techniques (e.g., OpenCV) to analyze the movement and position of players and the ball in the video. This analysis allows for player tracking, ball location detection, and identification of important events such as goals and points scored.

[0204] Based on the analysis results, the server automatically generates commentary using a natural language generation algorithm (e.g., GPT-4). The generated commentary is associated with a specific event and managed with a timestamp. For example, a commentary such as "Player No. 8 is dribbling the ball" may be generated.

[0205] Furthermore, the server uses an emotion engine (e.g., Microsoft® Azure® Emotion API) to recognize the user's emotions. The emotion engine analyzes the user's facial expressions and movements based on the smartphone camera footage and recognizes their emotions in real time. Based on the recognized emotion data, the content of the commentary is adjusted in real time. For example, if the user is excited, a more exciting commentary such as "Great dribbling! Player 8 is moving forward!" is generated.

[0206] The server then combines the generated commentary with the video and adds a text overlay of the commentary to the video stream. The combined video and commentary are then encoded again using the FFmpeg library and streamed in real time using the HLS protocol. Users can then watch the immersive live broadcast through a dedicated application or a web browser.

[0207] As a concrete example, consider a children's soccer match. The user films the match using a smartphone, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." Furthermore, if the server recognizes that the user is excited, it generates an exciting commentary such as "Great dribble! Player number 8 is moving forward!"

[0208] An example of a prompt to input to a generative AI model is as follows:

[0209] Prompt: "Generate a commentary about a soccer game where player 8 is dribbling the ball forward. If the user is excited, use more exciting language."

[0210] In this way, this system can provide a realistic live broadcast that reflects the user's emotions, making it possible for anyone to easily broadcast live sports without the need for expensive dedicated equipment or a specialized broadcasting team.

[0211] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0212] Step 1:

[0213] The user takes a video.

[0214] Specific operation: A user uses a smartphone to capture video of a sports game. The game is recorded using a high-quality camera, and preparations are made to broadcast the game live in real time. The input is the camera footage, and the output is the video data before encoding.

[0215] Step 2:

[0216] The terminal encodes the video data and sends it to the server.

[0217] Specific operation: The device encodes the captured video data in H.264 format and sends the encoded data to the server in real time using a low-latency protocol (e.g., RTMP). The input is the camera video data, and the output is the encoded video data.

[0218] Step 3:

[0219] The server decodes the video data and temporarily stores it in a buffer.

[0220] Specific operation: The server uses the FFmpeg library to decode the received video data. The decoded data is temporarily stored in a buffer. The input is the encoded video data, and the output is the decoded video data.

[0221] Step 4:

[0222] The server analyzes the video data to detect the player's movements and positions, as well as the position of the ball.

[0223] How it works: The server uses computer vision techniques (e.g., OpenCV) to analyze the movement and position of players and the ball in the video. It tracks players, detects the position of the ball, and identifies important events such as goals and points. The input is the decoded video data, and the output is the analyzed event data.

[0224] Step 5:

[0225] The server generates commentary based on the event data using a natural language generation algorithm.

[0226] Specific operation: The server generates commentary based on the analyzed data using a natural language generation algorithm (e.g., GPT-4). Commentary corresponding to each event is generated and managed along with a timestamp. The input is the analyzed event data, and the output is the generated commentary.

[0227] Step 6:

[0228] The server uses an emotion engine to recognize the user's emotions.

[0229] Specific operation: The server uses an emotion engine (e.g., Microsoft Azure Emotion API) to analyze the user's facial expressions and movements to recognize emotions. Emotional data such as whether the user is excited or not is obtained. The input is the user's camera image, and the output is the recognized emotional data.

[0230] Step 7:

[0231] The server adjusts the commentary based on the recognized emotion data.

[0232] Specific operation: The server adjusts the content and tone of the generated commentary in real time based on the recognized emotion data. If an excited state is detected, the commentary is changed to a more exciting one. The input is the generated commentary and the recognized emotion data, and the output is the adjusted commentary.

[0233] Step 8:

[0234] The server integrates the commentary and video and distributes them in real time.

[0235] Specific operation: The server integrates the generated commentary with the video, adds a text overlay of the commentary to the video stream, encodes the integrated video using the FFmpeg library, and delivers it in real time using the HLS protocol. The input is the adjusted commentary and decoded video data, and the output is the encoded integrated video.

[0236] Step 9:

[0237] The user watches the distributed video.

[0238] Specific operation: A user watches immersive video delivered in real time through a dedicated application or a web browser. The input is the delivered streaming video, and the output is the user's viewing experience.

[0239] (Application example 2)

[0240] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0241] Conventional sports commentary systems require professional commentators and expensive equipment, and are difficult to provide realistic commentary that reflects the user's emotions. Furthermore, it is not possible for users to add appropriate commentary in real time to footage they have shot themselves. For this reason, there is a demand for a system that allows anyone to easily enjoy realistic live broadcasts in real time, even for sporting events with small audiences or children's games.

[0242] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring video, means for analyzing the acquired video data and detecting the movements and positions of players and the position of the ball, means for generating commentary in natural language based on the detected data, means for integrating the generated commentary with the video and distributing it in real time, and means for analyzing the user's emotions using an emotion engine and adjusting the generated commentary. This enables a realistic live broadcast that reflects the emotions of the user in real time in the video that the user has shot.

[0243] "Means for acquiring video" refers to devices or software that capture video data shot by users in real time and transmit it to the system.

[0244] "Means for analysing captured video data" refers to technologies and algorithms for detecting the movements and positions of players and the ball within the captured video and extracting relevant information.

[0245] "Means for detecting player movements and positions, and ball position" refers to a system that uses computer vision technology to track and detect the position of each element in the video.

[0246] "Means for generating commentary in natural language" refers to algorithms or technologies that automatically generate commentary in appropriate natural language in real time based on analyzed data.

[0247] "Means for integrating generated commentary with video and distributing in real time" refers to a system that overlays generated commentary on video, encodes it in real time, and distributes it through streaming.

[0248] "Means of analyzing user emotions using an emotion engine" refers to technology that uses computer vision technology to analyze the user's facial expressions and movements in video and recognize their emotions.

[0249] "Means for adjusting generated commentary" refers to algorithms or techniques that dynamically change the content or tone of already-generated commentary based on perceived user sentiment.

[0250] The present invention provides a system for providing real-time live broadcasts based on sports videos taken by users. The system includes a video acquisition unit, a video data analysis unit, a natural language generation unit, a sentiment analysis unit, a synthesis unit, and a distribution unit.

[0251] The server first receives sports videos in real time, which are captured by users using their smartphones. The smartphones act as a means of capturing the video and transmit the captured video data to the server using a low-latency protocol (e.g., WebRTC). The server also encodes the video data using FFmpeg before transmitting it.

[0252] The server decodes the received video data and analyzes the movements and positions of players and the ball in the video using computer vision technology (for example, using TensorFlow), which allows it to identify important events such as goals and points scored.

[0253] Next, a natural language generation algorithm written in Python is used to automatically generate commentary in real time based on the detected data, which is then saved along with a timestamp.

[0254] Furthermore, an emotion engine (also using TensorFlow) analyzes the user's facial expressions and movements from the camera footage to recognize their emotions. Based on this emotional data, the server adjusts the content and tone of the commentary it generates in real time. For example, if the user is excited, the tone of the commentary will be changed to an exciting one.

[0255] The server then combines the generated commentary with the video and adds it as a text overlay to the video. This combined data is then encoded using FFmpeg and streamed in real time. This process allows users to provide emotionally charged commentary on their own footage.

[0256] As a concrete example, consider a scene in a children's soccer game where player number 8 is dribbling towards the goal. If the user is excited, the system will generate a commentary in real time such as, "Great dribble! Player number 8 is approaching the goal!"

[0257] An example prompt is:

[0258] "Video data: sports_video_stream.mp4",

[0259] "User emotion: Facial expression: Excitement, Action: Applause"

[0260] "Event: Player ID: 8, Action: Dribble, Position: Towards Goal"

[0261] In this way, by combining an emotion engine and real-time distribution technology, the present invention provides a system that allows users to easily broadcast live sports with a sense of realism.

[0262] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0263] Step 1:

[0264] A user uses a smartphone to film a sports game. The video data captured by the user is saved on the device in real time. The input is the video data captured in real time, and the output is the video data stored on the device.

[0265] Step 2:

[0266] The device sends the captured video data to the server using a low-latency protocol. In this process, the video data is encoded using FFmpeg and sent as a real-time stream to the server using the WebRTC protocol. The input is the video data in the device, and the output is the encoded video data sent to the server.

[0267] Step 3:

[0268] The server decodes the received video data and temporarily stores it in a buffer. It uses FFmpeg to decode the video data and make it playable. The input is the encoded video data sent to the server, and the output is the decoded video data stored in the buffer.

[0269] Step 4:

[0270] The server begins video analysis. Using computer vision techniques, it uses a TensorFlow model to analyze the position and movement of players and the ball in the video. This analysis involves tracking players, locating the ball, and identifying important events such as goals and points scored. The input is the decoded video data stored in the buffer, and the output is the analyzed event data.

[0271] Step 5:

[0272] The server generates commentary in natural language based on the parsed data. It uses a natural language generation algorithm written in Python to generate commentary corresponding to specific events along with timestamps. The input is the parsed event data, and the output is the generated commentary.

[0273] Step 6:

[0274] The server collects camera footage to analyze the user's emotions using an emotion engine. Using a TensorFlow model, it analyzes the user's facial expressions and movements to recognize their emotions. The input is the user's camera footage, and the output is the recognized emotion data.

[0275] Step 7:

[0276] The server adjusts the content of the commentary generated based on the recognized emotion data. For example, if the user is excited, the tone of the commentary can be made more exciting. The input is the generated commentary and the recognized emotion data, and the output is the adjusted commentary.

[0277] Step 8:

[0278] The server merges the generated commentary with the video, adding a text overlay of the commentary to the video stream and using FFmpeg to re-encode it. The input is the adjusted commentary and decoded video data, and the output is the merged video data.

[0279] Step 9:

[0280] The server re-encodes the merged video data and delivers it using a real-time streaming protocol, allowing users to watch immersive, live coverage in real time. The input is the merged video data, and the output is a real-time stream delivered to the user.

[0281] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0282] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0283] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0284] [Second embodiment]

[0285] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0286] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0287] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0288] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0289] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0290] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0291] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0292] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0293] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0294] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0295] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0296] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0297] This paper describes a system for automatically broadcasting live sports events, and explains its method in detail. This system performs all steps from video capture and analysis to natural language commentary generation and real-time distribution.

[0298] First, a user uses a smartphone to capture video of a sports game. The device transmits the captured video data to a server in real time. The video data is then encoded and sent to the server using a low-latency protocol.

[0299] The server decodes the received video data and stores the video stream in a buffer. The server then uses computer vision technology to analyze the movement and position of players and the ball in the video. This includes player tracking and ball location detection, and identifies significant events such as goals and points scored.

[0300] Based on the event data generated by the analysis, the server automatically generates commentary using a natural language generation algorithm (e.g., GPT-4). This commentary corresponds to a specific event and is managed along with a timestamp.

[0301] The server then integrates the generated commentary with the video, adding the commentary as a text overlay to the video stream, and then encodes the integrated video and commentary for delivery in real time using a streaming protocol.

[0302] As a concrete example, let's take a children's soccer match. A user uses a smartphone to film the match, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." If player number 8 then scores a goal, the server generates a commentary such as "Goal! Player number 8 scores a great shot!" These commentaries are integrated with the video and distributed to viewers in real time.

[0303] This system allows anyone to easily broadcast live sports without the need for expensive dedicated equipment or a professional broadcasting team. It is particularly useful for sports with small audiences and children's games.

[0304] The processing flow will be explained below.

[0305] Step 1:

[0306] Users use their smartphone cameras to capture footage of sports matches, and the device acquires the footage in real time.

[0307] Step 2:

[0308] The device compresses and encodes the acquired video data and sends it to the server using a low-latency protocol (e.g., RTSP, WebRTC).

[0309] Step 3:

[0310] The server receives the video data sent from the terminal, decodes it, and temporarily stores it in a buffer.

[0311] Step 4:

[0312] The server analyzes the stored video data using computer vision technology (e.g., OpenCV, TensorFlow), which detects the movements and positions of players and the position of the ball.

[0313] Step 5:

[0314] Based on the analysis results, the server generates data about player and ball movements and identifies important events (e.g. goals, shots, etc.).

[0315] Step 6:

[0316] The server uses the generated event data to apply a natural language generation algorithm (e.g., GPT-4) to automatically generate commentary.

[0317] Step 7:

[0318] The server timestamps the generated commentary and synchronizes it with the actual game time.

[0319] Step 8:

[0320] The server adds the commentary as an overlay to the video stream, integrating the video and commentary.

[0321] Step 9:

[0322] The server encodes the aggregated video stream and converts it into a streaming format (e.g., HLS, DASH).

[0323] Step 10:

[0324] The server delivers the encoded video stream in real time, allowing users to watch the game by accessing the live stream on their smartphones or computers.

[0325] These processing steps allow anyone to easily enjoy live coverage of sports games in real time.

[0326] Example 1

[0327] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0328] Conventional live broadcasts of sporting events require expensive equipment and a specialized broadcasting team, and preparation and operation are costly and labor-intensive. Furthermore, it is difficult to broadcast professionally when the audience is small, such as for small-scale sporting events or children's games. For this reason, there is a demand for a system that allows anyone to easily broadcast live sports at low cost.

[0329] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0330] In this invention, the server includes means for acquiring video, means for encoding the acquired video data and transmitting it to the server using a low-latency protocol, means for decoding the received video data and storing it in a buffer, means for analyzing the video data and detecting the movements and positions of players and objects in the sporting event, means for generating a natural language commentary using a generative AI model that creates generated text based on the detected data, means for integrating the generated commentary with the video and adding it as a text overlay, and means for encoding the integrated video and commentary and distributing it in real time using a streaming protocol, thereby enabling anyone to easily broadcast a sporting event in real time without the need for expensive equipment.

[0331] "Means for acquiring video" refers to means for capturing video in real time using a photographic device such as a smartphone or camera.

[0332] "Means for encoding acquired video data and transmitting it to a server using a low-latency protocol" refers to means for encoding video data based on a compression method and transmitting it to a server using a low-latency data transmission protocol such as WebRTC or RTMP.

[0333] "Means for decoding received video data and storing it in a buffer" refers to means for the server to decode encoded video data received from a client terminal and store it in a temporary memory space (buffer).

[0334] "Means for analyzing the video data and detecting the movements and positions of players and objects in the sporting event" refers to means for identifying the positions and movements of players and the ball in the video using computer vision technology and machine learning algorithms.

[0335] A "means for generating natural language commentary using a generative AI model that creates generated text based on detected data" means a means for using the analyzed data as input and a generative AI model (e.g., a deep learning model) to create a natural language commentary that includes details of the event.

[0336] The "means for integrating the generated commentary with the video and adding it as a text overlay" refers to a means for visually displaying the generated commentary as text overlaid at a specific position on the video stream.

[0337] "Means for encoding the integrated video and commentary and delivering it in real time using a streaming protocol" refers to a means for re-encoding the video stream with the added text overlay and delivering it to viewers in real time using a streaming protocol such as HLS or DASH.

[0338] This invention is a system for automatically broadcasting live sports events, and is realized using specific devices and algorithms. This system performs all processes from video capture and analysis to natural language commentary generation and real-time distribution.

[0339] First, a user uses a smartphone to capture video of a sports game. The smartphone used here must be equipped with a camera function and a streaming application. The captured video data is sent from the device to a server in real time. At this time, the device encodes the video data using a low-latency codec such as H.264 or H.265, and sends it to the server using a low-latency protocol such as WebRTC or RTMP.

[0340] The server receives the encoded video data, decodes it using a decoding library such as FFmpeg, and temporarily stores the video stream in a buffer. The server then uses computer vision techniques to analyze the movement and position of players and the ball in the video. Specific algorithms used include OpenCV and YOLO. This analysis process includes player tracking and ball position detection, and identifies important events such as goals and points.

[0341] Based on the analysis data, the server uses a generative AI model (such as GPT-4) to generate commentary. The generated commentary is managed along with a timestamp. For example, if the analysis detects that player number 8 is dribbling the ball, the prompt "Player number 8 is dribbling the ball" is generated. If player number 8 scores a goal, the commentary will read "Goal! Player number 8 made a great shot!"

[0342] The server then uses a library such as FFmpeg to add the generated commentary as a text overlay to the video stream. The commentary is integrated into the video at the appropriate time based on the timestamp. After integration, the server distributes the encoded video stream in real time using a streaming protocol such as HLS or DASH. Viewers can then watch the distributed video in real time using devices such as smartphones and PCs.

[0343] As a concrete example, let's take a children's soccer match. A user uses a smartphone to film the match, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." If player number 8 then scores a goal, the server generates a commentary such as "Goal! Player number 8 scores a great shot!" These commentaries are integrated with the video and distributed to viewers in real time.

[0344] As a result, this system allows anyone to easily broadcast live sports without the need for expensive dedicated equipment or a specialized broadcasting team. This is particularly useful for sports with small audiences or for children's games.

[0345] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0346] Step 1:

[0347] Video shooting by users

[0348] Input: Sports event footage, smartphone

[0349] Processing: A user uses a smartphone to capture video of a sporting event in real time, where the smartphone has a camera function and an application installed to capture the video.

[0350] Output: Real-time video data captured by a smartphone camera

[0351] Step 2:

[0352] Video encoding and transmission by the device

[0353] Input: Real-time video data

[0354] Processing: The device encodes the captured video using a codec such as H.264 or H.265 to compress the video data. It then transmits the data to the server using a low-latency protocol such as WebRTC or RTMP. It also monitors the network connection status to minimize latency.

[0355] Output: The encoded video data is sent to the server.

[0356] Step 3:

[0357] Video reception and decoding by the server

[0358] Input: Encoded video data sent from the device

[0359] Processing: The server receives the encoded video data sent from the device, decodes the video data using a decoding library such as FFmpeg, and stores it in a buffer memory in real time. This information is used for later analysis and integration processing.

[0360] Output: Decoded video data stored in a buffer

[0361] Step 4:

[0362] Server-based video analysis

[0363] Input: Decoded video data stored in a buffer

[0364] Processing: The server uses computer vision techniques (e.g., OpenCV or YOLO) to analyze the video data. Specifically, it detects the tracking, movement, and position of players and the ball within the video. This analysis identifies important events, such as goals scored.

[0365] Output: Analysis data (player position, ball position, event information, etc.)

[0366] Step 5:

[0367] Server-based natural language generation

[0368] Input: Analysis data (player position, ball position, event information, etc.)

[0369] Processing: The server uses a generative AI model (e.g., GPT-4) to generate commentary in natural language based on the analyzed data. For each specific event, a prompt is provided as input, and a natural-sounding commentary is generated based on that. For example, the commentary generated might be, "Player No. 8 is dribbling the ball."

[0370] Output: Generated commentary (with timestamp)

[0371] Step 6:

[0372] Server-based integration of video and commentary

[0373] Input: Decoded video data, generated commentary

[0374] Processing: The server adds the generated commentary to the video stream as a text overlay, using FFmpeg filters to insert the text at the right time and in the right position, based on the timestamps.

[0375] Output: Video stream with text overlay added

[0376] Step 7:

[0377] Server-based video content distribution

[0378] Input: Video stream with text overlay added

[0379] Processing: The server re-encodes the combined video stream and delivers it in real time using a streaming protocol such as HLS or DASH, allowing viewers to enjoy the video and commentary smoothly without delay.

[0380] Output: Real-time integrated video stream

[0381] (Application example 1)

[0382] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0383] Conventional monitoring systems require manual confirmation, making it difficult to respond quickly. It is also difficult to generate warning messages in real time or automatically recognize suspicious activity, making it difficult to implement effective security measures. The goal of this project is to solve these issues and provide a more effective and efficient security system.

[0384] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0385] In this invention, the server includes a means for acquiring video, a means for analyzing the acquired video data to detect the movement and position of a target or the position of an object, a means for generating a warning message in natural language based on the detected data, and a means for integrating the generated warning message with the video and distributing it in real time, thereby enabling the rapid detection of suspicious movements and the immediate issuance of a warning.

[0386] "Means for acquiring video" refers to devices that capture video data in real time using cameras, sensors, etc.

[0387] "Means for analyzing acquired video data and detecting the movement and position of a target or object" refers to a device or software that uses computer vision technology and algorithms to detect the movement and position of a specific target or object in a video and analyze its behavior.

[0388] A "means for generating natural language warning text based on detected data" is a device or program that uses generative AI models or natural language processing techniques to automatically generate appropriate warning messages based on detected events or situations.

[0389] "Means for integrating generated warning text with video and delivering it in real time" refers to a technology or system that overlays generated warning text onto video data and delivers it to users in real time.

[0390] The "means for encoding video data and transmitting it to the server" refers to a device or software that converts (encodes) the captured video data into an appropriate format and transmits it to the server using a low-latency protocol.

[0391] "Means for detecting the position of a target or object using computer vision technology" refers to technologies or programs that use image processing algorithms or machine learning models to accurately detect the position of a specific target or object within a video.

[0392] The purpose of this invention is to automate and improve the efficiency of a monitoring system. We will now explain in detail how the server, terminals, and users cooperate to realize this system.

[0393] First, a surveillance camera, smartphone, or smart glasses camera captures video in real time. This is the "means of video capture." The device captures the video data and sends it to a streaming server using a low-latency protocol (e.g., RTMP).

[0394] The server then analyzes the received video data. Specifically, it uses computer vision technology such as OpenCV to detect specific objects in the video, such as people, vehicles, or suspicious movements. This is "a means of analyzing acquired video data and detecting the movement and location of objects."

[0395] Based on the detected data, a generative AI model (e.g., GPT-4) is used to generate a natural language warning. This is a "means of generating a natural language warning based on the detected data." The following prompts can be used:

[0396] "We've detected suspicious activity. What should we do?"

[0397] "I detected suspicious activity at night. What is the appropriate response?"

[0398] The generated warning text is integrated with the video data. Specifically, the warning text is added as a text overlay to the video stream. This is the "means of integrating the generated warning text with the video and delivering it in real time." This integrated data is delivered using a real-time streaming protocol (such as HLS or DASH), allowing users to check the monitoring results in real time.

[0399] Additionally, when important events occur, emergency response teams can be alerted using push notification services such as Firebase Cloud Messaging.

[0400] For example, if a surveillance camera detects motion at night, the server analyzes the video data to detect suspicious activity. Using this information, a generative AI model generates a warning message: "Suspicious activity detected. Please check immediately." This message is overlaid on the video stream and distributed in real time, simultaneously notifying emergency response teams.

[0401] In this way, the present invention realizes an efficient monitoring system and effective security measures.

[0402] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0403] Step 1:

[0404] The device (smartphone, smart glasses) captures video in real time using a camera. The input is the camera image, and the output is the captured video data. This video data is sent to a streaming server using a low-latency protocol (e.g., RTMP).

[0405] Step 2:

[0406] The server decodes the received video data. The input is the encoded video data, and the output is the decoded video frames. This is handled efficiently by the streaming protocol, for example, using the FFmpeg library for decoding.

[0407] Step 3:

[0408] The server analyzes the decoded video frames using computer vision technology. The input is the video frame, and the output is detection data such as the movement and position of the target and the location of the object. For example, OpenCV is used to detect suspicious movements and abnormal behavior. Specifically, it uses background subtraction to extract moving objects, and performs facial recognition and motion detection.

[0409] Step 4:

[0410] The server generates a warning in natural language based on the detection data. The input is the detection data and the output is the warning. The warning is generated using a generative AI model (e.g., GPT-4). As a specific example, an appropriate warning is generated based on the prompt, "Suspicious activity has been detected. What should be done?"

[0411] Step 5:

[0412] The server overlays the generated warning text onto the video frame. The input is the warning text and the video frame, and the output is the video with the warning text overlaid. Specifically, we add text to the video using OpenCV.

[0413] Step 6:

[0414] The server re-encodes the overlaid video and delivers it in real time. The input is the overlaid video, and the output is the streamed video. It is delivered to users using a streaming protocol (e.g., HLS or DASH).

[0415] Step 7:

[0416] The server sends a push notification when an important event occurs. The input is the event data, and the output is the notification message. A push notification service such as Firebase Cloud Messaging is used to send an alert to an emergency response team. Specifically, the alert text is set as the message and a Firebase API is called.

[0417] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0418] This invention is a system for automatically broadcasting live sports commentary, and combines it with an emotion engine that recognizes user emotions. This system enables video capture and analysis, commentary generation in natural language, real-time distribution, and even more realistic commentary that incorporates user emotions.

[0419] First, a user uses a smartphone to capture video of a sports game. The device transmits the captured video data to a server in real time, where it is encoded and transmitted to the server using a low-latency protocol.

[0420] The server decodes the received video data and temporarily stores it in a buffer. The server then uses computer vision technology to analyze the movement and position of players and the ball in the video. This analysis allows for player tracking, ball location detection, and identification of significant events such as goals and points scored.

[0421] The server then uses a natural language generation algorithm to automatically generate commentary based on the analysis results, which corresponds to a specific event and is maintained with a timestamp.

[0422] The server then uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial expressions and movements based on the camera footage to recognize their emotions. Based on the recognized emotion data, the content of the commentary can be adjusted in real time. For example, if the user is excited, the tone and content of the commentary can be made more exciting.

[0423] The server merges the generated commentary with the video, adds a text overlay of the commentary to the video stream, and then encodes the merged video and commentary for delivery in real time using a streaming protocol.

[0424] As a concrete example, let's take a children's soccer game. The user uses a smartphone to film the game, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." Furthermore, if the server recognizes that the user is excited, it generates an exciting commentary such as "Great dribbling! Player number 8 is moving forward!"

[0425] In this way, by using the emotion engine, this system can provide more realistic live broadcasts that reflect the user's emotions. This makes it possible for anyone to easily broadcast live sports without the need for expensive dedicated equipment or a specialized broadcasting team. This is particularly useful for sports with small audiences and children's games.

[0426] The processing flow will be explained below.

[0427] Step 1:

[0428] Users use their smartphone cameras to capture footage of sports matches, and the device captures the footage in real time and stores it in an internal buffer.

[0429] Step 2:

[0430] The device compresses and encodes the captured video data and sequentially transmits it to the server using a low-latency protocol (e.g., RTSP, WebRTC).

[0431] Step 3:

[0432] The server receives the video data sent from the terminal, decodes it, and stores the decoded video data in the server's buffer.

[0433] Step 4:

[0434] The server uses computer vision techniques (e.g., OpenCV, TensorFlow) to analyze the video data, determining the movements and positions of players and the position of the ball.

[0435] Step 5:

[0436] Based on the analysis results, the server generates event data about player and ball movements, e.g., "Player No. 8 is dribbling the ball" or "Player No. 8 has scored a goal."

[0437] Step 6:

[0438] The server uses a natural language generation algorithm (e.g., GPT-4) to generate commentary based on the generated event data. Examples include, "Player 8 is dribbling the ball," and "Goal! Player 8 made a great shot!"

[0439] Step 7:

[0440] The server recognizes the user's emotions using an emotion engine within the server, which analyzes the video data of the user's face and voice to identify his emotional state (e.g., excitement, joy, surprise).

[0441] Step 8:

[0442] The server adjusts the content and tone of the commentary based on the user's emotional data. For example, if the user is excited, the server will generate an exciting commentary such as, "Player No. 8 is showing off some great dribbling skills!"

[0443] Step 9:

[0444] The server combines the generated commentary with the decoded video data, and the commentary is overlaid onto the video stream with a timestamp.

[0445] Step 10:

[0446] The server compresses and encodes the aggregated video stream and delivers it in real time using a streaming protocol (e.g., HLS, DASH).

[0447] Step 11:

[0448] Users can watch the video stream in real time using their smartphones or PCs, and can enjoy a realistic commentary that reflects their own emotions.

[0449] By using these specific processing steps, the present invention can realize a live sports broadcast that is full of realism and incorporates the user's emotions.

[0450] Example 2

[0451] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0452] Conventional sports commentary systems have been unable to reflect the user's emotions in the acquisition and analysis of video footage, and in the generation and distribution of commentary, resulting in a lack of realism and a sense of unity. In particular, for sports with small audiences or children's games, expensive dedicated equipment and a specialized broadcasting team are required, making it difficult to easily provide live commentary.

[0453] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0454] In this invention, the server includes means for acquiring video, means for analyzing the acquired video data and detecting the movements and positions of players and the position of the ball, means for generating commentary in natural language based on the detected data, means for integrating the generated commentary with the video and distributing it in real time, and means for recognizing the user's emotions and adjusting the commentary based on the recognized emotion data. This makes it possible to easily provide a realistic live broadcast that reflects the user's emotions.

[0455] "Means for acquiring video footage" refers to the technology and methods for capturing video footage of sports games using devices such as cameras and smartphones.

[0456] "Means for analyzing acquired video data and detecting the movement and position of players and the position of the ball" refers to techniques and methods that use computer vision technology to analyze and identify the movement and position of players and the ball in the video.

[0457] "Means for generating commentary in natural language based on detected data" refers to technologies and methods that utilize artificial intelligence and natural language generation algorithms to automatically generate commentary on sports based on analyzed data.

[0458] "Means for integrating generated commentary with video and distributing it in real time" refers to a technology or method for overlaying generated commentary on video data, encoding it, and streaming it in real time using a distribution protocol.

[0459] "Means for recognizing the user's emotions and adjusting the commentary text based on the recognized emotion data" refers to a technology or method that uses an emotion recognition engine to analyze the user's facial expressions and movements, and changes the content and tone of the commentary text in real time according to the emotions.

[0460] The present invention is a system for automatically broadcasting live sports games, providing a realistic commentary by recognizing the user's emotions. This system involves capturing and analyzing video footage, generating commentary in natural language, delivering it in real time, and adjusting the commentary to incorporate the user's emotions. Specific embodiments are described below.

[0461] First, a user uses a smartphone to capture video of a sports game. The smartphone has a camera function and can capture high-quality images. The captured video data is encoded in H.264 format and sent to a server in real time using a low-latency protocol (e.g., RTMP).

[0462] The device encodes the video and sends the encoded video data to the server. The data is compressed and sent with low latency, maintaining real-time performance.

[0463] The server decodes the received video data and temporarily stores it in a buffer. The decoding is performed using the FFmpeg library for efficient data processing. The server then uses computer vision techniques (e.g., OpenCV) to analyze the movement and position of players and the ball in the video. This analysis allows for player tracking, ball location detection, and identification of important events such as goals and points scored.

[0464] Based on the analysis results, the server automatically generates commentary using a natural language generation algorithm (e.g., GPT-4). The generated commentary is associated with a specific event and managed with a timestamp. For example, a commentary such as "Player No. 8 is dribbling the ball" may be generated.

[0465] Furthermore, the server uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the user's emotions. The emotion engine analyzes the user's facial expressions and movements based on the smartphone camera footage and recognizes their emotions in real time. Based on the recognized emotion data, the content of the commentary is adjusted in real time. For example, if the user is excited, a more exciting commentary such as "Great dribbling! Player 8 is moving forward!" is generated.

[0466] The server then combines the generated commentary with the video and adds a text overlay of the commentary to the video stream. The combined video and commentary are then encoded again using the FFmpeg library and streamed in real time using the HLS protocol. Users can then watch the immersive live broadcast through a dedicated application or a web browser.

[0467] As a concrete example, consider a children's soccer match. The user films the match using a smartphone, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." Furthermore, if the server recognizes that the user is excited, it generates an exciting commentary such as "Great dribble! Player number 8 is moving forward!"

[0468] An example of a prompt to input to a generative AI model is as follows:

[0469] Prompt: "Generate a commentary about a soccer game where player 8 is dribbling the ball forward. If the user is excited, use more exciting language."

[0470] In this way, this system can provide a realistic live broadcast that reflects the user's emotions, making it possible for anyone to easily broadcast live sports without the need for expensive dedicated equipment or a specialized broadcasting team.

[0471] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0472] Step 1:

[0473] The user takes a video.

[0474] Specific operation: A user uses a smartphone to capture video of a sports game. The game is recorded using a high-quality camera, and preparations are made to broadcast the game live in real time. The input is the camera footage, and the output is the video data before encoding.

[0475] Step 2:

[0476] The terminal encodes the video data and sends it to the server.

[0477] Specific operation: The device encodes the captured video data in H.264 format and sends the encoded data to the server in real time using a low-latency protocol (e.g., RTMP). The input is the camera video data, and the output is the encoded video data.

[0478] Step 3:

[0479] The server decodes the video data and temporarily stores it in a buffer.

[0480] Specific operation: The server uses the FFmpeg library to decode the received video data. The decoded data is temporarily stored in a buffer. The input is the encoded video data, and the output is the decoded video data.

[0481] Step 4:

[0482] The server analyzes the video data to detect the player's movements and positions, as well as the position of the ball.

[0483] How it works: The server uses computer vision techniques (e.g., OpenCV) to analyze the movement and position of players and the ball in the video. It tracks players, detects the position of the ball, and identifies important events such as goals and points. The input is the decoded video data, and the output is the analyzed event data.

[0484] Step 5:

[0485] The server generates commentary based on the event data using a natural language generation algorithm.

[0486] Specific operation: The server generates commentary based on the analyzed data using a natural language generation algorithm (e.g., GPT-4). Commentary corresponding to each event is generated and managed along with a timestamp. The input is the analyzed event data, and the output is the generated commentary.

[0487] Step 6:

[0488] The server uses an emotion engine to recognize the user's emotions.

[0489] Specific operation: The server uses an emotion engine (e.g., Microsoft Azure Emotion API) to analyze the user's facial expressions and movements to recognize emotions. Emotional data such as whether the user is excited or not is obtained. The input is the user's camera image, and the output is the recognized emotional data.

[0490] Step 7:

[0491] The server adjusts the commentary based on the recognized emotion data.

[0492] Specific operation: The server adjusts the content and tone of the generated commentary in real time based on the recognized emotion data. If an excited state is detected, the commentary is changed to a more exciting one. The input is the generated commentary and the recognized emotion data, and the output is the adjusted commentary.

[0493] Step 8:

[0494] The server integrates the commentary and video and distributes them in real time.

[0495] Specific operation: The server integrates the generated commentary with the video, adds a text overlay of the commentary to the video stream, encodes the integrated video using the FFmpeg library, and delivers it in real time using the HLS protocol. The input is the adjusted commentary and decoded video data, and the output is the encoded integrated video.

[0496] Step 9:

[0497] The user watches the distributed video.

[0498] Specific operation: A user watches immersive video delivered in real time through a dedicated application or a web browser. The input is the delivered streaming video, and the output is the user's viewing experience.

[0499] (Application example 2)

[0500] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0501] Conventional sports commentary systems require professional commentators and expensive equipment, and are difficult to provide realistic commentary that reflects the user's emotions. Furthermore, it is not possible for users to add appropriate commentary in real time to footage they have shot themselves. For this reason, there is a demand for a system that allows anyone to easily enjoy realistic live broadcasts in real time, even for sporting events with small audiences or children's games.

[0502] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring video, means for analyzing the acquired video data and detecting the movements and positions of players and the position of the ball, means for generating commentary in natural language based on the detected data, means for integrating the generated commentary with the video and distributing it in real time, and means for analyzing the user's emotions using an emotion engine and adjusting the generated commentary. This enables a realistic live broadcast that reflects the emotions of the user in real time in the video that the user has shot.

[0503] "Means for acquiring video" refers to devices or software that capture video data shot by users in real time and transmit it to the system.

[0504] "Means for analysing captured video data" refers to technologies and algorithms for detecting the movements and positions of players and the ball within the captured video and extracting relevant information.

[0505] "Means for detecting player movements and positions, and ball position" refers to a system that uses computer vision technology to track and detect the position of each element in the video.

[0506] "Means for generating commentary in natural language" refers to algorithms or technologies that automatically generate commentary in appropriate natural language in real time based on analyzed data.

[0507] "Means for integrating generated commentary with video and distributing in real time" refers to a system that overlays generated commentary on video, encodes it in real time, and distributes it through streaming.

[0508] "Means of analyzing user emotions using an emotion engine" refers to technology that uses computer vision technology to analyze the user's facial expressions and movements in video and recognize their emotions.

[0509] "Means for adjusting generated commentary" refers to algorithms or techniques that dynamically change the content or tone of already-generated commentary based on perceived user sentiment.

[0510] The present invention provides a system for providing real-time live broadcasts based on sports videos taken by users. The system includes a video acquisition unit, a video data analysis unit, a natural language generation unit, a sentiment analysis unit, a synthesis unit, and a distribution unit.

[0511] The server first receives sports videos in real time, which are captured by users using their smartphones. The smartphones act as a means of capturing the video and transmit the captured video data to the server using a low-latency protocol (e.g., WebRTC). The server also encodes the video data using FFmpeg before transmitting it.

[0512] The server decodes the received video data and analyzes the movements and positions of players and the ball in the video using computer vision technology (for example, using TensorFlow), which allows it to identify important events such as goals and points scored.

[0513] Next, a natural language generation algorithm written in Python is used to automatically generate commentary in real time based on the detected data, which is then saved along with a timestamp.

[0514] Furthermore, an emotion engine (also using TensorFlow) analyzes the user's facial expressions and movements from the camera footage to recognize their emotions. Based on this emotional data, the server adjusts the content and tone of the commentary it generates in real time. For example, if the user is excited, the tone of the commentary will be changed to an exciting one.

[0515] The server then combines the generated commentary with the video and adds it as a text overlay to the video. This combined data is then encoded using FFmpeg and streamed in real time. This process allows users to provide emotionally charged commentary on their own footage.

[0516] As a concrete example, consider a scene in a children's soccer game where player number 8 is dribbling towards the goal. If the user is excited, the system will generate a commentary in real time such as, "Great dribble! Player number 8 is approaching the goal!"

[0517] An example prompt is:

[0518] "Video data: sports_video_stream.mp4",

[0519] "User emotion: Facial expression: Excitement, Action: Applause"

[0520] "Event: Player ID: 8, Action: Dribble, Position: Towards Goal"

[0521] In this way, by combining an emotion engine and real-time distribution technology, the present invention provides a system that allows users to easily broadcast live sports with a sense of realism.

[0522] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0523] Step 1:

[0524] A user uses a smartphone to film a sports game. The video data captured by the user is saved on the device in real time. The input is the video data captured in real time, and the output is the video data stored on the device.

[0525] Step 2:

[0526] The device sends the captured video data to the server using a low-latency protocol. In this process, the video data is encoded using FFmpeg and sent as a real-time stream to the server using the WebRTC protocol. The input is the video data in the device, and the output is the encoded video data sent to the server.

[0527] Step 3:

[0528] The server decodes the received video data and temporarily stores it in a buffer. It uses FFmpeg to decode the video data and make it playable. The input is the encoded video data sent to the server, and the output is the decoded video data stored in the buffer.

[0529] Step 4:

[0530] The server begins video analysis. Using computer vision techniques, it uses a TensorFlow model to analyze the position and movement of players and the ball in the video. This analysis involves tracking players, locating the ball, and identifying important events such as goals and points scored. The input is the decoded video data stored in the buffer, and the output is the analyzed event data.

[0531] Step 5:

[0532] The server generates commentary in natural language based on the parsed data. It uses a natural language generation algorithm written in Python to generate commentary corresponding to specific events along with timestamps. The input is the parsed event data, and the output is the generated commentary.

[0533] Step 6:

[0534] The server collects camera footage to analyze the user's emotions using an emotion engine. Using a TensorFlow model, it analyzes the user's facial expressions and movements to recognize their emotions. The input is the user's camera footage, and the output is the recognized emotion data.

[0535] Step 7:

[0536] The server adjusts the content of the commentary generated based on the recognized emotion data. For example, if the user is excited, the tone of the commentary can be made more exciting. The input is the generated commentary and the recognized emotion data, and the output is the adjusted commentary.

[0537] Step 8:

[0538] The server merges the generated commentary with the video, adding a text overlay of the commentary to the video stream and using FFmpeg to re-encode it. The input is the adjusted commentary and decoded video data, and the output is the merged video data.

[0539] Step 9:

[0540] The server re-encodes the merged video data and delivers it using a real-time streaming protocol, allowing users to watch immersive, live coverage in real time. The input is the merged video data, and the output is a real-time stream delivered to the user.

[0541] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0542] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0543] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0544] [Third embodiment]

[0545] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0546] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0547] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0548] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0549] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0550] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0551] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0552] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0553] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0554] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0555] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0556] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0557] This paper describes a system for automatically broadcasting live sports events, and explains its method in detail. This system performs all steps from video capture and analysis to natural language commentary generation and real-time distribution.

[0558] First, a user uses a smartphone to capture video of a sports game. The device transmits the captured video data to a server in real time. The video data is then encoded and sent to the server using a low-latency protocol.

[0559] The server decodes the received video data and stores the video stream in a buffer. The server then uses computer vision technology to analyze the movement and position of players and the ball in the video. This includes player tracking and ball location detection, and identifies significant events such as goals and points scored.

[0560] Based on the event data generated by the analysis, the server automatically generates commentary using a natural language generation algorithm (e.g., GPT-4). This commentary corresponds to a specific event and is managed along with a timestamp.

[0561] The server then integrates the generated commentary with the video, adding the commentary as a text overlay to the video stream, and then encodes the integrated video and commentary for delivery in real time using a streaming protocol.

[0562] As a concrete example, let's take a children's soccer match. A user uses a smartphone to film the match, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." If player number 8 then scores a goal, the server generates a commentary such as "Goal! Player number 8 scores a great shot!" These commentaries are integrated with the video and distributed to viewers in real time.

[0563] This system allows anyone to easily broadcast live sports without the need for expensive dedicated equipment or a professional broadcasting team. It is particularly useful for sports with small audiences and children's games.

[0564] The processing flow will be explained below.

[0565] Step 1:

[0566] Users use their smartphone cameras to capture footage of sports matches, and the device acquires the footage in real time.

[0567] Step 2:

[0568] The device compresses and encodes the acquired video data and sends it to the server using a low-latency protocol (e.g., RTSP, WebRTC).

[0569] Step 3:

[0570] The server receives the video data sent from the terminal, decodes it, and temporarily stores it in a buffer.

[0571] Step 4:

[0572] The server analyzes the stored video data using computer vision technology (e.g., OpenCV, TensorFlow), which detects the movements and positions of players and the position of the ball.

[0573] Step 5:

[0574] Based on the analysis results, the server generates data about player and ball movements and identifies important events (e.g. goals, shots, etc.).

[0575] Step 6:

[0576] The server uses the generated event data to apply a natural language generation algorithm (e.g., GPT-4) to automatically generate commentary.

[0577] Step 7:

[0578] The server timestamps the generated commentary and synchronizes it with the actual game time.

[0579] Step 8:

[0580] The server adds the commentary as an overlay to the video stream, integrating the video and commentary.

[0581] Step 9:

[0582] The server encodes the aggregated video stream and converts it into a streaming format (e.g., HLS, DASH).

[0583] Step 10:

[0584] The server delivers the encoded video stream in real time, allowing users to watch the game by accessing the live stream on their smartphones or computers.

[0585] These processing steps allow anyone to easily enjoy live coverage of sports games in real time.

[0586] Example 1

[0587] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0588] Conventional live broadcasts of sporting events require expensive equipment and a specialized broadcasting team, and preparation and operation are costly and labor-intensive. Furthermore, it is difficult to broadcast professionally when the audience is small, such as for small-scale sporting events or children's games. For this reason, there is a demand for a system that allows anyone to easily broadcast live sports at low cost.

[0589] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0590] In this invention, the server includes means for acquiring video, means for encoding the acquired video data and transmitting it to the server using a low-latency protocol, means for decoding the received video data and storing it in a buffer, means for analyzing the video data and detecting the movements and positions of players and objects in the sporting event, means for generating a natural language commentary using a generative AI model that creates generated text based on the detected data, means for integrating the generated commentary with the video and adding it as a text overlay, and means for encoding the integrated video and commentary and distributing it in real time using a streaming protocol, thereby enabling anyone to easily broadcast a sporting event in real time without the need for expensive equipment.

[0591] "Means for acquiring video" refers to means for capturing video in real time using a photographic device such as a smartphone or camera.

[0592] "Means for encoding acquired video data and transmitting it to a server using a low-latency protocol" refers to means for encoding video data based on a compression method and transmitting it to a server using a low-latency data transmission protocol such as WebRTC or RTMP.

[0593] "Means for decoding received video data and storing it in a buffer" refers to means for the server to decode encoded video data received from a client terminal and store it in a temporary memory space (buffer).

[0594] "Means for analyzing the video data and detecting the movements and positions of players and objects in the sporting event" refers to means for identifying the positions and movements of players and the ball in the video using computer vision technology and machine learning algorithms.

[0595] A "means for generating natural language commentary using a generative AI model that creates generated text based on detected data" means a means for using the analyzed data as input and a generative AI model (e.g., a deep learning model) to create a natural language commentary that includes details of the event.

[0596] The "means for integrating the generated commentary with the video and adding it as a text overlay" refers to a means for visually displaying the generated commentary as text overlaid at a specific position on the video stream.

[0597] "Means for encoding the integrated video and commentary and delivering it in real time using a streaming protocol" refers to a means for re-encoding the video stream with the added text overlay and delivering it to viewers in real time using a streaming protocol such as HLS or DASH.

[0598] This invention is a system for automatically broadcasting live sports events, and is realized using specific devices and algorithms. This system performs all processes from video capture and analysis to natural language commentary generation and real-time distribution.

[0599] First, a user uses a smartphone to capture video of a sports game. The smartphone used here must be equipped with a camera function and a streaming application. The captured video data is sent from the device to a server in real time. At this time, the device encodes the video data using a low-latency codec such as H.264 or H.265, and sends it to the server using a low-latency protocol such as WebRTC or RTMP.

[0600] The server receives the encoded video data, decodes it using a decoding library such as FFmpeg, and temporarily stores the video stream in a buffer. The server then uses computer vision techniques to analyze the movement and position of players and the ball in the video. Specific algorithms used include OpenCV and YOLO. This analysis process includes player tracking and ball position detection, and identifies important events such as goals and points.

[0601] Based on the analysis data, the server uses a generative AI model (such as GPT-4) to generate commentary. The generated commentary is managed along with a timestamp. For example, if the analysis detects that player number 8 is dribbling the ball, the prompt "Player number 8 is dribbling the ball" is generated. If player number 8 scores a goal, the commentary will read "Goal! Player number 8 made a great shot!"

[0602] The server then uses a library such as FFmpeg to add the generated commentary as a text overlay to the video stream. The commentary is integrated into the video at the appropriate time based on the timestamp. After integration, the server distributes the encoded video stream in real time using a streaming protocol such as HLS or DASH. Viewers can then watch the distributed video in real time using devices such as smartphones and PCs.

[0603] As a concrete example, let's take a children's soccer match. A user uses a smartphone to film the match, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." If player number 8 then scores a goal, the server generates a commentary such as "Goal! Player number 8 scores a great shot!" These commentaries are integrated with the video and distributed to viewers in real time.

[0604] As a result, this system allows anyone to easily broadcast live sports without the need for expensive dedicated equipment or a specialized broadcasting team. This is particularly useful for sports with small audiences or for children's games.

[0605] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0606] Step 1:

[0607] Video shooting by users

[0608] Input: Sports event footage, smartphone

[0609] Processing: A user uses a smartphone to capture video of a sporting event in real time, where the smartphone has a camera function and an application installed to capture the video.

[0610] Output: Real-time video data captured by a smartphone camera

[0611] Step 2:

[0612] Video encoding and transmission by the device

[0613] Input: Real-time video data

[0614] Processing: The device encodes the captured video using a codec such as H.264 or H.265 to compress the video data. It then transmits the data to the server using a low-latency protocol such as WebRTC or RTMP. It also monitors the network connection status to minimize latency.

[0615] Output: The encoded video data is sent to the server.

[0616] Step 3:

[0617] Video reception and decoding by the server

[0618] Input: Encoded video data sent from the device

[0619] Processing: The server receives the encoded video data sent from the device, decodes the video data using a decoding library such as FFmpeg, and stores it in a buffer memory in real time. This information is used for later analysis and integration processing.

[0620] Output: Decoded video data stored in a buffer

[0621] Step 4:

[0622] Server-based video analysis

[0623] Input: Decoded video data stored in a buffer

[0624] Processing: The server uses computer vision techniques (e.g., OpenCV or YOLO) to analyze the video data. Specifically, it detects the tracking, movement, and position of players and the ball within the video. This analysis identifies important events, such as goals scored.

[0625] Output: Analysis data (player position, ball position, event information, etc.)

[0626] Step 5:

[0627] Server-based natural language generation

[0628] Input: Analysis data (player position, ball position, event information, etc.)

[0629] Processing: The server uses a generative AI model (e.g., GPT-4) to generate commentary in natural language based on the analyzed data. For each specific event, a prompt is provided as input, and a natural-sounding commentary is generated based on that. For example, the commentary generated might be, "Player No. 8 is dribbling the ball."

[0630] Output: Generated commentary (with timestamp)

[0631] Step 6:

[0632] Server-based integration of video and commentary

[0633] Input: Decoded video data, generated commentary

[0634] Processing: The server adds the generated commentary to the video stream as a text overlay, using FFmpeg filters to insert the text at the right time and in the right position, based on the timestamps.

[0635] Output: Video stream with text overlay added

[0636] Step 7:

[0637] Server-based video content distribution

[0638] Input: Video stream with text overlay added

[0639] Processing: The server re-encodes the combined video stream and delivers it in real time using a streaming protocol such as HLS or DASH, allowing viewers to enjoy the video and commentary smoothly without delay.

[0640] Output: Real-time integrated video stream

[0641] (Application example 1)

[0642] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0643] Conventional monitoring systems require manual confirmation, making it difficult to respond quickly. It is also difficult to generate warning messages in real time or automatically recognize suspicious activity, making it difficult to implement effective security measures. The goal of this project is to solve these issues and provide a more effective and efficient security system.

[0644] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0645] In this invention, the server includes a means for acquiring video, a means for analyzing the acquired video data to detect the movement and position of a target or the position of an object, a means for generating a warning message in natural language based on the detected data, and a means for integrating the generated warning message with the video and distributing it in real time, thereby enabling the rapid detection of suspicious movements and the immediate issuance of a warning.

[0646] "Means for acquiring video" refers to devices that capture video data in real time using cameras, sensors, etc.

[0647] "Means for analyzing acquired video data and detecting the movement and position of a target or object" refers to a device or software that uses computer vision technology and algorithms to detect the movement and position of a specific target or object in a video and analyze its behavior.

[0648] A "means for generating natural language warning text based on detected data" is a device or program that uses generative AI models or natural language processing techniques to automatically generate appropriate warning messages based on detected events or situations.

[0649] "Means for integrating generated warning text with video and delivering it in real time" refers to a technology or system that overlays generated warning text onto video data and delivers it to users in real time.

[0650] The "means for encoding video data and transmitting it to the server" refers to a device or software that converts (encodes) the captured video data into an appropriate format and transmits it to the server using a low-latency protocol.

[0651] "Means for detecting the position of a target or object using computer vision technology" refers to technologies or programs that use image processing algorithms or machine learning models to accurately detect the position of a specific target or object within a video.

[0652] The purpose of this invention is to automate and improve the efficiency of a monitoring system. We will now explain in detail how the server, terminals, and users cooperate to realize this system.

[0653] First, a surveillance camera, smartphone, or smart glasses camera captures video in real time. This is the "means of video capture." The device captures the video data and sends it to a streaming server using a low-latency protocol (e.g., RTMP).

[0654] The server then analyzes the received video data. Specifically, it uses computer vision technology such as OpenCV to detect specific objects in the video, such as people, vehicles, or suspicious movements. This is "a means of analyzing acquired video data and detecting the movement and location of objects."

[0655] Based on the detected data, a generative AI model (e.g., GPT-4) is used to generate a natural language warning. This is a "means of generating a natural language warning based on the detected data." The following prompts can be used:

[0656] "We've detected suspicious activity. What should we do?"

[0657] "I detected suspicious activity at night. What is the appropriate response?"

[0658] The generated warning text is integrated with the video data. Specifically, the warning text is added as a text overlay to the video stream. This is the "means of integrating the generated warning text with the video and delivering it in real time." This integrated data is delivered using a real-time streaming protocol (such as HLS or DASH), allowing users to check the monitoring results in real time.

[0659] Additionally, when important events occur, emergency response teams can be alerted using push notification services such as Firebase Cloud Messaging.

[0660] For example, if a surveillance camera detects motion at night, the server analyzes the video data to detect suspicious activity. Using this information, a generative AI model generates a warning message: "Suspicious activity detected. Please check immediately." This message is overlaid on the video stream and distributed in real time, simultaneously notifying emergency response teams.

[0661] In this way, the present invention realizes an efficient monitoring system and effective security measures.

[0662] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0663] Step 1:

[0664] The device (smartphone, smart glasses) captures video in real time using a camera. The input is the camera image, and the output is the captured video data. This video data is sent to a streaming server using a low-latency protocol (e.g., RTMP).

[0665] Step 2:

[0666] The server decodes the received video data. The input is the encoded video data, and the output is the decoded video frames. This is handled efficiently by the streaming protocol, for example, using the FFmpeg library for decoding.

[0667] Step 3:

[0668] The server analyzes the decoded video frames using computer vision technology. The input is the video frame, and the output is detection data such as the movement and position of the target and the location of the object. For example, OpenCV is used to detect suspicious movements and abnormal behavior. Specifically, it uses background subtraction to extract moving objects, and performs facial recognition and motion detection.

[0669] Step 4:

[0670] The server generates a warning in natural language based on the detection data. The input is the detection data and the output is the warning. The warning is generated using a generative AI model (e.g., GPT-4). As a specific example, an appropriate warning is generated based on the prompt, "Suspicious activity has been detected. What should be done?"

[0671] Step 5:

[0672] The server overlays the generated warning text onto the video frame. The input is the warning text and the video frame, and the output is the video with the warning text overlaid. Specifically, we add text to the video using OpenCV.

[0673] Step 6:

[0674] The server re-encodes the overlaid video and delivers it in real time. The input is the overlaid video, and the output is the streamed video. It is delivered to users using a streaming protocol (e.g., HLS or DASH).

[0675] Step 7:

[0676] The server sends a push notification when an important event occurs. The input is the event data, and the output is the notification message. A push notification service such as Firebase Cloud Messaging is used to send an alert to an emergency response team. Specifically, the alert text is set as the message and a Firebase API is called.

[0677] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0678] This invention is a system for automatically broadcasting live sports commentary, and combines it with an emotion engine that recognizes user emotions. This system enables video capture and analysis, commentary generation in natural language, real-time distribution, and even more realistic commentary that incorporates user emotions.

[0679] First, a user uses a smartphone to capture video of a sports game. The device transmits the captured video data to a server in real time, where it is encoded and transmitted to the server using a low-latency protocol.

[0680] The server decodes the received video data and temporarily stores it in a buffer. The server then uses computer vision technology to analyze the movement and position of players and the ball in the video. This analysis allows for player tracking, ball location detection, and identification of significant events such as goals and points scored.

[0681] The server then uses a natural language generation algorithm to automatically generate commentary based on the analysis results, which corresponds to a specific event and is maintained with a timestamp.

[0682] The server then uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial expressions and movements based on the camera footage to recognize their emotions. Based on the recognized emotion data, the content of the commentary can be adjusted in real time. For example, if the user is excited, the tone and content of the commentary can be made more exciting.

[0683] The server merges the generated commentary with the video, adds a text overlay of the commentary to the video stream, and then encodes the merged video and commentary for delivery in real time using a streaming protocol.

[0684] As a concrete example, let's take a children's soccer game. The user uses a smartphone to film the game, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." Furthermore, if the server recognizes that the user is excited, it generates an exciting commentary such as "Great dribbling! Player number 8 is moving forward!"

[0685] In this way, by using the emotion engine, this system can provide more realistic live broadcasts that reflect the user's emotions. This makes it possible for anyone to easily broadcast live sports without the need for expensive dedicated equipment or a specialized broadcasting team. This is particularly useful for sports with small audiences and children's games.

[0686] The processing flow will be explained below.

[0687] Step 1:

[0688] Users use their smartphone cameras to capture footage of sports matches, and the device captures the footage in real time and stores it in an internal buffer.

[0689] Step 2:

[0690] The device compresses and encodes the captured video data and sequentially transmits it to the server using a low-latency protocol (e.g., RTSP, WebRTC).

[0691] Step 3:

[0692] The server receives the video data sent from the terminal, decodes it, and stores the decoded video data in the server's buffer.

[0693] Step 4:

[0694] The server uses computer vision techniques (e.g., OpenCV, TensorFlow) to analyze the video data, determining the movements and positions of players and the position of the ball.

[0695] Step 5:

[0696] Based on the analysis results, the server generates event data about player and ball movements, e.g., "Player No. 8 is dribbling the ball" or "Player No. 8 has scored a goal."

[0697] Step 6:

[0698] The server uses a natural language generation algorithm (e.g., GPT-4) to generate commentary based on the generated event data. Examples include, "Player 8 is dribbling the ball," and "Goal! Player 8 made a great shot!"

[0699] Step 7:

[0700] The server recognizes the user's emotions using an emotion engine within the server, which analyzes the video data of the user's face and voice to identify his emotional state (e.g., excitement, joy, surprise).

[0701] Step 8:

[0702] The server adjusts the content and tone of the commentary based on the user's emotional data. For example, if the user is excited, the server will generate an exciting commentary such as, "Player No. 8 is showing off some great dribbling skills!"

[0703] Step 9:

[0704] The server combines the generated commentary with the decoded video data, and the commentary is overlaid onto the video stream with a timestamp.

[0705] Step 10:

[0706] The server compresses and encodes the aggregated video stream and delivers it in real time using a streaming protocol (e.g., HLS, DASH).

[0707] Step 11:

[0708] Users can watch the video stream in real time using their smartphones or PCs, and can enjoy a realistic commentary that reflects their own emotions.

[0709] By using these specific processing steps, the present invention can realize a live sports broadcast that is full of realism and incorporates the user's emotions.

[0710] Example 2

[0711] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0712] Conventional sports commentary systems have been unable to reflect the user's emotions in the acquisition and analysis of video footage, and in the generation and distribution of commentary, resulting in a lack of realism and a sense of unity. In particular, for sports with small audiences or children's games, expensive dedicated equipment and a specialized broadcasting team are required, making it difficult to easily provide live commentary.

[0713] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0714] In this invention, the server includes means for acquiring video, means for analyzing the acquired video data and detecting the movements and positions of players and the position of the ball, means for generating commentary in natural language based on the detected data, means for integrating the generated commentary with the video and distributing it in real time, and means for recognizing the user's emotions and adjusting the commentary based on the recognized emotion data. This makes it possible to easily provide a realistic live broadcast that reflects the user's emotions.

[0715] "Means for acquiring video footage" refers to the technology and methods for capturing video footage of sports games using devices such as cameras and smartphones.

[0716] "Means for analyzing acquired video data and detecting the movement and position of players and the position of the ball" refers to techniques and methods that use computer vision technology to analyze and identify the movement and position of players and the ball in the video.

[0717] "Means for generating commentary in natural language based on detected data" refers to technologies and methods that utilize artificial intelligence and natural language generation algorithms to automatically generate commentary on sports based on analyzed data.

[0718] "Means for integrating generated commentary with video and distributing it in real time" refers to a technology or method for overlaying generated commentary on video data, encoding it, and streaming it in real time using a distribution protocol.

[0719] "Means for recognizing the user's emotions and adjusting the commentary text based on the recognized emotion data" refers to a technology or method that uses an emotion recognition engine to analyze the user's facial expressions and movements, and changes the content and tone of the commentary text in real time according to the emotions.

[0720] The present invention is a system for automatically broadcasting live sports games, providing a realistic commentary by recognizing the user's emotions. This system involves capturing and analyzing video footage, generating commentary in natural language, delivering it in real time, and adjusting the commentary to incorporate the user's emotions. Specific embodiments are described below.

[0721] First, a user uses a smartphone to capture video of a sports game. The smartphone has a camera function and can capture high-quality images. The captured video data is encoded in H.264 format and sent to a server in real time using a low-latency protocol (e.g., RTMP).

[0722] The device encodes the video and sends the encoded video data to the server. The data is compressed and sent with low latency, maintaining real-time performance.

[0723] The server decodes the received video data and temporarily stores it in a buffer. The decoding is performed using the FFmpeg library for efficient data processing. The server then uses computer vision techniques (e.g., OpenCV) to analyze the movement and position of players and the ball in the video. This analysis allows for player tracking, ball location detection, and identification of important events such as goals and points scored.

[0724] Based on the analysis results, the server automatically generates commentary using a natural language generation algorithm (e.g., GPT-4). The generated commentary is associated with a specific event and managed with a timestamp. For example, a commentary such as "Player No. 8 is dribbling the ball" may be generated.

[0725] Furthermore, the server uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the user's emotions. The emotion engine analyzes the user's facial expressions and movements based on the smartphone camera footage and recognizes their emotions in real time. Based on the recognized emotion data, the content of the commentary is adjusted in real time. For example, if the user is excited, a more exciting commentary such as "Great dribbling! Player 8 is moving forward!" is generated.

[0726] The server then combines the generated commentary with the video and adds a text overlay of the commentary to the video stream. The combined video and commentary are then encoded again using the FFmpeg library and streamed in real time using the HLS protocol. Users can then watch the immersive live broadcast through a dedicated application or a web browser.

[0727] As a concrete example, consider a children's soccer match. The user films the match using a smartphone, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." Furthermore, if the server recognizes that the user is excited, it generates an exciting commentary such as "Great dribble! Player number 8 is moving forward!"

[0728] An example of a prompt to input to a generative AI model is as follows:

[0729] Prompt: "Generate a commentary about a soccer game where player 8 is dribbling the ball forward. If the user is excited, use more exciting language."

[0730] In this way, this system can provide a realistic live broadcast that reflects the user's emotions, making it possible for anyone to easily broadcast live sports without the need for expensive dedicated equipment or a specialized broadcasting team.

[0731] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0732] Step 1:

[0733] The user takes a video.

[0734] Specific operation: A user uses a smartphone to capture video of a sports game. The game is recorded using a high-quality camera, and preparations are made to broadcast the game live in real time. The input is the camera footage, and the output is the video data before encoding.

[0735] Step 2:

[0736] The terminal encodes the video data and sends it to the server.

[0737] Specific operation: The device encodes the captured video data in H.264 format and sends the encoded data to the server in real time using a low-latency protocol (e.g., RTMP). The input is the camera video data, and the output is the encoded video data.

[0738] Step 3:

[0739] The server decodes the video data and temporarily stores it in a buffer.

[0740] Specific operation: The server uses the FFmpeg library to decode the received video data. The decoded data is temporarily stored in a buffer. The input is the encoded video data, and the output is the decoded video data.

[0741] Step 4:

[0742] The server analyzes the video data to detect the player's movements and positions, as well as the position of the ball.

[0743] How it works: The server uses computer vision techniques (e.g., OpenCV) to analyze the movement and position of players and the ball in the video. It tracks players, detects the position of the ball, and identifies important events such as goals and points. The input is the decoded video data, and the output is the analyzed event data.

[0744] Step 5:

[0745] The server generates commentary based on the event data using a natural language generation algorithm.

[0746] Specific operation: The server generates commentary based on the analyzed data using a natural language generation algorithm (e.g., GPT-4). Commentary corresponding to each event is generated and managed along with a timestamp. The input is the analyzed event data, and the output is the generated commentary.

[0747] Step 6:

[0748] The server uses an emotion engine to recognize the user's emotions.

[0749] Specific operation: The server uses an emotion engine (e.g., Microsoft Azure Emotion API) to analyze the user's facial expressions and movements to recognize emotions. Emotional data such as whether the user is excited or not is obtained. The input is the user's camera image, and the output is the recognized emotional data.

[0750] Step 7:

[0751] The server adjusts the commentary based on the recognized emotion data.

[0752] Specific operation: The server adjusts the content and tone of the generated commentary in real time based on the recognized emotion data. If an excited state is detected, the commentary is changed to a more exciting one. The input is the generated commentary and the recognized emotion data, and the output is the adjusted commentary.

[0753] Step 8:

[0754] The server integrates the commentary and video and distributes them in real time.

[0755] Specific operation: The server integrates the generated commentary with the video, adds a text overlay of the commentary to the video stream, encodes the integrated video using the FFmpeg library, and delivers it in real time using the HLS protocol. The input is the adjusted commentary and decoded video data, and the output is the encoded integrated video.

[0756] Step 9:

[0757] The user watches the distributed video.

[0758] Specific operation: A user watches immersive video delivered in real time through a dedicated application or a web browser. The input is the delivered streaming video, and the output is the user's viewing experience.

[0759] (Application example 2)

[0760] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0761] Conventional sports commentary systems require professional commentators and expensive equipment, and are difficult to provide realistic commentary that reflects the user's emotions. Furthermore, it is not possible for users to add appropriate commentary in real time to footage they have shot themselves. For this reason, there is a demand for a system that allows anyone to easily enjoy realistic live broadcasts in real time, even for sporting events with small audiences or children's games.

[0762] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring video, means for analyzing the acquired video data and detecting the movements and positions of players and the position of the ball, means for generating commentary in natural language based on the detected data, means for integrating the generated commentary with the video and distributing it in real time, and means for analyzing the user's emotions using an emotion engine and adjusting the generated commentary. This enables a realistic live broadcast that reflects the emotions of the user in real time in the video that the user has shot.

[0763] "Means for acquiring video" refers to devices or software that capture video data shot by users in real time and transmit it to the system.

[0764] "Means for analysing captured video data" refers to technologies and algorithms for detecting the movements and positions of players and the ball within the captured video and extracting relevant information.

[0765] "Means for detecting player movements and positions, and ball position" refers to a system that uses computer vision technology to track and detect the position of each element in the video.

[0766] "Means for generating commentary in natural language" refers to algorithms or technologies that automatically generate commentary in appropriate natural language in real time based on analyzed data.

[0767] "Means for integrating generated commentary with video and distributing in real time" refers to a system that overlays generated commentary on video, encodes it in real time, and distributes it through streaming.

[0768] "Means of analyzing user emotions using an emotion engine" refers to technology that uses computer vision technology to analyze the user's facial expressions and movements in video and recognize their emotions.

[0769] "Means for adjusting generated commentary" refers to algorithms or techniques that dynamically change the content or tone of already-generated commentary based on perceived user sentiment.

[0770] The present invention provides a system for providing real-time live broadcasts based on sports videos taken by users. The system includes a video acquisition unit, a video data analysis unit, a natural language generation unit, a sentiment analysis unit, a synthesis unit, and a distribution unit.

[0771] The server first receives sports videos in real time, which are captured by users using their smartphones. The smartphones act as a means of capturing the video and transmit the captured video data to the server using a low-latency protocol (e.g., WebRTC). The server also encodes the video data using FFmpeg before transmitting it.

[0772] The server decodes the received video data and analyzes the movements and positions of players and the ball in the video using computer vision technology (for example, using TensorFlow), which allows it to identify important events such as goals and points scored.

[0773] Next, a natural language generation algorithm written in Python is used to automatically generate commentary in real time based on the detected data, which is then saved along with a timestamp.

[0774] Furthermore, an emotion engine (also using TensorFlow) analyzes the user's facial expressions and movements from the camera footage to recognize their emotions. Based on this emotional data, the server adjusts the content and tone of the commentary it generates in real time. For example, if the user is excited, the tone of the commentary will be changed to an exciting one.

[0775] The server then combines the generated commentary with the video and adds it as a text overlay to the video. This combined data is then encoded using FFmpeg and streamed in real time. This process allows users to provide emotionally charged commentary on their own footage.

[0776] As a concrete example, consider a scene in a children's soccer game where player number 8 is dribbling towards the goal. If the user is excited, the system will generate a commentary in real time such as, "Great dribble! Player number 8 is approaching the goal!"

[0777] An example prompt is:

[0778] "Video data: sports_video_stream.mp4",

[0779] "User emotion: Facial expression: Excitement, Action: Applause"

[0780] "Event: Player ID: 8, Action: Dribble, Position: Towards Goal"

[0781] In this way, by combining an emotion engine and real-time distribution technology, the present invention provides a system that allows users to easily broadcast live sports with a sense of realism.

[0782] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0783] Step 1:

[0784] A user uses a smartphone to film a sports game. The video data captured by the user is saved on the device in real time. The input is the video data captured in real time, and the output is the video data stored on the device.

[0785] Step 2:

[0786] The device sends the captured video data to the server using a low-latency protocol. In this process, the video data is encoded using FFmpeg and sent as a real-time stream to the server using the WebRTC protocol. The input is the video data in the device, and the output is the encoded video data sent to the server.

[0787] Step 3:

[0788] The server decodes the received video data and temporarily stores it in a buffer. It uses FFmpeg to decode the video data and make it playable. The input is the encoded video data sent to the server, and the output is the decoded video data stored in the buffer.

[0789] Step 4:

[0790] The server begins video analysis. Using computer vision techniques, it uses a TensorFlow model to analyze the position and movement of players and the ball in the video. This analysis involves tracking players, locating the ball, and identifying important events such as goals and points scored. The input is the decoded video data stored in the buffer, and the output is the analyzed event data.

[0791] Step 5:

[0792] The server generates commentary in natural language based on the parsed data. It uses a natural language generation algorithm written in Python to generate commentary corresponding to specific events along with timestamps. The input is the parsed event data, and the output is the generated commentary.

[0793] Step 6:

[0794] The server collects camera footage to analyze the user's emotions using an emotion engine. Using a TensorFlow model, it analyzes the user's facial expressions and movements to recognize their emotions. The input is the user's camera footage, and the output is the recognized emotion data.

[0795] Step 7:

[0796] The server adjusts the content of the commentary generated based on the recognized emotion data. For example, if the user is excited, the tone of the commentary can be made more exciting. The input is the generated commentary and the recognized emotion data, and the output is the adjusted commentary.

[0797] Step 8:

[0798] The server merges the generated commentary with the video, adding a text overlay of the commentary to the video stream and using FFmpeg to re-encode it. The input is the adjusted commentary and decoded video data, and the output is the merged video data.

[0799] Step 9:

[0800] The server re-encodes the merged video data and delivers it using a real-time streaming protocol, allowing users to watch immersive, live coverage in real time. The input is the merged video data, and the output is a real-time stream delivered to the user.

[0801] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0802] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0803] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0804] [Fourth embodiment]

[0805] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0806] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0807] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0808] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0809] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0810] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0811] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0812] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0813] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0814] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0815] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0816] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0817] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0818] This paper describes a system for automatically broadcasting live sports events, and explains its method in detail. This system performs all steps from video capture and analysis to natural language commentary generation and real-time distribution.

[0819] First, a user uses a smartphone to capture video of a sports game. The device transmits the captured video data to a server in real time. The video data is then encoded and sent to the server using a low-latency protocol.

[0820] The server decodes the received video data and stores the video stream in a buffer. The server then uses computer vision technology to analyze the movement and position of players and the ball in the video. This includes player tracking and ball location detection, and identifies significant events such as goals and points scored.

[0821] Based on the event data generated by the analysis, the server automatically generates commentary using a natural language generation algorithm (e.g., GPT-4). This commentary corresponds to a specific event and is managed along with a timestamp.

[0822] The server then integrates the generated commentary with the video, adding the commentary as a text overlay to the video stream, and then encodes the integrated video and commentary for delivery in real time using a streaming protocol.

[0823] As a concrete example, let's take a children's soccer match. A user uses a smartphone to film the match, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." If player number 8 then scores a goal, the server generates a commentary such as "Goal! Player number 8 scores a great shot!" These commentaries are integrated with the video and distributed to viewers in real time.

[0824] This system allows anyone to easily broadcast live sports without the need for expensive dedicated equipment or a professional broadcasting team. It is particularly useful for sports with small audiences and children's games.

[0825] The processing flow will be explained below.

[0826] Step 1:

[0827] Users use their smartphone cameras to capture footage of sports matches, and the device acquires the footage in real time.

[0828] Step 2:

[0829] The device compresses and encodes the acquired video data and sends it to the server using a low-latency protocol (e.g., RTSP, WebRTC).

[0830] Step 3:

[0831] The server receives the video data sent from the terminal, decodes it, and temporarily stores it in a buffer.

[0832] Step 4:

[0833] The server analyzes the stored video data using computer vision technology (e.g., OpenCV, TensorFlow), which detects the movements and positions of players and the position of the ball.

[0834] Step 5:

[0835] Based on the analysis results, the server generates data about player and ball movements and identifies important events (e.g. goals, shots, etc.).

[0836] Step 6:

[0837] The server uses the generated event data to apply a natural language generation algorithm (e.g., GPT-4) to automatically generate commentary.

[0838] Step 7:

[0839] The server timestamps the generated commentary and synchronizes it with the actual game time.

[0840] Step 8:

[0841] The server adds the commentary as an overlay to the video stream, integrating the video and commentary.

[0842] Step 9:

[0843] The server encodes the aggregated video stream and converts it into a streaming format (e.g., HLS, DASH).

[0844] Step 10:

[0845] The server delivers the encoded video stream in real time, allowing users to watch the game by accessing the live stream on their smartphones or computers.

[0846] These processing steps allow anyone to easily enjoy live coverage of sports games in real time.

[0847] Example 1

[0848] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0849] Conventional live broadcasts of sporting events require expensive equipment and a specialized broadcasting team, and preparation and operation are costly and labor-intensive. Furthermore, it is difficult to broadcast professionally when the audience is small, such as for small-scale sporting events or children's games. For this reason, there is a demand for a system that allows anyone to easily broadcast live sports at low cost.

[0850] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0851] In this invention, the server includes means for acquiring video, means for encoding the acquired video data and transmitting it to the server using a low-latency protocol, means for decoding the received video data and storing it in a buffer, means for analyzing the video data and detecting the movements and positions of players and objects in the sporting event, means for generating a natural language commentary using a generative AI model that creates generated text based on the detected data, means for integrating the generated commentary with the video and adding it as a text overlay, and means for encoding the integrated video and commentary and distributing it in real time using a streaming protocol, thereby enabling anyone to easily broadcast a sporting event in real time without the need for expensive equipment.

[0852] "Means for acquiring video" refers to means for capturing video in real time using a photographic device such as a smartphone or camera.

[0853] "Means for encoding acquired video data and transmitting it to a server using a low-latency protocol" refers to means for encoding video data based on a compression method and transmitting it to a server using a low-latency data transmission protocol such as WebRTC or RTMP.

[0854] "Means for decoding received video data and storing it in a buffer" refers to means for the server to decode encoded video data received from a client terminal and store it in a temporary memory space (buffer).

[0855] "Means for analyzing the video data and detecting the movements and positions of players and objects in the sporting event" refers to means for identifying the positions and movements of players and the ball in the video using computer vision technology and machine learning algorithms.

[0856] A "means for generating natural language commentary using a generative AI model that creates generated text based on detected data" means a means for using the analyzed data as input and a generative AI model (e.g., a deep learning model) to create a natural language commentary that includes details of the event.

[0857] The "means for integrating the generated commentary with the video and adding it as a text overlay" refers to a means for visually displaying the generated commentary as text overlaid at a specific position on the video stream.

[0858] "Means for encoding the integrated video and commentary and delivering it in real time using a streaming protocol" refers to a means for re-encoding the video stream with the added text overlay and delivering it to viewers in real time using a streaming protocol such as HLS or DASH.

[0859] This invention is a system for automatically broadcasting live sports events, and is realized using specific devices and algorithms. This system performs all processes from video capture and analysis to natural language commentary generation and real-time distribution.

[0860] First, a user uses a smartphone to capture video of a sports game. The smartphone used here must be equipped with a camera function and a streaming application. The captured video data is sent from the device to a server in real time. At this time, the device encodes the video data using a low-latency codec such as H.264 or H.265, and sends it to the server using a low-latency protocol such as WebRTC or RTMP.

[0861] The server receives the encoded video data, decodes it using a decoding library such as FFmpeg, and temporarily stores the video stream in a buffer. The server then uses computer vision techniques to analyze the movement and position of players and the ball in the video. Specific algorithms used include OpenCV and YOLO. This analysis process includes player tracking and ball position detection, and identifies important events such as goals and points.

[0862] Based on the analysis data, the server uses a generative AI model (such as GPT-4) to generate commentary. The generated commentary is managed along with a timestamp. For example, if the analysis detects that player number 8 is dribbling the ball, the prompt "Player number 8 is dribbling the ball" is generated. If player number 8 scores a goal, the commentary will read "Goal! Player number 8 made a great shot!"

[0863] The server then uses a library such as FFmpeg to add the generated commentary as a text overlay to the video stream. The commentary is integrated into the video at the appropriate time based on the timestamp. After integration, the server distributes the encoded video stream in real time using a streaming protocol such as HLS or DASH. Viewers can then watch the distributed video in real time using devices such as smartphones and PCs.

[0864] As a concrete example, let's take a children's soccer match. A user uses a smartphone to film the match, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." If player number 8 then scores a goal, the server generates a commentary such as "Goal! Player number 8 scores a great shot!" These commentaries are integrated with the video and distributed to viewers in real time.

[0865] As a result, this system allows anyone to easily broadcast live sports without the need for expensive dedicated equipment or a specialized broadcasting team. This is particularly useful for sports with small audiences or for children's games.

[0866] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0867] Step 1:

[0868] Video shooting by users

[0869] Input: Sports event footage, smartphone

[0870] Processing: A user uses a smartphone to capture video of a sporting event in real time, where the smartphone has a camera function and an application installed to capture the video.

[0871] Output: Real-time video data captured by a smartphone camera

[0872] Step 2:

[0873] Video encoding and transmission by the device

[0874] Input: Real-time video data

[0875] Processing: The device encodes the captured video using a codec such as H.264 or H.265 to compress the video data. It then transmits the data to the server using a low-latency protocol such as WebRTC or RTMP. It also monitors the network connection status to minimize latency.

[0876] Output: The encoded video data is sent to the server.

[0877] Step 3:

[0878] Video reception and decoding by the server

[0879] Input: Encoded video data sent from the device

[0880] Processing: The server receives the encoded video data sent from the device, decodes the video data using a decoding library such as FFmpeg, and stores it in a buffer memory in real time. This information is used for later analysis and integration processing.

[0881] Output: Decoded video data stored in a buffer

[0882] Step 4:

[0883] Server-based video analysis

[0884] Input: Decoded video data stored in a buffer

[0885] Processing: The server uses computer vision techniques (e.g., OpenCV or YOLO) to analyze the video data. Specifically, it detects the tracking, movement, and position of players and the ball within the video. This analysis identifies important events, such as goals scored.

[0886] Output: Analysis data (player position, ball position, event information, etc.)

[0887] Step 5:

[0888] Server-based natural language generation

[0889] Input: Analysis data (player position, ball position, event information, etc.)

[0890] Processing: The server uses a generative AI model (e.g., GPT-4) to generate commentary in natural language based on the analyzed data. For each specific event, a prompt is provided as input, and a natural-sounding commentary is generated based on that. For example, the commentary generated might be, "Player No. 8 is dribbling the ball."

[0891] Output: Generated commentary (with timestamp)

[0892] Step 6:

[0893] Server-based integration of video and commentary

[0894] Input: Decoded video data, generated commentary

[0895] Processing: The server adds the generated commentary to the video stream as a text overlay, using FFmpeg filters to insert the text at the right time and in the right position, based on the timestamps.

[0896] Output: Video stream with text overlay added

[0897] Step 7:

[0898] Server-based video content distribution

[0899] Input: Video stream with text overlay added

[0900] Processing: The server re-encodes the combined video stream and delivers it in real time using a streaming protocol such as HLS or DASH, allowing viewers to enjoy the video and commentary smoothly without delay.

[0901] Output: Real-time integrated video stream

[0902] (Application example 1)

[0903] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0904] Conventional monitoring systems require manual confirmation, making it difficult to respond quickly. It is also difficult to generate warning messages in real time or automatically recognize suspicious activity, making it difficult to implement effective security measures. The goal of this project is to solve these issues and provide a more effective and efficient security system.

[0905] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0906] In this invention, the server includes a means for acquiring video, a means for analyzing the acquired video data to detect the movement and position of a target or the position of an object, a means for generating a warning message in natural language based on the detected data, and a means for integrating the generated warning message with the video and distributing it in real time, thereby enabling the rapid detection of suspicious movements and the immediate issuance of a warning.

[0907] "Means for acquiring video" refers to devices that capture video data in real time using cameras, sensors, etc.

[0908] "Means for analyzing acquired video data and detecting the movement and position of a target or object" refers to a device or software that uses computer vision technology and algorithms to detect the movement and position of a specific target or object in a video and analyze its behavior.

[0909] A "means for generating natural language warning text based on detected data" is a device or program that uses generative AI models or natural language processing techniques to automatically generate appropriate warning messages based on detected events or situations.

[0910] "Means for integrating generated warning text with video and delivering it in real time" refers to a technology or system that overlays generated warning text onto video data and delivers it to users in real time.

[0911] The "means for encoding video data and transmitting it to the server" refers to a device or software that converts (encodes) the captured video data into an appropriate format and transmits it to the server using a low-latency protocol.

[0912] "Means for detecting the position of a target or object using computer vision technology" refers to technologies or programs that use image processing algorithms or machine learning models to accurately detect the position of a specific target or object within a video.

[0913] The purpose of this invention is to automate and improve the efficiency of a monitoring system. We will now explain in detail how the server, terminals, and users cooperate to realize this system.

[0914] First, a surveillance camera, smartphone, or smart glasses camera captures video in real time. This is the "means of video capture." The device captures the video data and sends it to a streaming server using a low-latency protocol (e.g., RTMP).

[0915] The server then analyzes the received video data. Specifically, it uses computer vision technology such as OpenCV to detect specific objects in the video, such as people, vehicles, or suspicious movements. This is "a means of analyzing acquired video data and detecting the movement and location of objects."

[0916] Based on the detected data, a generative AI model (e.g., GPT-4) is used to generate a natural language warning. This is a "means of generating a natural language warning based on the detected data." The following prompts can be used:

[0917] "We've detected suspicious activity. What should we do?"

[0918] "I detected suspicious activity at night. What is the appropriate response?"

[0919] The generated warning text is integrated with the video data. Specifically, the warning text is added as a text overlay to the video stream. This is the "means of integrating the generated warning text with the video and delivering it in real time." This integrated data is delivered using a real-time streaming protocol (such as HLS or DASH), allowing users to check the monitoring results in real time.

[0920] Additionally, when important events occur, emergency response teams can be alerted using push notification services such as Firebase Cloud Messaging.

[0921] For example, if a surveillance camera detects motion at night, the server analyzes the video data to detect suspicious activity. Using this information, a generative AI model generates a warning message: "Suspicious activity detected. Please check immediately." This message is overlaid on the video stream and distributed in real time, simultaneously notifying emergency response teams.

[0922] In this way, the present invention realizes an efficient monitoring system and effective security measures.

[0923] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0924] Step 1:

[0925] The device (smartphone, smart glasses) captures video in real time using a camera. The input is the camera image, and the output is the captured video data. This video data is sent to a streaming server using a low-latency protocol (e.g., RTMP).

[0926] Step 2:

[0927] The server decodes the received video data. The input is the encoded video data, and the output is the decoded video frames. This is handled efficiently by the streaming protocol, for example, using the FFmpeg library for decoding.

[0928] Step 3:

[0929] The server analyzes the decoded video frames using computer vision technology. The input is the video frame, and the output is detection data such as the movement and position of the target and the location of the object. For example, OpenCV is used to detect suspicious movements and abnormal behavior. Specifically, it uses background subtraction to extract moving objects, and performs facial recognition and motion detection.

[0930] Step 4:

[0931] The server generates a warning in natural language based on the detection data. The input is the detection data and the output is the warning. The warning is generated using a generative AI model (e.g., GPT-4). As a specific example, an appropriate warning is generated based on the prompt, "Suspicious activity has been detected. What should be done?"

[0932] Step 5:

[0933] The server overlays the generated warning text onto the video frame. The input is the warning text and the video frame, and the output is the video with the warning text overlaid. Specifically, we add text to the video using OpenCV.

[0934] Step 6:

[0935] The server re-encodes the overlaid video and delivers it in real time. The input is the overlaid video, and the output is the streamed video. It is delivered to users using a streaming protocol (e.g., HLS or DASH).

[0936] Step 7:

[0937] The server sends a push notification when an important event occurs. The input is the event data, and the output is the notification message. A push notification service such as Firebase Cloud Messaging is used to send an alert to an emergency response team. Specifically, the alert text is set as the message and a Firebase API is called.

[0938] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0939] This invention is a system for automatically broadcasting live sports commentary, and combines it with an emotion engine that recognizes user emotions. This system enables video capture and analysis, commentary generation in natural language, real-time distribution, and even more realistic commentary that incorporates user emotions.

[0940] First, a user uses a smartphone to capture video of a sports game. The device transmits the captured video data to a server in real time, where it is encoded and transmitted to the server using a low-latency protocol.

[0941] The server decodes the received video data and temporarily stores it in a buffer. The server then uses computer vision technology to analyze the movement and position of players and the ball in the video. This analysis allows for player tracking, ball location detection, and identification of significant events such as goals and points scored.

[0942] The server then uses a natural language generation algorithm to automatically generate commentary based on the analysis results, which corresponds to a specific event and is maintained with a timestamp.

[0943] The server then uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial expressions and movements based on the camera footage to recognize their emotions. Based on the recognized emotion data, the content of the commentary can be adjusted in real time. For example, if the user is excited, the tone and content of the commentary can be made more exciting.

[0944] The server merges the generated commentary with the video, adds a text overlay of the commentary to the video stream, and then encodes the merged video and commentary for delivery in real time using a streaming protocol.

[0945] As a concrete example, let's take a children's soccer game. The user uses a smartphone to film the game, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." Furthermore, if the server recognizes that the user is excited, it generates an exciting commentary such as "Great dribbling! Player number 8 is moving forward!"

[0946] In this way, by using the emotion engine, this system can provide more realistic live broadcasts that reflect the user's emotions. This makes it possible for anyone to easily broadcast live sports without the need for expensive dedicated equipment or a specialized broadcasting team. This is particularly useful for sports with small audiences and children's games.

[0947] The processing flow will be explained below.

[0948] Step 1:

[0949] Users use their smartphone cameras to capture footage of sports matches, and the device captures the footage in real time and stores it in an internal buffer.

[0950] Step 2:

[0951] The device compresses and encodes the captured video data and sequentially transmits it to the server using a low-latency protocol (e.g., RTSP, WebRTC).

[0952] Step 3:

[0953] The server receives the video data sent from the terminal, decodes it, and stores the decoded video data in the server's buffer.

[0954] Step 4:

[0955] The server uses computer vision techniques (e.g., OpenCV, TensorFlow) to analyze the video data, determining the movements and positions of players and the position of the ball.

[0956] Step 5:

[0957] Based on the analysis results, the server generates event data about player and ball movements, e.g., "Player No. 8 is dribbling the ball" or "Player No. 8 has scored a goal."

[0958] Step 6:

[0959] The server uses a natural language generation algorithm (e.g., GPT-4) to generate commentary based on the generated event data. Examples include, "Player 8 is dribbling the ball," and "Goal! Player 8 made a great shot!"

[0960] Step 7:

[0961] The server recognizes the user's emotions using an emotion engine within the server, which analyzes the video data of the user's face and voice to identify his emotional state (e.g., excitement, joy, surprise).

[0962] Step 8:

[0963] The server adjusts the content and tone of the commentary based on the user's emotional data. For example, if the user is excited, the server will generate an exciting commentary such as, "Player No. 8 is showing off some great dribbling skills!"

[0964] Step 9:

[0965] The server combines the generated commentary with the decoded video data, and the commentary is overlaid onto the video stream with a timestamp.

[0966] Step 10:

[0967] The server compresses and encodes the aggregated video stream and delivers it in real time using a streaming protocol (e.g., HLS, DASH).

[0968] Step 11:

[0969] Users can watch the video stream in real time using their smartphones or PCs, and can enjoy a realistic commentary that reflects their own emotions.

[0970] By using these specific processing steps, the present invention can realize a live sports broadcast that is full of realism and incorporates the user's emotions.

[0971] Example 2

[0972] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0973] Conventional sports commentary systems have been unable to reflect the user's emotions in the acquisition and analysis of video footage, and in the generation and distribution of commentary, resulting in a lack of realism and a sense of unity. In particular, for sports with small audiences or children's games, expensive dedicated equipment and a specialized broadcasting team are required, making it difficult to easily provide live commentary.

[0974] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0975] In this invention, the server includes means for acquiring video, means for analyzing the acquired video data and detecting the movements and positions of players and the position of the ball, means for generating commentary in natural language based on the detected data, means for integrating the generated commentary with the video and distributing it in real time, and means for recognizing the user's emotions and adjusting the commentary based on the recognized emotion data. This makes it possible to easily provide a realistic live broadcast that reflects the user's emotions.

[0976] "Means for acquiring video footage" refers to the technology and methods for capturing video footage of sports games using devices such as cameras and smartphones.

[0977] "Means for analyzing acquired video data and detecting the movement and position of players and the position of the ball" refers to techniques and methods that use computer vision technology to analyze and identify the movement and position of players and the ball in the video.

[0978] "Means for generating commentary in natural language based on detected data" refers to technologies and methods that utilize artificial intelligence and natural language generation algorithms to automatically generate commentary on sports based on analyzed data.

[0979] "Means for integrating generated commentary with video and distributing it in real time" refers to a technology or method for overlaying generated commentary on video data, encoding it, and streaming it in real time using a distribution protocol.

[0980] "Means for recognizing the user's emotions and adjusting the commentary text based on the recognized emotion data" refers to a technology or method that uses an emotion recognition engine to analyze the user's facial expressions and movements, and changes the content and tone of the commentary text in real time according to the emotions.

[0981] The present invention is a system for automatically broadcasting live sports games, providing a realistic commentary by recognizing the user's emotions. This system involves capturing and analyzing video footage, generating commentary in natural language, delivering it in real time, and adjusting the commentary to incorporate the user's emotions. Specific embodiments are described below.

[0982] First, a user uses a smartphone to capture video of a sports game. The smartphone has a camera function and can capture high-quality images. The captured video data is encoded in H.264 format and sent to a server in real time using a low-latency protocol (e.g., RTMP).

[0983] The device encodes the video and sends the encoded video data to the server. The data is compressed and sent with low latency, maintaining real-time performance.

[0984] The server decodes the received video data and temporarily stores it in a buffer. The decoding is performed using the FFmpeg library for efficient data processing. The server then uses computer vision techniques (e.g., OpenCV) to analyze the movement and position of players and the ball in the video. This analysis allows for player tracking, ball location detection, and identification of important events such as goals and points scored.

[0985] Based on the analysis results, the server automatically generates commentary using a natural language generation algorithm (e.g., GPT-4). The generated commentary is associated with a specific event and managed with a timestamp. For example, a commentary such as "Player No. 8 is dribbling the ball" may be generated.

[0986] Furthermore, the server uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the user's emotions. The emotion engine analyzes the user's facial expressions and movements based on the smartphone camera footage and recognizes their emotions in real time. Based on the recognized emotion data, the content of the commentary is adjusted in real time. For example, if the user is excited, a more exciting commentary such as "Great dribbling! Player 8 is moving forward!" is generated.

[0987] The server then combines the generated commentary with the video and adds a text overlay of the commentary to the video stream. The combined video and commentary are then encoded again using the FFmpeg library and streamed in real time using the HLS protocol. Users can then watch the immersive live broadcast through a dedicated application or a web browser.

[0988] As a concrete example, consider a children's soccer match. The user films the match using a smartphone, and the device sends the video data to a server. The server detects that player number 8 is dribbling the ball in the video and generates a commentary such as "Player number 8 is dribbling the ball." Furthermore, if the server recognizes that the user is excited, it generates an exciting commentary such as "Great dribble! Player number 8 is moving forward!"

[0989] An example of a prompt to input to a generative AI model is as follows:

[0990] Prompt: "Generate a commentary about a soccer game where player 8 is dribbling the ball forward. If the user is excited, use more exciting language."

[0991] In this way, this system can provide a realistic live broadcast that reflects the user's emotions, making it possible for anyone to easily broadcast live sports without the need for expensive dedicated equipment or a specialized broadcasting team.

[0992] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0993] Step 1:

[0994] The user takes a video.

[0995] Specific operation: A user uses a smartphone to capture video of a sports game. The game is recorded using a high-quality camera, and preparations are made to broadcast the game live in real time. The input is the camera footage, and the output is the video data before encoding.

[0996] Step 2:

[0997] The terminal encodes the video data and sends it to the server.

[0998] Specific operation: The device encodes the captured video data in H.264 format and sends the encoded data to the server in real time using a low-latency protocol (e.g., RTMP). The input is the camera video data, and the output is the encoded video data.

[0999] Step 3:

[1000] The server decodes the video data and temporarily stores it in a buffer.

[1001] Specific operation: The server uses the FFmpeg library to decode the received video data. The decoded data is temporarily stored in a buffer. The input is the encoded video data, and the output is the decoded video data.

[1002] Step 4:

[1003] The server analyzes the video data to detect the player's movements and positions, as well as the position of the ball.

[1004] How it works: The server uses computer vision techniques (e.g., OpenCV) to analyze the movement and position of players and the ball in the video. It tracks players, detects the position of the ball, and identifies important events such as goals and points. The input is the decoded video data, and the output is the analyzed event data.

[1005] Step 5:

[1006] The server generates commentary based on the event data using a natural language generation algorithm.

[1007] Specific operation: The server generates commentary based on the analyzed data using a natural language generation algorithm (e.g., GPT-4). Commentary corresponding to each event is generated and managed along with a timestamp. The input is the analyzed event data, and the output is the generated commentary.

[1008] Step 6:

[1009] The server uses an emotion engine to recognize the user's emotions.

[1010] Specific operation: The server uses an emotion engine (e.g., Microsoft Azure Emotion API) to analyze the user's facial expressions and movements to recognize emotions. Emotional data such as whether the user is excited or not is obtained. The input is the user's camera image, and the output is the recognized emotional data.

[1011] Step 7:

[1012] The server adjusts the commentary based on the recognized emotion data.

[1013] Specific operation: The server adjusts the content and tone of the generated commentary in real time based on the recognized emotion data. If an excited state is detected, the commentary is changed to a more exciting one. The input is the generated commentary and the recognized emotion data, and the output is the adjusted commentary.

[1014] Step 8:

[1015] The server integrates the commentary and video and distributes them in real time.

[1016] Specific operation: The server integrates the generated commentary with the video, adds a text overlay of the commentary to the video stream, encodes the integrated video using the FFmpeg library, and delivers it in real time using the HLS protocol. The input is the adjusted commentary and decoded video data, and the output is the encoded integrated video.

[1017] Step 9:

[1018] The user watches the distributed video.

[1019] Specific operation: A user watches immersive video delivered in real time through a dedicated application or a web browser. The input is the delivered streaming video, and the output is the user's viewing experience.

[1020] (Application example 2)

[1021] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1022] Conventional sports commentary systems require professional commentators and expensive equipment, and are difficult to provide realistic commentary that reflects the user's emotions. Furthermore, it is not possible for users to add appropriate commentary in real time to footage they have shot themselves. For this reason, there is a demand for a system that allows anyone to easily enjoy realistic live broadcasts in real time, even for sporting events with small audiences or children's games.

[1023] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring video, means for analyzing the acquired video data and detecting the movements and positions of players and the position of the ball, means for generating commentary in natural language based on the detected data, means for integrating the generated commentary with the video and distributing it in real time, and means for analyzing the user's emotions using an emotion engine and adjusting the generated commentary. This enables a realistic live broadcast that reflects the emotions of the user in real time in the video that the user has shot.

[1024] "Means for acquiring video" refers to devices or software that capture video data shot by users in real time and transmit it to the system.

[1025] "Means for analysing captured video data" refers to technologies and algorithms for detecting the movements and positions of players and the ball within the captured video and extracting relevant information.

[1026] "Means for detecting player movements and positions, and ball position" refers to a system that uses computer vision technology to track and detect the position of each element in the video.

[1027] "Means for generating commentary in natural language" refers to algorithms or technologies that automatically generate commentary in appropriate natural language in real time based on analyzed data.

[1028] "Means for integrating generated commentary with video and distributing in real time" refers to a system that overlays generated commentary on video, encodes it in real time, and distributes it through streaming.

[1029] "Means of analyzing user emotions using an emotion engine" refers to technology that uses computer vision technology to analyze the user's facial expressions and movements in video and recognize their emotions.

[1030] "Means for adjusting generated commentary" refers to algorithms or techniques that dynamically change the content or tone of already-generated commentary based on perceived user sentiment.

[1031] The present invention provides a system for providing real-time live broadcasts based on sports videos taken by users. The system includes a video acquisition unit, a video data analysis unit, a natural language generation unit, a sentiment analysis unit, a synthesis unit, and a distribution unit.

[1032] The server first receives sports videos in real time, which are captured by users using their smartphones. The smartphones act as a means of capturing the video and transmit the captured video data to the server using a low-latency protocol (e.g., WebRTC). The server also encodes the video data using FFmpeg before transmitting it.

[1033] The server decodes the received video data and analyzes the movements and positions of players and the ball in the video using computer vision technology (for example, using TensorFlow), which allows it to identify important events such as goals and points scored.

[1034] Next, a natural language generation algorithm written in Python is used to automatically generate commentary in real time based on the detected data, which is then saved along with a timestamp.

[1035] Furthermore, an emotion engine (also using TensorFlow) analyzes the user's facial expressions and movements from the camera footage to recognize their emotions. Based on this emotional data, the server adjusts the content and tone of the commentary it generates in real time. For example, if the user is excited, the tone of the commentary will be changed to an exciting one.

[1036] The server then combines the generated commentary with the video and adds it as a text overlay to the video. This combined data is then encoded using FFmpeg and streamed in real time. This process allows users to provide emotionally charged commentary on their own footage.

[1037] As a concrete example, consider a scene in a children's soccer game where player number 8 is dribbling towards the goal. If the user is excited, the system will generate a commentary in real time such as, "Great dribble! Player number 8 is approaching the goal!"

[1038] An example prompt is:

[1039] "Video data: sports_video_stream.mp4",

[1040] "User emotion: Facial expression: Excitement, Action: Applause"

[1041] "Event: Player ID: 8, Action: Dribble, Position: Towards Goal"

[1042] In this way, by combining an emotion engine and real-time distribution technology, the present invention provides a system that allows users to easily broadcast live sports with a sense of realism.

[1043] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1044] Step 1:

[1045] A user uses a smartphone to film a sports game. The video data captured by the user is saved on the device in real time. The input is the video data captured in real time, and the output is the video data stored on the device.

[1046] Step 2:

[1047] The device sends the captured video data to the server using a low-latency protocol. In this process, the video data is encoded using FFmpeg and sent as a real-time stream to the server using the WebRTC protocol. The input is the video data in the device, and the output is the encoded video data sent to the server.

[1048] Step 3:

[1049] The server decodes the received video data and temporarily stores it in a buffer. It uses FFmpeg to decode the video data and make it playable. The input is the encoded video data sent to the server, and the output is the decoded video data stored in the buffer.

[1050] Step 4:

[1051] The server begins video analysis. Using computer vision techniques, it uses a TensorFlow model to analyze the position and movement of players and the ball in the video. This analysis involves tracking players, locating the ball, and identifying important events such as goals and points scored. The input is the decoded video data stored in the buffer, and the output is the analyzed event data.

[1052] Step 5:

[1053] The server generates commentary in natural language based on the parsed data. It uses a natural language generation algorithm written in Python to generate commentary corresponding to specific events along with timestamps. The input is the parsed event data, and the output is the generated commentary.

[1054] Step 6:

[1055] The server collects camera footage to analyze the user's emotions using an emotion engine. Using a TensorFlow model, it analyzes the user's facial expressions and movements to recognize their emotions. The input is the user's camera footage, and the output is the recognized emotion data.

[1056] Step 7:

[1057] The server adjusts the content of the commentary generated based on the recognized emotion data. For example, if the user is excited, the tone of the commentary can be made more exciting. The input is the generated commentary and the recognized emotion data, and the output is the adjusted commentary.

[1058] Step 8:

[1059] The server merges the generated commentary with the video, adding a text overlay of the commentary to the video stream and using FFmpeg to re-encode it. The input is the adjusted commentary and decoded video data, and the output is the merged video data.

[1060] Step 9:

[1061] The server re-encodes the merged video data and delivers it using a real-time streaming protocol, allowing users to watch immersive, live coverage in real time. The input is the merged video data, and the output is a real-time stream delivered to the user.

[1062] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1063] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1064] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1065] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1066] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1067] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1068] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1069] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1070] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1071] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1072] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1073] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1074] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1075] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1076] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1077] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1078] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1079] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1080] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1081] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1082] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1083] The following is further disclosed regarding the above embodiment.

[1084] (Claim 1)

[1085] a means for acquiring the image;

[1086] A means for analyzing the acquired video data and detecting the player's movements and positions, as well as the position of the ball;

[1087] means for generating a commentary in natural language based on the detected data;

[1088] A means to integrate the generated commentary with the video and distribute it in real time,

[1089] A system including:

[1090] (Claim 2)

[1091] 10. The system of claim 1, further comprising means for encoding the video data and transmitting it to the server.

[1092] (Claim 3)

[1093] 10. The system of claim 1, further comprising means for detecting the position of a player and a ball using computer vision techniques.

[1094] "Example 1"

[1095] (Claim 1)

[1096] a means for acquiring the image;

[1097] means for encoding the captured video data and transmitting it to a server using a low latency protocol;

[1098] means for decoding the received video data and storing it in a buffer;

[1099] means for analyzing the video data to detect the movements and positions of players and objects in the sporting event;

[1100] means for generating a natural language commentary using a generative AI model that creates generated text based on the detected data;

[1101] A means to integrate the generated commentary with the video and add it as a text overlay;

[1102] a means for encoding and distributing the integrated video and commentary in real time using a streaming protocol;

[1103] A system including:

[1104] (Claim 2)

[1105] 10. The system of claim 1, further comprising means for encoding the video data and transmitting it to the server.

[1106] (Claim 3)

[1107] 10. The system of claim 1, further comprising means for detecting the locations of players and objects in a sporting event using computer vision techniques.

[1108] "Application Example 1"

[1109] (Claim 1)

[1110] a means for acquiring the image;

[1111] A means for analyzing the acquired video data and detecting the movement and position of the target and the position of the object;

[1112] means for generating a natural language warning based on the detected data;

[1113] A means to integrate the generated warning text and video and distribute them in real time;

[1114] A system including:

[1115] (Claim 2)

[1116] 10. The system of claim 1, further comprising means for encoding the video data and transmitting it to the server.

[1117] (Claim 3)

[1118] 10. The system of claim 1, further comprising means for detecting the location of an object or body using computer vision techniques.

[1119] "Example 2: Combining Emotion Engines"

[1120] (Claim 1)

[1121] a means for acquiring the image;

[1122] A means for analyzing the acquired video data and detecting the player's movements and positions, as well as the position of the ball;

[1123] means for generating a commentary in natural language based on the detected data;

[1124] A means to integrate the generated commentary with the video and distribute it in real time,

[1125] means for recognizing a user's emotion and adjusting commentary based on the recognized emotion data;

[1126] A system including:

[1127] (Claim 2)

[1128] 10. The system of claim 1, further comprising means for encoding the video data and transmitting it to the server.

[1129] (Claim 3)

[1130] 10. The system of claim 1, further comprising means for detecting the position of a player and a ball using computer vision techniques.

[1131] "Application example 2 when combining emotion engines"

[1132] (Claim 1)

[1133] a means for acquiring the image;

[1134] A means for analyzing the acquired video data and detecting the player's movements and positions, as well as the position of the ball;

[1135] means for generating a commentary in natural language based on the detected data;

[1136] A means to integrate the generated commentary with the video and distribute it in real time,

[1137] A means for analyzing a user's emotions using an emotion engine and adjusting the generated commentary;

[1138] A system including:

[1139] (Claim 2)

[1140] 10. The system of claim 1, further comprising means for encoding the video data and transmitting it to the server.

[1141] (Claim 3)

[1142] 10. The system of claim 1, further comprising means for detecting the position of a player and a ball using computer vision techniques. [Explanation of symbols]

[1143] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for acquiring the image; A means for analyzing the acquired video data and detecting the player's movements and positions, as well as the position of the ball; means for generating a commentary in natural language based on the detected data; A means to integrate the generated commentary with the video and distribute it in real time, A system including:

2. 10. The system of claim 1, further comprising means for encoding the video data and transmitting it to the server.

3. 10. The system of claim 1, further comprising means for detecting the position of a player and a ball using computer vision techniques.

Images