System

Abstract

To provide a system.SOLUTION: The system includes: means for acquiring live video data; means for preprocessing the acquired video data; means for analyzing the preprocessed video data using an image recognition technique; means for specifying a score scene on the basis of an analysis result; means for generating a prompt sentence that causes an explanation for the specified score scene to be generated; means for generating the explanation by inputting the generated prompt sentence to a generative AI model; and means for outputting the generated explanation in the form of voice or text.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] In live amateur sports videos, it is difficult to automatically display the score, identify the scoring scenes, and generate audio and text commentary for them. [Means for solving the problem]

[0005] The present invention provides a system that analyzes live video of amateur sports, generates a score display based on the analysis results, identifies scoring scenes, and generates commentary in audio and text for the identified scoring scenes. In particular, the analysis is characterized by using image recognition technology, and the commentary is generated using natural language generation technology. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] An example of the present invention is a system that analyzes live videos of amateur soccer matches. This system uses image recognition technology to analyze video data obtained from the live videos and generate a display of the match score. Specifically, the system identifies goal scenes and scorers, and keeps track of the match time. When a goal-scoring scene is identified, the system generates a detailed commentary of that scene in voice and text. Natural language generation technology is used to generate this commentary.

[0029] "Example 2"

[0030] Another example is a system that analyzes live videos of amateur basketball games. This system uses image recognition technology to analyze video data obtained from the live videos and generate a display of the game score. Specifically, it identifies basketball goal scenes and scorers, and keeps track of the game time. In addition, when a scoring scene is identified, it generates a detailed commentary of that scene in voice and text. Natural language generation technology is used to generate this commentary.

[0031] The process flow of each embodiment will be described below.

[0032] "Example 1"

[0033] Step 1: Get live amateur football video.

[0034] Step 2: Extract video data from the captured live video and analyze it using image recognition technology. This analysis includes identifying goal scenes, scorers, and game time.

[0035] Step 3: Generate a score display based on the analysis results.

[0036] Step 4: If a scoring scene is identified, a detailed commentary of that scene is generated in audio and text format using natural language generation technology.

[0037] "Example 2"

[0038] Step 1: Get live amateur basketball video.

[0039] Step 2: Extract video data from the captured live video and analyze it using image recognition technology. This analysis includes identifying basketball goal scenes, scorers, and game time.

[0040] Step 3: Generate a score display based on the analysis results.

[0041] Step 4: If a scoring scene is identified, a detailed commentary of that scene is generated in audio and text format using natural language generation technology.

[0042] Example 1

[0043] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0044] In conventional live video analysis systems for amateur sports, displaying the game score and identifying the scoring scenes are often done manually, which means that they lack real-timeness and accuracy. In addition, commentary on the scoring scenes is also done manually, which makes it difficult to provide information quickly. This makes it difficult for spectators and officials to accurately and quickly grasp the progress of the game.

[0045] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0046] In this invention, the server includes a means for acquiring live video data, a means for preprocessing the acquired video data, a means for analyzing the preprocessed video data using image recognition technology, a means for identifying a scoring scene based on the analysis result, a means for generating a prompt sentence for generating a commentary for the identified scoring scene, a means for inputting the generated prompt sentence into a generation AI model to generate a commentary, and a means for outputting the generated commentary in voice or text. This automates the display of the score of the game and the identification of the scoring scene, making it possible to provide accurate information in real time. In addition, detailed commentary of the scoring scene is also quickly generated, making it possible for spectators and related parties to accurately and quickly grasp the progress of the game.

[0047] "Live video data" refers to video information acquired in real time.

[0048] "Preprocessing" refers to processes such as data resizing, noise removal, and frame extraction that are carried out to make the acquired video data easier to analyze.

[0049] "Image recognition technology" is a technology for detecting and identifying specific objects and actions from video data.

[0050] "Analysis results" refers to the analysis results of data obtained using image recognition technology.

[0051] A "scoring scene" is a video of the exact moment a goal is scored in a sports game.

[0052] A "prompt sentence" is an instruction sentence to be input into a generative AI model.

[0053] A "generative AI model" is an artificial intelligence model that generates natural language text based on a given prompt.

[0054] "Commentary" refers to detailed explanations and comments about the scoring scene.

[0055] "Audio output" refers to providing the generated description in audio form.

[0056] "Text output" refers to providing the generated commentary in text form.

[0057] This invention is a system that analyzes live videos of amateur sports and automatically generates game scores and detailed commentary of scoring scenes. This system works in cooperation with three entities: a server, a terminal, and a user.

[0058] First, the server acquires live video data. The video data is transmitted in real time from devices such as cameras and smartphones. The server streams the video data from the cameras using RTSP (Real-Time Streaming Protocol).

[0059] The server then pre-processes the captured video data, which includes image resizing, noise removal, frame extraction, etc. Specifically, the server uses OpenCV to resize the video data to 640x480 pixels and apply a noise removal filter.

[0060] The pre-processed video data is then analyzed using image recognition technology. The server uses libraries such as TENSOFLOW (registered trademark) and YOLO (You Only Look Once) to detect players and the ball in each frame. Based on the analysis results, the server identifies the goal-scoring scene. To identify the goal scene, the server detects the movement of the ball and the shaking of the goal net. The server detects the moment the ball crosses the goal line and saves that frame.

[0061] When a scoring scene is identified, the server generates a prompt sentence for generating commentary. For example, the server generates a prompt sentence such as, "Please generate a detailed commentary on the scene where Player A scored a goal." The server inputs this prompt sentence into a generative AI model (for example, GPT-3 (registered trademark)) and generates a commentary. The generative AI model generates a commentary such as, "Player A received a cross from the right side and scored a goal with a brilliant volley shot."

[0062] The generated commentary is output by the server in the form of voice or text. For voice output, Google® Text-to-Speech API or the like is used. The server sends the generated text commentary to Google® Text-to-Speech API to generate an audio file. The server then sends the audio file to the user's device. The user can watch or view the commentary through the device.

[0063] As a concrete example, consider a case where a server analyzes a live video and identifies the scene where player A scored a goal. The server inputs a prompt sentence, "Please generate a detailed commentary on the scene where player A scored a goal," into the generative AI model. The generative AI model generates a commentary, "Player A received a cross from the right side and scored a goal with a brilliant volley shot." The server converts the generated commentary into audio, which the user can listen to on their device.

[0064] The server also keeps track of the game time. For example, the server detects that the game time has passed 45 minutes, and inputs a prompt message to the generative AI model saying, "Please generate a message announcing the end of the first half." The generative AI model generates an explanation saying, "The first half has ended. The current score is 1-0." The server displays the generated explanation as text, and the user can view it on their device.

[0065] The above is an embodiment of the present invention.

[0066] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0067] Step 1:

[0068] The server acquires the live video data.

[0069] Input: Real-time video data sent from devices such as cameras and smartphones.

[0070] Specific operation: The server streams video data from the camera using RTSP (Real-Time Streaming Protocol).

[0071] Output: Acquired live video data.

[0072] Step 2:

[0073] The server pre-processes the acquired video data.

[0074] Input: Acquired live video data.

[0075] Specific operation: The server uses OpenCV to resize the video data to 640x480 pixels and apply a noise reduction filter.

[0076] Output: Pre-processed video data.

[0077] Step 3:

[0078] The server analyzes the pre-processed video data using image recognition technology.

[0079] Input: Preprocessed video data.

[0080] Specific operation: The server uses libraries such as TENSOFLOW (registered trademark) and YOLO (You Only Look Once) to detect players and the ball in each frame.

[0081] Output: Analysis results (positions of players and the ball, etc.).

[0082] Step 4:

[0083] The server identifies the scoring scene based on the analysis result.

[0084] Input: The analysis results.

[0085] What happens: The server detects the moment the ball crosses the goal line and saves that frame.

[0086] Output: Goal specific information (scoring frame).

[0087] Step 5:

[0088] The server generates a prompt sentence for generating an explanation for the scoring scene.

[0089] Input: Scoring scene specific information.

[0090] Specific operation: The server generates a prompt sentence saying, "Please generate a detailed commentary of the scene where player A scored a goal."

[0091] Output: The prompt statement.

[0092] Step 6:

[0093] The server inputs the generated prompt sentence into a generative AI model to generate an explanation.

[0094] Input: The prompt statement.

[0095] Specific operation: The server sends a prompt sentence to a generative AI model (e.g., GPT-3 (registered trademark)) to generate an explanation.

[0096] Output: The generated description (in text format).

[0097] Step 7:

[0098] The server outputs the generated commentary in audio and / or text.

[0099] Input: The generated description (in text format).

[0100] Specific operation: The server sends the generated text commentary to the Google® Text-to-Speech API to generate an audio file, and then sends the audio file to the user's device.

[0101] Output: Audio files and text description.

[0102] The above is the flow of processing of the program of this system.

[0103] (Application example 1)

[0104] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] In live video distribution of amateur sports, it is difficult to provide detailed commentary of game scores and scoring scenes in real time. In addition, when viewers watch games using smart devices, the user interface is not optimized, resulting in a poor viewing experience. Furthermore, there is also the issue that it is difficult for viewers to understand the flow of the game because commentary of scoring scenes cannot be provided by audio.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0107] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in voice or text for the identified scoring scene, a means for providing a user interface optimized for a smart device, a means for measuring the game time in real time, and a means for converting detailed commentary of the scoring scene into voice using a voice synthesis technology. This makes it possible to provide detailed commentary of the game score and the scoring scene in real time, allowing viewers to comfortably watch the game using their smart devices. Also, by providing commentary of the scoring scene in voice, viewers can have a deeper understanding of the flow of the game.

[0108] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[0109] "Live video" is video that is filmed in real time and streamed simultaneously.

[0110] "Analysis" means examining data or information in detail to clarify its structure and meaning.

[0111] A "score display" is a visual indication of the score and progress of a match.

[0112] A "scoring scene" is a video of the moment when a goal is scored in a sporting event.

[0113] "Means for generating commentary in audio or text" refers to technology for explaining scoring scenes and the situation of the game in audio or text.

[0114] A "smart device" is an electronic device with advanced functions, such as a smartphone, tablet, or head-mounted display.

[0115] A "user interface" is a screen or method of operation that allows a user to interact with a system or application.

[0116] "Means for measuring match time in real time" refers to technology that measures and displays the progress of a match in real time.

[0117] "Speech synthesis technology" is a technology that converts text data into speech.

[0118] "Image recognition technology" is a technology that analyzes image data and recognizes specific objects or scenes.

[0119] "Natural language generation technology" is a technology that allows computers to generate human language.

[0120] A system for carrying out the present invention analyzes live video of amateur sports and provides real-time scores and commentary on scoring scenes. A specific embodiment of this system will be described below.

[0121] System Configuration

[0122] 1. Hardware:

[0123] Server: A server with high-performance computing power (e.g. AWS EC2 instance)

[0124] Devices: Smartphones (e.g. iPhone (registered trademark), ANDROID (registered trademark) devices), head-mounted displays (e.g. Oculus Quest, HoloLens (registered trademark))

[0125] 2. Software:

[0126] Image recognition technology: OpenCV, TENSOFLOW (registered trademark)

[0127] Natural language generation technology: GPT-3 (registered trademark), BERT

[0128] Speech synthesis technology: Google(R) Text-to-Speech

[0129] Streaming services: AWS Elemental MediaLive, YouTube(R) Live

[0130] Data processing and calculation

[0131] 1. Live video analysis:

[0132] The server captures frames of live video and uses image recognition technology (OpenCV, TENSOFLOW (registered trademark)) to identify goal scenes and scorers.

[0133] The server times the match in real time and generates a score display.

[0134] 2. Commentary generation for goal scoring scenes:

[0135] When a scoring scene is identified, the server generates a detailed description of the scene using natural language generation technology (GPT-3 (registered trademark), BERT).

[0136] The generated commentary is converted into audio using speech synthesis technology (Google® Text-to-Speech).

[0137] 3. Providing the user interface:

[0138] The terminal provides a user interface optimized for smart devices, allowing viewers to watch the game comfortably.

[0139] Examples

[0140] For example, when an amateur soccer match is being broadcast live, users watch the match through their smartphones or head-mounted displays. When a goal is scored, the server automatically identifies the scene and displays the name of the scorer and the details of the goal in real time. In addition, commentary of the goal is provided in audio and text, allowing users to gain a deeper understanding of the flow of the match.

[0141] Examples of prompt statements

[0142] "The current match time is 45 minutes. A goal has been scored. The scorer is Tanaka. Please generate a detailed explanation of this scene."

[0143] By inputting this prompt into a generative AI model, a detailed explanation of the scoring scene is generated.

[0144] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0145] Step 1:

[0146] The server captures frames of live video. The input is the live video stream and the output is the individual frame images. The server receives the live video using a streaming service (e.g. AWS Elemental MediaLive) and splits it into frames.

[0147] Step 2:

[0148] The server analyzes the captured frame images using image recognition technology (e.g., OpenCV, TENSOFLOW (registered trademark)). The input is the frame image, and the output is the goal scene and the scorer identification result. The server identifies specific actions and people in the frame and identifies the goal scene and the scorer.

[0149] Step 3:

[0150] The server measures the match time in real time. The input is the live video stream and the output is the current match time. The server calculates the elapsed time based on the start time of the video and updates the match time in real time.

[0151] Step 4:

[0152] The server generates a score display based on the analysis results. The inputs are goal scenes, scorer identification results, and match time, and the output is score display data. The server updates the score when a goal occurs and generates data to display it visually.

[0153] Step 5:

[0154] When a scoring scene is identified, the server generates a detailed commentary of the scene using natural language generation technology (e.g., GPT-3 (registered trademark), BERT). The input is the result of identifying the scoring scene and the game time, and the output is the commentary text. The server inputs the prompt sentence into the generative AI model and generates a detailed commentary.

[0155] Step 6:

[0156] The server converts the generated commentary text into audio using voice synthesis technology (e.g., Google (registered trademark) Text-to-Speech). The input is the commentary text, and the output is audio data. The server converts the text data into audio data and provides it to the viewer.

[0157] Step 7:

[0158] The terminal provides a user interface optimized for smart devices. The input is score display data and audio data, and the output is visual and audio information. The terminal displays the score and commentary through the user interface and plays the audio commentary.

[0159] Step 8:

[0160] Users use smart devices to watch games. The input is the visual and auditory information provided by the device, and the output is the user's viewing experience. Users understand the flow of the game through real-time updated scores and commentary.

[0161] Example 2

[0162] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0163] In live amateur sports videos, it is difficult to provide detailed commentary on the progress of the game and scoring scenes in real time. In particular, identifying scoring scenes and generating commentary requires advanced technology, and conventional systems have not been able to do this efficiently.

[0164] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0165] In this invention, the server includes a means for acquiring live video, a means for analyzing the acquired video data using image recognition technology, a means for generating a score display based on the analysis result, a means for identifying scoring scenes based on the analysis result, a means for generating commentary in voice or text for the identified scoring scenes using natural language generation technology, a means for converting the generated commentary into a voice file using voice synthesis technology and providing it to the user, and a means for displaying the generated commentary in text format, thereby making it possible to provide detailed commentary on the progress of the game and scoring scenes in real time.

[0166] "Live video" refers to video data that is filmed in real time and instantly distributed via the Internet or other means.

[0167] "Video data" is digital data that includes visual information captured by a photographing device such as a camera.

[0168] "Image recognition technology" is a technology that uses computer vision technology to identify specific objects or patterns from images or videos.

[0169] "Score display" is a display format that visually shows the score and progress of a game.

[0170] A "scoring scene" is the moment when a goal is scored in a sports game or the scenes before and after that moment.

[0171] "Natural language generation technology" is a technology that enables computers to generate human language, and generates sentences based on specific input data.

[0172] "Speech synthesis technology" is a technology that converts text data into voice data.

[0173] "User" refers to any person or organization that uses this system.

[0174] A "text format" is a format of digital data that contains text information.

[0175] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0176] The present invention is a system that analyzes live video of amateur sports and provides detailed commentary of the progress of the game and scoring scenes in real time. A specific embodiment of this system will be described below.

[0177] Live video capture

[0178] The server acquires live video in real time. Specifically, it uses a high-performance video capture device to capture the video of the match, and transmits the data to the server using a video streaming tool such as FFmpeg. The server temporarily stores the received video data and prepares it for analysis.

[0179] Video data analysis

[0180] The server analyzes the stored video data. Specifically, it uses image recognition libraries such as OpenCV and TENSOFLOW (registered trademark) to identify basketball goal scenes and scorers in the video. For example, the server processes the images frame by frame to track the movement of the basketball. It also recognizes the uniform numbers of the players and identifies the scorers.

[0181] Generate score display

[0182] The server generates a score display for the match based on the analysis results. When a point is scored, the server updates the scoreboard data and displays it to the user in real time. Specifically, the server displays the scores on a web page using HTML and JavaScript (registered trademark). It also uses WebSocket to send updates in real time to notify users of changes in the score.

[0183] Identifying goal-scoring scenes and generating commentary

[0184] When a scoring scene is identified, the server generates a detailed commentary on that scene. This commentary generation uses natural language generation technology such as GPT-3 (registered trademark). Specifically, information such as the scorer's name, the scoring method, and the game situation is input into the generation AI model as a prompt sentence to generate a detailed commentary. For example, the server generates the following prompt sentence:

[0185] "With 2 minutes and 30 seconds left in the third quarter, 23 players made a three-point shot. With this score, Team A took a five-point lead over Team B."

[0186] The generative AI model generates a detailed explanation based on this prompt.

[0187] Providing commentary

[0188] The server provides the generated commentary to the user in the form of voice or text. For voice synthesis, Google (registered trademark) Text-to-Speech API or the like is used, and the generated audio file is streamed to the user. The text commentary is displayed on a web page or application. For example, the server embeds the generated commentary in a web page in HTML format, allowing the user to view it in real time.

[0189] In this way, users can follow the progress of the game and get detailed commentary on the scoring scenes in real time. This system is an effective means of improving the viewing experience of amateur sports.

[0190] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0191] Step 1:

[0192] Live video capture

[0193] The server acquires live video in real time. Specifically, it uses a high-performance video capture device to capture the video of the match, and sends the data to the server using a video streaming tool such as FFmpeg. The input is the live video, and the output is the video data stored on the server. The server temporarily stores the received video data and prepares it for analysis.

[0194] Step 2:

[0195] Video data analysis

[0196] The server analyzes the stored video data. Specifically, it uses image recognition libraries such as OpenCV and TENSOFLOW (registered trademark) to identify basketball goal scenes and scorers in the video. The input is the stored video data, and the output is the analysis results (information on the scoring scene and scorer). The server processes the images frame by frame and tracks the movement of the basketball. It also recognizes the players' uniform numbers and identifies the scorers.

[0197] Step 3:

[0198] Generate score display

[0199] The server generates a display of the match score based on the analysis results. The input is the analysis results, and the output is the updated scoreboard data. When a point is scored, the server updates the scoreboard data and displays it to the user in real time. Specifically, it displays the scores on a webpage using HTML and JavaScript. It also uses WebSockets to send updates in real time to notify users of score changes.

[0200] Step 4:

[0201] Identifying goal-scoring scenes and generating commentary

[0202] When a scoring scene is identified, the server generates a detailed commentary on that scene. The input is the information on the scoring scene, and the output is the generated commentary. This commentary generation uses natural language generation technology such as GPT-3 (registered trademark). Specifically, information such as the scorer's name, the scoring method, and the situation of the game is input into the generation AI model as a prompt sentence to generate a detailed commentary. For example, the server generates a prompt sentence like the following:

[0203] "With 2 minutes and 30 seconds left in the third quarter, 23 players made a three-point shot. With this score, Team A took a five-point lead over Team B."

[0204] The generative AI model generates a detailed explanation based on this prompt.

[0205] Step 5:

[0206] Providing commentary

[0207] The server provides the generated commentary to the user in the form of voice or text. The input is the generated commentary, and the output is an audio file and text data. For voice synthesis, Google (registered trademark) Text-to-Speech API or the like is used, and the generated audio file is streamed to the user. The text commentary is displayed on a web page or application. For example, the server embeds the generated commentary in a web page in HTML format, allowing the user to view it in real time.

[0208] (Application example 2)

[0209] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0210] In conventional live video distribution systems for amateur sports, the display of game scores and the identification of scoring scenes are often done manually, which causes problems such as a lack of real-timeness and accuracy. In addition, it is difficult to provide detailed commentary on scoring scenes, which makes it difficult to increase viewer satisfaction. This has led to the issue that the viewing experience of amateur sports is inferior to that of professional sports.

[0211] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0212] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in voice or text for the identified scoring scene, a means for streaming the live video, and a means for generating a detailed commentary of the scoring scene using a voice synthesis technique. This makes it possible to display an accurate score and identify a scoring scene in real time, and further provide a detailed commentary in voice or text, thereby improving the viewing experience of the viewer.

[0213] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[0214] "Live video" is video content that is filmed and distributed in real time.

[0215] "Analysis" is the process of examining data in detail and extracting specific information.

[0216] The "score display" is a display that visually shows the score and progress of the game.

[0217] A "scoring scene" is a video of the moment when a goal is scored in a sporting event.

[0218] "Audio and text commentary" refers to providing audio or text explanations or comments for a video.

[0219] "Streaming" is a technology for delivering video and audio in real time over the Internet.

[0220] "Speech synthesis technology" is a technology that converts text data into speech.

[0221] "Image recognition technology" is a technology that extracts and analyzes specific information from image data.

[0222] "Natural language generation technology" is a technology that allows computers to generate human language.

[0223] The system for carrying out the present invention analyzes live video of amateur sports, displays the score, identifies the scoring scenes, and provides detailed commentary in real time. Specific embodiments are described below.

[0224] Hardware and Software Configuration

[0225] The server uses the following hardware and software:

[0226] Hardware: Servers with powerful processors, memory, storage, and network interfaces.

[0227] software:

[0228] OpenCV: Image processing library

[0229] TENSOFLOW(registered trademark) / Keras: Framework for image recognition models

[0230] Transformers (Hugging Face): A Framework for Natural Language Generation Models

[0231] pyttsx3: A speech synthesis library

[0232] Data processing and calculation

[0233] 1. Live video streaming: The server receives and streams live video in real time over the Internet.

[0234] 2. Frame capture and pre-processing: The server captures frames from the live video and performs pre-processing such as resizing and normalization using OpenCV.

[0235] 3. Image Recognition: Analyze the pre-processed frames using TENSOFLOW® / Keras to identify scoring scenes and players.

[0236] 4. Identifying scoring scenes: Based on the analysis results, Identify scoring scenes.

[0237] 5. Natural Language Generation: When a scoring scene is identified, we use Transformers to generate a detailed commentary. An example prompt is "Tanaka scored a point in a basketball game. Please generate a detailed commentary."

[0238] 6. Speech synthesis: The generated explanation is converted into speech using pyttsx3 and provided to the user.

[0239] Examples

[0240] For example, if a scene such as "Tanaka makes a three-point shot!" is detected during a game, the server will perform the following process.

[0241] 1. Capture and pre-process frames from live video.

[0242] 2. Use an image recognition model to identify scoring scenes.

[0243] 3. If a scoring scene is identified, generate an explanation using a natural language generation model using the prompt sentence, "Tanaka scored a point in the basketball game. Please generate a detailed explanation."

[0244] 4. The generated commentary is converted into audio using speech synthesis technology and played back, for example, "Tanaka made a great three-point shot, widening the team's lead."

[0245] In this way, accurate scores can be displayed in real time, scoring scenes can be identified, and detailed commentary can be provided in audio and text, improving the viewer's viewing experience.

[0246] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0247] Step 1:

[0248] The server receives live video over the Internet and performs streaming. The input is the live video stream data, and the output is frames captured in real time. Specifically, the server splits the video stream and processes each frame.

[0249] Step 2:

[0250] The server preprocesses the captured frames using OpenCV. The input is the captured frame and the output is the preprocessed frame. Specific operations include resizing the frame (e.g. to 224x224 pixels) and normalizing it (scaling pixel values ​​to the range 0 to 1).

[0251] Step 3:

[0252] The server analyzes the preprocessed frames using TENSOFLOW (registered trademark) / Keras to identify scoring scenes and players. The input is the preprocessed frames, and the output is the analysis results (whether or not a scoring scene occurred and the identification of the player). In concrete terms, the server inputs the frames into an image recognition model and obtains the probability of a scoring scene and the identification of the player.

[0253] Step 4:

[0254] The server identifies a scoring scene based on the analysis result. The input is the analysis result, and the output is the result of identifying the scoring scene. In concrete terms, if the probability of a scoring scene exceeds a certain threshold, the server identifies the scene as a scoring scene.

[0255] Step 5:

[0256] When a scoring scene is identified, the server generates a prompt sentence and generates a detailed commentary using Transformers. The input is the result of identifying the scoring scene and the prompt sentence, and the output is the generated commentary text. In concrete terms, the prompt sentence "A player scored a point in a basketball game. Please generate a detailed commentary" is input into the natural language generation model, and commentary text is generated.

[0257] Step 6:

[0258] The server converts the generated commentary text into audio using pyttsx3 and provides it to the user. The input is the generated commentary text, and the output is audio data. In concrete terms, the commentary text is input to a speech synthesis engine, audio data is generated, and the audio data is played through a speaker.

[0259] In this way, the server can analyze live video in real time, identify scoring moments and generate detailed commentary to serve to viewers.

[0260] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0261] "Example 1"

[0262] In one embodiment of the present invention, a system for analyzing live amateur sports videos, generating score displays based on the analysis results, identifying scoring scenes, and generating commentary in audio or text for the identified scoring scenes is combined with an emotion engine that recognizes the emotions of a user. The emotion engine recognizes emotions from the user's facial expressions and tone of voice, and adjusts the generation of commentary based on the recognized emotions. For example, if the user shows a happy expression, the emotion engine feeds that information back to the analysis means, and the commentary generation means generates a more exciting commentary. Conversely, if the user shows a disappointed expression, the commentary generation means generates a more calm tone of commentary. This provides a personalized experience according to the user's emotions.

[0263] "Example 2"

[0264] In one embodiment of the present invention, a system for analyzing live amateur sports videos, generating score displays based on the analysis results, identifying scoring scenes, and generating commentary in audio or text for the identified scoring scenes is combined with an emotion engine that recognizes the emotions of a user. The emotion engine recognizes emotions from the user's facial expressions and tone of voice, and adjusts the generation of commentary based on the recognized emotions. For example, if the user shows a happy expression, the emotion engine feeds that information back to the analysis means, and the commentary generation means generates a more exciting commentary. Conversely, if the user shows a disappointed expression, the commentary generation means generates a more calm tone of commentary. This provides a personalized experience according to the user's emotions.

[0265] The process flow of each embodiment will be described below.

[0266] "Example 1"

[0267] Step 1: Live amateur sports video is entered into the system.

[0268] Step 2: The system analyzes the live video input and generates a score display.

[0269] Step 3: The system identifies scoring moments and generates audio and / or text commentary for the identified scoring moments.

[0270] Step 4: At the same time, the emotion engine recognizes the user's emotions from their facial expressions and tone of voice.

[0271] Step 5: The emotion engine adjusts the commentary generation based on the recognized emotion. For example, if the user shows a happy expression, the emotion engine feeds that information back to the analysis means, and the commentary generation means generates a more exciting commentary. Conversely, if the user shows a disappointed expression, the commentary generation means generates a more calm-toned commentary.

[0272] "Example 2"

[0273] Step 1: Live amateur sports video is entered into the system.

[0274] Step 2: The system analyzes the live video input and generates a score display.

[0275] Step 3: The system identifies scoring moments and generates audio and / or text commentary for the identified scoring moments.

[0276] Step 4: At the same time, the emotion engine recognizes the user's emotions from their facial expressions and tone of voice.

[0277] Step 5: The emotion engine adjusts the commentary generation based on the recognized emotion. For example, if the user shows a happy expression, the emotion engine feeds that information back to the analysis means, and the commentary generation means generates a more exciting commentary. Conversely, if the user shows a disappointed expression, the commentary generation means generates a more calm-toned commentary.

[0278] Example 1

[0279] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0280] Conventional live video analysis systems for amateur sports were able to display the score of a match and identify the scoring scenes, but they were unable to provide personalized commentary according to the user's emotions. As a result, the user experience was uniform, and flexible responses according to individual emotions were required.

[0281] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for acquiring live video of amateur sports, a means for analyzing the acquired video data using image recognition technology, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in voice or text for the identified scoring scene using natural language generation technology, a means for recognizing the user's emotion, and a means for adjusting the tone of the commentary based on the recognized emotion. This makes it possible to provide a personalized commentary according to the user's emotion.

[0282] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[0283] "Live video" is video data that is shot and distributed in real time.

[0284] "Means for acquiring" refers to a method or device for receiving live video in real time and importing it as data.

[0285] "Video data" is digital data that includes visual information captured by a photographic device such as a camera.

[0286] "Image recognition technology" refers to a technology that uses computer vision technology to identify specific objects or events from video data.

[0287] "Means for analyzing" refers to a method or device for processing acquired video data and extracting specific information.

[0288] A "score display" is an information display that visually shows the score and progress of a game.

[0289] A "generating means" is a method or device for producing new data or displays based on specific information.

[0290] A "scoring scene" refers to the moment or situation in which a goal is scored in a sporting event.

[0291] The "means for identifying" refers to a method or device for identifying a particular event or object from the analysis results.

[0292] "Natural language generation technology" is technology that enables computers to generate human language.

[0293] A "means for generating audio or textual descriptions" is a method or apparatus for generating descriptions in audio or text form using natural language generation techniques.

[0294] The "means for recognizing a user's emotion" refers to a method or device for analyzing a user's facial expression and tone of voice to identify the emotion.

[0295] A "means for adjusting the tone of commentary" is a method or device for modifying the manner in which commentary is presented based on perceived emotions.

[0296] This invention is a system that analyzes live amateur sports videos and provides personalized commentary according to the user's emotions. This system consists of three main components: a server, a terminal, and a user.

[0297] Hardware and Software Used

[0298] Hardware: High-performance servers, user devices (PCs, smartphones, etc.)

[0299] Software: Image recognition technology (e.g., OpenCV), natural language generation technology (e.g., GPT-3 (registered trademark)), emotion recognition engine (e.g., Microsoft (registered trademark) Azure (registered trademark) Emotion API)

[0300] Specific operation of the system

[0301] 1. Acquiring live video

[0302] The server obtains live amateur sports videos in real time. Specifically, it receives video data from streaming services (e.g., YouTube Live, Twitch). The server starts the live stream from a specified URL and stores the video data in a buffer so that it can be processed in real time.

[0303] 2. Analysis of video data

[0304] The server analyzes the acquired video data using image recognition technology (e.g. OpenCV). Specifically, it identifies goal scenes and scorers, and keeps track of the game time. The server performs image analysis for each frame, detects goal scenes, and identifies the scorers by identifying the players' jersey numbers and uniform colors. It also reads the game time from the timer in the video.

[0305] 3. Generate score display

[0306] The server generates a display of the match score based on the analysis results, e.g., updates the score when a goal is recognized. The server increments the score the moment a goal is detected and generates a text overlay to display the scorer's name and the time of the goal.

[0307] 4. Generating commentary on goal-scoring scenes

[0308] When a scoring scene is identified, the server generates a detailed commentary of that scene using natural language generation technology (e.g., GPT-3 (registered trademark)). The commentary is provided in audio and text. The server analyzes the video data of the scoring scene and extracts important events (e.g., assists, type of shot). A prompt sentence is generated based on the extracted information and input into the generative AI model. The generated commentary is converted into audio using speech synthesis technology (e.g., Google (registered trademark) Text-to-Speech).

[0309] 5. User Emotion Recognition

[0310] The device analyzes the user's facial expressions and tone of voice with an emotion recognition engine (e.g., Microsoft Azure Emotion API) to recognize the user's emotions. The device uses a camera and microphone to capture the user's facial expressions and voice, and sends the captured data to the emotion recognition engine to analyze the emotions. The analysis results are then sent to the server.

[0311] 6. Adjust your commentary based on emotions

[0312] The server adjusts the tone of the commentary based on the recognized user's emotions. For example, if the user shows a happy expression, it generates a more exciting commentary. Conversely, if the user shows a disappointed expression, it generates a more calm tone of commentary. The server receives the user's emotion data and feeds it back into the commentary generation process. It adjusts the prompt sentence based on the emotion data and re-inputs it into the generative AI model. The adjusted commentary is regenerated using speech synthesis technology and provided to the user.

[0313] Examples and prompts

[0314] Examples

[0315] Scenario: A goal is scored during an amateur soccer match.

[0316] Process flow:

[0317] 1. The server recognizes goal scenes from live video.

[0318] 2. The server updates the score and displays the scorer information.

[0319] 3. The server generates commentary of the scoring scene and provides it in audio and text.

[0320] 4. The device analyzes the user's facial expressions and recognizes expressions of joy.

[0321] 5. The server reproduces the commentary in an exciting tone and provides it to the user.

[0322] Examples of prompt statements

[0323] "Describe a scene where a goal is scored during an amateur soccer match. If the user shows signs of joy, generate commentary in an exciting tone."

[0324] The above is an embodiment of the present invention.

[0325] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0326] Step 1: Getting live video

[0327] The server retrieves live amateur sports videos in real time. As input, it receives the URL of a streaming service. Specifically, the server starts the live stream from the specified URL and stores the video data in a buffer. As output, it obtains video data that can be processed in real time.

[0328] Step 2: Analyze the video data

[0329] The server analyzes the acquired video data using image recognition technology (e.g. OpenCV). It receives frame data of live video as input. Specifically, the server performs image analysis for each frame, detects goal scenes, and identifies the scorers by identifying the players' jersey numbers and uniform colors. It also reads the match time from a timer in the video. It obtains the goal scene detection results, scorer information, and match time as output.

[0330] Step 3: Generate the score display

[0331] The server generates a match score display based on the analysis results. As input, it receives goal detection results, scorer information, and match time. Specifically, the server increments the score the moment a goal is detected and generates a text overlay to display the scorer's name and the time of the goal. As output, it gets an updated score display.

[0332] Step 4: Generate commentary on goal scoring scenes

[0333] When a scoring scene is identified, the server generates a detailed commentary of the scene using natural language generation technology (e.g., GPT-3 (registered trademark)). As input, the server receives video data of the scoring scene and important event information (e.g., assists, type of shot). In specific operations, the server generates a prompt sentence based on the extracted information and inputs it into the generative AI model. The generated commentary is converted into audio using speech synthesis technology (e.g., Google (registered trademark) Text-to-Speech). As output, the server obtains a commentary in audio and text format.

[0334] Step 5: Recognizing User Emotions

[0335] The device analyzes the user's facial expressions and tone of voice using an emotion recognition engine (e.g. Microsoft Azure Emotion API) to recognize the user's emotions. As input, it receives the user's facial expressions and voice data captured by the camera and microphone. Specifically, the device sends the captured data to the emotion recognition engine and analyzes the emotions. It then sends the analysis results to the server. As output, it obtains the user's emotional data.

[0336] Step 6: Adjust your commentary based on emotion

[0337] The server adjusts the tone of the commentary based on the recognized user emotion. As input, it receives the user's emotion data. In concrete terms, the server adjusts the prompt sentence based on the emotion data and re-inputs it into the generative AI model. The adjusted commentary is regenerated by speech synthesis technology and provided to the user. As output, a personalized commentary according to the user's emotion is obtained.

[0338] (Application example 1)

[0339] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0340] Conventional live video analysis systems for amateur sports display game scores, identify scoring scenes, and generate commentary, but they are unable to provide personalized commentary based on the user's emotions. This results in a uniform viewing experience for users, making it difficult to provide optimal commentary based on individual emotions.

[0341] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0342] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying scoring scenes based on the analysis result, a means for generating commentary in audio or text for the identified scoring scenes, a means for recognizing a user's emotion, and a means for adjusting the tone of the commentary based on the recognized emotion, thereby making it possible to provide a personalized commentary according to the user's emotion.

[0343] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[0344] "Live video" is video that is filmed in real time and streamed simultaneously.

[0345] "Analysis" is the process of examining data or information in detail to derive specific results or meanings.

[0346] The "score display" is a display that visually shows the score and progress of the game.

[0347] A "scoring scene" is a video or scene showing the exact moment a goal is scored in a sporting event.

[0348] A "commentary" is an explanation or commentary on a particular event or situation.

[0349] "Speech" refers to auditory information such as speech and music.

[0350] "Text" is information or writing written in text.

[0351] "User" means an individual or organization that uses the system or service.

[0352] Emotions are human psychological states and feelings.

[0353] "Cognition" is the process of understanding and grasping specific information or a situation.

[0354] "Tone" is the mood or atmosphere of a voice or text.

[0355] "Adaptation" is the process of changing or modifying something to suit a specific purpose.

[0356] "Personalization" means customizing to suit the characteristics and preferences of individual users.

[0357] As an embodiment of the present invention, a system is constructed that analyzes live videos of amateur sports and provides personalized commentary according to the user's emotions. A specific embodiment is described below.

[0358] Hardware and Software Configuration

[0359] Hardware

[0360] Server: Analyzes live video, generates score displays, identifies scoring moments, generates commentary, and recognizes emotions.

[0361] Device: The device (smartphone, tablet, PC, etc.) through which a user watches live video.

[0362] Camera: A device for capturing live video.

[0363] Microphone: A device for collecting the user's voice.

[0364] software

[0365] OpenCV: A library for live video streaming and frame processing.

[0366] Keras: A library that provides an image recognition model used to identify goal scoring scenes.

[0367] Transformers (Hugging Face): A library that generates commentary using the GPT-2 model.

[0368] SpeechRecognition: A library that converts the user's speech into text.

[0369] EmotionRecognition: A library for recognizing user emotions.

[0370] Data processing and calculation

[0371] Server Processing

[0372] The server receives the live video from the camera and analyzes the frames in real time using OpenCV. Based on the analysis, it generates a score display of the match and identifies the scoring scenes. If a scoring scene is identified, it uses the GPT-2 model to generate commentary for that scene.

[0373] User Emotion Recognition

[0374] It collects voice data through the microphone connected to the user's device and converts the voice to text using SpeechRecognition. It analyzes the converted text with the EmotionRecognition library to recognize the user's emotions. It adjusts the tone of the commentary based on the recognized emotions.

[0375] Examples

[0376] Live video URL

[0377] Example: http: / / example.com / live_stream

[0378] Explanation of the goal

[0379] Example: Player X scores a goal

[0380] User audio files

[0381] Example: user_audio.wav

[0382] Examples of prompt statements

[0383] Your goal is to develop a system that analyzes live amateur soccer videos, displays match scores, identifies goal-scoring moments, and generates commentary. The system will recognize the user's emotions and provide commentary that reflects those emotions. Specifically, your goal is to create a smartphone application that has the following functions:

[0384] 1. Live video streaming

[0385] 2. Score display

[0386] 3. Identifying the goal-scoring moment

[0387] 4. Description Generation

[0388] 5. Emotion recognition

[0389] In this way, a system can be realized that provides personalized commentary according to the user's emotions.

[0390] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0391] Step 1:

[0392] The server receives the live video sent from the camera. The input is the live video stream data, and the output is the image data for each frame. Using OpenCV, we grab the frames in real time and prepare them for analysis.

[0393] Step 2:

[0394] The server analyzes the captured frames and generates a display of the match score. The input is image data for each frame, and the output is text data that displays the score. Image recognition technology is used to determine the score and the progress of the match, and to update the score.

[0395] Step 3:

[0396] The server identifies the goal-scoring scene based on the analysis results. The input is the analyzed frame data, and the output is the score scene identification information. Using Keras, we detect the goal scene and identify the scene.

[0397] Step 4:

[0398] The server generates a commentary in voice or text for the identified scoring scene. The input is the specific information of the scoring scene, and the output is the text data or audio data of the commentary. The commentary is generated using natural language generation technology using the GPT-2 model of Transformers (Hugging Face).

[0399] Step 5:

[0400] The terminal collects the user's voice. The input is the user's voice data, and the output is an audio file. The terminal uses a microphone to record the user's voice and saves it as an audio file.

[0401] Step 6:

[0402] The server analyzes the collected voice data and recognizes the user's emotions. The input is the audio file and the output is the user's emotion information. It uses SpeechRecognition to convert the voice to text and the EmotionRecognition library to recognize emotions.

[0403] Step 7:

[0404] The server adjusts the tone of the commentary based on the recognized emotion. The input is the user's emotion information and the generated commentary data, and the output is the adjusted commentary data. Based on the emotion information, the tone of the commentary is changed to exciting or calm.

[0405] Step 8:

[0406] The terminal provides the adjusted commentary to the user. The input is the adjusted commentary data, and the output is display or playback of the commentary to the user. The commentary is provided to the user using the screen and speaker of the terminal.

[0407] Example 2

[0408] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0409] Conventional live video analysis systems for amateur sports can display the score of a match and identify the scoring scenes, but they cannot provide personalized commentary according to the user's emotions. Therefore, in order to enrich the user experience, a function that recognizes the user's emotions and adjusts the commentary based on them is required.

[0410] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0411] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying scoring scenes based on the analysis result, a means for generating commentary in audio or text for the identified scoring scenes, a means for recognizing a user's emotion, and a means for adjusting the commentary based on the recognized emotion, thereby making it possible to provide a personalized commentary according to the user's emotion.

[0412] "Amateur sports" refers to sports competitions played by non-professional individuals or teams.

[0413] "Live video" refers to video footage that is filmed in real time and streamed simultaneously.

[0414] "Analysis" is the process of examining acquired data in detail and extracting specific information.

[0415] A "score display" is a display that visually shows the score and progress of a game.

[0416] A "scoring scene" refers to footage of the exact moment a goal is scored in a sporting event.

[0417] "Means for generating commentary in audio or text" refers to technology that creates audio or text explanations of scoring scenes or game situations.

[0418] "Means for recognizing a user's emotions" refers to technology that determines emotions from a user's facial expressions, tone of voice, etc.

[0419] "Means for tailoring commentary" refers to techniques that change the content and tone of commentary based on recognized emotions.

[0420] "Image recognition technology" refers to the technology of analyzing image data to detect specific objects or actions.

[0421] "Natural language generation technology" refers to technology that allows computers to generate human language.

[0422] This invention is a system that analyzes live videos of amateur sports, generates score displays based on the analysis results, identifies scoring scenes, and generates audio and text commentary for the identified scoring scenes. It also has a function of recognizing the user's emotions and adjusting the commentary based on those emotions.

[0423] Hardware and Software Used

[0424] Hardware: High performance servers, cameras, microphones

[0425] Software: Image recognition technology (e.g. OpenCV), natural language generation technology (e.g. GPT-3 (registered trademark)), emotion recognition engine (e.g. Affectiva)

[0426] Specific operation of the system

[0427] Live video capture

[0428] The server receives live video from the camera.

[0429] The server connects to the camera and receives the video data in real time.

[0430] Specific operation: The server obtains the video stream from the camera using the RTSP protocol.

[0431] Video data analysis

[0432] The server analyzes the acquired video data using image recognition technology.

[0433] The server uses OpenCV to identify basketball goal scenes and scorers in the video.

[0434] What it does: The server processes the images frame by frame and runs an algorithm to detect goal moments.

[0435] Generate score display

[0436] The server generates a score display for the match based on the analysis.

[0437] The server updates the scoreboard every time a point is scored.

[0438] Specific operation: When the server detects a score event, it records the score in the database and sends an instruction to update the score to the front end.

[0439] Commentary generation for scoring scenes

[0440] When a scoring scene is identified, the server generates a detailed commentary of the scene in audio and / or text.

[0441] The server uses GPT-3® to generate explanations in natural language.

[0442] Specific operation: The server inputs the information about the scoring scene into GPT-3 (registered trademark) and generates an explanatory text such as "That was a great shot!"

[0443] User Emotion Recognition

[0444] The device recognizes the user's emotions from their facial expressions and tone of voice.

[0445] The device uses Affectiva to analyze the user's emotions.

[0446] How it works: The device uses the camera and microphone to capture the user's facial expressions and voice, and calls Affectiva's API to obtain emotion data.

[0447] Adjustment of explanation

[0448] The server adjusts the commentary generation based on the recognized emotions.

[0449] The server adjusts the commentary to an exciting tone if the user is happy.

[0450] Specific operation: The server analyzes the emotion data and re-inputs parameters to GPT-3 (registered trademark) to change the tone of the commentary.

[0451] Examples and prompts

[0452] Examples:

[0453] As the user watches a game, the camera captures a goal scene.

[0454] The server uses OpenCV to detect goal scenes and identify the scorer.

[0455] How it works: The server analyzes the movement within the frame and detects the moment the ball enters the goal.

[0456] The server uses GPT-3(registered trademark) to generate an explanation: "That was a great shot!"

[0457] Specific operation: The server inputs the scoring scene information into GPT-3 (registered trademark) and generates an explanatory text.

[0458] The device uses Affectiva to analyze the user's facial expressions and recognize when the user is happy.

[0459] Specific operation: The device captures the user's smile with the camera and calls Affectiva's API to obtain emotion data.

[0460] The server adjusts his commentary to an exciting tone: "This shot could change the course of the game!"

[0461] Specific operation: The server analyzes the emotion data and re-inputs parameters to GPT-3 (registered trademark) to change the tone of the commentary.

[0462] Example prompt:

[0463] "When a user is watching a game and the camera captures a goal, how do you identify the scorer and generate commentary? Also, how do you adjust the commentary depending on the user's emotions?"

[0464] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0465] Step 1:

[0466] Live video capture

[0467] The server receives live video from the camera.

[0468] Input: Real-time video stream from the camera

[0469] Specific operation: The server obtains the video stream from the camera using the RTSP protocol.

[0470] Output: Live video data

[0471] Step 2:

[0472] Video data analysis

[0473] The server analyzes the acquired video data using image recognition technology.

[0474] Input: Live video data

[0475] What it does: The server uses OpenCV to process the images frame by frame and run algorithms to identify goals and scorers.

[0476] Output: Goal scene detection results, scorer identification results

[0477] Step 3:

[0478] Generate score display

[0479] The server generates a score display for the match based on the analysis.

[0480] Input: Goal scene detection results, goal scorer identification results

[0481] Specific operation: When the server detects a score event, it records the score in the database and sends an instruction to update the score to the front end.

[0482] Output: Updated score display

[0483] Step 4:

[0484] Commentary generation for scoring scenes

[0485] When a scoring scene is identified, the server generates a detailed commentary of the scene in audio and / or text.

[0486] Input: Goal scene detection results, goal scorer identification results

[0487] Specific operation: The server inputs the scoring scene information into GPT-3 (registered trademark) and generates an explanatory text in natural language.

[0488] Output: Description (e.g. "That was a great shot!")

[0489] Step 5:

[0490] User Emotion Recognition

[0491] The device recognizes the user's emotions from their facial expressions and tone of voice.

[0492] Input: User's facial expression data, voice tone data

[0493] How it works: The device uses the camera and microphone to capture the user's facial expressions and voice, and calls Affectiva's API to obtain emotion data.

[0494] Output: User emotion data

[0495] Step 6:

[0496] Adjustment of explanation

[0497] The server adjusts the commentary generation based on the recognized emotions.

[0498] Input: User emotion data, explanatory text

[0499] Specific operation: The server analyzes the emotion data and re-inputs parameters to GPT-3 (registered trademark) to change the tone of the commentary.

[0500] Output: Tailored explanatory text (e.g. "This shot could change the game!")

[0501] (Application example 2)

[0502] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0503] When watching live amateur sports videos, it is difficult for viewers to grasp the progress of the game and the scoring scenes in real time. In addition, because personalized commentary based on the viewer's emotions is not provided, the viewing experience is uniform and it is not possible to increase the satisfaction of each individual viewer. This makes it difficult to maintain the viewer's interest, and there is a problem that the quality of the viewing experience is reduced.

[0504] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0505] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in audio or text for the identified scoring scene, a means for recognizing a user's emotion, and a means for adjusting the generation of the commentary based on the recognized emotion. This not only enables a viewer to grasp the progress of the game and the scoring scenes in real time, but also provides a personalized commentary according to the viewer's emotion, thereby improving the quality of the viewing experience and increasing the viewer's satisfaction.

[0506] "Amateur sports" are sporting events in which non-professional individuals or teams participate.

[0507] "Live video" is video content that is filmed in real time and streamed simultaneously.

[0508] "Analysis" is the process of examining data in detail to understand its structure and meaning.

[0509] A "score display" is an information display that visually shows the score and progress of a match.

[0510] A "scoring scene" refers to the image or circumstances of the moment when a goal is scored in a sporting event.

[0511] "Generating commentary in audio or text" means creating audio or text explanations about the situation in a game or scoring scenes.

[0512] "Recognizing the user's emotions" means judging the viewer's emotional state from facial expressions, tone of voice, and the like.

[0513] "Adjusting commentary production" refers to changing the content and tone of commentary based on the recognized emotions.

[0514] "Image recognition technology" is a technology that analyzes image data and identifies specific objects or scenes.

[0515] "Natural language generation technology" is technology that enables computers to generate human language.

[0516] The system for carrying out the present invention analyzes live video of amateur sports, generates a score display based on the analysis results, identifies scoring scenes, generates audio and text commentary for the identified scoring scenes, and has the function of recognizing the user's emotions and adjusting the generation of commentary based on the emotions.

[0517] Hardware and Software Used

[0518] Hardware:

[0519] It uses the smartphone's camera and microphone for live video capture and user emotion recognition.

[0520] The server provides computational resources for video analysis and commentary generation.

[0521] software:

[0522] OpenCV: A library for capturing and displaying live video.

[0523] DeepFace: A library for recognizing emotions from user facial expressions.

[0524] Transformers (Hugging Face): A library for generating explanations using natural language generation techniques.

[0525] gTTS: A library for converting generated text descriptions into speech.

[0526] playsound: A library for playing generated sounds.

[0527] System Operation

[0528] 1. Live video analysis:

[0529] The server analyzes live video captured by the smartphone camera in real time.

[0530] Using OpenCV, we capture video frames and use image recognition techniques to identify scoring scenes and scorers.

[0531] 2. Generate the score display:

[0532] The server updates the match score in real time based on the analysis results and displays it on the smartphone screen.

[0533] 3. Emotion recognition:

[0534] It uses your smartphone's camera and microphone to capture your facial expressions and tone of voice.

[0535] Use DeepFace to recognize user emotions in real time.

[0536] 4. Description Generation:

[0537] If a scoring scene is identified, the server generates a detailed explanation of the scene.

[0538] It uses Transformers' natural language generation model to generate explanations based on the user's emotions.

[0539] The generated commentary is converted into audio using gTTS and played through the smartphone's speakers.

[0540] Examples

[0541] For example, suppose a user is watching an amateur basketball game on their smartphone. If a goal is scored during the game and the user shows a joyful expression, the system recognizes the emotion and generates an exciting commentary. The generated commentary is generated using the following prompt sentence:

[0542] Examples of prompt statements

[0543] If the user indicates the emotion of joy, generate an exciting description of the following scene: Basketball Goal Scene

[0544] In this way, a personalized commentary is provided according to the user's emotions, improving the quality of the viewing experience and increasing viewer satisfaction.

[0545] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0546] Step 1:

[0547] The server captures live video from the smartphone camera. The input is the live video stream and the output is the image data for each frame. OpenCV is used to get the video frames in real time.

[0548] Step 2:

[0549] The server analyzes the captured video frames and identifies the scoring scenes and scorers. The input is the image data of the video frames, and the output is the result of identifying the scoring scenes. Image recognition technology is used to identify the basketball goal scenes and scorers.

[0550] Step 3:

[0551] The server updates the score of the game based on the analysis results and displays it on the smartphone screen. The input is the result of identifying the scoring scene, and the output is an updated score display. The score display generation means updates the score in real time.

[0552] Step 4:

[0553] The device uses the smartphone's camera and microphone to capture the user's facial expressions and tone of voice. The input is the user's facial expression images and voice data, and the output is data for emotion recognition. The camera and microphone are used to capture the user's emotions in real time.

[0554] Step 5:

[0555] The server analyzes the captured data and recognizes the user's emotions. The input is the user's facial expression images and voice data, and the output is the recognized emotions. DeepFace is used to analyze the user's emotions.

[0556] Step 6:

[0557] The server generates a detailed commentary of a scoring scene when it is identified. The input is the scoring scene identification result and the recognized emotion, and the output is the generated commentary text. The commentary is generated based on the prompt sentence using a natural language generation model from Transformers.

[0558] Step 7:

[0559] The server converts the generated commentary text into speech and plays it through the smartphone speaker. The input is the generated commentary text and the output is the audio data. gTTS is used to convert the text to speech and plays the audio using playsound.

[0560] In this way, the system can analyze live video in real time and provide personalized commentary according to the user's emotions.

[0561] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0562] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0563] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google(登録商標).com ?hl="ja">) are mentioned.

[0564] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0565] [Second embodiment]

[0566] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0567] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0568] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0569] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0570] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0571] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0572] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0573] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0574] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0575] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0576] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0577] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0578] "Example 1"

[0579] An example of the present invention is a system that analyzes live videos of amateur soccer matches. This system uses image recognition technology to analyze video data obtained from the live videos and generate a display of the match score. Specifically, the system identifies goal scenes and scorers, and keeps track of the match time. When a goal-scoring scene is identified, the system generates a detailed commentary of that scene in voice and text. Natural language generation technology is used to generate this commentary.

[0580] "Example 2"

[0581] Another example is a system that analyzes live videos of amateur basketball games. This system uses image recognition technology to analyze video data obtained from the live videos and generate a display of the game score. Specifically, it identifies basketball goal scenes and scorers, and keeps track of the game time. In addition, when a scoring scene is identified, it generates a detailed commentary of that scene in voice and text. Natural language generation technology is used to generate this commentary.

[0582] The process flow of each embodiment will be described below.

[0583] "Example 1"

[0584] Step 1: Get live amateur football video.

[0585] Step 2: Extract video data from the captured live video and analyze it using image recognition technology. This analysis includes identifying goal scenes, scorers, and game time.

[0586] Step 3: Generate a score display based on the analysis results.

[0587] Step 4: If a scoring scene is identified, a detailed commentary of that scene is generated in audio and text format using natural language generation technology.

[0588] "Example 2"

[0589] Step 1: Get live amateur basketball video.

[0590] Step 2: Extract video data from the captured live video and analyze it using image recognition technology. This analysis includes identifying basketball goal scenes, scorers, and game time.

[0591] Step 3: Generate a score display based on the analysis results.

[0592] Step 4: If a scoring scene is identified, a detailed commentary of that scene is generated in audio and text format using natural language generation technology.

[0593] Example 1

[0594] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0595] In conventional live video analysis systems for amateur sports, displaying the game score and identifying the scoring scenes are often done manually, which means that they lack real-timeness and accuracy. In addition, commentary on the scoring scenes is also done manually, which makes it difficult to provide information quickly. This makes it difficult for spectators and officials to accurately and quickly grasp the progress of the game.

[0596] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0597] In this invention, the server includes a means for acquiring live video data, a means for preprocessing the acquired video data, a means for analyzing the preprocessed video data using image recognition technology, a means for identifying a scoring scene based on the analysis result, a means for generating a prompt sentence for generating a commentary for the identified scoring scene, a means for inputting the generated prompt sentence into a generation AI model to generate a commentary, and a means for outputting the generated commentary in voice or text. This automates the display of the score of the game and the identification of the scoring scene, making it possible to provide accurate information in real time. In addition, detailed commentary of the scoring scene is also quickly generated, making it possible for spectators and related parties to accurately and quickly grasp the progress of the game.

[0598] "Live video data" refers to video information acquired in real time.

[0599] "Preprocessing" refers to processes such as data resizing, noise removal, and frame extraction that are carried out to make the acquired video data easier to analyze.

[0600] "Image recognition technology" is a technology for detecting and identifying specific objects and actions from video data.

[0601] "Analysis results" refers to the analysis results of data obtained using image recognition technology.

[0602] A "scoring scene" is a video of the exact moment a goal is scored in a sports game.

[0603] A "prompt sentence" is an instruction sentence to be input into a generative AI model.

[0604] A "generative AI model" is an artificial intelligence model that generates natural language text based on a given prompt.

[0605] "Commentary" refers to detailed explanations and comments about the scoring scene.

[0606] "Audio output" refers to providing the generated description in audio form.

[0607] "Text output" refers to providing the generated commentary in text form.

[0608] This invention is a system that analyzes live videos of amateur sports and automatically generates game scores and detailed commentary of scoring scenes. This system works in cooperation with three entities: a server, a terminal, and a user.

[0609] First, the server acquires live video data. The video data is transmitted in real time from devices such as cameras and smartphones. The server streams the video data from the cameras using RTSP (Real-Time Streaming Protocol).

[0610] The server then pre-processes the captured video data, which includes image resizing, noise removal, frame extraction, etc. Specifically, the server uses OpenCV to resize the video data to 640x480 pixels and apply a noise removal filter.

[0611] The pre-processed video data is then analyzed using image recognition technology. The server uses libraries such as TENSOFLOW (registered trademark) and YOLO (You Only Look Once) to detect players and the ball in each frame. Based on the analysis results, the server identifies the goal-scoring scene. To identify the goal scene, the server detects the movement of the ball and the shaking of the goal net. The server detects the moment the ball crosses the goal line and saves that frame.

[0612] When a scoring scene is identified, the server generates a prompt sentence for generating commentary. For example, the server generates a prompt sentence such as, "Please generate a detailed commentary on the scene where Player A scored a goal." The server inputs this prompt sentence into a generative AI model (for example, GPT-3 (registered trademark)) and generates a commentary. The generative AI model generates a commentary such as, "Player A received a cross from the right side and scored a goal with a brilliant volley shot."

[0613] The generated commentary is output by the server in the form of voice or text. For voice output, Google® Text-to-Speech API or the like is used. The server sends the generated text commentary to Google® Text-to-Speech API to generate an audio file. The server then sends the audio file to the user's device. The user can watch or view the commentary through the device.

[0614] As a concrete example, consider a case where a server analyzes a live video and identifies the scene where player A scored a goal. The server inputs a prompt sentence, "Please generate a detailed commentary on the scene where player A scored a goal," into the generative AI model. The generative AI model generates a commentary, "Player A received a cross from the right side and scored a goal with a brilliant volley shot." The server converts the generated commentary into audio, which the user can listen to on their device.

[0615] The server also keeps track of the game time. For example, the server detects that the game time has passed 45 minutes, and inputs a prompt message to the generative AI model saying, "Please generate a message announcing the end of the first half." The generative AI model generates an explanation saying, "The first half has ended. The current score is 1-0." The server displays the generated explanation as text, and the user can view it on their device.

[0616] The above is an embodiment of the present invention.

[0617] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0618] Step 1:

[0619] The server acquires the live video data.

[0620] Input: Real-time video data sent from devices such as cameras and smartphones.

[0621] Specific operation: The server streams video data from the camera using RTSP (Real-Time Streaming Protocol).

[0622] Output: Acquired live video data.

[0623] Step 2:

[0624] The server pre-processes the acquired video data.

[0625] Input: Acquired live video data.

[0626] Specific operation: The server uses OpenCV to resize the video data to 640x480 pixels and apply a noise reduction filter.

[0627] Output: Pre-processed video data.

[0628] Step 3:

[0629] The server analyzes the pre-processed video data using image recognition technology.

[0630] Input: Preprocessed video data.

[0631] Specific operation: The server uses libraries such as TENSOFLOW (registered trademark) and YOLO (You Only Look Once) to detect players and the ball in each frame.

[0632] Output: Analysis results (positions of players and the ball, etc.).

[0633] Step 4:

[0634] The server identifies the scoring scene based on the analysis result.

[0635] Input: The analysis results.

[0636] What happens: The server detects the moment the ball crosses the goal line and saves that frame.

[0637] Output: Goal specific information (scoring frame).

[0638] Step 5:

[0639] The server generates a prompt sentence for generating an explanation for the scoring scene.

[0640] Input: Scoring scene specific information.

[0641] Specific operation: The server generates a prompt sentence saying, "Please generate a detailed commentary of the scene where player A scored a goal."

[0642] Output: The prompt statement.

[0643] Step 6:

[0644] The server inputs the generated prompt sentence into a generative AI model to generate an explanation.

[0645] Input: The prompt statement.

[0646] Specific operation: The server sends a prompt sentence to a generative AI model (e.g., GPT-3 (registered trademark)) to generate an explanation.

[0647] Output: The generated description (in text format).

[0648] Step 7:

[0649] The server outputs the generated commentary in audio and / or text.

[0650] Input: The generated description (in text format).

[0651] Specific operation: The server sends the generated text commentary to the Google® Text-to-Speech API to generate an audio file, and then sends the audio file to the user's device.

[0652] Output: Audio files and text description.

[0653] The above is the flow of processing of the program of this system.

[0654] (Application example 1)

[0655] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0656] In live video distribution of amateur sports, it is difficult to provide detailed commentary of game scores and scoring scenes in real time. In addition, when viewers watch games using smart devices, the user interface is not optimized, resulting in a poor viewing experience. Furthermore, there is also the issue that it is difficult for viewers to understand the flow of the game because commentary of scoring scenes cannot be provided by audio.

[0657] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0658] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in voice or text for the identified scoring scene, a means for providing a user interface optimized for a smart device, a means for measuring the game time in real time, and a means for converting detailed commentary of the scoring scene into voice using a voice synthesis technology. This makes it possible to provide detailed commentary of the game score and the scoring scene in real time, allowing viewers to comfortably watch the game using their smart devices. Also, by providing commentary of the scoring scene in voice, viewers can have a deeper understanding of the flow of the game.

[0659] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[0660] "Live video" is video that is filmed in real time and streamed simultaneously.

[0661] "Analysis" means examining data or information in detail to clarify its structure and meaning.

[0662] A "score display" is a visual indication of the score and progress of a match.

[0663] A "scoring scene" is a video of the moment when a goal is scored in a sporting event.

[0664] "Means for generating commentary in audio or text" refers to technology for explaining scoring scenes and the situation of the game in audio or text.

[0665] A "smart device" is an electronic device with advanced functions, such as a smartphone, tablet, or head-mounted display.

[0666] A "user interface" is a screen or method of operation that allows a user to interact with a system or application.

[0667] "Means for measuring match time in real time" refers to technology that measures and displays the progress of a match in real time.

[0668] "Speech synthesis technology" is a technology that converts text data into speech.

[0669] "Image recognition technology" is a technology that analyzes image data and recognizes specific objects or scenes.

[0670] "Natural language generation technology" is a technology that allows computers to generate human language.

[0671] A system for carrying out the present invention analyzes live video of amateur sports and provides real-time scores and commentary on scoring scenes. A specific embodiment of this system will be described below.

[0672] System Configuration

[0673] 1. Hardware:

[0674] Server: A server with powerful computing power (e.g. AWS EC2 instance)

[0675] Devices: Smartphones, head-mounted displays (e.g. Oculus Quest, HoloLens)

[0676] 2. Software:

[0677] Image recognition technology: OpenCV, TENSOFLOW (registered trademark)

[0678] Natural language generation technology: GPT-3 (registered trademark), BERT

[0679] Speech synthesis technology: Google(R) Text-to-Speech

[0680] Streaming services: AWS Elemental MediaLive, YouTube(R) Live

[0681] Data processing and calculation

[0682] 1. Live video analysis:

[0683] The server captures frames of live video and uses image recognition technology (OpenCV, TENSOFLOW (registered trademark)) to identify goal scenes and scorers.

[0684] The server times the match in real time and generates a score display.

[0685] 2. Commentary generation for goal scoring scenes:

[0686] When a scoring scene is identified, the server generates a detailed description of the scene using natural language generation technology (GPT-3 (registered trademark), BERT).

[0687] The generated commentary is converted into audio using speech synthesis technology (Google® Text-to-Speech).

[0688] 3. Providing the user interface:

[0689] The terminal provides a user interface optimized for smart devices, allowing viewers to watch the game comfortably.

[0690] Examples

[0691] For example, when an amateur soccer match is being broadcast live, users watch the match through their smartphones or head-mounted displays. When a goal is scored, the server automatically identifies the scene and displays the name of the scorer and the details of the goal in real time. In addition, commentary of the goal is provided in audio and text, allowing users to gain a deeper understanding of the flow of the match.

[0692] Examples of prompt statements

[0693] "The current match time is 45 minutes. A goal has been scored. The scorer is Tanaka. Please generate a detailed explanation of this scene."

[0694] By inputting this prompt into a generative AI model, a detailed explanation of the scoring scene is generated.

[0695] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0696] Step 1:

[0697] The server captures frames of live video. The input is the live video stream and the output is the individual frame images. The server receives the live video using a streaming service (e.g. AWS Elemental MediaLive) and splits it into frames.

[0698] Step 2:

[0699] The server analyzes the captured frame images using image recognition technology (e.g., OpenCV, TENSOFLOW (registered trademark)). The input is the frame image, and the output is the goal scene and the scorer identification result. The server identifies specific actions and people in the frame and identifies the goal scene and the scorer.

[0700] Step 3:

[0701] The server measures the match time in real time. The input is the live video stream and the output is the current match time. The server calculates the elapsed time based on the start time of the video and updates the match time in real time.

[0702] Step 4:

[0703] The server generates a score display based on the analysis results. The inputs are goal scenes, scorer identification results, and match time, and the output is score display data. The server updates the score when a goal occurs and generates data to display it visually.

[0704] Step 5:

[0705] When a scoring scene is identified, the server generates a detailed commentary of the scene using natural language generation technology (e.g., GPT-3 (registered trademark), BERT). The input is the result of identifying the scoring scene and the game time, and the output is the commentary text. The server inputs the prompt sentence into the generative AI model and generates a detailed commentary.

[0706] Step 6:

[0707] The server converts the generated commentary text into audio using voice synthesis technology (e.g., Google (registered trademark) Text-to-Speech). The input is the commentary text, and the output is audio data. The server converts the text data into audio data and provides it to the viewer.

[0708] Step 7:

[0709] The terminal provides a user interface optimized for smart devices. The input is score display data and audio data, and the output is visual and audio information. The terminal displays the score and commentary through the user interface and plays the audio commentary.

[0710] Step 8:

[0711] Users use smart devices to watch games. The input is the visual and auditory information provided by the device, and the output is the user's viewing experience. Users understand the flow of the game through real-time updated scores and commentary.

[0712] Example 2

[0713] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0714] In live amateur sports videos, it is difficult to provide detailed commentary on the progress of the game and scoring scenes in real time. In particular, identifying scoring scenes and generating commentary requires advanced technology, and conventional systems have not been able to do this efficiently.

[0715] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0716] In this invention, the server includes a means for acquiring live video, a means for analyzing the acquired video data using image recognition technology, a means for generating a score display based on the analysis result, a means for identifying scoring scenes based on the analysis result, a means for generating commentary in voice or text for the identified scoring scenes using natural language generation technology, a means for converting the generated commentary into a voice file using voice synthesis technology and providing it to the user, and a means for displaying the generated commentary in text format, thereby making it possible to provide detailed commentary on the progress of the game and scoring scenes in real time.

[0717] "Live video" refers to video data that is filmed in real time and instantly distributed via the Internet or other means.

[0718] "Video data" is digital data that includes visual information captured by a photographing device such as a camera.

[0719] "Image recognition technology" is a technology that uses computer vision technology to identify specific objects or patterns from images or videos.

[0720] "Score display" is a display format that visually shows the score and progress of a game.

[0721] A "scoring scene" is the moment when a goal is scored in a sports game or the scenes before and after that moment.

[0722] "Natural language generation technology" is a technology that enables computers to generate human language, and generates sentences based on specific input data.

[0723] "Speech synthesis technology" is a technology that converts text data into voice data.

[0724] "User" refers to any person or organization that uses this system.

[0725] A "text format" is a format of digital data that contains text information.

[0726] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0727] The present invention is a system that analyzes live video of amateur sports and provides detailed commentary of the progress of the game and scoring scenes in real time. A specific embodiment of this system will be described below.

[0728] Live video capture

[0729] The server acquires live video in real time. Specifically, it uses a high-performance video capture device to capture the video of the match, and transmits the data to the server using a video streaming tool such as FFmpeg. The server temporarily stores the received video data and prepares it for analysis.

[0730] Video data analysis

[0731] The server analyzes the stored video data. Specifically, it uses image recognition libraries such as OpenCV and TENSOFLOW (registered trademark) to identify basketball goal scenes and scorers in the video. For example, the server processes the images frame by frame to track the movement of the basketball. It also recognizes the uniform numbers of the players and identifies the scorers.

[0732] Generate score display

[0733] The server generates a score display for the match based on the analysis results. When a point is scored, the server updates the scoreboard data and displays it to the user in real time. Specifically, the server displays the score on a web page using HTML and JavaScript. It also uses WebSocket to send updates in real time to notify users of score changes.

[0734] Identifying goal-scoring scenes and generating commentary

[0735] When a scoring scene is identified, the server generates a detailed commentary on that scene. This commentary generation uses natural language generation technology such as GPT-3 (registered trademark). Specifically, information such as the scorer's name, the scoring method, and the game situation is input into the generation AI model as a prompt sentence to generate a detailed commentary. For example, the server generates the following prompt sentence:

[0736] "With 2 minutes and 30 seconds left in the third quarter, 23 players made a three-point shot. With this score, Team A took a five-point lead over Team B."

[0737] The generative AI model generates a detailed explanation based on this prompt.

[0738] Providing commentary

[0739] The server provides the generated commentary to the user in the form of voice or text. For voice synthesis, Google (registered trademark) Text-to-Speech API or the like is used, and the generated audio file is streamed to the user. The text commentary is displayed on a web page or application. For example, the server embeds the generated commentary in a web page in HTML format, allowing the user to view it in real time.

[0740] In this way, users can follow the progress of the game and get detailed commentary on the scoring scenes in real time. This system is an effective means of improving the viewing experience of amateur sports.

[0741] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0742] Step 1:

[0743] Live video capture

[0744] The server acquires live video in real time. Specifically, it uses a high-performance video capture device to capture the video of the match, and sends the data to the server using a video streaming tool such as FFmpeg. The input is the live video, and the output is the video data stored on the server. The server temporarily stores the received video data and prepares it for analysis.

[0745] Step 2:

[0746] Video data analysis

[0747] The server analyzes the stored video data. Specifically, it uses image recognition libraries such as OpenCV and TENSOFLOW (registered trademark) to identify basketball goal scenes and scorers in the video. The input is the stored video data, and the output is the analysis results (information on the scoring scene and scorer). The server processes the images frame by frame and tracks the movement of the basketball. It also recognizes the players' uniform numbers and identifies the scorers.

[0748] Step 3:

[0749] Generate score display

[0750] The server generates a display of the match score based on the analysis results. The input is the analysis results, and the output is the updated scoreboard data. When a point is scored, the server updates the scoreboard data and displays it to the user in real time. Specifically, it displays the scores on a webpage using HTML and JavaScript. It also uses WebSockets to send updates in real time to notify users of score changes.

[0751] Step 4:

[0752] Identifying goal-scoring scenes and generating commentary

[0753] When a scoring scene is identified, the server generates a detailed commentary on that scene. The input is the information on the scoring scene, and the output is the generated commentary. This commentary generation uses natural language generation technology such as GPT-3 (registered trademark). Specifically, information such as the scorer's name, the scoring method, and the situation of the game is input into the generation AI model as a prompt sentence to generate a detailed commentary. For example, the server generates a prompt sentence like the following:

[0754] "With 2 minutes and 30 seconds left in the third quarter, 23 players made a three-point shot. With this score, Team A took a five-point lead over Team B."

[0755] The generative AI model generates a detailed explanation based on this prompt.

[0756] Step 5:

[0757] Providing commentary

[0758] The server provides the generated commentary to the user in the form of voice or text. The input is the generated commentary, and the output is an audio file and text data. For voice synthesis, Google (registered trademark) Text-to-Speech API or the like is used, and the generated audio file is streamed to the user. The text commentary is displayed on a web page or application. For example, the server embeds the generated commentary in a web page in HTML format, allowing the user to view it in real time.

[0759] (Application example 2)

[0760] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0761] In conventional live video distribution systems for amateur sports, the display of game scores and the identification of scoring scenes are often done manually, which causes problems such as a lack of real-timeness and accuracy. In addition, it is difficult to provide detailed commentary on scoring scenes, which makes it difficult to increase viewer satisfaction. This has led to the issue that the viewing experience of amateur sports is inferior to that of professional sports.

[0762] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0763] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in voice or text for the identified scoring scene, a means for streaming the live video, and a means for generating a detailed commentary of the scoring scene using a voice synthesis technique. This makes it possible to display an accurate score and identify a scoring scene in real time, and further provide a detailed commentary in voice or text, thereby improving the viewing experience of the viewer.

[0764] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[0765] "Live video" is video content that is filmed and distributed in real time.

[0766] "Analysis" is the process of examining data in detail and extracting specific information.

[0767] The "score display" is a display that visually shows the score and progress of the game.

[0768] A "scoring scene" is a video of the moment when a goal is scored in a sporting event.

[0769] "Audio and text commentary" refers to providing audio or text explanations or comments for a video.

[0770] "Streaming" is a technology for delivering video and audio in real time over the Internet.

[0771] "Speech synthesis technology" is a technology that converts text data into speech.

[0772] "Image recognition technology" is a technology that extracts and analyzes specific information from image data.

[0773] "Natural language generation technology" is a technology that allows computers to generate human language.

[0774] The system for carrying out the present invention analyzes live video of amateur sports, displays the score, identifies the scoring scenes, and provides detailed commentary in real time. Specific embodiments are described below.

[0775] Hardware and Software Configuration

[0776] The server uses the following hardware and software:

[0777] Hardware: Servers with powerful processors, memory, storage, and network interfaces.

[0778] software:

[0779] OpenCV: Image processing library

[0780] TENSOFLOW(registered trademark) / Keras: Framework for image recognition models

[0781] Transformers (Hugging Face): A Framework for Natural Language Generation Models

[0782] pyttsx3: A speech synthesis library

[0783] Data processing and calculation

[0784] 1. Live video streaming: The server receives and streams live video in real time over the Internet.

[0785] 2. Frame capture and pre-processing: The server captures frames from the live video and performs pre-processing such as resizing and normalization using OpenCV.

[0786] 3. Image Recognition: Analyze the pre-processed frames using TENSOFLOW® / Keras to identify scoring scenes and players.

[0787] 4. Identifying scoring scenes: Based on the analysis results, Identify scoring scenes.

[0788] 5. Natural Language Generation: When a scoring scene is identified, we use Transformers to generate a detailed commentary. An example prompt is "Tanaka scored a point in a basketball game. Please generate a detailed commentary."

[0789] 6. Speech synthesis: The generated explanation is converted into speech using pyttsx3 and provided to the user.

[0790] Examples

[0791] For example, if a scene such as "Tanaka makes a three-point shot!" is detected during a game, the server will perform the following process.

[0792] 1. Capture and pre-process frames from live video.

[0793] 2. Use an image recognition model to identify scoring scenes.

[0794] 3. If a scoring scene is identified, generate an explanation using a natural language generation model using the prompt sentence, "Tanaka scored a point in the basketball game. Please generate a detailed explanation."

[0795] 4. The generated commentary is converted into audio using speech synthesis technology and played back, for example, "Tanaka made a great three-point shot, widening the team's lead."

[0796] In this way, accurate scores can be displayed in real time, scoring scenes can be identified, and detailed commentary can be provided in audio and text, improving the viewer's viewing experience.

[0797] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0798] Step 1:

[0799] The server receives live video over the Internet and performs streaming. The input is the live video stream data, and the output is frames captured in real time. Specifically, the server splits the video stream and processes each frame.

[0800] Step 2:

[0801] The server preprocesses the captured frames using OpenCV. The input is the captured frame and the output is the preprocessed frame. Specific operations include resizing the frame (e.g. to 224x224 pixels) and normalizing it (scaling pixel values ​​to the range 0 to 1).

[0802] Step 3:

[0803] The server analyzes the preprocessed frames using TENSOFLOW (registered trademark) / Keras to identify scoring scenes and players. The input is the preprocessed frames, and the output is the analysis results (whether or not a scoring scene occurred and the identification of the player). In concrete terms, the server inputs the frames into an image recognition model and obtains the probability of a scoring scene and the identification of the player.

[0804] Step 4:

[0805] The server identifies a scoring scene based on the analysis result. The input is the analysis result, and the output is the result of identifying the scoring scene. In concrete terms, if the probability of a scoring scene exceeds a certain threshold, the server identifies the scene as a scoring scene.

[0806] Step 5:

[0807] When a scoring scene is identified, the server generates a prompt sentence and generates a detailed commentary using Transformers. The input is the result of identifying the scoring scene and the prompt sentence, and the output is the generated commentary text. In concrete terms, the prompt sentence "A player scored a point in a basketball game. Please generate a detailed commentary" is input into the natural language generation model, and commentary text is generated.

[0808] Step 6:

[0809] The server converts the generated commentary text into audio using pyttsx3 and provides it to the user. The input is the generated commentary text, and the output is audio data. In concrete terms, the commentary text is input to a speech synthesis engine, audio data is generated, and the audio data is played through a speaker.

[0810] In this way, the server can analyze live video in real time, identify scoring moments and generate detailed commentary to serve to viewers.

[0811] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0812] "Example 1"

[0813] In one embodiment of the present invention, a system for analyzing live amateur sports videos, generating score displays based on the analysis results, identifying scoring scenes, and generating commentary in audio or text for the identified scoring scenes is combined with an emotion engine that recognizes the emotions of a user. The emotion engine recognizes emotions from the user's facial expressions and tone of voice, and adjusts the generation of commentary based on the recognized emotions. For example, if the user shows a happy expression, the emotion engine feeds that information back to the analysis means, and the commentary generation means generates a more exciting commentary. Conversely, if the user shows a disappointed expression, the commentary generation means generates a more calm tone of commentary. This provides a personalized experience according to the user's emotions.

[0814] "Example 2"

[0815] In one embodiment of the present invention, a system for analyzing live amateur sports videos, generating score displays based on the analysis results, identifying scoring scenes, and generating commentary in audio or text for the identified scoring scenes is combined with an emotion engine that recognizes the emotions of a user. The emotion engine recognizes emotions from the user's facial expressions and tone of voice, and adjusts the generation of commentary based on the recognized emotions. For example, if the user shows a happy expression, the emotion engine feeds that information back to the analysis means, and the commentary generation means generates a more exciting commentary. Conversely, if the user shows a disappointed expression, the commentary generation means generates a more calm tone of commentary. This provides a personalized experience according to the user's emotions.

[0816] The process flow of each embodiment will be described below.

[0817] "Example 1"

[0818] Step 1: Live amateur sports video is entered into the system.

[0819] Step 2: The system analyzes the live video input and generates a score display.

[0820] Step 3: The system identifies scoring moments and generates audio and / or text commentary for the identified scoring moments.

[0821] Step 4: At the same time, the emotion engine recognizes the user's emotions from their facial expressions and tone of voice.

[0822] Step 5: The emotion engine adjusts the commentary generation based on the recognized emotion. For example, if the user shows a happy expression, the emotion engine feeds that information back to the analysis means, and the commentary generation means generates a more exciting commentary. Conversely, if the user shows a disappointed expression, the commentary generation means generates a more calm-toned commentary.

[0823] "Example 1"

[0824] Step 1: Live amateur sports video is entered into the system.

[0825] Step 2: The system analyzes the live video input and generates a score display.

[0826] Step 3: The system identifies scoring moments and generates audio and / or text commentary for the identified scoring moments.

[0827] Step 4: At the same time, the emotion engine recognizes the user's emotions from their facial expressions and tone of voice.

[0828] Step 5: The emotion engine adjusts the commentary generation based on the recognized emotion. For example, if the user shows a happy expression, the emotion engine feeds that information back to the analysis means, and the commentary generation means generates a more exciting commentary. Conversely, if the user shows a disappointed expression, the commentary generation means generates a more calm-toned commentary.

[0829] Example 1

[0830] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0831] Conventional live video analysis systems for amateur sports were able to display the score of a match and identify the scoring scenes, but they were unable to provide personalized commentary according to the user's emotions. As a result, the user experience was uniform, and flexible responses according to individual emotions were required.

[0832] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for acquiring live video of amateur sports, a means for analyzing the acquired video data using image recognition technology, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in voice or text for the identified scoring scene using natural language generation technology, a means for recognizing the user's emotion, and a means for adjusting the tone of the commentary based on the recognized emotion. This makes it possible to provide a personalized commentary according to the user's emotion.

[0833] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[0834] "Live video" is video data that is shot and distributed in real time.

[0835] "Means for acquiring" refers to a method or device for receiving live video in real time and importing it as data.

[0836] "Video data" is digital data that includes visual information captured by a photographic device such as a camera.

[0837] "Image recognition technology" refers to a technology that uses computer vision technology to identify specific objects or events from video data.

[0838] "Means for analyzing" refers to a method or device for processing acquired video data and extracting specific information.

[0839] A "score display" is an information display that visually shows the score and progress of a game.

[0840] A "generating means" is a method or device for producing new data or displays based on specific information.

[0841] A "scoring scene" refers to the moment or situation in which a goal is scored in a sporting event.

[0842] The "means for identifying" refers to a method or device for identifying a particular event or object from the analysis results.

[0843] "Natural language generation technology" is technology that enables computers to generate human language.

[0844] A "means for generating audio or textual descriptions" is a method or apparatus for generating descriptions in audio or text form using natural language generation techniques.

[0845] The "means for recognizing a user's emotion" refers to a method or device for analyzing a user's facial expression and tone of voice to identify the emotion.

[0846] A "means for adjusting the tone of commentary" is a method or device for modifying the manner in which commentary is presented based on perceived emotions.

[0847] This invention is a system that analyzes live amateur sports videos and provides personalized commentary according to the user's emotions. This system consists of three main components: a server, a terminal, and a user.

[0848] Hardware and Software Used

[0849] Hardware: High-performance servers, user devices (PCs, smartphones, etc.)

[0850] Software: Image recognition technology (e.g., OpenCV), natural language generation technology (e.g., GPT-3 (registered trademark)), emotion recognition engine (e.g., Microsoft Azure Emotion API)

[0851] Specific operation of the system

[0852] 1. Acquiring live video

[0853] The server obtains live amateur sports videos in real time. Specifically, it receives video data from streaming services (e.g., YouTube Live, Twitch). The server starts the live stream from a specified URL and stores the video data in a buffer so that it can be processed in real time.

[0854] 2. Analysis of video data

[0855] The server analyzes the acquired video data using image recognition technology (e.g. OpenCV). Specifically, it identifies goal scenes and scorers, and keeps track of the game time. The server performs image analysis for each frame, detects goal scenes, and identifies the scorers by identifying the players' jersey numbers and uniform colors. It also reads the game time from the timer in the video.

[0856] 3. Generate score display

[0857] The server generates a display of the match score based on the analysis results, e.g., updates the score when a goal is recognized. The server increments the score the moment a goal is detected and generates a text overlay to display the scorer's name and the time of the goal.

[0858] 4. Generating commentary on goal-scoring scenes

[0859] When a scoring scene is identified, the server generates a detailed commentary of that scene using natural language generation technology (e.g., GPT-3 (registered trademark)). The commentary is provided in audio and text. The server analyzes the video data of the scoring scene and extracts important events (e.g., assists, type of shot). A prompt sentence is generated based on the extracted information and input into the generative AI model. The generated commentary is converted into audio using speech synthesis technology (e.g., Google (registered trademark) Text-to-Speech).

[0860] 5. User Emotion Recognition

[0861] The device analyzes the user's facial expressions and tone of voice with an emotion recognition engine (e.g., Microsoft Azure Emotion API) to recognize the user's emotions. The device uses a camera and microphone to capture the user's facial expressions and voice, and sends the captured data to the emotion recognition engine to analyze the emotions. The analysis results are then sent to the server.

[0862] 6. Adjust your commentary based on emotions

[0863] The server adjusts the tone of the commentary based on the recognized user's emotions. For example, if the user shows a happy expression, it generates a more exciting commentary. Conversely, if the user shows a disappointed expression, it generates a more calm tone of commentary. The server receives the user's emotion data and feeds it back into the commentary generation process. It adjusts the prompt sentence based on the emotion data and re-inputs it into the generative AI model. The adjusted commentary is regenerated using speech synthesis technology and provided to the user.

[0864] Examples and prompts

[0865] Examples

[0866] Scenario: A goal is scored during an amateur soccer match.

[0867] Process flow:

[0868] 1. The server recognizes goal scenes from live video.

[0869] 2. The server updates the score and displays the scorer information.

[0870] 3. The server generates commentary of the scoring scene and provides it in audio and text.

[0871] 4. The device analyzes the user's facial expressions and recognizes expressions of joy.

[0872] 5. The server reproduces the commentary in an exciting tone and provides it to the user.

[0873] Examples of prompt statements

[0874] "Describe a scene where a goal is scored during an amateur soccer match. If the user shows signs of joy, generate commentary in an exciting tone."

[0875] The above is an embodiment of the present invention.

[0876] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0877] Step 1: Getting live video

[0878] The server retrieves live amateur sports videos in real time. As input, it receives the URL of a streaming service. Specifically, the server starts the live stream from the specified URL and stores the video data in a buffer. As output, it obtains video data that can be processed in real time.

[0879] Step 2: Analyze the video data

[0880] The server analyzes the acquired video data using image recognition technology (e.g. OpenCV). It receives frame data of live video as input. Specifically, the server performs image analysis for each frame, detects goal scenes, and identifies the scorers by identifying the players' jersey numbers and uniform colors. It also reads the match time from a timer in the video. It obtains the goal scene detection results, scorer information, and match time as output.

[0881] Step 3: Generate the score display

[0882] The server generates a match score display based on the analysis results. As input, it receives goal detection results, scorer information, and match time. Specifically, the server increments the score the moment a goal is detected and generates a text overlay to display the scorer's name and the time of the goal. As output, it gets an updated score display.

[0883] Step 4: Generate commentary on goal scoring scenes

[0884] When a scoring scene is identified, the server generates a detailed commentary of the scene using natural language generation technology (e.g., GPT-3 (registered trademark)). As input, the server receives video data of the scoring scene and important event information (e.g., assists, type of shot). In specific operations, the server generates a prompt sentence based on the extracted information and inputs it into the generative AI model. The generated commentary is converted into audio using speech synthesis technology (e.g., Google (registered trademark) Text-to-Speech). As output, the server obtains a commentary in audio and text format.

[0885] Step 5: Recognizing User Emotions

[0886] The device analyzes the user's facial expressions and tone of voice using an emotion recognition engine (e.g. Microsoft Azure Emotion API) to recognize the user's emotions. As input, it receives the user's facial expressions and voice data captured by the camera and microphone. Specifically, the device sends the captured data to the emotion recognition engine and analyzes the emotions. It then sends the analysis results to the server. As output, it obtains the user's emotional data.

[0887] Step 6: Adjust your commentary based on emotion

[0888] The server adjusts the tone of the commentary based on the recognized user emotion. As input, it receives the user's emotion data. In concrete terms, the server adjusts the prompt sentence based on the emotion data and re-inputs it into the generative AI model. The adjusted commentary is regenerated by speech synthesis technology and provided to the user. As output, a personalized commentary according to the user's emotion is obtained.

[0889] (Application example 1)

[0890] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0891] Conventional live video analysis systems for amateur sports display game scores, identify scoring scenes, and generate commentary, but they are unable to provide personalized commentary based on the user's emotions. This results in a uniform viewing experience for users, making it difficult to provide optimal commentary based on individual emotions.

[0892] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0893] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying scoring scenes based on the analysis result, a means for generating commentary in audio or text for the identified scoring scenes, a means for recognizing a user's emotion, and a means for adjusting the tone of the commentary based on the recognized emotion, thereby making it possible to provide a personalized commentary according to the user's emotion.

[0894] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[0895] "Live video" is video that is filmed in real time and streamed simultaneously.

[0896] "Analysis" is the process of examining data or information in detail to derive specific results or meanings.

[0897] The "score display" is a display that visually shows the score and progress of the game.

[0898] A "scoring scene" is a video or scene showing the exact moment a goal is scored in a sporting event.

[0899] A "commentary" is an explanation or commentary on a particular event or situation.

[0900] "Speech" refers to auditory information such as speech and music.

[0901] "Text" is information or writing written in text.

[0902] "User" means an individual or organization that uses the system or service.

[0903] Emotions are human psychological states and feelings.

[0904] "Cognition" is the process of understanding and grasping specific information or a situation.

[0905] "Tone" is the mood or atmosphere of a voice or text.

[0906] "Adaptation" is the process of changing or modifying something to suit a specific purpose.

[0907] "Personalization" means customizing to suit the characteristics and preferences of individual users.

[0908] As an embodiment of the present invention, a system is constructed that analyzes live videos of amateur sports and provides personalized commentary according to the user's emotions. A specific embodiment is described below.

[0909] Hardware and Software Configuration

[0910] Hardware

[0911] Server: Analyzes live video, generates score displays, identifies scoring moments, generates commentary, and recognizes emotions.

[0912] Device: The device (smartphone, tablet, PC, etc.) through which a user watches live video.

[0913] Camera: A device for capturing live video.

[0914] Microphone: A device for collecting the user's voice.

[0915] software

[0916] OpenCV: A library for live video streaming and frame processing.

[0917] Keras: A library that provides an image recognition model used to identify goal scoring scenes.

[0918] Transformers (Hugging Face): A library that generates commentary using the GPT-2 model.

[0919] SpeechRecognition: A library that converts the user's speech into text.

[0920] EmotionRecognition: A library for recognizing user emotions.

[0921] Data processing and calculation

[0922] Server Processing

[0923] The server receives the live video from the camera and analyzes the frames in real time using OpenCV. Based on the analysis, it generates a score display of the match and identifies the scoring scenes. If a scoring scene is identified, it uses the GPT-2 model to generate commentary for that scene.

[0924] User Emotion Recognition

[0925] It collects voice data through the microphone connected to the user's device and converts the voice to text using SpeechRecognition. It analyzes the converted text with the EmotionRecognition library to recognize the user's emotions. It adjusts the tone of the commentary based on the recognized emotions.

[0926] Examples

[0927] Live video URL

[0928] Example: http: / / example.com / live_stream

[0929] Explanation of the goal

[0930] Example: Player X scores a goal

[0931] User audio files

[0932] Example: user_audio.wav

[0933] Examples of prompt statements

[0934] Your goal is to develop a system that analyzes live amateur soccer videos, displays match scores, identifies goal-scoring moments, and generates commentary. The system will recognize the user's emotions and provide commentary that reflects those emotions. Specifically, your goal is to create a smartphone application that has the following functions:

[0935] 1. Live video streaming

[0936] 2. Score display

[0937] 3. Identifying the goal-scoring moment

[0938] 4. Description Generation

[0939] 5. Emotion recognition

[0940] In this way, a system can be realized that provides personalized commentary according to the user's emotions.

[0941] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0942] Step 1:

[0943] The server receives the live video sent from the camera. The input is the live video stream data, and the output is the image data for each frame. Using OpenCV, we grab the frames in real time and prepare them for analysis.

[0944] Step 2:

[0945] The server analyzes the captured frames and generates a display of the match score. The input is image data for each frame, and the output is text data that displays the score. Image recognition technology is used to determine the score and the progress of the match, and to update the score.

[0946] Step 3:

[0947] The server identifies the goal-scoring scene based on the analysis results. The input is the analyzed frame data, and the output is the score scene identification information. Using Keras, we detect the goal scene and identify the scene.

[0948] Step 4:

[0949] The server generates a commentary in voice or text for the identified scoring scene. The input is the specific information of the scoring scene, and the output is the text data or audio data of the commentary. The commentary is generated using natural language generation technology using the GPT-2 model of Transformers (Hugging Face).

[0950] Step 5:

[0951] The terminal collects the user's voice. The input is the user's voice data, and the output is an audio file. The terminal uses a microphone to record the user's voice and saves it as an audio file.

[0952] Step 6:

[0953] The server analyzes the collected voice data and recognizes the user's emotions. The input is the audio file and the output is the user's emotion information. It uses SpeechRecognition to convert the voice to text and the EmotionRecognition library to recognize emotions.

[0954] Step 7:

[0955] The server adjusts the tone of the commentary based on the recognized emotion. The input is the user's emotion information and the generated commentary data, and the output is the adjusted commentary data. Based on the emotion information, the tone of the commentary is changed to exciting or calm.

[0956] Step 8:

[0957] The terminal provides the adjusted commentary to the user. The input is the adjusted commentary data, and the output is display or playback of the commentary to the user. The commentary is provided to the user using the screen and speaker of the terminal.

[0958] Example 2

[0959] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0960] Conventional live video analysis systems for amateur sports can display the score of a match and identify the scoring scenes, but they cannot provide personalized commentary according to the user's emotions. Therefore, in order to enrich the user experience, a function that recognizes the user's emotions and adjusts the commentary based on them is required.

[0961] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0962] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying scoring scenes based on the analysis result, a means for generating commentary in audio or text for the identified scoring scenes, a means for recognizing a user's emotion, and a means for adjusting the commentary based on the recognized emotion, thereby making it possible to provide a personalized commentary according to the user's emotion.

[0963] "Amateur sports" refers to sports competitions played by non-professional individuals or teams.

[0964] "Live video" refers to video footage that is filmed in real time and streamed simultaneously.

[0965] "Analysis" is the process of examining acquired data in detail and extracting specific information.

[0966] A "score display" is a display that visually shows the score and progress of a game.

[0967] A "scoring scene" refers to footage of the exact moment a goal is scored in a sporting event.

[0968] "Means for generating commentary in audio or text" refers to technology that creates audio or text explanations of scoring scenes or game situations.

[0969] "Means for recognizing a user's emotions" refers to technology that determines emotions from a user's facial expressions, tone of voice, etc.

[0970] "Means for tailoring commentary" refers to techniques that change the content and tone of commentary based on recognized emotions.

[0971] "Image recognition technology" refers to the technology of analyzing image data to detect specific objects or actions.

[0972] "Natural language generation technology" refers to technology that allows computers to generate human language.

[0973] This invention is a system that analyzes live videos of amateur sports, generates score displays based on the analysis results, identifies scoring scenes, and generates audio and text commentary for the identified scoring scenes. It also has a function of recognizing the user's emotions and adjusting the commentary based on those emotions.

[0974] Hardware and Software Used

[0975] Hardware: High performance servers, cameras, microphones

[0976] Software: Image recognition technology (e.g. OpenCV), natural language generation technology (e.g. GPT-3 (registered trademark)), emotion recognition engine (e.g. Affectiva)

[0977] Specific operation of the system

[0978] Live video capture

[0979] The server receives live video from the camera.

[0980] The server connects to the camera and receives the video data in real time.

[0981] Specific operation: The server obtains the video stream from the camera using the RTSP protocol.

[0982] Video data analysis

[0983] The server analyzes the acquired video data using image recognition technology.

[0984] The server uses OpenCV to identify basketball goal scenes and scorers in the video.

[0985] What it does: The server processes the images frame by frame and runs an algorithm to detect goal moments.

[0986] Generate score display

[0987] The server generates a score display for the match based on the analysis.

[0988] The server updates the scoreboard every time a point is scored.

[0989] Specific operation: When the server detects a score event, it records the score in the database and sends an instruction to update the score to the front end.

[0990] Commentary generation for scoring scenes

[0991] When a scoring scene is identified, the server generates a detailed commentary of the scene in audio and / or text.

[0992] The server uses GPT-3® to generate explanations in natural language.

[0993] Specific operation: The server inputs the information about the scoring scene into GPT-3 (registered trademark) and generates an explanatory text such as "That was a great shot!"

[0994] User Emotion Recognition

[0995] The device recognizes the user's emotions from their facial expressions and tone of voice.

[0996] The device uses Affectiva to analyze the user's emotions.

[0997] How it works: The device uses the camera and microphone to capture the user's facial expressions and voice, and calls Affectiva's API to obtain emotion data.

[0998] Adjustment of explanation

[0999] The server adjusts the commentary generation based on the recognized emotions.

[1000] The server adjusts the commentary to an exciting tone if the user is happy.

[1001] Specific operation: The server analyzes the emotion data and re-inputs parameters to GPT-3 (registered trademark) to change the tone of the commentary.

[1002] Examples and prompts

[1003] Examples:

[1004] As the user watches a game, the camera captures a goal scene.

[1005] The server uses OpenCV to detect goal scenes and identify the scorer.

[1006] How it works: The server analyzes the movement within the frame and detects the moment the ball enters the goal.

[1007] The server uses GPT-3(registered trademark) to generate an explanation: "That was a great shot!"

[1008] Specific operation: The server inputs the scoring scene information into GPT-3 (registered trademark) and generates an explanatory text.

[1009] The device uses Affectiva to analyze the user's facial expressions and recognize when the user is happy.

[1010] Specific operation: The device captures the user's smile with the camera and calls Affectiva's API to obtain emotion data.

[1011] The server adjusts his commentary to an exciting tone: "This shot could change the course of the game!"

[1012] Specific operation: The server analyzes the emotion data and re-inputs parameters to GPT-3 (registered trademark) to change the tone of the commentary.

[1013] Example prompt:

[1014] "When a user is watching a game and the camera captures a goal, how do you identify the scorer and generate commentary? Also, how do you adjust the commentary depending on the user's emotions?"

[1015] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1016] Step 1:

[1017] Live video capture

[1018] The server receives live video from the camera.

[1019] Input: Real-time video stream from the camera

[1020] Specific operation: The server obtains the video stream from the camera using the RTSP protocol.

[1021] Output: Live video data

[1022] Step 2:

[1023] Video data analysis

[1024] The server analyzes the acquired video data using image recognition technology.

[1025] Input: Live video data

[1026] What it does: The server uses OpenCV to process the images frame by frame and run algorithms to identify goals and scorers.

[1027] Output: Goal scene detection results, scorer identification results

[1028] Step 3:

[1029] Generate score display

[1030] The server generates a score display for the match based on the analysis.

[1031] Input: Goal scene detection results, goal scorer identification results

[1032] Specific operation: When the server detects a score event, it records the score in the database and sends an instruction to update the score to the front end.

[1033] Output: Updated score display

[1034] Step 4:

[1035] Commentary generation for scoring scenes

[1036] When a scoring scene is identified, the server generates a detailed commentary of the scene in audio and / or text.

[1037] Input: Goal scene detection results, goal scorer identification results

[1038] Specific operation: The server inputs the scoring scene information into GPT-3 (registered trademark) and generates an explanatory text in natural language.

[1039] Output: Description (e.g. "That was a great shot!")

[1040] Step 5:

[1041] User Emotion Recognition

[1042] The device recognizes the user's emotions from their facial expressions and tone of voice.

[1043] Input: User's facial expression data, voice tone data

[1044] How it works: The device uses the camera and microphone to capture the user's facial expressions and voice, and calls Affectiva's API to obtain emotion data.

[1045] Output: User emotion data

[1046] Step 6:

[1047] Adjustment of explanation

[1048] The server adjusts the commentary generation based on the recognized emotions.

[1049] Input: User emotion data, explanatory text

[1050] Specific operation: The server analyzes the emotion data and re-inputs parameters to GPT-3 (registered trademark) to change the tone of the commentary.

[1051] Output: Tailored explanatory text (e.g. "This shot could change the game!")

[1052] (Application example 2)

[1053] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1054] When watching live amateur sports videos, it is difficult for viewers to grasp the progress of the game and the scoring scenes in real time. In addition, because personalized commentary based on the viewer's emotions is not provided, the viewing experience is uniform and it is not possible to increase the satisfaction of each individual viewer. This makes it difficult to maintain the viewer's interest, and there is a problem that the quality of the viewing experience is reduced.

[1055] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1056] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in audio or text for the identified scoring scene, a means for recognizing a user's emotion, and a means for adjusting the generation of the commentary based on the recognized emotion. This not only enables a viewer to grasp the progress of the game and the scoring scenes in real time, but also provides a personalized commentary according to the viewer's emotion, thereby improving the quality of the viewing experience and increasing the viewer's satisfaction.

[1057] "Amateur sports" are sporting events in which non-professional individuals or teams participate.

[1058] "Live video" is video content that is filmed in real time and streamed simultaneously.

[1059] "Analysis" is the process of examining data in detail to understand its structure and meaning.

[1060] A "score display" is an information display that visually shows the score and progress of a match.

[1061] A "scoring scene" refers to the image or circumstances of the moment when a goal is scored in a sporting event.

[1062] "Generating commentary in audio or text" means creating audio or text explanations about the situation in a game or scoring scenes.

[1063] "Recognizing the user's emotions" means judging the viewer's emotional state from facial expressions, tone of voice, and the like.

[1064] "Adjusting commentary production" refers to changing the content and tone of commentary based on the recognized emotions.

[1065] "Image recognition technology" is a technology that analyzes image data and identifies specific objects or scenes.

[1066] "Natural language generation technology" is technology that enables computers to generate human language.

[1067] The system for carrying out the present invention analyzes live video of amateur sports, generates a score display based on the analysis results, identifies scoring scenes, generates audio and text commentary for the identified scoring scenes, and has the function of recognizing the user's emotions and adjusting the generation of commentary based on the emotions.

[1068] Hardware and Software Used

[1069] Hardware:

[1070] It uses the smartphone's camera and microphone for live video capture and user emotion recognition.

[1071] The server provides computational resources for video analysis and commentary generation.

[1072] software:

[1073] OpenCV: A library for capturing and displaying live video.

[1074] DeepFace: A library for recognizing emotions from user facial expressions.

[1075] Transformers (Hugging Face): A library for generating explanations using natural language generation techniques.

[1076] gTTS: A library for converting generated text descriptions into speech.

[1077] playsound: A library for playing generated sounds.

[1078] System Operation

[1079] 1. Live video analysis:

[1080] The server analyzes live video captured by the smartphone camera in real time.

[1081] Using OpenCV, we capture video frames and use image recognition techniques to identify scoring scenes and scorers.

[1082] 2. Generate the score display:

[1083] The server updates the match score in real time based on the analysis results and displays it on the smartphone screen.

[1084] 3. Emotion recognition:

[1085] It uses your smartphone's camera and microphone to capture your facial expressions and tone of voice.

[1086] Use DeepFace to recognize user emotions in real time.

[1087] 4. Description Generation:

[1088] If a scoring scene is identified, the server generates a detailed explanation of the scene.

[1089] It uses Transformers' natural language generation model to generate explanations based on the user's emotions.

[1090] The generated commentary is converted into audio using gTTS and played through the smartphone's speakers.

[1091] Examples

[1092] For example, suppose a user is watching an amateur basketball game on their smartphone. If a goal is scored during the game and the user shows a joyful expression, the system recognizes the emotion and generates an exciting commentary. The generated commentary is generated using the following prompt sentence:

[1093] Examples of prompt statements

[1094] If the user indicates the emotion of joy, generate an exciting description of the following scene: Basketball Goal Scene

[1095] In this way, a personalized commentary is provided according to the user's emotions, improving the quality of the viewing experience and increasing viewer satisfaction.

[1096] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1097] Step 1:

[1098] The server captures live video from the smartphone camera. The input is the live video stream and the output is the image data for each frame. OpenCV is used to get the video frames in real time.

[1099] Step 2:

[1100] The server analyzes the captured video frames and identifies the scoring scenes and scorers. The input is the image data of the video frames, and the output is the result of identifying the scoring scenes. Image recognition technology is used to identify the basketball goal scenes and scorers.

[1101] Step 3:

[1102] The server updates the score of the game based on the analysis results and displays it on the smartphone screen. The input is the result of identifying the scoring scene, and the output is an updated score display. The score display generation means updates the score in real time.

[1103] Step 4:

[1104] The device uses the smartphone's camera and microphone to capture the user's facial expressions and tone of voice. The input is the user's facial expression images and voice data, and the output is data for emotion recognition. The camera and microphone are used to capture the user's emotions in real time.

[1105] Step 5:

[1106] The server analyzes the captured data and recognizes the user's emotions. The input is the user's facial expression images and voice data, and the output is the recognized emotions. DeepFace is used to analyze the user's emotions.

[1107] Step 6:

[1108] The server generates a detailed commentary of a scoring scene when it is identified. The input is the scoring scene identification result and the recognized emotion, and the output is the generated commentary text. The commentary is generated based on the prompt sentence using a natural language generation model from Transformers.

[1109] Step 7:

[1110] The server converts the generated commentary text into speech and plays it through the smartphone speaker. The input is the generated commentary text and the output is the audio data. gTTS is used to convert the text to speech and plays the audio using playsound.

[1111] In this way, the system can analyze live video in real time and provide personalized commentary according to the user's emotions.

[1112] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1113] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1114] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google(登録商標).com ?hl="ja">) are mentioned.

[1115] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1116] [Third embodiment]

[1117] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1118] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1119] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1120] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1121] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1122] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1123] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1124] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1125] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1126] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1127] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1128] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1129] "Example 1"

[1130] An example of the present invention is a system that analyzes live videos of amateur soccer matches. This system uses image recognition technology to analyze video data obtained from the live videos and generate a display of the match score. Specifically, the system identifies goal scenes and scorers, and keeps track of the match time. When a goal-scoring scene is identified, the system generates a detailed commentary of that scene in voice and text. Natural language generation technology is used to generate this commentary.

[1131] "Example 2"

[1132] Another example is a system that analyzes live videos of amateur basketball games. This system uses image recognition technology to analyze video data obtained from the live videos and generate a display of the game score. Specifically, it identifies basketball goal scenes and scorers, and keeps track of the game time. In addition, when a scoring scene is identified, it generates a detailed commentary of that scene in voice and text. Natural language generation technology is used to generate this commentary.

[1133] The process flow of each embodiment will be described below.

[1134] "Example 1"

[1135] Step 1: Get live amateur football video.

[1136] Step 2: Extract video data from the captured live video and analyze it using image recognition technology. This analysis includes identifying goal scenes, scorers, and game time.

[1137] Step 3: Generate a score display based on the analysis results.

[1138] Step 4: If a scoring scene is identified, a detailed commentary of that scene is generated in audio and text format using natural language generation technology.

[1139] "Example 2"

[1140] Step 1: Get live amateur basketball video.

[1141] Step 2: Extract video data from the captured live video and analyze it using image recognition technology. This analysis includes identifying basketball goal scenes, scorers, and game time.

[1142] Step 3: Generate a score display based on the analysis results.

[1143] Step 4: If a scoring scene is identified, a detailed commentary of that scene is generated in audio and text format using natural language generation technology.

[1144] Example 1

[1145] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1146] In conventional live video analysis systems for amateur sports, displaying the game score and identifying the scoring scenes are often done manually, which means that they lack real-timeness and accuracy. In addition, commentary on the scoring scenes is also done manually, which makes it difficult to provide information quickly. This makes it difficult for spectators and officials to accurately and quickly grasp the progress of the game.

[1147] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1148] In this invention, the server includes a means for acquiring live video data, a means for preprocessing the acquired video data, a means for analyzing the preprocessed video data using image recognition technology, a means for identifying a scoring scene based on the analysis result, a means for generating a prompt sentence for generating a commentary for the identified scoring scene, a means for inputting the generated prompt sentence into a generation AI model to generate a commentary, and a means for outputting the generated commentary in voice or text. This automates the display of the score of the game and the identification of the scoring scene, making it possible to provide accurate information in real time. In addition, detailed commentary of the scoring scene is also quickly generated, making it possible for spectators and related parties to accurately and quickly grasp the progress of the game.

[1149] "Live video data" refers to video information acquired in real time.

[1150] "Preprocessing" refers to processes such as data resizing, noise removal, and frame extraction that are carried out to make the acquired video data easier to analyze.

[1151] "Image recognition technology" is a technology for detecting and identifying specific objects and actions from video data.

[1152] "Analysis results" refers to the analysis results of data obtained using image recognition technology.

[1153] A "scoring scene" is a video of the exact moment a goal is scored in a sports game.

[1154] A "prompt sentence" is an instruction sentence to be input into a generative AI model.

[1155] A "generative AI model" is an artificial intelligence model that generates natural language text based on a given prompt.

[1156] "Commentary" refers to detailed explanations and comments about the scoring scene.

[1157] "Audio output" refers to providing the generated description in audio form.

[1158] "Text output" refers to providing the generated commentary in text form.

[1159] This invention is a system that analyzes live videos of amateur sports and automatically generates game scores and detailed commentary of scoring scenes. This system works in cooperation with three entities: a server, a terminal, and a user.

[1160] First, the server acquires live video data. The video data is transmitted in real time from devices such as cameras and smartphones. The server streams the video data from the cameras using RTSP (Real-Time Streaming Protocol).

[1161] The server then pre-processes the captured video data, which includes image resizing, noise removal, frame extraction, etc. Specifically, the server uses OpenCV to resize the video data to 640x480 pixels and apply a noise removal filter.

[1162] The pre-processed video data is then analyzed using image recognition technology. The server uses libraries such as TENSOFLOW (registered trademark) and YOLO (You Only Look Once) to detect players and the ball in each frame. Based on the analysis results, the server identifies the goal-scoring scene. To identify the goal scene, the server detects the movement of the ball and the shaking of the goal net. The server detects the moment the ball crosses the goal line and saves that frame.

[1163] When a scoring scene is identified, the server generates a prompt sentence for generating commentary. For example, the server generates a prompt sentence such as, "Please generate a detailed commentary on the scene where Player A scored a goal." The server inputs this prompt sentence into a generative AI model (for example, GPT-3 (registered trademark)) and generates a commentary. The generative AI model generates a commentary such as, "Player A received a cross from the right side and scored a goal with a brilliant volley shot."

[1164] The generated commentary is output by the server in the form of voice or text. For voice output, Google® Text-to-Speech API or the like is used. The server sends the generated text commentary to Google® Text-to-Speech API to generate an audio file. The server then sends the audio file to the user's device. The user can watch or view the commentary through the device.

[1165] As a concrete example, consider a case where a server analyzes a live video and identifies the scene where player A scored a goal. The server inputs a prompt sentence, "Please generate a detailed commentary on the scene where player A scored a goal," into the generative AI model. The generative AI model generates a commentary, "Player A received a cross from the right side and scored a goal with a brilliant volley shot." The server converts the generated commentary into audio, which the user can listen to on their device.

[1166] The server also keeps track of the game time. For example, the server detects that the game time has passed 45 minutes, and inputs a prompt message to the generative AI model saying, "Please generate a message announcing the end of the first half." The generative AI model generates an explanation saying, "The first half has ended. The current score is 1-0." The server displays the generated explanation as text, and the user can view it on their device.

[1167] The above is an embodiment of the present invention.

[1168] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1169] Step 1:

[1170] The server acquires the live video data.

[1171] Input: Real-time video data sent from devices such as cameras and smartphones.

[1172] Specific operation: The server streams video data from the camera using RTSP (Real-Time Streaming Protocol).

[1173] Output: Acquired live video data.

[1174] Step 2:

[1175] The server pre-processes the acquired video data.

[1176] Input: Acquired live video data.

[1177] Specific operation: The server uses OpenCV to resize the video data to 640x480 pixels and apply a noise reduction filter.

[1178] Output: Pre-processed video data.

[1179] Step 3:

[1180] The server analyzes the pre-processed video data using image recognition technology.

[1181] Input: Preprocessed video data.

[1182] Specific operation: The server uses libraries such as TENSOFLOW (registered trademark) and YOLO (You Only Look Once) to detect players and the ball in each frame.

[1183] Output: Analysis results (positions of players and the ball, etc.).

[1184] Step 4:

[1185] The server identifies the scoring scene based on the analysis result.

[1186] Input: The analysis results.

[1187] What happens: The server detects the moment the ball crosses the goal line and saves that frame.

[1188] Output: Goal specific information (scoring frame).

[1189] Step 5:

[1190] The server generates a prompt sentence for generating an explanation for the scoring scene.

[1191] Input: Scoring scene specific information.

[1192] Specific operation: The server generates a prompt sentence saying, "Please generate a detailed commentary of the scene where player A scored a goal."

[1193] Output: The prompt statement.

[1194] Step 6:

[1195] The server inputs the generated prompt sentence into a generative AI model to generate an explanation.

[1196] Input: The prompt statement.

[1197] Specific operation: The server sends a prompt sentence to a generative AI model (e.g., GPT-3 (registered trademark)) to generate an explanation.

[1198] Output: The generated description (in text format).

[1199] Step 7:

[1200] The server outputs the generated commentary in audio and / or text.

[1201] Input: The generated description (in text format).

[1202] Specific operation: The server sends the generated text commentary to the Google® Text-to-Speech API to generate an audio file, and then sends the audio file to the user's device.

[1203] Output: Audio files and text description.

[1204] The above is the flow of processing of the program of this system.

[1205] (Application example 1)

[1206] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1207] In live video distribution of amateur sports, it is difficult to provide detailed commentary of game scores and scoring scenes in real time. In addition, when viewers watch games using smart devices, the user interface is not optimized, resulting in a poor viewing experience. Furthermore, there is also the issue that it is difficult for viewers to understand the flow of the game because commentary of scoring scenes cannot be provided by audio.

[1208] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1209] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in voice or text for the identified scoring scene, a means for providing a user interface optimized for a smart device, a means for measuring the game time in real time, and a means for converting detailed commentary of the scoring scene into voice using a voice synthesis technology. This makes it possible to provide detailed commentary of the game score and the scoring scene in real time, allowing viewers to comfortably watch the game using their smart devices. Also, by providing commentary of the scoring scene in voice, viewers can have a deeper understanding of the flow of the game.

[1210] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[1211] "Live video" is video that is filmed in real time and streamed simultaneously.

[1212] "Analysis" means examining data or information in detail to clarify its structure and meaning.

[1213] A "score display" is a visual indication of the score and progress of a match.

[1214] A "scoring scene" is a video of the moment when a goal is scored in a sporting event.

[1215] "Means for generating commentary in audio or text" refers to technology for explaining scoring scenes and the situation of the game in audio or text.

[1216] A "smart device" is an electronic device with advanced functions, such as a smartphone, tablet, or head-mounted display.

[1217] A "user interface" is a screen or method of operation that allows a user to interact with a system or application.

[1218] "Means for measuring match time in real time" refers to technology that measures and displays the progress of a match in real time.

[1219] "Speech synthesis technology" is a technology that converts text data into speech.

[1220] "Image recognition technology" is a technology that analyzes image data and recognizes specific objects or scenes.

[1221] "Natural language generation technology" is a technology that allows computers to generate human language.

[1222] A system for carrying out the present invention analyzes live video of amateur sports and provides real-time scores and commentary on scoring scenes. A specific embodiment of this system will be described below.

[1223] System Configuration

[1224] 1. Hardware:

[1225] Server: A server with powerful computing power (e.g. AWS EC2 instance)

[1226] Devices: Smartphones, head-mounted displays (e.g. Oculus Quest, HoloLens)

[1227] 2. Software:

[1228] Image recognition technology: OpenCV, TENSOFLOW (registered trademark)

[1229] Natural language generation technology: GPT-3 (registered trademark), BERT

[1230] Speech synthesis technology: Google(R) Text-to-Speech

[1231] Streaming services: AWS Elemental MediaLive, YouTube(R) Live

[1232] Data processing and calculation

[1233] 1. Live video analysis:

[1234] The server captures frames of live video and uses image recognition technology (OpenCV, TENSOFLOW (registered trademark)) to identify goal scenes and scorers.

[1235] The server times the match in real time and generates a score display.

[1236] 2. Commentary generation for goal scoring scenes:

[1237] When a scoring scene is identified, the server generates a detailed description of the scene using natural language generation technology (GPT-3 (registered trademark), BERT).

[1238] The generated commentary is converted into audio using speech synthesis technology (Google® Text-to-Speech).

[1239] 3. Providing the user interface:

[1240] The terminal provides a user interface optimized for smart devices, allowing viewers to watch the game comfortably.

[1241] Examples

[1242] For example, when an amateur soccer match is being broadcast live, users watch the match through their smartphones or head-mounted displays. When a goal is scored, the server automatically identifies the scene and displays the name of the scorer and the details of the goal in real time. In addition, commentary of the goal is provided in audio and text, allowing users to gain a deeper understanding of the flow of the match.

[1243] Examples of prompt statements

[1244] "The current match time is 45 minutes. A goal has been scored. The scorer is Tanaka. Please generate a detailed explanation of this scene."

[1245] By inputting this prompt into a generative AI model, a detailed explanation of the scoring scene is generated.

[1246] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1247] Step 1:

[1248] The server captures frames of live video. The input is the live video stream and the output is the individual frame images. The server receives the live video using a streaming service (e.g. AWS Elemental MediaLive) and splits it into frames.

[1249] Step 2:

[1250] The server analyzes the captured frame images using image recognition technology (e.g., OpenCV, TENSOFLOW (registered trademark)). The input is the frame image, and the output is the goal scene and the scorer identification result. The server identifies specific actions and people in the frame and identifies the goal scene and the scorer.

[1251] Step 3:

[1252] The server measures the match time in real time. The input is the live video stream and the output is the current match time. The server calculates the elapsed time based on the start time of the video and updates the match time in real time.

[1253] Step 4:

[1254] The server generates a score display based on the analysis results. The inputs are goal scenes, scorer identification results, and match time, and the output is score display data. The server updates the score when a goal occurs and generates data to display it visually.

[1255] Step 5:

[1256] When a scoring scene is identified, the server generates a detailed commentary of the scene using natural language generation technology (e.g., GPT-3 (registered trademark), BERT). The input is the result of identifying the scoring scene and the game time, and the output is the commentary text. The server inputs the prompt sentence into the generative AI model and generates a detailed commentary.

[1257] Step 6:

[1258] The server converts the generated commentary text into audio using voice synthesis technology (e.g., Google (registered trademark) Text-to-Speech). The input is the commentary text, and the output is audio data. The server converts the text data into audio data and provides it to the viewer.

[1259] Step 7:

[1260] The terminal provides a user interface optimized for smart devices. The input is score display data and audio data, and the output is visual and audio information. The terminal displays the score and commentary through the user interface and plays the audio commentary.

[1261] Step 8:

[1262] Users use smart devices to watch games. The input is the visual and auditory information provided by the device, and the output is the user's viewing experience. Users understand the flow of the game through real-time updated scores and commentary.

[1263] Example 2

[1264] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1265] In live amateur sports videos, it is difficult to provide detailed commentary on the progress of the game and scoring scenes in real time. In particular, identifying scoring scenes and generating commentary requires advanced technology, and conventional systems have not been able to do this efficiently.

[1266] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1267] In this invention, the server includes a means for acquiring live video, a means for analyzing the acquired video data using image recognition technology, a means for generating a score display based on the analysis result, a means for identifying scoring scenes based on the analysis result, a means for generating commentary in voice or text for the identified scoring scenes using natural language generation technology, a means for converting the generated commentary into a voice file using voice synthesis technology and providing it to the user, and a means for displaying the generated commentary in text format, thereby making it possible to provide detailed commentary on the progress of the game and scoring scenes in real time.

[1268] "Live video" refers to video data that is filmed in real time and instantly distributed via the Internet or other means.

[1269] "Video data" is digital data that includes visual information captured by a photographing device such as a camera.

[1270] "Image recognition technology" is a technology that uses computer vision technology to identify specific objects or patterns from images or videos.

[1271] "Score display" is a display format that visually shows the score and progress of a game.

[1272] A "scoring scene" is the moment when a goal is scored in a sports game or the scenes before and after that moment.

[1273] "Natural language generation technology" is a technology that enables computers to generate human language, and generates sentences based on specific input data.

[1274] "Speech synthesis technology" is a technology that converts text data into voice data.

[1275] "User" refers to any person or organization that uses this system.

[1276] A "text format" is a format of digital data that contains text information.

[1277] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1278] The present invention is a system that analyzes live video of amateur sports and provides detailed commentary of the progress of the game and scoring scenes in real time. A specific embodiment of this system will be described below.

[1279] Live video capture

[1280] The server acquires live video in real time. Specifically, it uses a high-performance video capture device to capture the video of the match, and transmits the data to the server using a video streaming tool such as FFmpeg. The server temporarily stores the received video data and prepares it for analysis.

[1281] Video data analysis

[1282] The server analyzes the stored video data. Specifically, it uses image recognition libraries such as OpenCV and TENSOFLOW (registered trademark) to identify basketball goal scenes and scorers in the video. For example, the server processes the images frame by frame to track the movement of the basketball. It also recognizes the uniform numbers of the players and identifies the scorers.

[1283] Generate score display

[1284] The server generates a score display for the match based on the analysis results. When a point is scored, the server updates the scoreboard data and displays it to the user in real time. Specifically, the server displays the score on a web page using HTML and JavaScript. It also uses WebSocket to send updates in real time to notify users of score changes.

[1285] Identifying goal-scoring scenes and generating commentary

[1286] When a scoring scene is identified, the server generates a detailed commentary on that scene. This commentary generation uses natural language generation technology such as GPT-3 (registered trademark). Specifically, information such as the scorer's name, the scoring method, and the game situation is input into the generation AI model as a prompt sentence to generate a detailed commentary. For example, the server generates the following prompt sentence:

[1287] "With 2 minutes and 30 seconds left in the third quarter, 23 players made a three-point shot. With this score, Team A took a five-point lead over Team B."

[1288] The generative AI model generates a detailed explanation based on this prompt.

[1289] Providing commentary

[1290] The server provides the generated commentary to the user in the form of voice or text. For voice synthesis, Google (registered trademark) Text-to-Speech API or the like is used, and the generated audio file is streamed to the user. The text commentary is displayed on a web page or application. For example, the server embeds the generated commentary in a web page in HTML format, allowing the user to view it in real time.

[1291] In this way, users can follow the progress of the game and get detailed commentary on the scoring scenes in real time. This system is an effective means of improving the viewing experience of amateur sports.

[1292] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1293] Step 1:

[1294] Live video capture

[1295] The server acquires live video in real time. Specifically, it uses a high-performance video capture device to capture the video of the match, and sends the data to the server using a video streaming tool such as FFmpeg. The input is the live video, and the output is the video data stored on the server. The server temporarily stores the received video data and prepares it for analysis.

[1296] Step 2:

[1297] Video data analysis

[1298] The server analyzes the stored video data. Specifically, it uses image recognition libraries such as OpenCV and TENSOFLOW (registered trademark) to identify basketball goal scenes and scorers in the video. The input is the stored video data, and the output is the analysis results (information on the scoring scene and scorer). The server processes the images frame by frame and tracks the movement of the basketball. It also recognizes the players' uniform numbers and identifies the scorers.

[1299] Step 3:

[1300] Generate score display

[1301] The server generates a display of the match score based on the analysis results. The input is the analysis results, and the output is the updated scoreboard data. When a point is scored, the server updates the scoreboard data and displays it to the user in real time. Specifically, it displays the scores on a webpage using HTML and JavaScript. It also uses WebSockets to send updates in real time to notify users of score changes.

[1302] Step 4:

[1303] Identifying goal-scoring scenes and generating commentary

[1304] When a scoring scene is identified, the server generates a detailed commentary on that scene. The input is the information on the scoring scene, and the output is the generated commentary. This commentary generation uses natural language generation technology such as GPT-3 (registered trademark). Specifically, information such as the scorer's name, the scoring method, and the situation of the game is input into the generation AI model as a prompt sentence to generate a detailed commentary. For example, the server generates a prompt sentence like the following:

[1305] "With 2 minutes and 30 seconds left in the third quarter, 23 players made a three-point shot. With this score, Team A took a five-point lead over Team B."

[1306] The generative AI model generates a detailed explanation based on this prompt.

[1307] Step 5:

[1308] Providing commentary

[1309] The server provides the generated commentary to the user in the form of voice or text. The input is the generated commentary, and the output is an audio file and text data. For voice synthesis, Google (registered trademark) Text-to-Speech API or the like is used, and the generated audio file is streamed to the user. The text commentary is displayed on a web page or application. For example, the server embeds the generated commentary in a web page in HTML format, allowing the user to view it in real time.

[1310] (Application example 2)

[1311] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1312] In conventional live video distribution systems for amateur sports, the display of game scores and the identification of scoring scenes are often done manually, which causes problems such as a lack of real-timeness and accuracy. In addition, it is difficult to provide detailed commentary on scoring scenes, which makes it difficult to increase viewer satisfaction. This has led to the issue that the viewing experience of amateur sports is inferior to that of professional sports.

[1313] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1314] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in voice or text for the identified scoring scene, a means for streaming the live video, and a means for generating a detailed commentary of the scoring scene using a voice synthesis technique. This makes it possible to display an accurate score and identify a scoring scene in real time, and further provide a detailed commentary in voice or text, thereby improving the viewing experience of the viewer.

[1315] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[1316] "Live video" is video content that is filmed and distributed in real time.

[1317] "Analysis" is the process of examining data in detail and extracting specific information.

[1318] The "score display" is a display that visually shows the score and progress of the game.

[1319] A "scoring scene" is a video of the moment when a goal is scored in a sporting event.

[1320] "Audio and text commentary" refers to providing audio or text explanations or comments for a video.

[1321] "Streaming" is a technology for delivering video and audio in real time over the Internet.

[1322] "Speech synthesis technology" is a technology that converts text data into speech.

[1323] "Image recognition technology" is a technology that extracts and analyzes specific information from image data.

[1324] "Natural language generation technology" is a technology that allows computers to generate human language.

[1325] The system for carrying out the present invention analyzes live video of amateur sports, displays the score, identifies the scoring scenes, and provides detailed commentary in real time. Specific embodiments are described below.

[1326] Hardware and Software Configuration

[1327] The server uses the following hardware and software:

[1328] Hardware: Servers with powerful processors, memory, storage, and network interfaces.

[1329] software:

[1330] OpenCV: Image processing library

[1331] TENSOFLOW(registered trademark) / Keras: Framework for image recognition models

[1332] Transformers (Hugging Face): A Framework for Natural Language Generation Models

[1333] pyttsx3: A speech synthesis library

[1334] Data processing and calculation

[1335] 1. Live video streaming: The server receives and streams live video in real time over the Internet.

[1336] 2. Frame capture and pre-processing: The server captures frames from the live video and performs pre-processing such as resizing and normalization using OpenCV.

[1337] 3. Image Recognition: Analyze the pre-processed frames using TENSOFLOW® / Keras to identify scoring scenes and players.

[1338] 4. Identifying scoring scenes: Based on the analysis results, Identify scoring scenes.

[1339] 5. Natural Language Generation: When a scoring scene is identified, we use Transformers to generate a detailed commentary. An example prompt is "Tanaka scored a point in a basketball game. Please generate a detailed commentary."

[1340] 6. Speech synthesis: The generated explanation is converted into speech using pyttsx3 and provided to the user.

[1341] Examples

[1342] For example, if a scene such as "Tanaka makes a three-point shot!" is detected during a game, the server will perform the following process.

[1343] 1. Capture and pre-process frames from live video.

[1344] 2. Use an image recognition model to identify scoring scenes.

[1345] 3. If a scoring scene is identified, generate an explanation using a natural language generation model using the prompt sentence, "Tanaka scored a point in the basketball game. Please generate a detailed explanation."

[1346] 4. The generated commentary is converted into audio using speech synthesis technology and played back, for example, "Tanaka made a great three-point shot, widening the team's lead."

[1347] In this way, accurate scores can be displayed in real time, scoring scenes can be identified, and detailed commentary can be provided in audio and text, improving the viewer's viewing experience.

[1348] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1349] Step 1:

[1350] The server receives live video over the Internet and performs streaming. The input is the live video stream data, and the output is frames captured in real time. Specifically, the server splits the video stream and processes each frame.

[1351] Step 2:

[1352] The server preprocesses the captured frames using OpenCV. The input is the captured frame and the output is the preprocessed frame. Specific operations include resizing the frame (e.g. to 224x224 pixels) and normalizing it (scaling pixel values ​​to the range 0 to 1).

[1353] Step 3:

[1354] The server analyzes the preprocessed frames using TENSOFLOW (registered trademark) / Keras to identify scoring scenes and players. The input is the preprocessed frames, and the output is the analysis results (whether or not a scoring scene occurred and the identification of the player). In concrete terms, the server inputs the frames into an image recognition model and obtains the probability of a scoring scene and the identification of the player.

[1355] Step 4:

[1356] The server identifies a scoring scene based on the analysis result. The input is the analysis result, and the output is the result of identifying the scoring scene. In concrete terms, if the probability of a scoring scene exceeds a certain threshold, the server identifies the scene as a scoring scene.

[1357] Step 5:

[1358] When a scoring scene is identified, the server generates a prompt sentence and generates a detailed commentary using Transformers. The input is the result of identifying the scoring scene and the prompt sentence, and the output is the generated commentary text. In concrete terms, the prompt sentence "A player scored a point in a basketball game. Please generate a detailed commentary" is input into the natural language generation model, and commentary text is generated.

[1359] Step 6:

[1360] The server converts the generated commentary text into audio using pyttsx3 and provides it to the user. The input is the generated commentary text, and the output is audio data. In concrete terms, the commentary text is input to a speech synthesis engine, audio data is generated, and the audio data is played through a speaker.

[1361] In this way, the server can analyze live video in real time, identify scoring moments and generate detailed commentary to serve to viewers.

[1362] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1363] "Example 1"

[1364] In one embodiment of the present invention, a system for analyzing live amateur sports videos, generating score displays based on the analysis results, identifying scoring scenes, and generating commentary in audio or text for the identified scoring scenes is combined with an emotion engine that recognizes the emotions of a user. The emotion engine recognizes emotions from the user's facial expressions and tone of voice, and adjusts the generation of commentary based on the recognized emotions. For example, if the user shows a happy expression, the emotion engine feeds that information back to the analysis means, and the commentary generation means generates a more exciting commentary. Conversely, if the user shows a disappointed expression, the commentary generation means generates a more calm tone of commentary. This provides a personalized experience according to the user's emotions.

[1365] "Example 2"

[1366] In one embodiment of the present invention, a system for analyzing live amateur sports videos, generating score displays based on the analysis results, identifying scoring scenes, and generating commentary in audio or text for the identified scoring scenes is combined with an emotion engine that recognizes the emotions of a user. The emotion engine recognizes emotions from the user's facial expressions and tone of voice, and adjusts the generation of commentary based on the recognized emotions. For example, if the user shows a happy expression, the emotion engine feeds that information back to the analysis means, and the commentary generation means generates a more exciting commentary. Conversely, if the user shows a disappointed expression, the commentary generation means generates a more calm tone of commentary. This provides a personalized experience according to the user's emotions.

[1367] The process flow of each embodiment will be described below.

[1368] "Example 1"

[1369] Step 1: Live amateur sports video is entered into the system.

[1370] Step 2: The system analyzes the live video input and generates a score display.

[1371] Step 3: The system identifies scoring moments and generates audio and / or text commentary for the identified scoring moments.

[1372] Step 4: At the same time, the emotion engine recognizes the user's emotions from their facial expressions and tone of voice.

[1373] Step 5: The emotion engine adjusts the commentary generation based on the recognized emotion. For example, if the user shows a happy expression, the emotion engine feeds that information back to the analysis means, and the commentary generation means generates a more exciting commentary. Conversely, if the user shows a disappointed expression, the commentary generation means generates a more calm-toned commentary.

[1374] "Example 2"

[1375] Step 1: Live amateur sports video is entered into the system.

[1376] Step 2: The system analyzes the live video input and generates a score display.

[1377] Step 3: The system identifies scoring moments and generates audio and / or text commentary for the identified scoring moments.

[1378] Step 4: At the same time, the emotion engine recognizes the user's emotions from their facial expressions and tone of voice.

[1379] Step 5: The emotion engine adjusts the commentary generation based on the recognized emotion. For example, if the user shows a happy expression, the emotion engine feeds that information back to the analysis means, and the commentary generation means generates a more exciting commentary. Conversely, if the user shows a disappointed expression, the commentary generation means generates a more calm-toned commentary.

[1380] Example 1

[1381] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1382] Conventional live video analysis systems for amateur sports were able to display the score of a match and identify the scoring scenes, but they were unable to provide personalized commentary according to the user's emotions. As a result, the user experience was uniform, and flexible responses according to individual emotions were required.

[1383] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for acquiring live video of amateur sports, a means for analyzing the acquired video data using image recognition technology, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in voice or text for the identified scoring scene using natural language generation technology, a means for recognizing the user's emotion, and a means for adjusting the tone of the commentary based on the recognized emotion. This makes it possible to provide a personalized commentary according to the user's emotion.

[1384] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[1385] "Live video" is video data that is shot and distributed in real time.

[1386] "Means for acquiring" refers to a method or device for receiving live video in real time and importing it as data.

[1387] "Video data" is digital data that includes visual information captured by a photographic device such as a camera.

[1388] "Image recognition technology" refers to a technology that uses computer vision technology to identify specific objects or events from video data.

[1389] "Means for analyzing" refers to a method or device for processing acquired video data and extracting specific information.

[1390] A "score display" is an information display that visually shows the score and progress of a game.

[1391] A "generating means" is a method or device for producing new data or displays based on specific information.

[1392] A "scoring scene" refers to the moment or situation in which a goal is scored in a sporting event.

[1393] The "means for identifying" refers to a method or device for identifying a particular event or object from the analysis results.

[1394] "Natural language generation technology" is technology that enables computers to generate human language.

[1395] A "means for generating audio or textual descriptions" is a method or apparatus for generating descriptions in audio or text form using natural language generation techniques.

[1396] The "means for recognizing a user's emotion" refers to a method or device for analyzing a user's facial expression and tone of voice to identify the emotion.

[1397] A "means for adjusting the tone of commentary" is a method or device for modifying the manner in which commentary is presented based on perceived emotions.

[1398] This invention is a system that analyzes live amateur sports videos and provides personalized commentary according to the user's emotions. This system consists of three main components: a server, a terminal, and a user.

[1399] Hardware and Software Used

[1400] Hardware: High-performance servers, user devices (PCs, smartphones, etc.)

[1401] Software: Image recognition technology (e.g., OpenCV), natural language generation technology (e.g., GPT-3 (registered trademark)), emotion recognition engine (e.g., Microsoft Azure Emotion API)

[1402] Specific operation of the system

[1403] 1. Acquiring live video

[1404] The server obtains live amateur sports videos in real time. Specifically, it receives video data from streaming services (e.g., YouTube Live, Twitch). The server starts the live stream from a specified URL and stores the video data in a buffer so that it can be processed in real time.

[1405] 2. Analysis of video data

[1406] The server analyzes the acquired video data using image recognition technology (e.g. OpenCV). Specifically, it identifies goal scenes and scorers, and keeps track of the game time. The server performs image analysis for each frame, detects goal scenes, and identifies the scorers by identifying the players' jersey numbers and uniform colors. It also reads the game time from the timer in the video.

[1407] 3. Generate score display

[1408] The server generates a display of the match score based on the analysis results, e.g., updates the score when a goal is recognized. The server increments the score the moment a goal is detected and generates a text overlay to display the scorer's name and the time of the goal.

[1409] 4. Generating commentary on goal-scoring scenes

[1410] When a scoring scene is identified, the server generates a detailed commentary of that scene using natural language generation technology (e.g., GPT-3 (registered trademark)). The commentary is provided in audio and text. The server analyzes the video data of the scoring scene and extracts important events (e.g., assists, type of shot). A prompt sentence is generated based on the extracted information and input into the generative AI model. The generated commentary is converted into audio using speech synthesis technology (e.g., Google (registered trademark) Text-to-Speech).

[1411] 5. User Emotion Recognition

[1412] The device analyzes the user's facial expressions and tone of voice with an emotion recognition engine (e.g., Microsoft Azure Emotion API) to recognize the user's emotions. The device uses a camera and microphone to capture the user's facial expressions and voice, and sends the captured data to the emotion recognition engine to analyze the emotions. The analysis results are then sent to the server.

[1413] 6. Adjust your commentary based on emotions

[1414] The server adjusts the tone of the commentary based on the recognized user's emotions. For example, if the user shows a happy expression, it generates a more exciting commentary. Conversely, if the user shows a disappointed expression, it generates a more calm tone of commentary. The server receives the user's emotion data and feeds it back into the commentary generation process. It adjusts the prompt sentence based on the emotion data and re-inputs it into the generative AI model. The adjusted commentary is regenerated using speech synthesis technology and provided to the user.

[1415] Examples and prompts

[1416] Examples

[1417] Scenario: A goal is scored during an amateur soccer match.

[1418] Process flow:

[1419] 1. The server recognizes goal scenes from live video.

[1420] 2. The server updates the score and displays the scorer information.

[1421] 3. The server generates commentary of the scoring scene and provides it in audio and text.

[1422] 4. The device analyzes the user's facial expressions and recognizes expressions of joy.

[1423] 5. The server reproduces the commentary in an exciting tone and provides it to the user.

[1424] Examples of prompt statements

[1425] "Describe a scene where a goal is scored during an amateur soccer match. If the user shows signs of joy, generate commentary in an exciting tone."

[1426] The above is an embodiment of the present invention.

[1427] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1428] Step 1: Getting live video

[1429] The server retrieves live amateur sports videos in real time. As input, it receives the URL of a streaming service. Specifically, the server starts the live stream from the specified URL and stores the video data in a buffer. As output, it obtains video data that can be processed in real time.

[1430] Step 2: Analyze the video data

[1431] The server analyzes the acquired video data using image recognition technology (e.g. OpenCV). It receives frame data of live video as input. Specifically, the server performs image analysis for each frame, detects goal scenes, and identifies the scorers by identifying the players' jersey numbers and uniform colors. It also reads the match time from a timer in the video. It obtains the goal scene detection results, scorer information, and match time as output.

[1432] Step 3: Generate the score display

[1433] The server generates a match score display based on the analysis results. As input, it receives goal detection results, scorer information, and match time. Specifically, the server increments the score the moment a goal is detected and generates a text overlay to display the scorer's name and the time of the goal. As output, it gets an updated score display.

[1434] Step 4: Generate commentary on goal scoring scenes

[1435] When a scoring scene is identified, the server generates a detailed commentary of the scene using natural language generation technology (e.g., GPT-3 (registered trademark)). As input, the server receives video data of the scoring scene and important event information (e.g., assists, type of shot). In specific operations, the server generates a prompt sentence based on the extracted information and inputs it into the generative AI model. The generated commentary is converted into audio using speech synthesis technology (e.g., Google (registered trademark) Text-to-Speech). As output, the server obtains a commentary in audio and text format.

[1436] Step 5: Recognizing User Emotions

[1437] The device analyzes the user's facial expressions and tone of voice using an emotion recognition engine (e.g. Microsoft Azure Emotion API) to recognize the user's emotions. As input, it receives the user's facial expressions and voice data captured by the camera and microphone. Specifically, the device sends the captured data to the emotion recognition engine and analyzes the emotions. It then sends the analysis results to the server. As output, it obtains the user's emotional data.

[1438] Step 6: Adjust your commentary based on emotion

[1439] The server adjusts the tone of the commentary based on the recognized user emotion. As input, it receives the user's emotion data. In concrete terms, the server adjusts the prompt sentence based on the emotion data and re-inputs it into the generative AI model. The adjusted commentary is regenerated by speech synthesis technology and provided to the user. As output, a personalized commentary according to the user's emotion is obtained.

[1440] (Application example 1)

[1441] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1442] Conventional live video analysis systems for amateur sports display game scores, identify scoring scenes, and generate commentary, but they are unable to provide personalized commentary based on the user's emotions. This results in a uniform viewing experience for users, making it difficult to provide optimal commentary based on individual emotions.

[1443] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1444] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying scoring scenes based on the analysis result, a means for generating commentary in audio or text for the identified scoring scenes, a means for recognizing a user's emotion, and a means for adjusting the tone of the commentary based on the recognized emotion, thereby making it possible to provide a personalized commentary according to the user's emotion.

[1445] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[1446] "Live video" is video that is filmed in real time and streamed simultaneously.

[1447] "Analysis" is the process of examining data or information in detail to derive specific results or meanings.

[1448] The "score display" is a display that visually shows the score and progress of the game.

[1449] A "scoring scene" is a video or scene showing the exact moment a goal is scored in a sporting event.

[1450] A "commentary" is an explanation or commentary on a particular event or situation.

[1451] "Speech" refers to auditory information such as speech and music.

[1452] "Text" is information or writing written in text.

[1453] "User" means an individual or organization that uses the system or service.

[1454] Emotions are human psychological states and feelings.

[1455] "Cognition" is the process of understanding and grasping specific information or a situation.

[1456] "Tone" is the mood or atmosphere of a voice or text.

[1457] "Adaptation" is the process of changing or modifying something to suit a specific purpose.

[1458] "Personalization" means customizing to suit the characteristics and preferences of individual users.

[1459] As an embodiment of the present invention, a system is constructed that analyzes live videos of amateur sports and provides personalized commentary according to the user's emotions. A specific embodiment is described below.

[1460] Hardware and Software Configuration

[1461] Hardware

[1462] Server: Analyzes live video, generates score displays, identifies scoring moments, generates commentary, and recognizes emotions.

[1463] Device: The device (smartphone, tablet, PC, etc.) through which a user watches live video.

[1464] Camera: A device for capturing live video.

[1465] Microphone: A device for collecting the user's voice.

[1466] software

[1467] OpenCV: A library for live video streaming and frame processing.

[1468] Keras: A library that provides an image recognition model used to identify goal scoring scenes.

[1469] Transformers (Hugging Face): A library that generates commentary using the GPT-2 model.

[1470] SpeechRecognition: A library that converts the user's speech into text.

[1471] EmotionRecognition: A library for recognizing user emotions.

[1472] Data processing and calculation

[1473] Server Processing

[1474] The server receives the live video from the camera and analyzes the frames in real time using OpenCV. Based on the analysis, it generates a score display of the match and identifies the scoring scenes. If a scoring scene is identified, it uses the GPT-2 model to generate commentary for that scene.

[1475] User Emotion Recognition

[1476] It collects voice data through the microphone connected to the user's device and converts the voice to text using SpeechRecognition. It analyzes the converted text with the EmotionRecognition library to recognize the user's emotions. It adjusts the tone of the commentary based on the recognized emotions.

[1477] Examples

[1478] Live video URL

[1479] Example: http: / / example.com / live_stream

[1480] Explanation of the goal

[1481] Example: Player X scores a goal

[1482] User audio files

[1483] Example: user_audio.wav

[1484] Examples of prompt statements

[1485] Your goal is to develop a system that analyzes live amateur soccer videos, displays match scores, identifies goal-scoring moments, and generates commentary. The system will recognize the user's emotions and provide commentary that reflects those emotions. Specifically, your goal is to create a smartphone application that has the following functions:

[1486] 1. Live video streaming

[1487] 2. Score display

[1488] 3. Identifying the goal-scoring moment

[1489] 4. Description Generation

[1490] 5. Emotion recognition

[1491] In this way, a system can be realized that provides personalized commentary according to the user's emotions.

[1492] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1493] Step 1:

[1494] The server receives the live video sent from the camera. The input is the live video stream data, and the output is the image data for each frame. Using OpenCV, we grab the frames in real time and prepare them for analysis.

[1495] Step 2:

[1496] The server analyzes the captured frames and generates a display of the match score. The input is image data for each frame, and the output is text data that displays the score. Image recognition technology is used to determine the score and the progress of the match, and to update the score.

[1497] Step 3:

[1498] The server identifies the goal-scoring scene based on the analysis results. The input is the analyzed frame data, and the output is the score scene identification information. Using Keras, we detect the goal scene and identify the scene.

[1499] Step 4:

[1500] The server generates a commentary in voice or text for the identified scoring scene. The input is the specific information of the scoring scene, and the output is the text data or audio data of the commentary. The commentary is generated using natural language generation technology using the GPT-2 model of Transformers (Hugging Face).

[1501] Step 5:

[1502] The terminal collects the user's voice. The input is the user's voice data, and the output is an audio file. The terminal uses a microphone to record the user's voice and saves it as an audio file.

[1503] Step 6:

[1504] The server analyzes the collected voice data and recognizes the user's emotions. The input is the audio file and the output is the user's emotion information. It uses SpeechRecognition to convert the voice to text and the EmotionRecognition library to recognize emotions.

[1505] Step 7:

[1506] The server adjusts the tone of the commentary based on the recognized emotion. The input is the user's emotion information and the generated commentary data, and the output is the adjusted commentary data. Based on the emotion information, the tone of the commentary is changed to exciting or calm.

[1507] Step 8:

[1508] The terminal provides the adjusted commentary to the user. The input is the adjusted commentary data, and the output is display or playback of the commentary to the user. The commentary is provided to the user using the screen and speaker of the terminal.

[1509] Example 2

[1510] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1511] Conventional live video analysis systems for amateur sports can display the score of a match and identify the scoring scenes, but they cannot provide personalized commentary according to the user's emotions. Therefore, in order to enrich the user experience, a function that recognizes the user's emotions and adjusts the commentary based on them is required.

[1512] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1513] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying scoring scenes based on the analysis result, a means for generating commentary in audio or text for the identified scoring scenes, a means for recognizing a user's emotion, and a means for adjusting the commentary based on the recognized emotion, thereby making it possible to provide a personalized commentary according to the user's emotion.

[1514] "Amateur sports" refers to sports competitions played by non-professional individuals or teams.

[1515] "Live video" refers to video footage that is filmed in real time and streamed simultaneously.

[1516] "Analysis" is the process of examining acquired data in detail and extracting specific information.

[1517] A "score display" is a display that visually shows the score and progress of a game.

[1518] A "scoring scene" refers to footage of the exact moment a goal is scored in a sporting event.

[1519] "Means for generating commentary in audio or text" refers to technology that creates audio or text explanations of scoring scenes or game situations.

[1520] "Means for recognizing a user's emotions" refers to technology that determines emotions from a user's facial expressions, tone of voice, etc.

[1521] "Means for tailoring commentary" refers to techniques that change the content and tone of commentary based on recognized emotions.

[1522] "Image recognition technology" refers to the technology of analyzing image data to detect specific objects or actions.

[1523] "Natural language generation technology" refers to technology that allows computers to generate human language.

[1524] This invention is a system that analyzes live videos of amateur sports, generates score displays based on the analysis results, identifies scoring scenes, and generates audio and text commentary for the identified scoring scenes. It also has a function of recognizing the user's emotions and adjusting the commentary based on those emotions.

[1525] Hardware and Software Used

[1526] Hardware: High performance servers, cameras, microphones

[1527] Software: Image recognition technology (e.g. OpenCV), natural language generation technology (e.g. GPT-3 (registered trademark)), emotion recognition engine (e.g. Affectiva)

[1528] Specific operation of the system

[1529] Live video capture

[1530] The server receives live video from the camera.

[1531] The server connects to the camera and receives the video data in real time.

[1532] Specific operation: The server obtains the video stream from the camera using the RTSP protocol.

[1533] Video data analysis

[1534] The server analyzes the acquired video data using image recognition technology.

[1535] The server uses OpenCV to identify basketball goal scenes and scorers in the video.

[1536] What it does: The server processes the images frame by frame and runs an algorithm to detect goal moments.

[1537] Generate score display

[1538] The server generates a score display for the match based on the analysis.

[1539] The server updates the scoreboard every time a point is scored.

[1540] Specific operation: When the server detects a score event, it records the score in the database and sends an instruction to update the score to the front end.

[1541] Commentary generation for scoring scenes

[1542] When a scoring scene is identified, the server generates a detailed commentary of the scene in audio and / or text.

[1543] The server uses GPT-3® to generate explanations in natural language.

[1544] Specific operation: The server inputs the information about the scoring scene into GPT-3 (registered trademark) and generates an explanatory text such as "That was a great shot!"

[1545] User Emotion Recognition

[1546] The device recognizes the user's emotions from their facial expressions and tone of voice.

[1547] The device uses Affectiva to analyze the user's emotions.

[1548] How it works: The device uses the camera and microphone to capture the user's facial expressions and voice, and calls Affectiva's API to obtain emotion data.

[1549] Adjustment of explanation

[1550] The server adjusts the commentary generation based on the recognized emotions.

[1551] The server adjusts the commentary to an exciting tone if the user is happy.

[1552] Specific operation: The server analyzes the emotion data and re-inputs parameters to GPT-3 (registered trademark) to change the tone of the commentary.

[1553] Examples and prompts

[1554] Examples:

[1555] As the user watches a game, the camera captures a goal scene.

[1556] The server uses OpenCV to detect goal scenes and identify the scorer.

[1557] How it works: The server analyzes the movement within the frame and detects the moment the ball enters the goal.

[1558] The server uses GPT-3(registered trademark) to generate an explanation: "That was a great shot!"

[1559] Specific operation: The server inputs the scoring scene information into GPT-3 (registered trademark) and generates an explanatory text.

[1560] The device uses Affectiva to analyze the user's facial expressions and recognize when the user is happy.

[1561] Specific operation: The device captures the user's smile with the camera and calls Affectiva's API to obtain emotion data.

[1562] The server adjusts his commentary to an exciting tone: "This shot could change the course of the game!"

[1563] Specific operation: The server analyzes the emotion data and re-inputs parameters to GPT-3 (registered trademark) to change the tone of the commentary.

[1564] Example prompt:

[1565] "When a user is watching a game and the camera captures a goal, how do you identify the scorer and generate commentary? Also, how do you adjust the commentary depending on the user's emotions?"

[1566] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1567] Step 1:

[1568] Live video capture

[1569] The server receives live video from the camera.

[1570] Input: Real-time video stream from the camera

[1571] Specific operation: The server obtains the video stream from the camera using the RTSP protocol.

[1572] Output: Live video data

[1573] Step 2:

[1574] Video data analysis

[1575] The server analyzes the acquired video data using image recognition technology.

[1576] Input: Live video data

[1577] What it does: The server uses OpenCV to process the images frame by frame and run algorithms to identify goals and scorers.

[1578] Output: Goal scene detection results, scorer identification results

[1579] Step 3:

[1580] Generate score display

[1581] The server generates a score display for the match based on the analysis.

[1582] Input: Goal scene detection results, goal scorer identification results

[1583] Specific operation: When the server detects a score event, it records the score in the database and sends an instruction to update the score to the front end.

[1584] Output: Updated score display

[1585] Step 4:

[1586] Commentary generation for scoring scenes

[1587] When a scoring scene is identified, the server generates a detailed commentary of the scene in audio and / or text.

[1588] Input: Goal scene detection results, goal scorer identification results

[1589] Specific operation: The server inputs the scoring scene information into GPT-3 (registered trademark) and generates an explanatory text in natural language.

[1590] Output: Description (e.g. "That was a great shot!")

[1591] Step 5:

[1592] User Emotion Recognition

[1593] The device recognizes the user's emotions from their facial expressions and tone of voice.

[1594] Input: User's facial expression data, voice tone data

[1595] How it works: The device uses the camera and microphone to capture the user's facial expressions and voice, and calls Affectiva's API to obtain emotion data.

[1596] Output: User emotion data

[1597] Step 6:

[1598] Adjustment of explanation

[1599] The server adjusts the commentary generation based on the recognized emotions.

[1600] Input: User emotion data, explanatory text

[1601] Specific operation: The server analyzes the emotion data and re-inputs parameters to GPT-3 (registered trademark) to change the tone of the commentary.

[1602] Output: Tailored explanatory text (e.g. "This shot could change the game!")

[1603] (Application example 2)

[1604] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1605] When watching live amateur sports videos, it is difficult for viewers to grasp the progress of the game and the scoring scenes in real time. In addition, because personalized commentary based on the viewer's emotions is not provided, the viewing experience is uniform and it is not possible to increase the satisfaction of each individual viewer. This makes it difficult to maintain the viewer's interest, and there is a problem that the quality of the viewing experience is reduced.

[1606] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1607] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in audio or text for the identified scoring scene, a means for recognizing a user's emotion, and a means for adjusting the generation of the commentary based on the recognized emotion. This not only enables a viewer to grasp the progress of the game and the scoring scenes in real time, but also provides a personalized commentary according to the viewer's emotion, thereby improving the quality of the viewing experience and increasing the viewer's satisfaction.

[1608] "Amateur sports" are sporting events in which non-professional individuals or teams participate.

[1609] "Live video" is video content that is filmed in real time and streamed simultaneously.

[1610] "Analysis" is the process of examining data in detail to understand its structure and meaning.

[1611] A "score display" is an information display that visually shows the score and progress of a match.

[1612] A "scoring scene" refers to the image or circumstances of the moment when a goal is scored in a sporting event.

[1613] "Generating commentary in audio or text" means creating audio or text explanations about the situation in a game or scoring scenes.

[1614] "Recognizing the user's emotions" means judging the viewer's emotional state from facial expressions, tone of voice, and the like.

[1615] "Adjusting commentary production" refers to changing the content and tone of commentary based on the recognized emotions.

[1616] "Image recognition technology" is a technology that analyzes image data and identifies specific objects or scenes.

[1617] "Natural language generation technology" is technology that enables computers to generate human language.

[1618] The system for carrying out the present invention analyzes live video of amateur sports, generates a score display based on the analysis results, identifies scoring scenes, generates audio and text commentary for the identified scoring scenes, and has the function of recognizing the user's emotions and adjusting the generation of commentary based on the emotions.

[1619] Hardware and Software Used

[1620] Hardware:

[1621] It uses the smartphone's camera and microphone for live video capture and user emotion recognition.

[1622] The server provides computational resources for video analysis and commentary generation.

[1623] software:

[1624] OpenCV: A library for capturing and displaying live video.

[1625] DeepFace: A library for recognizing emotions from user facial expressions.

[1626] Transformers (Hugging Face): A library for generating explanations using natural language generation techniques.

[1627] gTTS: A library for converting generated text descriptions into speech.

[1628] playsound: A library for playing generated sounds.

[1629] System Operation

[1630] 1. Live video analysis:

[1631] The server analyzes live video captured by the smartphone camera in real time.

[1632] Using OpenCV, we capture video frames and use image recognition techniques to identify scoring scenes and scorers.

[1633] 2. Generate the score display:

[1634] The server updates the match score in real time based on the analysis results and displays it on the smartphone screen.

[1635] 3. Emotion recognition:

[1636] It uses your smartphone's camera and microphone to capture your facial expressions and tone of voice.

[1637] Use DeepFace to recognize user emotions in real time.

[1638] 4. Description Generation:

[1639] If a scoring scene is identified, the server generates a detailed explanation of the scene.

[1640] It uses Transformers' natural language generation model to generate explanations based on the user's emotions.

[1641] The generated commentary is converted into audio using gTTS and played through the smartphone's speakers.

[1642] Examples

[1643] For example, suppose a user is watching an amateur basketball game on their smartphone. If a goal is scored during the game and the user shows a joyful expression, the system recognizes the emotion and generates an exciting commentary. The generated commentary is generated using the following prompt sentence:

[1644] Examples of prompt statements

[1645] If the user indicates the emotion of joy, generate an exciting description of the following scene: Basketball Goal Scene

[1646] In this way, a personalized commentary is provided according to the user's emotions, improving the quality of the viewing experience and increasing viewer satisfaction.

[1647] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1648] Step 1:

[1649] The server captures live video from the smartphone camera. The input is the live video stream and the output is the image data for each frame. OpenCV is used to get the video frames in real time.

[1650] Step 2:

[1651] The server analyzes the captured video frames and identifies the scoring scenes and scorers. The input is the image data of the video frames, and the output is the result of identifying the scoring scenes. Image recognition technology is used to identify the basketball goal scenes and scorers.

[1652] Step 3:

[1653] The server updates the score of the game based on the analysis results and displays it on the smartphone screen. The input is the result of identifying the scoring scene, and the output is an updated score display. The score display generation means updates the score in real time.

[1654] Step 4:

[1655] The device uses the smartphone's camera and microphone to capture the user's facial expressions and tone of voice. The input is the user's facial expression images and voice data, and the output is data for emotion recognition. The camera and microphone are used to capture the user's emotions in real time.

[1656] Step 5:

[1657] The server analyzes the captured data and recognizes the user's emotions. The input is the user's facial expression images and voice data, and the output is the recognized emotions. DeepFace is used to analyze the user's emotions.

[1658] Step 6:

[1659] The server generates a detailed commentary of a scoring scene when it is identified. The input is the scoring scene identification result and the recognized emotion, and the output is the generated commentary text. The commentary is generated based on the prompt sentence using a natural language generation model from Transformers.

[1660] Step 7:

[1661] The server converts the generated commentary text into speech and plays it through the smartphone speaker. The input is the generated commentary text and the output is the audio data. gTTS is used to convert the text to speech and plays the audio using playsound.

[1662] In this way, the system can analyze live video in real time and provide personalized commentary according to the user's emotions.

[1663] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1664] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1665] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google(登録商標).com ?hl="ja">) are mentioned.

[1666] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1667] [Fourth embodiment]

[1668] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1669] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1670] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1671] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1672] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1673] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1674] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1675] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1676] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1677] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1678] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1679] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1680] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1681] "Example 1"

[1682] An example of the present invention is a system that analyzes live videos of amateur soccer matches. This system uses image recognition technology to analyze video data obtained from the live videos and generate a display of the match score. Specifically, the system identifies goal scenes and scorers, and keeps track of the match time. When a goal-scoring scene is identified, the system generates a detailed commentary of that scene in voice and text. Natural language generation technology is used to generate this commentary.

[1683] "Example 2"

[1684] Another example is a system that analyzes live videos of amateur basketball games. This system uses image recognition technology to analyze video data obtained from the live videos and generate a display of the game score. Specifically, it identifies basketball goal scenes and scorers, and keeps track of the game time. In addition, when a scoring scene is identified, it generates a detailed commentary of that scene in voice and text. Natural language generation technology is used to generate this commentary.

[1685] The process flow of each embodiment will be described below.

[1686] "Example 1"

[1687] Step 1: Get live amateur football video.

[1688] Step 2: Extract video data from the captured live video and analyze it using image recognition technology. This analysis includes identifying goal scenes, scorers, and game time.

[1689] Step 3: Generate a score display based on the analysis results.

[1690] Step 4: If a scoring scene is identified, a detailed commentary of that scene is generated in audio and text format using natural language generation technology.

[1691] "Example 2"

[1692] Step 1: Get live amateur basketball video.

[1693] Step 2: Extract video data from the captured live video and analyze it using image recognition technology. This analysis includes identifying basketball goal scenes, scorers, and game time.

[1694] Step 3: Generate a score display based on the analysis results.

[1695] Step 4: If a scoring scene is identified, a detailed commentary of that scene is generated in audio and text format using natural language generation technology.

[1696] Example 1

[1697] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1698] In conventional live video analysis systems for amateur sports, displaying the game score and identifying the scoring scenes are often done manually, which means that they lack real-timeness and accuracy. In addition, commentary on the scoring scenes is also done manually, which makes it difficult to provide information quickly. This makes it difficult for spectators and officials to accurately and quickly grasp the progress of the game.

[1699] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1700] In this invention, the server includes a means for acquiring live video data, a means for preprocessing the acquired video data, a means for analyzing the preprocessed video data using image recognition technology, a means for identifying a scoring scene based on the analysis result, a means for generating a prompt sentence for generating a commentary for the identified scoring scene, a means for inputting the generated prompt sentence into a generation AI model to generate a commentary, and a means for outputting the generated commentary in voice or text. This automates the display of the score of the game and the identification of the scoring scene, making it possible to provide accurate information in real time. In addition, detailed commentary of the scoring scene is also quickly generated, making it possible for spectators and related parties to accurately and quickly grasp the progress of the game.

[1701] "Live video data" refers to video information acquired in real time.

[1702] "Preprocessing" refers to processes such as data resizing, noise removal, and frame extraction that are carried out to make the acquired video data easier to analyze.

[1703] "Image recognition technology" is a technology for detecting and identifying specific objects and actions from video data.

[1704] "Analysis results" refers to the analysis results of data obtained using image recognition technology.

[1705] A "scoring scene" is a video of the exact moment a goal is scored in a sports game.

[1706] A "prompt sentence" is an instruction sentence to be input into a generative AI model.

[1707] A "generative AI model" is an artificial intelligence model that generates natural language text based on a given prompt.

[1708] "Commentary" refers to detailed explanations and comments about the scoring scene.

[1709] "Audio output" refers to providing the generated description in audio form.

[1710] "Text output" refers to providing the generated commentary in text form.

[1711] This invention is a system that analyzes live videos of amateur sports and automatically generates game scores and detailed commentary of scoring scenes. This system works in cooperation with three entities: a server, a terminal, and a user.

[1712] First, the server acquires live video data. The video data is transmitted in real time from devices such as cameras and smartphones. The server streams the video data from the cameras using RTSP (Real-Time Streaming Protocol).

[1713] The server then pre-processes the captured video data, which includes image resizing, noise removal, frame extraction, etc. Specifically, the server uses OpenCV to resize the video data to 640x480 pixels and apply a noise removal filter.

[1714] The pre-processed video data is then analyzed using image recognition technology. The server uses libraries such as TENSOFLOW (registered trademark) and YOLO (You Only Look Once) to detect players and the ball in each frame. Based on the analysis results, the server identifies the goal-scoring scene. To identify the goal scene, the server detects the movement of the ball and the shaking of the goal net. The server detects the moment the ball crosses the goal line and saves that frame.

[1715] When a scoring scene is identified, the server generates a prompt sentence for generating commentary. For example, the server generates a prompt sentence such as, "Please generate a detailed commentary on the scene where Player A scored a goal." The server inputs this prompt sentence into a generative AI model (for example, GPT-3 (registered trademark)) and generates a commentary. The generative AI model generates a commentary such as, "Player A received a cross from the right side and scored a goal with a brilliant volley shot."

[1716] The generated commentary is output by the server in the form of voice or text. For voice output, Google® Text-to-Speech API or the like is used. The server sends the generated text commentary to Google® Text-to-Speech API to generate an audio file. The server then sends the audio file to the user's device. The user can watch or view the commentary through the device.

[1717] As a concrete example, consider a case where a server analyzes a live video and identifies the scene where player A scored a goal. The server inputs a prompt sentence, "Please generate a detailed commentary on the scene where player A scored a goal," into the generative AI model. The generative AI model generates a commentary, "Player A received a cross from the right side and scored a goal with a brilliant volley shot." The server converts the generated commentary into audio, which the user can listen to on their device.

[1718] The server also keeps track of the game time. For example, the server detects that the game time has passed 45 minutes, and inputs a prompt message to the generative AI model saying, "Please generate a message announcing the end of the first half." The generative AI model generates an explanation saying, "The first half has ended. The current score is 1-0." The server displays the generated explanation as text, and the user can view it on their device.

[1719] The above is an embodiment of the present invention.

[1720] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1721] Step 1:

[1722] The server acquires the live video data.

[1723] Input: Real-time video data sent from devices such as cameras and smartphones.

[1724] Specific operation: The server streams video data from the camera using RTSP (Real-Time Streaming Protocol).

[1725] Output: Acquired live video data.

[1726] Step 2:

[1727] The server pre-processes the acquired video data.

[1728] Input: Acquired live video data.

[1729] Specific operation: The server uses OpenCV to resize the video data to 640x480 pixels and apply a noise reduction filter.

[1730] Output: Pre-processed video data.

[1731] Step 3:

[1732] The server analyzes the pre-processed video data using image recognition technology.

[1733] Input: Preprocessed video data.

[1734] Specific operation: The server uses libraries such as TENSOFLOW (registered trademark) and YOLO (You Only Look Once) to detect players and the ball in each frame.

[1735] Output: Analysis results (positions of players and the ball, etc.).

[1736] Step 4:

[1737] The server identifies the scoring scene based on the analysis result.

[1738] Input: The analysis results.

[1739] What happens: The server detects the moment the ball crosses the goal line and saves that frame.

[1740] Output: Goal specific information (scoring frame).

[1741] Step 5:

[1742] The server generates a prompt sentence for generating an explanation for the scoring scene.

[1743] Input: Scoring scene specific information.

[1744] Specific operation: The server generates a prompt sentence saying, "Please generate a detailed commentary of the scene where player A scored a goal."

[1745] Output: The prompt statement.

[1746] Step 6:

[1747] The server inputs the generated prompt sentence into a generative AI model to generate an explanation.

[1748] Input: The prompt statement.

[1749] Specific operation: The server sends a prompt sentence to a generative AI model (e.g., GPT-3 (registered trademark)) to generate an explanation.

[1750] Output: The generated description (in text format).

[1751] Step 7:

[1752] The server outputs the generated commentary in audio and / or text.

[1753] Input: The generated description (in text format).

[1754] Specific operation: The server sends the generated text commentary to the Google® Text-to-Speech API to generate an audio file, and then sends the audio file to the user's device.

[1755] Output: Audio files and text description.

[1756] The above is the flow of processing of the program of this system.

[1757] (Application example 1)

[1758] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1759] In live video distribution of amateur sports, it is difficult to provide detailed commentary of game scores and scoring scenes in real time. In addition, when viewers watch games using smart devices, the user interface is not optimized, resulting in a poor viewing experience. Furthermore, there is also the issue that it is difficult for viewers to understand the flow of the game because commentary of scoring scenes cannot be provided by audio.

[1760] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1761] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in voice or text for the identified scoring scene, a means for providing a user interface optimized for a smart device, a means for measuring the game time in real time, and a means for converting detailed commentary of the scoring scene into voice using a voice synthesis technology. This makes it possible to provide detailed commentary of the game score and the scoring scene in real time, allowing viewers to comfortably watch the game using their smart devices. Also, by providing commentary of the scoring scene in voice, viewers can have a deeper understanding of the flow of the game.

[1762] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[1763] "Live video" is video that is filmed in real time and streamed simultaneously.

[1764] "Analysis" means examining data or information in detail to clarify its structure and meaning.

[1765] A "score display" is a visual indication of the score and progress of a match.

[1766] A "scoring scene" is a video of the moment when a goal is scored in a sporting event.

[1767] "Means for generating commentary in audio or text" refers to technology for explaining scoring scenes and the situation of the game in audio or text.

[1768] A "smart device" is an electronic device with advanced functions, such as a smartphone, tablet, or head-mounted display.

[1769] A "user interface" is a screen or method of operation that allows a user to interact with a system or application.

[1770] "Means for measuring match time in real time" refers to technology that measures and displays the progress of a match in real time.

[1771] "Speech synthesis technology" is a technology that converts text data into speech.

[1772] "Image recognition technology" is a technology that analyzes image data and recognizes specific objects or scenes.

[1773] "Natural language generation technology" is a technology that allows computers to generate human language.

[1774] A system for carrying out the present invention analyzes live video of amateur sports and provides real-time scores and commentary on scoring scenes. A specific embodiment of this system will be described below.

[1775] System Configuration

[1776] 1. Hardware:

[1777] Server: A server with powerful computing power (e.g. AWS EC2 instance)

[1778] Devices: Smartphones, head-mounted displays (e.g. Oculus Quest, HoloLens)

[1779] 2. Software:

[1780] Image recognition technology: OpenCV, TENSOFLOW (registered trademark)

[1781] Natural language generation technology: GPT-3 (registered trademark), BERT

[1782] Speech synthesis technology: Google(R) Text-to-Speech

[1783] Streaming services: AWS Elemental MediaLive, YouTube(R) Live

[1784] Data processing and calculation

[1785] 1. Live video analysis:

[1786] The server captures frames of live video and uses image recognition technology (OpenCV, TENSOFLOW (registered trademark)) to identify goal scenes and scorers.

[1787] The server times the match in real time and generates a score display.

[1788] 2. Commentary generation for goal scoring scenes:

[1789] When a scoring scene is identified, the server generates a detailed description of the scene using natural language generation technology (GPT-3 (registered trademark), BERT).

[1790] The generated commentary is converted into audio using speech synthesis technology (Google® Text-to-Speech).

[1791] 3. Providing the user interface:

[1792] The terminal provides a user interface optimized for smart devices, allowing viewers to watch the game comfortably.

[1793] Examples

[1794] For example, when an amateur soccer match is being broadcast live, users watch the match through their smartphones or head-mounted displays. When a goal is scored, the server automatically identifies the scene and displays the name of the scorer and the details of the goal in real time. In addition, commentary of the goal is provided in audio and text, allowing users to gain a deeper understanding of the flow of the match.

[1795] Examples of prompt statements

[1796] "The current match time is 45 minutes. A goal has been scored. The scorer is Tanaka. Please generate a detailed explanation of this scene."

[1797] By inputting this prompt into a generative AI model, a detailed explanation of the scoring scene is generated.

[1798] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1799] Step 1:

[1800] The server captures frames of live video. The input is the live video stream and the output is the individual frame images. The server receives the live video using a streaming service (e.g. AWS Elemental MediaLive) and splits it into frames.

[1801] Step 2:

[1802] The server analyzes the captured frame images using image recognition technology (e.g., OpenCV, TENSOFLOW (registered trademark)). The input is the frame image, and the output is the goal scene and the scorer identification result. The server identifies specific actions and people in the frame and identifies the goal scene and the scorer.

[1803] Step 3:

[1804] The server measures the match time in real time. The input is the live video stream and the output is the current match time. The server calculates the elapsed time based on the start time of the video and updates the match time in real time.

[1805] Step 4:

[1806] The server generates a score display based on the analysis results. The inputs are goal scenes, scorer identification results, and match time, and the output is score display data. The server updates the score when a goal occurs and generates data to display it visually.

[1807] Step 5:

[1808] When a scoring scene is identified, the server generates a detailed commentary of the scene using natural language generation technology (e.g., GPT-3 (registered trademark), BERT). The input is the result of identifying the scoring scene and the game time, and the output is the commentary text. The server inputs the prompt sentence into the generative AI model and generates a detailed commentary.

[1809] Step 6:

[1810] The server converts the generated commentary text into audio using voice synthesis technology (e.g., Google (registered trademark) Text-to-Speech). The input is the commentary text, and the output is audio data. The server converts the text data into audio data and provides it to the viewer.

[1811] Step 7:

[1812] The terminal provides a user interface optimized for smart devices. The input is score display data and audio data, and the output is visual and audio information. The terminal displays the score and commentary through the user interface and plays the audio commentary.

[1813] Step 8:

[1814] Users use smart devices to watch games. The input is the visual and auditory information provided by the device, and the output is the user's viewing experience. Users understand the flow of the game through real-time updated scores and commentary.

[1815] Example 2

[1816] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1817] In live amateur sports videos, it is difficult to provide detailed commentary on the progress of the game and scoring scenes in real time. In particular, identifying scoring scenes and generating commentary requires advanced technology, and conventional systems have not been able to do this efficiently.

[1818] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1819] In this invention, the server includes a means for acquiring live video, a means for analyzing the acquired video data using image recognition technology, a means for generating a score display based on the analysis result, a means for identifying scoring scenes based on the analysis result, a means for generating commentary in voice or text for the identified scoring scenes using natural language generation technology, a means for converting the generated commentary into a voice file using voice synthesis technology and providing it to the user, and a means for displaying the generated commentary in text format, thereby making it possible to provide detailed commentary on the progress of the game and scoring scenes in real time.

[1820] "Live video" refers to video data that is filmed in real time and instantly distributed via the Internet or other means.

[1821] "Video data" is digital data that includes visual information captured by a photographing device such as a camera.

[1822] "Image recognition technology" is a technology that uses computer vision technology to identify specific objects or patterns from images or videos.

[1823] "Score display" is a display format that visually shows the score and progress of a game.

[1824] A "scoring scene" is the moment when a goal is scored in a sports game or the scenes before and after that moment.

[1825] "Natural language generation technology" is a technology that enables computers to generate human language, and generates sentences based on specific input data.

[1826] "Speech synthesis technology" is a technology that converts text data into voice data.

[1827] "User" refers to any person or organization that uses this system.

[1828] A "text format" is a format of digital data that contains text information.

[1829] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1830] The present invention is a system that analyzes live video of amateur sports and provides detailed commentary of the progress of the game and scoring scenes in real time. A specific embodiment of this system will be described below.

[1831] Live video capture

[1832] The server acquires live video in real time. Specifically, it uses a high-performance video capture device to capture the video of the match, and transmits the data to the server using a video streaming tool such as FFmpeg. The server temporarily stores the received video data and prepares it for analysis.

[1833] Video data analysis

[1834] The server analyzes the stored video data. Specifically, it uses image recognition libraries such as OpenCV and TENSOFLOW (registered trademark) to identify basketball goal scenes and scorers in the video. For example, the server processes the images frame by frame to track the movement of the basketball. It also recognizes the uniform numbers of the players and identifies the scorers.

[1835] Generate score display

[1836] The server generates a score display for the match based on the analysis results. When a point is scored, the server updates the scoreboard data and displays it to the user in real time. Specifically, the server displays the score on a web page using HTML and JavaScript. It also uses WebSocket to send updates in real time to notify users of score changes.

[1837] Identifying goal-scoring scenes and generating commentary

[1838] When a scoring scene is identified, the server generates a detailed commentary on that scene. This commentary generation uses natural language generation technology such as GPT-3 (registered trademark). Specifically, information such as the scorer's name, the scoring method, and the game situation is input into the generation AI model as a prompt sentence to generate a detailed commentary. For example, the server generates the following prompt sentence:

[1839] "With 2 minutes and 30 seconds left in the third quarter, 23 players made a three-point shot. With this score, Team A took a five-point lead over Team B."

[1840] The generative AI model generates a detailed explanation based on this prompt.

[1841] Providing commentary

[1842] The server provides the generated commentary to the user in the form of voice or text. For voice synthesis, Google (registered trademark) Text-to-Speech API or the like is used, and the generated audio file is streamed to the user. The text commentary is displayed on a web page or application. For example, the server embeds the generated commentary in a web page in HTML format, allowing the user to view it in real time.

[1843] In this way, users can follow the progress of the game and get detailed commentary on the scoring scenes in real time. This system is an effective means of improving the viewing experience of amateur sports.

[1844] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1845] Step 1:

[1846] Live video capture

[1847] The server acquires live video in real time. Specifically, it uses a high-performance video capture device to capture the video of the match, and sends the data to the server using a video streaming tool such as FFmpeg. The input is the live video, and the output is the video data stored on the server. The server temporarily stores the received video data and prepares it for analysis.

[1848] Step 2:

[1849] Video data analysis

[1850] The server analyzes the stored video data. Specifically, it uses image recognition libraries such as OpenCV and TENSOFLOW (registered trademark) to identify basketball goal scenes and scorers in the video. The input is the stored video data, and the output is the analysis results (information on the scoring scene and scorer). The server processes the images frame by frame and tracks the movement of the basketball. It also recognizes the players' uniform numbers and identifies the scorers.

[1851] Step 3:

[1852] Generate score display

[1853] The server generates a display of the match score based on the analysis results. The input is the analysis results, and the output is the updated scoreboard data. When a point is scored, the server updates the scoreboard data and displays it to the user in real time. Specifically, it displays the scores on a webpage using HTML and JavaScript. It also uses WebSockets to send updates in real time to notify users of score changes.

[1854] Step 4:

[1855] Identifying goal-scoring scenes and generating commentary

[1856] When a scoring scene is identified, the server generates a detailed commentary on that scene. The input is the information on the scoring scene, and the output is the generated commentary. This commentary generation uses natural language generation technology such as GPT-3 (registered trademark). Specifically, information such as the scorer's name, the scoring method, and the situation of the game is input into the generation AI model as a prompt sentence to generate a detailed commentary. For example, the server generates a prompt sentence like the following:

[1857] "With 2 minutes and 30 seconds left in the third quarter, 23 players made a three-point shot. With this score, Team A took a five-point lead over Team B."

[1858] The generative AI model generates a detailed explanation based on this prompt.

[1859] Step 5:

[1860] Providing commentary

[1861] The server provides the generated commentary to the user in the form of voice or text. The input is the generated commentary, and the output is an audio file and text data. For voice synthesis, Google (registered trademark) Text-to-Speech API or the like is used, and the generated audio file is streamed to the user. The text commentary is displayed on a web page or application. For example, the server embeds the generated commentary in a web page in HTML format, allowing the user to view it in real time.

[1862] (Application example 2)

[1863] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1864] In conventional live video distribution systems for amateur sports, the display of game scores and the identification of scoring scenes are often done manually, which causes problems such as a lack of real-timeness and accuracy. In addition, it is difficult to provide detailed commentary on scoring scenes, which makes it difficult to increase viewer satisfaction. This has led to the issue that the viewing experience of amateur sports is inferior to that of professional sports.

[1865] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1866] In this invention, the server includes a means for analyzing live video of amateur sports, a means for generating a score display based on the analysis result, a means for identifying a scoring scene based on the analysis result, a means for generating a commentary in voice or text for the identified scoring scene, a means for streaming the live video, and a means for generating a detailed commentary of the scoring scene using a voice synthesis technique. This makes it possible to display an accurate score and identify a scoring scene in real time, and further provide a detailed commentary in voice or text, thereby improving the viewing experience of the viewer.

[1867] "Amateur sports" are sports competitions played by non-professional individuals or teams.

[1868] "Live video" is video content that is filmed and distributed in real time.

[1869] "Analysis" is the process of examining data in detail and extracting specific information.

[1870] The "score display" is a display that visually shows the score and progress of the game.

[1871] A "scoring scene" is a video of the moment when a goal is scored in a sporting event.

[1872] "Audio and text commentary" refers to providing audio or text explanations or comments for a video.

[1873] "Streaming" is a technology for delivering video and audio in real time over the Internet.

[1874] "Speech synthesis technology" is a technology that converts text data into speech.

[1875] "Image recognition technology" is a technology that extracts and analyzes specific information from image data.

[1876] "Natural language generation technology" is a technology that allows computers to generate human language.

[1877] The system for carrying out the present invention analyzes live video of amateur sports, displays the score, identifies the scoring scenes, and provides detailed commentary in real time. Specific embodiments are described below.

[1878] Hardware and Software Configuration

[1879] The server uses the following hardware and software:

[1880] Hardware: Servers with powerful processors, memory, storage, and network interfaces.

[1881] software:

[1882] OpenCV: Image processing library

[1883] TENSOFLOW(registered trademark) / Keras: Framework for image recognition models

[1884] Transformers (Hugging Face): A Framework for Natural Language Generation Models

[1885] pyttsx3: A speech synthesis library

[1886] Data processing and calculation

[1887] 1. Live video streaming: The server receives and streams live video in real time over the Internet.

[1888] 2. Frame capture and pre-processing: The server captures frames from the live video and performs pre-processing such as resizing and normalization using OpenCV.

[1889] 3. Image Recognition: Analyze the pre-processed frames using TENSOFLOW® / Keras to identify scoring scenes and players.

[1890] 4. Identifying scoring scenes: Based on the analysis results, Identify scoring scenes.

[1891] 5. Natural Language Generation: When a scoring scene is identified, we use Transformers to generate a detailed commentary. An example prompt is "Tanaka scored a point in a basketball game. Please generate a detailed commentary."

[1892] 6. Speech synthesis: The generated explanation is converted into speech using pyttsx3 and provided to the user.

[1893] Examples

[1894] For example, if a scene such as "Tanaka makes a three-point shot!" is detected during a game, the server will perform the following process.

[1895] 1. Capture and pre-process frames from live video.

[1896] 2. Use an image recognition model to identify scoring scenes.

[1897] 3. If a scoring scene is identified, generate an explanation using a natural language generation model using the prompt sentence, "Tanaka scored a point in the basketball game. Please generate a detailed explanation."

[1898] 4. The generated commentary is converted into audio using speech synthesis technology and played back, for example, "Tanaka made a great three-point shot, widening the team's lead."

[1899] In this way, accurate scores can be displayed in real time, scoring scenes can be identified, and detailed commentary can be provided in audio and text, improving the viewer's viewing experience.

[1900] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1901] Step 1:

[1902] The server receives live video over the Internet and performs streaming. The input is the live video stream data, and the output is frames captured in real time. Specifically, the server splits the video stream and processes each frame.

[1903] Step 2:

[1904] The server preprocesses the captured frames using OpenCV. The input is the captured frame and the output is the preprocessed frame. Specific operations include resizing the frame (e.g. to 224x224 pixels) and normalizing it (scaling pixel values ​​to the range 0 to 1).

[1905] Step 3:

[1906] The server analyzes the preprocessed frames using TENSOFLOW (registered trademark) / Keras to identify scoring scenes and players. The input is the preprocessed frames, and the output is the analysis results (whether or not a scoring scene occurred and the identification of the player). In concrete terms, the server inputs the frames into an image recognition model and obtains the probability of a scoring scene and the identification of the player.

[1907] Step 4:

[1908] The server identifies a scoring scene based on the analysis result. The input is the analysis result, and the output is the result of identifying the scoring scene. In concrete terms, if the probability of a scoring scene exceeds a certain threshold, the server identifies the scene as a scoring scene.

[1909] Step 5:

[1910] When a scoring scene is identified, the server generates a prompt sentence and generates a detailed commentary using Transformers. The input is the result of identifying the scoring scene and the prompt sentence, and the output is the generated commentary text. In concrete terms, the prompt sentence "A player scored a point in a basketball game. Please generate a detailed commentary" is input into the natural language generation model, and commentary text is generated.

[1911] Step 6:

[1912] The server converts the generated commentary text into audio using pyttsx3 and provides it to the user. The input is the generated commentary text, and the output is audio data. In concrete terms, the commentary text is input to a speech synthesis engine, audio data is generated, and the audio data is played through a speaker.

[1913] In this way, the server can analyze live video in real time, identify scoring moments and generate detailed commentary to serve to viewers.

[1914] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1915] "Example 1"

[1916] In one embodiment of the present invention, a system for analyzing live amateur sports videos, generating score displays based on the analysis results, identifying scoring scenes, and generating commentary in audio or text for the identified scoring scenes is combined with an emotion engine that recognizes the emotions of a user. The emotion engine recognizes emotions from the user's facial expressions and tone of voice, and adjusts the generation of commentary based on the recognized emotions. For example, if the user shows a happy expression, the emotion engine feeds that information back to the analysis means, and the commentary generation means generates a more exciting commentary. Conversely, if the user shows a disappointed expression, the commentary generation means generates a more calm tone of commentary. This provides a personalized experience according to the user's emotions.

[1917] "Example 2"

[1918] In one embodiment of the present invention, a system for analyzing live amateur sports v...

Claims

1. A means for acquiring live video data; means for pre-processing the acquired video data; A means for analyzing the preprocessed video data using image recognition technology; A means for identifying a goal-scoring scene based on the analysis result; A means for generating a prompt sentence for generating an explanation for the identified scoring scene; A means for inputting the generated prompt sentence into a generative AI model to generate an explanation; The system includes means for outputting the generated description in audio and / or text.

2. A means for acquiring live video; A means for analyzing the acquired video data using image recognition technology; means for generating a score display based on the analysis results; A means for identifying a goal-scoring scene based on the analysis result; A means for generating a commentary in the form of at least one of voice and text using a natural language generation technique for the identified scoring scene; A means for converting the generated commentary into an audio file using a voice synthesis technology and providing the audio file; a means for displaying the generated commentary in text form; A system including:

3. A means to analyze live video; means for generating a score display based on the analysis results; A means for identifying a goal-scoring scene based on the analysis result; A means for generating commentary on the identified scoring scene using at least one of voice and text; A means for recognizing a user's emotion; A means to adjust the tone of commentary based on perceived emotions; and A system including:

Images