System

Abstract

A system is provided.SOLUTION: A system comprising: a server means for collecting training data; a server means for pre-processing the training data to train a machine-learning model; a server means for receiving a request and analyzing a video of a sporting event; a server means for generating a commentary text based on an analysis result; a server means for converting the commentary text into a narration voice by a speech synthesis engine; a server means for transmitting the narration voice to a user device; and a device means for synchronously playing the video and the narration voice on the user device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, live commentary for sporting events such as car races, horse races, and drone races is provided manually by experts, which is expensive and difficult to implement for sports other than major sports. This makes it difficult to understand the situation when watching in silence, and it is difficult to attract viewers' attention. Furthermore, due to a lack of commentators with specialized knowledge, live commentary is not provided for many sporting events. Given this background, there is a need for a low-cost, automated method to provide professional, real-time commentary. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means. First, a server means is provided that collects large datasets related to sporting events such as car races, horse races, and drone races from the Internet. Next, a server means is provided that preprocesses the collected data and trains a machine learning model. This allows video of a specific sporting event to be analyzed and important events and actions to be detected in real time. The present invention also includes a server means that automatically generates commentary text based on the detected information and converts it into narration using a speech synthesis engine. The generated narration audio is also transmitted to a user's device, and a function is provided for synchronous playback of the video and narration audio on the device. In this way, a low-cost automatic commentary system that does not require experts is realized, making it applicable to a wide range of sporting events.

[0006] The "server means" is a computer system that receives requests from users, collects learning data, preprocesses it, trains machine learning models, analyzes video, generates live text, and generates and transmits narration audio.

[0007] The "terminal means" is a device that allows a user to send a request to a server and play back the narration audio sent from the server in synchronization with the video.

[0008] A "user" is a spectator or viewer who requests a live commentary of a sporting event and uses a terminal means to view the video with the commentary.

[0009] "Learning data" refers to video footage, commentary, or other related data about past sporting events collected from the Internet.

[0010] "Preprocessing" refers to the process of converting collected training data into a format suitable for machine learning models, such as labeling video frames and tokenizing text.

[0011] A "machine learning model" is a set of algorithms that are trained using collected training data to analyze video footage of sporting events and generate commentary. For example, a Transformer model is used.

[0012] "Video analysis" is the process of analyzing user-provided footage of a sporting event frame by frame to recognize important events and actions.

[0013] "Commentary text" is text that describes the real-time commentary of a sporting event, generated based on video analysis.

[0014] A "speech synthesis engine" is software or a system for converting generated commentary text into speech.

[0015] "Narration voice" is data created by a voice synthesis engine that provides commentary in audio format.

[0016] "Sporting events" specifically refer to various competitions and races that attract spectators, such as car races, horse races, and drone races. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] System Overview

[0039] The present invention is a system that utilizes AI to provide automatic live commentary of sporting events. This system has a function that links a server and a terminal and provides users with real-time live commentary as audio. The specific processing of the program of the system of the present invention is explained below in natural language.

[0040] Learning Phase

[0041] server

[0042] 1. Data Collection

[0043] The server collects a large amount of video and live text from sporting events such as car races, horse races, and drone races from the Internet, which are used as large-scale training datasets for video analysis and natural language processing.

[0044] 2. Data Preprocessing

[0045] The server pre-processes the collected data: video data is labeled for each frame with important events and actions, and text data is tokenized and converted into a format that can be easily understood by machine learning models.

[0046] 3. Model training

[0047] The server uses the pre-processed data to train machine learning models, which are built using advanced algorithms such as Transformer models and trained to analyze video footage and generate commentary for sporting events.

[0048] 4. Save the model

[0049] The server stores the trained model and builds a framework for regularly updating it, enabling highly accurate analysis based on the latest information at all times.

[0050] Commentary Phase

[0051] User

[0052] 1. Submit a request

[0053] A user requests a live broadcast of a particular sporting event from their device, which sends a request for the live broadcast to the server.

[0054] Terminal

[0055] 1. Receiving a request

[0056] The device sends the user's request, which includes the live video or streaming link, to the server for analysis.

[0057] 2. Data reception and playback

[0058] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing the user to watch the video with commentary in real time.

[0059] server

[0060] 1. Request Processing

[0061] When the server receives a request from a user, it loads and prepares the appropriate trained model.

[0062] 2. Video Analysis

[0063] The server analyzes the real-time video provided by the user frame by frame. This analysis recognizes important events and actions. For example, in a car race, events such as "the first lap is over" and "vehicle collision" are detected.

[0064] 3. Live Text Generation

[0065] Based on the analysis results, the server uses a natural language generation model to generate live text, which is adjusted to include appropriate phrases and sentence structures depending on the situation.

[0066] 4. Speech Synthesis

[0067] The server sends the generated commentary text to a speech synthesis engine to generate a narration voice, which is tuned for high quality and fluency.

[0068] 5. Narration audio transmission

[0069] The server transmits the generated narration voice data to the user's terminal in real time.

[0070] Example: Car race commentary

[0071] Learning Phase

[0072] The server collects footage and commentary from past F1 races and rallies, preprocesses it, and then trains a machine learning model.

[0073] Commentary Phase

[0074] 1. User Request

[0075] The user requests a live broadcast of the F1 race currently underway from the terminal to the server.

[0076] 2. Server Processing

[0077] The server analyzes the live video data sent by the user and detects events such as "two cars collide" or "first lap completed."

[0078] Based on the detected events, the server generates commentary text such as "The first lap has now ended. Two cars have collided!" and generates narration audio using a speech synthesis engine.

[0079] The server transmits the generated voice to the user's terminal.

[0080] 3. Terminal Processing

[0081] The user's terminal plays back the live commentary audio data sent from the server in synchronization with the video, providing the user with a real-time commentary.

[0082] This system will enable viewers to enjoy sporting events such as car races, horse races, and drone races smoothly without the need for live commentary by experts. By providing specific commentary in real time, viewers can gain a deeper understanding of the action, providing a more immersive viewing experience.

[0083] The processing flow will be explained below.

[0084] Learning Phase

[0085] server

[0086] Step 1:

[0087] The server collects video and commentary data about sporting events from the Internet, including video footage of car races, horse races, drone races, and other events, along with commentary.

[0088] Step 2:

[0089] The server preprocesses the collected data, labeling the video data with important events and actions for each frame, and tokenizing the text data, converting it into a format that can be easily understood by machine learning models.

[0090] Step 3:

[0091] The server uses the pre-processed data to train machine learning models, which use advanced algorithms such as the Transformer architecture to improve video analysis and play-by-play text generation for sporting events.

[0092] Step 4:

[0093] The server stores the trained model and builds a framework to update the model as needed, enabling highly accurate analysis with the latest information.

[0094] Commentary Phase

[0095] User

[0096] Step 1:

[0097] A user uses his / her terminal to request a live broadcast of a particular sporting event from the server, providing a live video or streaming link.

[0098] Terminal

[0099] Step 2:

[0100] The terminal transmits the user's request and the video data to the server.

[0101] server

[0102] Step 3:

[0103] The server receives requests sent by users and loads the relevant trained models.

[0104] Step 4:

[0105] The server analyzes the provided video data frame by frame, detecting specific actions and events (e.g., "first lap completed," "vehicle collision").

[0106] Step 5:

[0107] The server uses a natural language generation model based on the analysis results to instantly generate live commentary text, such as "The first lap has now finished. Two cars have collided!"

[0108] Step 6:

[0109] The server sends the generated commentary text to a speech synthesis engine to create a narration voice that is natural and easy to understand.

[0110] Step 7:

[0111] The server transmits the generated narration voice data to the user's terminal in real time.

[0112] Terminal

[0113] Step 8:

[0114] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing users to enjoy the commentary in real time along with the video.

[0115] Example: Car race commentary

[0116] Learning Phase

[0117] Step 1:

[0118] The server collects footage and live commentary from past F1 races and rallies.

[0119] Step 2:

[0120] The server preprocesses the collected data, labels the video frames, and tokenizes the text.

[0121] Step 3:

[0122] The server trains a Transformer model with the preprocessed data, giving it high-precision video analysis and commentary generation capabilities.

[0123] Step 4:

[0124] The server stores the trained model and periodically updates it.

[0125] Commentary Phase

[0126] Step 1:

[0127] The user requests a live broadcast of the F1 race currently underway from the server via the terminal.

[0128] Step 2:

[0129] The terminal transmits the user's request and the live video data to the server.

[0130] Step 3:

[0131] The server receives the request and loads a model suitable for F1 racing.

[0132] Step 4:

[0133] The server analyzes the video and detects events such as "first lap completed" or "vehicle collision."

[0134] Step 5:

[0135] The server generates live text: "First lap now completed. Two cars have collided!"

[0136] Step 6:

[0137] The server sends the commentary text to a speech synthesis engine to generate the narration voice.

[0138] Step 7:

[0139] The server transmits the narration voice to the user's terminal.

[0140] Step 8:

[0141] The terminal plays back the narration audio in synchronization with the video, allowing the user to view the video with commentary in real time.

[0142] Example 1

[0143] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0144] Conventional live commentary of sporting events relied on human commentators, which had issues with real-timeness and accuracy. It was also difficult to secure commentators with specialized knowledge. Furthermore, there was a need to provide a realistic live commentary experience for users who could not attend the event in person.

[0145] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0146] In this invention, the server includes: means for collecting large amounts of video and commentary text related to sporting events such as car races, horse races, and drone races from the Internet; means for preprocessing the collected data, labeling important events and actions in each frame of the video data, and tokenizing the text data; means for training a machine learning model using the preprocessed data; means for periodically updating and saving the trained machine learning model; means for receiving requests from users and analyzing real-time sporting event video; means for generating commentary text based on the analysis results; means for converting the generated commentary text into narration using a speech synthesis engine; means for transmitting the narration text to a user's device; and means for synchronously playing back the video and narration text on the user's device. This enables automatic commentary with high real-time accuracy. This allows users without specialized knowledge to enjoy a realistic sports viewing experience, eliminating the need for a commentator.

[0147] "On the Internet" refers to a globally connected information and communications network and infrastructure for sending and receiving data.

[0148] "Car racing" is a competitive event involving automobiles traveling at high speeds, and is a sport in which multiple vehicles compete against each other for ranking.

[0149] "Horse racing" is a competitive event using horses, and is a sport in which multiple horses compete for rankings.

[0150] "Drone racing" is a competitive event using unmanned aerial vehicles (drones), and is a sport in which drones compete for rankings by driving around a course while avoiding obstacles.

[0151] "Footage" means real-time or recorded visual data of a sporting event provided in the form of video.

[0152] "Live commentary text" is a document that verbally explains the progress of a sporting event and important events, and is character data that serves as material for speech synthesis.

[0153] "Server means" refers to a computer system in general that has the function of managing and processing data via a network.

[0154] "Collecting" is the process of obtaining and storing data from the internet or other sources.

[0155] "Preprocessing" refers to the process of converting collected data into a form that is easier to analyze, and specifically includes labeling video frames and tokenizing text data.

[0156] A "machine learning model" is a set of algorithms or programs that can learn from collected data and perform specific tasks automatically.

[0157] A "Transformer model" is a deep learning model that demonstrates excellent performance in natural language processing and machine translation, and has the ability to generate output while taking into account the context of the input data.

[0158] "Analyzing" is the process of analyzing collected or received video or data to detect significant events or actions.

[0159] "Generating" refers to the process of creating new data or information based on the analysis results, and in this case refers to the creation of commentary text.

[0160] A "speech synthesis engine" is a set of software or hardware for converting text data into speech data.

[0161] "Narration voice" is voice data that sounds like a human being speaking, generated by a voice synthesis engine.

[0162] "Terminal means" refers to all electronic devices that are directly operated by users and used to input and output data.

[0163] MODE FOR CARRYING OUT THE INVENTION

[0164] The present invention is a system that uses AI to provide automatic commentary of sporting events. This system has a function that links a server and a terminal and provides users with real-time commentary as audio. Specific embodiments for carrying out the invention are described below.

[0165] server

[0166] The server collects a large amount of video and live text about sporting events from the Internet, using various sources such as YouTube and official sporting sites, and stores the collected data in a large database.

[0167] Next, we preprocess the collected data: for video data, we use a visual information analysis library (e.g., OpenCV) to label important events and actions for each frame, and for text data, we use a natural language processing (NLP) library (e.g., NLTK or SpaCy) to tokenize it.

[0168] Once the preprocessing is complete, the data is used to train a machine learning model. In this invention, advanced algorithms such as the Transformer model are used. This model is trained using machine learning frameworks such as TensorFlow and Scikit-learn. The trained model is periodically retrained with new data and updated to ensure the model's accuracy.

[0169] When a live video request is received from a user, the server loads the appropriate pre-trained model and analyzes the video in real time. Based on the analysis results, a natural language generation model is used to generate live commentary text. This text is then passed to a speech synthesis engine, which converts it into fluent narration audio in real time. For example, a text-to-speech (TTS) engine such as Amazon Polly is used for speech synthesis.

[0170] The narration audio is transmitted to the user's device in real time using protocols such as HTTP Live Streaming (HLS).

[0171] Terminal

[0172] The user's device receives the narration audio data sent from the server and plays it in sync with the live video. The user then watches the video with commentary in real time via a dedicated application or web browser.

[0173] User

[0174] Users request live coverage of a particular sporting event from their device. For example, if they want live coverage of a current F1 race, they click a request button in the application, enter the URL of the live video in the pop-up window, and send it to the server.

[0175] Specific examples

[0176] An example of such a prompt sentence is, "Please provide a live commentary of the F1 race currently being broadcast. Please give a detailed description of the events that occur from the start to the pit stop."

[0177] This system allows viewers to enjoy sporting events such as car races, horse races, and drone races smoothly without the need for live commentary by experts. By providing accurate commentary in real time, viewers can gain a deeper understanding of the action and enjoy a more immersive viewing experience.

[0178] keyword

[0179] Generative AI Models

[0180] Prompt statement

[0181] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0182] Step 1: Data collection

[0183] The server collects video and live commentary of sporting events such as car races, horse races, and drone races from the Internet. The input is a list of URLs on the Internet, and the output is the collected video and text data. Specifically, the server automatically downloads the video and text data from the specified URL list and stores them in a database.

[0184] Step 2: Data Preprocessing

[0185] The server preprocesses the collected data. For video data, it uses a visual information analysis library such as OpenCV to label important events and actions for each frame. For text data, it uses a natural language processing library such as NLTK or SpaCy to tokenize it. The input is the collected video and text data, and the output is preprocessed video frames and text tokens.

[0186] Step 3: Model training

[0187] The server uses the preprocessed data to train a machine learning model. Specifically, it uses a machine learning framework such as TensorFlow or Scikit-learn to train a Transformer model. The input is the preprocessed dataset, and the output is a trained machine learning model. The server uses video and text pairs to batch process the model. It updates the model parameters every epoch and minimizes the loss function.

[0188] Step 4: Save and update the model

[0189] The server saves the trained model in a database and periodically retrains it with new data. The input is the trained model and a new dataset, and the output is the updated model. Specifically, the server saves the parameters of the trained model and schedules periodic retraining with new data.

[0190] Step 5: Receiving the request

[0191] A user requests a live broadcast of a specific sporting event from their device. The input is the URL of the live video specified by the user, and the output is a request sent to the server. The user clicks the request button in the app, enters the URL in the pop-up window, and submits it.

[0192] Step 6: Request Processing

[0193] When the server receives a request from a user, it loads the appropriate trained model. The input is the user request and the URL of the live video, and the output is the loaded trained model. Specifically, the server analyzes the request and determines which model to load.

[0194] Step 7: Video Analysis

[0195] The server analyzes real-time video provided by the user frame by frame. The input is the live video stream, and the output is the analyzed event and action data. Specifically, the server receives the video stream and analyzes each frame using OpenCV.

[0196] Step 8: Live text generation

[0197] The server generates commentary text using a natural language generation model based on the analysis results. The input is the analyzed event data, and the output is the generated commentary text. Specifically, the server passes the analysis results as input to the natural language generation model to generate commentary text.

[0198] Step 9: Text-to-Speech

[0199] The server sends the generated commentary text to a speech synthesis engine to generate a narration voice. The input is the commentary text, and the output is the synthesized narration voice. Specifically, the server sends the generated text to the speech synthesis engine and converts it into voice data.

[0200] Step 10: Send narration audio

[0201] The server transmits the generated narration audio data to the user's terminal. The input is the narration audio, and the output is the transmission of audio data to the user's terminal. Specifically, the server streams the audio file to the terminal.

[0202] Step 11: Data reception and playback

[0203] The device receives the narration audio data sent from the server and plays it in sync with the live video. The input is the narration audio data and the live video, and the output is the synchronized playback of the live video. Specifically, the device receives an HTTP request and launches a player to play the streaming audio data.

[0204] (Application example 1)

[0205] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0206] Modern factories require real-time monitoring of the progress and abnormalities of production lines and robots. However, it is difficult for on-site workers to constantly monitor the situation, and they are unable to respond immediately when an abnormality occurs. Furthermore, visual information alone does not provide a sufficient understanding of the details of the production process, making it difficult to improve efficiency and control quality. To address these issues, the present invention aims to provide a system that provides real-time audio commentary on the progress and abnormalities of production lines and robots.

[0207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0208] In this invention, the server includes means for collecting training data, means for preprocessing the training data and training a machine learning model, means for accepting requests and analyzing video of a sporting event or a production line, means for generating commentary text based on the analysis results, means for converting the commentary text into narration audio using a speech synthesis engine, means for transmitting the narration audio to a user's terminal, means for synchronously playing back the video and narration audio on the user's terminal, and means for providing real-time commentary on the progress and abnormalities of the production line or robot. This enables real-time audio commentary on the status of the manufacturing process in a factory, allowing workers to perform their work efficiently.

[0209] The "server means" is a computer system that controls the functions of data collection, preprocessing, analysis, generation, synthesis, and transmission, and communicates with the user's terminal.

[0210] "Training Data" is a data set collected by the Server Means and used to train a Machine Learning Model.

[0211] "Preprocessing" is the process of converting collected training data into a format that is easy for machine learning models to understand, using techniques such as tokenization and labeling.

[0212] A "machine learning model" is an algorithm, particularly a Transformer model, that is trained on training data to perform a specific task.

[0213] A "request" is a communication signal that a user uses from his / her own terminal to request a live report of a specific event from the server means.

[0214] "Video analytics" is the process of analyzing real-time video data to detect important events or actions.

[0215] "Live text" is explanatory text generated based on the results of video analysis, which is converted into voice by a voice synthesis engine.

[0216] A "speech synthesis engine" is software or hardware for converting text data into narration voice.

[0217] The "terminal means" is a device used by a user, and has the function of playing back the narration audio and video transmitted from the server means in synchronization.

[0218] A "production line" is a system that shows the machines and work flow arranged to carry out production activities within a factory.

[0219] A "robot" is a mechanical device that performs tasks automatically or semi-automatically in a factory.

[0220] "Real-time" means that processing occurs immediately, with virtually no delay.

[0221] "Audio commentary" is the process of communicating analyzed information to the user in real time by voice.

[0222] "Progress" refers to the state and speed at which a production line or robot is working.

[0223] An "abnormality" refers to a state that deviates from normal operation, or a malfunction such as an error or breakdown.

[0224] The present invention provides a system that provides real-time audio commentary on the progress and abnormalities of manufacturing lines and robots. This system consists of three main elements: a server, a terminal, and a user.

[0225] server

[0226] 1. Data Collection and Learning

[0227] The server collects actual video and text data about manufacturing lines and robots collected from the Internet. This data is used as learning data to train machine learning models. Specific hardware used is a high-performance server.

[0228] 2. Data Preprocessing

[0229] The collected data is preprocessed by the server. For example, video data is broken down into frames and labeled. Text data is tokenized and converted into a format that is easy for machine learning models to understand. Open source software such as TensorFlow and OpenCV is used.

[0230] 3. Model training

[0231] Using the pre-processed data, the server trains machine learning models, built using advanced algorithms such as the Transformer model, capable of analyzing video footage of production lines and robots and generating live text.

[0232] 4. Model application and analysis

[0233] The server analyzes the video of the production line based on user requests. Video analysis includes real-time event detection. For example, it detects actions such as "the production line has stopped" or "a robot has encountered an error." The software used is TensorFlow and OpenCV.

[0234] 5. Live Text Generation and Speech Synthesis

[0235] Based on the analysis results, the server generates commentary text using a natural language generation model (such as GPT-3), which is then converted into narration audio by a speech synthesis engine (Amazon Polly, Google Text-to-Speech).

[0236] 6. Sending data to the user device

[0237] Once the narration voice data is generated, the server transmits it to the user's terminal in real time.

[0238] Terminal

[0239] 1. Data reception and synchronous playback

[0240] The user's device (such as smart glasses) receives the narration audio data sent from the server and plays it in sync with the video. This function allows the user to understand the status of the production line and robots in real time with audio commentary.

[0241] User

[0242] 1. Submit a request

[0243] Users can request live footage of a specific manufacturing line or robot from a terminal, which can include live video or streaming links.

[0244] Specific examples

[0245] For example, if a user sends a request from a terminal about the progress of a production line, the server generates a prompt sentence such as the following:

[0246] Example prompt sentence:

[0247] "The production line has stopped. What is the next action?"

[0248] In this way, by linking the server, terminal, and user, it becomes possible to provide real-time audio updates on the progress and abnormalities of the production line and robots, allowing workers to carry out their work efficiently.

[0249] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0250] Step 1: Data collection

[0251] The server collects large amounts of video and text data related to manufacturing lines and robots from the Internet. This data is used as a training dataset. The input is raw data related to the manufacturing process, and the output is the collected dataset. Specifically, the data is downloaded using a web scraping tool or API.

[0252] Step 2: Data Preprocessing

[0253] The server preprocesses the collected data. Specifically, the video data is decomposed into frames and important events are labeled. The text data is tokenized and formatted. The input is the collected video data and text data, and the output is the preprocessed data. Specifically, frame decomposition is performed using OpenCV and labeling is performed using TensorFlow.

[0254] Step 3: Model training

[0255] The server uses the preprocessed data to train a machine learning model. This model can then analyze video of production lines and robots and generate live text. The specific method used is a Transformer model. The input is the preprocessed data, and the output is the trained model. The specific operation is to train the Transformer model using the TensorFlow API.

[0256] Step 4: Submitting the request

[0257] A user requests a live report of a specific manufacturing line or robot from a terminal to a server. This request may include live video or a streaming link. The input is the user's request, and the output is a communication signal to the server. Specifically, the request is sent from the terminal's UI to the server.

[0258] Step 5: Video Analysis

[0259] The server analyzes the real-time video provided by the user frame by frame. This analysis recognizes important events and actions. For example, events such as "the production line has stopped" or "an error has occurred in a robot" can be detected. The input is the live video data, and the output is the analysis results. Specifically, the video analysis is performed using TensorFlow.

[0260] Step 6: Generate commentary text

[0261] The server generates commentary text using a natural language generation model based on the analysis results. The generated text has appropriate phrases and sentence structures depending on the situation. The input is the analysis results, and the output is the commentary text. Specifically, text generation is performed using GPT-3.

[0262] Step 7: Text-to-Speech

[0263] The server sends the generated commentary text to a speech synthesis engine to generate narration audio. This audio is high quality and fluent. The input is the commentary text, and the output is the narration audio data. Specifically, speech synthesis is performed using the Google Text-to-Speech API.

[0264] Step 8: Send narration audio

[0265] The server transmits the generated narration audio data to the user's device in real time. The input is the narration audio data, and the output is the audio data sent to the user's device. Specifically, the audio data is transmitted using a network protocol (e.g., WebSocket).

[0266] Step 9: Data reception and synchronous playback

[0267] The user's device receives the narration audio data sent from the server and plays it back in sync with the video. This function allows the user to understand the status of the production line or robot in real time with audio commentary. The input is the audio data sent from the server, and the output is synchronous playback of the video and audio. Specifically, the device's audio playback function is used to synchronize with the video data.

[0268] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0269] System Overview

[0270] The present invention is a system that utilizes AI to provide automatic commentary of sporting events. In addition, by combining it with an emotion engine that recognizes the user's emotions, it provides adaptive commentary that corresponds to the user's emotions. The server and terminals work together to provide commentary in the form of audio in real time. The specific processing of the program of the system of the present invention is explained below in natural language.

[0271] Learning Phase

[0272] server

[0273] 1. Data Collection

[0274] The server collects a large amount of video and live text from sporting events such as car races, horse races, and drone races from the Internet, which are used as large-scale training datasets for video analysis and natural language processing.

[0275] 2. Data Preprocessing

[0276] The server preprocesses the collected data: it labels important events and actions for each frame of video data, tokenizes text data and converts it into a format that machine learning models can easily understand, and collects and processes audio and facial expression data for the emotion engine.

[0277] 3. Model training

[0278] The server uses the pre-processed data to train machine learning models, which use advanced algorithms such as Transformer models, and train them to perform video analysis, live text generation, and emotion recognition.

[0279] 4. Save the model

[0280] The server will store the trained model and create a framework for periodic updates, enabling highly accurate analysis using the latest information.

[0281] Commentary Phase

[0282] User

[0283] 1. Submit a request

[0284] Users can use their devices to request live coverage of a particular sporting event from the server, providing live video or a streaming link, and voice and facial expression data is also collected to recognize the user's emotions.

[0285] Terminal

[0286] 1. Receiving a request

[0287] The terminal transmits the user's request, video data, and emotion data to the server.

[0288] server

[0289] 1. Request Processing

[0290] The server receives requests from users and loads and prepares the relevant trained models.

[0291] 2. Video Analysis

[0292] The server analyzes the provided video data frame by frame. This analysis allows it to recognize important events and actions. For example, in the case of a car race, events such as "the first lap is over" and "vehicle collision" can be detected.

[0293] 3. Emotion recognition

[0294] The server analyzes the voice and facial expression data sent by the user and uses an emotion engine to recognize the user's emotional state, including the user's tone of voice, facial expressions, and body movements.

[0295] 4. Live Text Generation

[0296] The server generates commentary text using a natural language generation model based on the video analysis results. The content and tone of the commentary text are adjusted according to the user's emotional state as recognized by the emotion engine. For example, if the user is excited, the server generates text that maintains the same sense of excitement.

[0297] 5. Speech Synthesis

[0298] The server sends the generated commentary text to a speech synthesis engine to create a narration voice, which is natural and easy to understand, and is provided in a tone that matches the user's emotions.

[0299] 6. Narration audio transmission

[0300] The server transmits the generated narration voice data to the user's terminal in real time.

[0301] Terminal

[0302] 1. Data reception and playback

[0303] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing users to enjoy real-time commentary based on their emotions along with the video.

[0304] Example: Car race commentary

[0305] Learning Phase

[0306] 1. Data Collection and Preprocessing

[0307] The server collects and preprocesses video footage from past F1 races and rallies, commentary text, and user emotion data, labeling video frames, tokenizing text, and processing audio and facial expression data for the emotion engine.

[0308] 2. Model training

[0309] The server trains a Transformer model on the preprocessed data, giving it high-precision video analysis, commentary generation, and emotion recognition capabilities.

[0310] 3. Save the model

[0311] The server stores the trained model and periodically updates it.

[0312] Commentary Phase

[0313] 1. User Request

[0314] The user requests a live report of the current F1 race from the server via their device, and data based on the user's emotional state is also collected at the same time.

[0315] 2. Sending and Receiving Devices

[0316] The device transmits the user's request, live video data, and emotion data to the server, and receives the live audio data sent from the server and plays it in sync with the video.

[0317] 3. Server Processing

[0318] The server receives the request, analyzes the video, and the emotion engine recognizes the emotional state. Events such as "First lap completed" and "Vehicle collision" are detected, and based on the user's emotional state, a live commentary text is generated, such as "The first lap has now completed. This is exciting! Two cars have collided!"

[0319] The server sends the commentary text to a speech synthesis engine to generate a narration voice that maintains an uplifting tone.

[0320] The server transmits the generated narration voice to the user's terminal.

[0321] 4. Playback of the device

[0322] The user's terminal plays back the narration audio in synchronization with the video, allowing the user to view a realistic commentary in real time.

[0323] This system allows viewers to watch sporting events in a way that adapts to their emotions, enabling real-time, adaptive commentary without the need for expert commentary, improving the sense of realism and entertainment, and making it easier to attract viewers' attention.

[0324] The processing flow will be explained below.

[0325] Learning Phase

[0326] server

[0327] Step 1:

[0328] The server collects videos, commentary text data, and user emotion data related to sporting events from the Internet. This includes videos of car races, horse races, drone races, and other events, along with commentary text and user voice and facial expression data.

[0329] Step 2:

[0330] The server preprocesses the collected data: it labels important events and actions for each frame of video data, tokenizes text data and converts it into a format that machine learning models can easily understand, and adds emotion labels from audio and facial expression data for the emotion engine.

[0331] Step 3:

[0332] The server uses the pre-processed data to train machine learning models, which are trained using advanced algorithms such as Transformer models to analyze video footage of sporting events, generate commentary, and recognize emotions.

[0333] Step 4:

[0334] The server stores the trained model and builds a framework to periodically update the model as needed, enabling highly accurate analysis using the latest information.

[0335] Commentary Phase

[0336] User

[0337] Step 1:

[0338] Users use their devices to request a live broadcast of a particular sporting event from the server, providing a live video or streaming link, and voice and facial expression data is also collected for emotion recognition.

[0339] Terminal

[0340] Step 2:

[0341] The terminal transmits the user's request, live video data, and emotion data to the server.

[0342] server

[0343] Step 3:

[0344] The server receives requests sent by users and loads the relevant trained models.

[0345] Step 4:

[0346] The server analyzes the received video data frame by frame to detect important events and actions (e.g., "first lap completed" or "vehicle collision").

[0347] Step 5:

[0348] The server analyzes the voice and facial expression data sent by the user and recognizes the user's emotional state (e.g., excitement, surprise) using an emotion engine.

[0349] Step 6:

[0350] The server generates commentary text using a natural language generation model based on the video analysis and emotion recognition results, and adjusts the content and tone of the commentary text to reflect the user's emotions.

[0351] Step 7:

[0352] The server sends the generated commentary text to a speech synthesis engine to create a narration voice, which is provided in a tone that matches the user's emotions.

[0353] Step 8:

[0354] The server transmits the generated narration voice data to the user's terminal in real time.

[0355] Terminal

[0356] Step 9:

[0357] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing users to enjoy real-time commentary based on their emotions along with the video.

[0358] Example: Car race commentary

[0359] Learning Phase

[0360] Step 1:

[0361] The server collects footage of past F1 races and rally competitions, commentary text, and user emotion data.

[0362] Step 2:

[0363] The server preprocesses the collected data, labels video frames, tokenizes text, and assigns emotion labels to audio and facial expression data for the emotion engine.

[0364] Step 3:

[0365] The server trains a Transformer model on the preprocessed data, giving it high-precision video analysis, commentary generation, and emotion recognition capabilities.

[0366] Step 4:

[0367] The server stores the trained model and periodically updates it.

[0368] Commentary Phase

[0369] Step 1:

[0370] The user requests a live report of the current F1 race from the server via their device, and data based on the user's emotional state is also collected at the same time.

[0371] Step 2:

[0372] The terminal transmits the user's request, live video data, and emotion data to the server.

[0373] Step 3:

[0374] The server receives the request and loads a model suitable for F1 racing.

[0375] Step 4:

[0376] The server analyzes the video and detects events such as "first lap completed" or "vehicle collision."

[0377] Step 5:

[0378] The server analyzes the user's voice and facial expression data, and the emotion engine recognizes the user's emotional state (e.g., the user is excited).

[0379] Step 6:

[0380] Based on the video analysis and emotion recognition results, the server generates live text such as, "The first lap is now complete. This is very exciting! Two cars have collided!"

[0381] Step 7:

[0382] The server sends the commentary text to a speech synthesis engine to generate a narration voice with a tone that reflects excitement.

[0383] Step 8:

[0384] The server transmits the generated narration voice to the user's terminal.

[0385] Step 9:

[0386] The terminal plays back the narration audio in synchronization with the video, allowing the user to view a realistic commentary in real time.

[0387] Example 2

[0388] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0389] In recent years, there has been a demand for real-time, realistic commentary when watching sporting events. However, professional commentators are not always available, so even general users lack the means to enjoy high-quality commentary. Furthermore, when it comes to realistic commentary that changes in real time based on the user's emotions, conventional systems have difficulty in handling this. Therefore, there is a need for a system that can adapt to the user's emotional state and provide high-quality commentary in real time.

[0390] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0391] In this invention, the server includes server means for collecting training data, server means for preprocessing the training data to train a machine learning model, server means for accepting requests and analyzing video of a sporting event, server means for generating commentary text based on the analysis results, server means for analyzing user emotion data to recognize the user's emotional state, server means for adjusting the tone of the commentary text based on the recognized emotional state, server means for converting the commentary text into narration using a speech synthesis engine, server means for transmitting the narration to the user's terminal, and terminal means for synchronously playing back the video and narration on the user's terminal, thereby enabling real-time, high-quality commentary that is adapted to the user's emotions.

[0392] "Training data" refers to data collected by the server and used to train machine learning models, including, among other things, footage and commentary about sporting events.

[0393] "Server means" means a server designed to perform a specific function, such as data collection, pre-processing, analysis, model training, text generation, speech synthesis, and emotion recognition.

[0394] "Terminal means" refers to a device used by a user, and has the function of receiving data from a server and playing back video and narration audio in sync.

[0395] A "machine learning model" is a collection of algorithms that can perform specific tasks by learning from data, and this system uses a Transformer model.

[0396] A "Transformer model" is a type of machine learning model with advanced pattern recognition and natural language generation capabilities.

[0397] "Emotion data" is data obtained from the user's voice, facial expressions, etc., and is used to analyze the user's emotional state.

[0398] "Emotional state" refers to the psychological state of the user as judged from their tone of voice, facial expressions, and physical movements.

[0399] A "play-by-play text" is a textual description of the situation at a sporting event in natural language, which is generated in a form that can be read or listened to by a user in real time.

[0400] A "speech synthesis engine" is a software or hardware system for converting text data into speech data.

[0401] "Narration voice" is a voice synthesized based on the generated commentary text, and is used to convey information to the user auditorily.

[0402] "Real-time" means that processing occurs and results are provided immediately, without delay.

[0403] MODE FOR CARRYING OUT THE INVENTION

[0404] The present invention is a system that uses AI to provide automatic commentary of sporting events. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides adaptive commentary that corresponds to the user's emotions. This system has a function that links a server and a terminal and provides commentary as audio in real time. Specific embodiments of the system of the present invention are described in detail below.

[0405] Server Roles

[0406] 1. Data Collection

[0407] The server collects a large amount of video and live commentary from the Internet about sporting events such as car races, horse races, and drone races, and uses these as a large-scale learning dataset for video analysis and natural language processing.

[0408] As a concrete example, the server collects F1 race footage and commentary text from YouTube and sports news sites.

[0409] 2. Data Preprocessing

[0410] The server labels the collected video data for important events and actions in each frame, and tokenizes the text data, converting it into a format that can be easily understood by machine learning models, and then stores it.

[0411] As a specific example, labels such as "first lap completed" or "vehicle collision" are added to video frames, and the text data is divided into words.

[0412] 3. Model training

[0413] The server uses the pre-processed data to train advanced algorithms such as Transformer models, which then have the capabilities of video analysis, live text generation, and emotion recognition.

[0414] As a concrete example, video analysis data and tokenized commentary text are input into the Transformer model to enable it to recognize "the scene from the start of a car race to the end of the first lap."

[0415] 4. Saving and updating the model

[0416] The server stores the trained model and periodically updates it to ensure that the latest information is always used for highly accurate analysis.

[0417] For example, new race data is added once a month to retrain the model and reflect the latest racing developments.

[0418] 5. Receiving and Processing Requests

[0419] The server receives requests from users and loads and prepares the relevant trained models.

[0420] As a concrete example, the server loads a "F1 race commentary model" and prepares it for analysis.

[0421] 6. Video Analysis

[0422] The server analyzes the provided video data frame by frame to recognize important events and actions.

[0423] As a specific example, events such as "first lap completed" and "vehicle collision" are detected in real time.

[0424] 7. Emotion recognition

[0425] The server analyzes the voice and facial expression data sent by the user and recognizes the user's emotional state via an emotion engine.

[0426] As a specific example, the emotional state of the user, such as "excited" or "relaxed," is analyzed from the tone of voice and facial expression.

[0427] 8. Live Text Generation

[0428] The server generates commentary text using a generative AI model based on the video analysis results, and adjusts the tone and content of the commentary text based on the emotion recognition results.

[0429] As a specific example, we generate a commentary such as "The first round is over. What an exciting development!"

[0430] 9. Speech Synthesis

[0431] The server sends the generated commentary text to a voice synthesis engine, which generates a narration voice in a tone that matches the user's emotions.

[0432] For example, the tone of the narration can be increased to further excite the user.

[0433] 10. Narration audio transmission

[0434] The server transmits the generated narration voice data to the user's terminal in real time.

[0435] As a specific example, narration audio data is sent to the user's smartphone in streaming format.

[0436] Device Role

[0437] 1. Submit a request

[0438] Users use their devices to request live coverage of a particular sporting event from the server, which in the process provides live video and streaming links, as well as voice and facial expression data for emotion recognition.

[0439] As a specific example, a user sends a request from a smartphone app saying, "Please give me a live report of the current F1 race."

[0440] 2. Data reception and playback

[0441] The terminal receives the narration audio data sent from the server and plays it in sync with the video.

[0442] As a specific example, a smartphone app receives narration audio and plays it simultaneously with the live video.

[0443] This system allows users to enjoy a real-time, emotionally adaptive commentary experience. Here are some example prompts:

[0444] "The first round has now finished. Please give us a running commentary on the next developments."

[0445] "Vehicle collision occurred. Please emphasize the tense situation."

[0446] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0447] Step 1: Data collection

[0448] The server collects a large amount of video and live text about sporting events such as car races, horse races, and drone races from the Internet. Specifically, the server uses a web crawler to download video and text data from video sharing sites and sports news sites. The input is a list of URLs, and the output is video files and text files stored in local storage.

[0449] Step 2: Preprocessing the data

[0450] The server splits the collected video data into frames and labels important events and actions. It tokenizes the text data, converts it into a format that is easy for machine learning models to understand, and stores it. Specifically, the server uses video analysis tools to extract video frames and uses a labeling algorithm to identify important events. The inputs are raw video data and live text data, and the output is labeled video frames and tokenized text data.

[0451] Step 3: Model training

[0452] The server uses the preprocessed data to train a machine learning model using advanced algorithms such as the Transformer model. Specifically, the server trains the model using a deep learning framework (e.g., TensorFlow) on a GPU cluster. The input is the preprocessed data, and the output is a trained machine learning model.

[0453] Step 4: Save and update the model

[0454] The server builds a framework to store the trained model and periodically update it. Specifically, the server saves the model in a database and retrains it whenever new data is added. The input is the trained model and the new data, and the output is the updated model.

[0455] Step 5: Submitting the request

[0456] A user uses a device to request a live broadcast of a specific sporting event from a server. In the process, live video and streaming links are provided, and voice and facial expression data for emotion recognition are also transmitted. Specifically, the user enters the necessary information into a request form in an application on the device and submits it. The input is the user's request and emotion data, and the output is the transmission of the request information.

[0457] Step 6: Receiving and Processing the Request

[0458] The server receives requests from users and loads and prepares the relevant trained models. Specifically, the server analyzes the content of the request and loads the appropriate model into memory. The input is the user request, and the output is the loaded model.

[0459] Step 7: Video Analysis

[0460] The server analyzes the provided video data frame by frame and recognizes important events and actions. Specifically, the server uses a video analysis algorithm to analyze the provided live video frame by frame in real time. The input is the live video data, and the output is recognized event information.

[0461] Step 8: Emotion Recognition

[0462] The server analyzes the voice and facial expression data sent by the user and recognizes the user's emotional state through an emotion engine. Specifically, the server analyzes the user's emotions using voice recognition software and facial recognition algorithms. The input is voice data and facial expression data, and the output is the recognized emotional state.

[0463] Step 9: Generate commentary text

[0464] The server generates commentary text using a generative AI model based on the video analysis results. The tone and content of the commentary text are adjusted based on the emotion recognition results. Specifically, the server sends input data to the generative AI model, which adapts the text content according to the emotional status. The input is event information and emotional state, and the output is commentary text.

[0465] Step 10: Text-to-Speech

[0466] The server sends the generated commentary text to a speech synthesis engine and generates a narration voice in a tone that matches the user's emotions. Specifically, the server creates a narration voice using a text-to-speech synthesis tool. The input is the commentary text, and the output is the narration voice.

[0467] Step 11: Send narration audio

[0468] The server transmits the generated narration audio data to the user's device in real time. Specifically, the server uses a protocol to transfer audio data in streaming format. The input is the narration audio, and the output is data transmission to the user's device.

[0469] Step 12: Data reception and playback

[0470] The device receives the narration audio data sent from the server and plays it in sync with the video. Specifically, the device stores the received audio data in a buffer and synchronizes it with the video playback. The input is the narration audio data, and the output is real-time playback.

[0471] (Application example 2)

[0472] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0473] Conventional production line monitoring systems require a human supervisor to constantly monitor the video feed, which requires a lot of effort and can delay the detection of abnormalities. Furthermore, in environments where a rapid response is required when an abnormality occurs, the supervisor's response may vary depending on their emotional state. If these issues are not resolved, the efficiency and safety of the production line may decrease.

[0474] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0475] In this invention, the server includes a computing device that collects learning data, a computing device that preprocesses the learning data and trains a machine learning model, a computing device that accepts requests and analyzes video of the production line, a computing device that generates alert text corresponding to anomaly detection and the manager's emotions based on the analysis results, a computing device that converts the alert text into narration using a speech synthesis engine, a computing device that sends the narration to the manager's terminal, and a terminal device that synchronously plays back the video and narration on the manager's terminal.This automates production line monitoring and enables adaptive alerts corresponding to the manager's emotions.

[0476] A "computing device" is a device that performs computational tasks such as collecting, analyzing, processing, and transmitting data.

[0477] "Training data" refers to input data for training a machine learning model, and can include video, text, and other forms of data.

[0478] "Preprocessing" is a series of operations that transform training data into a form suitable for analysis and model training.

[0479] A "machine learning model" is an algorithm that learns patterns from data and performs analysis and predictions.

[0480] "Video analytics" is the process of identifying information contained in images or video frames to detect specific events or anomalies.

[0481] "Anomaly detection" is the process of identifying unusual behavior or conditions on a manufacturing line.

[0482] "Emotion recognition" is a technology that determines the emotional state of an administrator from video and audio data.

[0483] An "alert text" is a text message generated to notify you of an abnormality or important event.

[0484] A "speech synthesis engine" is a system that converts text data into voice data.

[0485] "Narration voice" is a voice message generated by a voice synthesis engine.

[0486] "Terminal device" refers to a computer or smart device used by an administrator, and is a device for receiving and playing narration audio and video from the server.

[0487] Learning Phase

[0488] The system of the present invention functions through an initial training phase. During the training phase, a server collects video and text data related to manufacturing processes from the internet. The collected data is preprocessed into a format suitable for analysis and machine learning model training. This preprocessing includes frame-by-frame classification of video data and tokenization of text data. The server then uses the preprocessed data to train Transformer models and other machine learning algorithms to improve anomaly detection and emotion recognition capabilities. The trained models are stored on the server and periodically updated.

[0489] Execution Phase

[0490] Receiving a request

[0491] A user sends a request for monitoring the production line to the server through a terminal used by the manager. The request includes a streaming link of the live video and audio and video data for recognizing the manager's emotions. The terminal then sends this data to the server.

[0492] Video analysis and emotion recognition

[0493] The server analyzes the video data from the user frame by frame, allowing abnormal behavior and conditions to be detected in real time. For example, if the robot arm is not aligned correctly, the abnormality is immediately detected. At the same time, the server analyzes the administrator's voice and facial expression data, and uses an emotion engine to recognize the administrator's emotional state. This takes into account voice tone, facial expressions, body movements, and more.

[0494] Alert text generation and speech synthesis

[0495] The server generates adaptive alert text based on the results of video analysis and emotion recognition. The generated alert text is converted into a voice narration by a speech synthesis engine. This voice is natural and easy to understand, and the tone is adjusted to match the administrator's emotions.

[0496] Sending and playing narration audio

[0497] The server sends the generated narration audio data in real time to the administrator's terminal. The terminal receives the narration audio data sent from the server and plays it in sync with the video. This allows the administrator to respond to any abnormalities in real time while watching the video.

[0498] Specific examples

[0499] For example, if a manager is monitoring a production line and notices that a robot arm is not operating normally, the server will use video analysis to detect the abnormality. At the same time, if the manager is feeling nervous, the emotion engine will recognize this as well. As a result, an alert text is generated stating, "An abnormality has been detected in the robot arm's operation. Please be careful!", and an adaptive voice narration is created by the speech synthesis engine. The narration is sent to the manager's device in real time, allowing the manager to listen to the alert along with the video and respond quickly.

[0500] Prompt Sentence Examples

[0501] "Detect anomalies in the factory in real time and adaptively notify managers. Generate appropriate alerts based on the nature of the anomaly and the manager's sentiment."

[0502] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0503] Step 1: Collect training data

[0504] The server collects video and text data related to the manufacturing process from the internet. Specifically, the server obtains the data using web scraping or APIs. The collected data is used for subsequent analysis and training of machine learning models. The input is publicly available data on the internet, and the output is the collected dataset.

[0505] Step 2: Preprocessing the data

[0506] The server preprocesses the collected training data. Video data is classified by frame, and text data is tokenized. Data preprocessing also includes noise reduction and format conversion. Specific operations include segmenting video frames using OpenCV and tokenizing text using a natural language processing library. The input is the collected dataset, and the output is the preprocessed data.

[0507] Step 3: Train the machine learning model

[0508] The server uses the preprocessed data to train a machine learning model. Transformer models and image recognition models are typically used. Specifically, a multilayer neural network is trained using the dataset to create a model capable of anomaly detection and emotion recognition. The input is the preprocessed data, and the output is the trained model.

[0509] Step 4: Receiving the request

[0510] A user sends a request to monitor the production line to the server through a terminal used by the manager. The terminal then sends a request that includes a streaming link of the live video and audio and video data for recognizing the manager's emotions. Specifically, the user inputs the request using a dedicated application, and the terminal then sends it to the server. The input is the request data from the user, and the output is a confirmation that the request has been received.

[0511] Step 5: Video Analysis

[0512] The server analyzes the received video data frame by frame. Any abnormal behavior or state is detected. Specifically, video analysis is performed using OpenCV and anomaly detection is performed using a machine learning model. For example, if the robot arm is not aligned correctly, the anomaly is immediately detected. The input is the live video data, and the output is the anomaly detection result.

[0513] Step 6: Emotion Recognition

[0514] The server analyzes the received voice and facial expression data and recognizes the administrator's emotional state using an emotion engine. Specifically, voice and facial expression analysis are performed to determine the administrator's emotional state. The input is the user's voice and video data, and the output is the emotion recognition result.

[0515] Step 7: Generate the alert text

[0516] The server generates adaptive alert text based on the video analysis results and emotion recognition results. Specifically, the alert text is generated using a natural language generation model. For example, the text generated might read, "An abnormality has been detected in the robot arm's operation. Please be careful!" The input is the anomaly detection results and emotion recognition results, and the output is the alert text.

[0517] Step 8: Text-to-Speech

[0518] The server converts the generated alert text into a narration voice using a speech synthesis engine. Specifically, it converts the text into an audio file using gTTS. The audio is natural and easy to understand, and is provided in a tone that matches the administrator's emotions. The input is the alert text, and the output is the narration voice data.

[0519] Step 9: Send your narration audio

[0520] The server transmits the generated narration voice data to the administrator's terminal in real time. Specifically, the voice data is transmitted over the network. The input is the narration voice data, and the output is the transmitted voice data.

[0521] Step 10: Playing the narration audio

[0522] The device receives the narration audio data sent from the server and plays it in sync with the video. Specifically, the device receives the audio data and plays it back via an application for synchronous playback. The input is the narration audio data, and the output is the played back narration audio.

[0523] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0524] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search<url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0525] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0526] [Second embodiment]

[0527] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0528] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0529] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0530] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0531] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0532] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0533] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0534] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0535] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0536] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0537] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0538] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0539] System Overview

[0540] The present invention is a system that utilizes AI to provide automatic live commentary of sporting events. This system has a function that links a server and a terminal and provides users with real-time live commentary as audio. The specific processing of the program of the system of the present invention is explained below in natural language.

[0541] Learning Phase

[0542] server

[0543] 1. Data Collection

[0544] The server collects a large amount of video and live text from sporting events such as car races, horse races, and drone races from the Internet, which are used as large-scale training datasets for video analysis and natural language processing.

[0545] 2. Data Preprocessing

[0546] The server pre-processes the collected data: video data is labeled for each frame with important events and actions, and text data is tokenized and converted into a format that can be easily understood by machine learning models.

[0547] 3. Model training

[0548] The server uses the pre-processed data to train machine learning models, which are built using advanced algorithms such as Transformer models and trained to analyze video footage and generate commentary for sporting events.

[0549] 4. Save the model

[0550] The server stores the trained model and builds a framework for regularly updating it, enabling highly accurate analysis based on the latest information at all times.

[0551] Commentary Phase

[0552] User

[0553] 1. Submit a request

[0554] A user requests a live broadcast of a particular sporting event from their device, which sends a request for the live broadcast to the server.

[0555] Terminal

[0556] 1. Receiving a request

[0557] The device sends the user's request, which includes the live video or streaming link, to the server for analysis.

[0558] 2. Data reception and playback

[0559] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing the user to watch the video with commentary in real time.

[0560] server

[0561] 1. Request Processing

[0562] When the server receives a request from a user, it loads and prepares the appropriate trained model.

[0563] 2. Video Analysis

[0564] The server analyzes the real-time video provided by the user frame by frame. This analysis recognizes important events and actions. For example, in a car race, events such as "the first lap is over" and "vehicle collision" are detected.

[0565] 3. Live Text Generation

[0566] Based on the analysis results, the server uses a natural language generation model to generate live text, which is adjusted to include appropriate phrases and sentence structures depending on the situation.

[0567] 4. Speech Synthesis

[0568] The server sends the generated commentary text to a speech synthesis engine to generate a narration voice, which is tuned for high quality and fluency.

[0569] 5. Narration audio transmission

[0570] The server transmits the generated narration voice data to the user's terminal in real time.

[0571] Example: Car race commentary

[0572] Learning Phase

[0573] The server collects footage and commentary from past F1 races and rallies, preprocesses it, and then trains a machine learning model.

[0574] Commentary Phase

[0575] 1. User Request

[0576] The user requests a live broadcast of the F1 race currently underway from the terminal to the server.

[0577] 2. Server Processing

[0578] The server analyzes the live video data sent by the user and detects events such as "two cars collide" or "first lap completed."

[0579] Based on the detected events, the server generates commentary text such as "The first lap has now ended. Two cars have collided!" and generates narration audio using a speech synthesis engine.

[0580] The server transmits the generated voice to the user's terminal.

[0581] 3. Terminal Processing

[0582] The user's terminal plays back the live commentary audio data sent from the server in synchronization with the video, providing the user with a real-time commentary.

[0583] This system will enable viewers to enjoy sporting events such as car races, horse races, and drone races smoothly without the need for live commentary by experts. By providing specific commentary in real time, viewers can gain a deeper understanding of the action, providing a more immersive viewing experience.

[0584] The processing flow will be explained below.

[0585] Learning Phase

[0586] server

[0587] Step 1:

[0588] The server collects video and commentary data about sporting events from the Internet, including video footage of car races, horse races, drone races, and other events, along with commentary.

[0589] Step 2:

[0590] The server preprocesses the collected data, labeling the video data with important events and actions for each frame, and tokenizing the text data, converting it into a format that can be easily understood by machine learning models.

[0591] Step 3:

[0592] The server uses the pre-processed data to train machine learning models, which use advanced algorithms such as the Transformer architecture to improve video analysis and play-by-play text generation for sporting events.

[0593] Step 4:

[0594] The server stores the trained model and builds a framework to update the model as needed, enabling highly accurate analysis with the latest information.

[0595] Commentary Phase

[0596] User

[0597] Step 1:

[0598] A user uses his / her terminal to request a live broadcast of a particular sporting event from the server, providing a live video or streaming link.

[0599] Terminal

[0600] Step 2:

[0601] The terminal transmits the user's request and the video data to the server.

[0602] server

[0603] Step 3:

[0604] The server receives requests sent by users and loads the relevant trained models.

[0605] Step 4:

[0606] The server analyzes the provided video data frame by frame, detecting specific actions and events (e.g., "first lap completed," "vehicle collision").

[0607] Step 5:

[0608] The server uses a natural language generation model based on the analysis results to instantly generate live commentary text, such as "The first lap has now finished. Two cars have collided!"

[0609] Step 6:

[0610] The server sends the generated commentary text to a speech synthesis engine to create a narration voice that is natural and easy to understand.

[0611] Step 7:

[0612] The server transmits the generated narration voice data to the user's terminal in real time.

[0613] Terminal

[0614] Step 8:

[0615] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing users to enjoy the commentary in real time along with the video.

[0616] Example: Car race commentary

[0617] Learning Phase

[0618] Step 1:

[0619] The server collects footage and live commentary from past F1 races and rallies.

[0620] Step 2:

[0621] The server preprocesses the collected data, labels the video frames, and tokenizes the text.

[0622] Step 3:

[0623] The server trains a Transformer model with the preprocessed data, giving it high-precision video analysis and commentary generation capabilities.

[0624] Step 4:

[0625] The server stores the trained model and periodically updates it.

[0626] Commentary Phase

[0627] Step 1:

[0628] The user requests a live broadcast of the F1 race currently underway from the server via the terminal.

[0629] Step 2:

[0630] The terminal transmits the user's request and the live video data to the server.

[0631] Step 3:

[0632] The server receives the request and loads a model suitable for F1 racing.

[0633] Step 4:

[0634] The server analyzes the video and detects events such as "first lap completed" or "vehicle collision."

[0635] Step 5:

[0636] The server generates live text: "First lap now completed. Two cars have collided!"

[0637] Step 6:

[0638] The server sends the commentary text to a speech synthesis engine to generate the narration voice.

[0639] Step 7:

[0640] The server transmits the narration voice to the user's terminal.

[0641] Step 8:

[0642] The terminal plays back the narration audio in synchronization with the video, allowing the user to view the video with commentary in real time.

[0643] Example 1

[0644] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0645] Conventional live commentary of sporting events relied on human commentators, which had issues with real-timeness and accuracy. It was also difficult to secure commentators with specialized knowledge. Furthermore, there was a need to provide a realistic live commentary experience for users who could not attend the event in person.

[0646] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0647] In this invention, the server includes: means for collecting large amounts of video and commentary text related to sporting events such as car races, horse races, and drone races from the Internet; means for preprocessing the collected data, labeling important events and actions in each frame of the video data, and tokenizing the text data; means for training a machine learning model using the preprocessed data; means for periodically updating and saving the trained machine learning model; means for receiving requests from users and analyzing real-time sporting event video; means for generating commentary text based on the analysis results; means for converting the generated commentary text into narration using a speech synthesis engine; means for transmitting the narration text to a user's device; and means for synchronously playing back the video and narration text on the user's device. This enables automatic commentary with high real-time accuracy. This allows users without specialized knowledge to enjoy a realistic sports viewing experience, eliminating the need for a commentator.

[0648] "On the Internet" refers to a globally connected information and communications network and infrastructure for sending and receiving data.

[0649] "Car racing" is a competitive event involving automobiles traveling at high speeds, and is a sport in which multiple vehicles compete against each other for ranking.

[0650] "Horse racing" is a competitive event using horses, and is a sport in which multiple horses compete for rankings.

[0651] "Drone racing" is a competitive event using unmanned aerial vehicles (drones), and is a sport in which drones compete for rankings by driving around a course while avoiding obstacles.

[0652] "Footage" means real-time or recorded visual data of a sporting event provided in the form of video.

[0653] "Live commentary text" is a document that verbally explains the progress of a sporting event and important events, and is character data that serves as material for speech synthesis.

[0654] "Server means" refers to a computer system in general that has the function of managing and processing data via a network.

[0655] "Collecting" is the process of obtaining and storing data from the internet or other sources.

[0656] "Preprocessing" refers to the process of converting collected data into a form that is easier to analyze, and specifically includes labeling video frames and tokenizing text data.

[0657] A "machine learning model" is a set of algorithms or programs that can learn from collected data and perform specific tasks automatically.

[0658] A "Transformer model" is a deep learning model that demonstrates excellent performance in natural language processing and machine translation, and has the ability to generate output while taking into account the context of the input data.

[0659] "Analyzing" is the process of analyzing collected or received video or data to detect significant events or actions.

[0660] "Generating" refers to the process of creating new data or information based on the analysis results, and in this case refers to the creation of commentary text.

[0661] A "speech synthesis engine" is a set of software or hardware for converting text data into speech data.

[0662] "Narration voice" is voice data that sounds like a human being speaking, generated by a voice synthesis engine.

[0663] "Terminal means" refers to all electronic devices that are directly operated by users and used to input and output data.

[0664] MODE FOR CARRYING OUT THE INVENTION

[0665] The present invention is a system that uses AI to provide automatic commentary of sporting events. This system has a function that links a server and a terminal and provides users with real-time commentary as audio. Specific embodiments for carrying out the invention are described below.

[0666] server

[0667] The server collects a large amount of video and live text about sporting events from the Internet, using various sources such as YouTube and official sporting sites, and stores the collected data in a large database.

[0668] Next, we preprocess the collected data: for video data, we use a visual information analysis library (e.g., OpenCV) to label important events and actions for each frame, and for text data, we use a natural language processing (NLP) library (e.g., NLTK or SpaCy) to tokenize it.

[0669] Once the preprocessing is complete, the data is used to train a machine learning model. In this invention, advanced algorithms such as the Transformer model are used. This model is trained using machine learning frameworks such as TensorFlow and Scikit-learn. The trained model is periodically retrained with new data and updated to ensure the model's accuracy.

[0670] When a live video request is received from a user, the server loads the appropriate pre-trained model and analyzes the video in real time. Based on the analysis results, a natural language generation model is used to generate live commentary text. This text is then passed to a speech synthesis engine, which converts it into fluent narration audio in real time. For example, a text-to-speech (TTS) engine such as Amazon Polly is used for speech synthesis.

[0671] The narration audio is transmitted to the user's device in real time using protocols such as HTTP Live Streaming (HLS).

[0672] Terminal

[0673] The user's device receives the narration audio data sent from the server and plays it in sync with the live video. The user then watches the video with commentary in real time via a dedicated application or web browser.

[0674] User

[0675] Users request live coverage of a particular sporting event from their device. For example, if they want live coverage of a current F1 race, they click a request button in the application, enter the URL of the live video in the pop-up window, and send it to the server.

[0676] Specific examples

[0677] An example of such a prompt sentence is, "Please provide a live commentary of the F1 race currently being broadcast. Please give a detailed description of the events that occur from the start to the pit stop."

[0678] This system allows viewers to enjoy sporting events such as car races, horse races, and drone races smoothly without the need for live commentary by experts. By providing accurate commentary in real time, viewers can gain a deeper understanding of the action and enjoy a more immersive viewing experience.

[0679] keyword

[0680] Generative AI Models

[0681] Prompt statement

[0682] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0683] Step 1: Data collection

[0684] The server collects video and live commentary of sporting events such as car races, horse races, and drone races from the Internet. The input is a list of URLs on the Internet, and the output is the collected video and text data. Specifically, the server automatically downloads the video and text data from the specified URL list and stores them in a database.

[0685] Step 2: Data Preprocessing

[0686] The server preprocesses the collected data. For video data, it uses a visual information analysis library such as OpenCV to label important events and actions for each frame. For text data, it uses a natural language processing library such as NLTK or SpaCy to tokenize it. The input is the collected video and text data, and the output is preprocessed video frames and text tokens.

[0687] Step 3: Model training

[0688] The server uses the preprocessed data to train a machine learning model. Specifically, it uses a machine learning framework such as TensorFlow or Scikit-learn to train a Transformer model. The input is the preprocessed dataset, and the output is a trained machine learning model. The server uses video and text pairs to batch process the model. It updates the model parameters every epoch and minimizes the loss function.

[0689] Step 4: Save and update the model

[0690] The server saves the trained model in a database and periodically retrains it with new data. The input is the trained model and a new dataset, and the output is the updated model. Specifically, the server saves the parameters of the trained model and schedules periodic retraining with new data.

[0691] Step 5: Receiving the request

[0692] A user requests a live broadcast of a specific sporting event from their device. The input is the URL of the live video specified by the user, and the output is a request sent to the server. The user clicks the request button in the app, enters the URL in the pop-up window, and submits it.

[0693] Step 6: Request Processing

[0694] When the server receives a request from a user, it loads the appropriate trained model. The input is the user request and the URL of the live video, and the output is the loaded trained model. Specifically, the server analyzes the request and determines which model to load.

[0695] Step 7: Video Analysis

[0696] The server analyzes real-time video provided by the user frame by frame. The input is the live video stream, and the output is the analyzed event and action data. Specifically, the server receives the video stream and analyzes each frame using OpenCV.

[0697] Step 8: Live text generation

[0698] The server generates commentary text using a natural language generation model based on the analysis results. The input is the analyzed event data, and the output is the generated commentary text. Specifically, the server passes the analysis results as input to the natural language generation model to generate commentary text.

[0699] Step 9: Text-to-Speech

[0700] The server sends the generated commentary text to a speech synthesis engine to generate a narration voice. The input is the commentary text, and the output is the synthesized narration voice. Specifically, the server sends the generated text to the speech synthesis engine and converts it into voice data.

[0701] Step 10: Send narration audio

[0702] The server transmits the generated narration audio data to the user's terminal. The input is the narration audio, and the output is the transmission of audio data to the user's terminal. Specifically, the server streams the audio file to the terminal.

[0703] Step 11: Data reception and playback

[0704] The device receives the narration audio data sent from the server and plays it in sync with the live video. The input is the narration audio data and the live video, and the output is the synchronized playback of the live video. Specifically, the device receives an HTTP request and launches a player to play the streaming audio data.

[0705] (Application example 1)

[0706] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0707] Modern factories require real-time monitoring of the progress and abnormalities of production lines and robots. However, it is difficult for on-site workers to constantly monitor the situation, and they are unable to respond immediately when an abnormality occurs. Furthermore, visual information alone does not provide a sufficient understanding of the details of the production process, making it difficult to improve efficiency and control quality. To address these issues, the present invention aims to provide a system that provides real-time audio commentary on the progress and abnormalities of production lines and robots.

[0708] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0709] In this invention, the server includes means for collecting training data, means for preprocessing the training data and training a machine learning model, means for accepting requests and analyzing video of a sporting event or a production line, means for generating commentary text based on the analysis results, means for converting the commentary text into narration audio using a speech synthesis engine, means for transmitting the narration audio to a user's terminal, means for synchronously playing back the video and narration audio on the user's terminal, and means for providing real-time commentary on the progress and abnormalities of the production line or robot. This enables real-time audio commentary on the status of the manufacturing process in a factory, allowing workers to perform their work efficiently.

[0710] The "server means" is a computer system that controls the functions of data collection, preprocessing, analysis, generation, synthesis, and transmission, and communicates with the user's terminal.

[0711] "Training Data" is a data set collected by the Server Means and used to train a Machine Learning Model.

[0712] "Preprocessing" is the process of converting collected training data into a format that is easy for machine learning models to understand, using techniques such as tokenization and labeling.

[0713] A "machine learning model" is an algorithm, particularly a Transformer model, that is trained on training data to perform a specific task.

[0714] A "request" is a communication signal that a user uses from his / her own terminal to request a live report of a specific event from the server means.

[0715] "Video analytics" is the process of analyzing real-time video data to detect important events or actions.

[0716] "Live text" is explanatory text generated based on the results of video analysis, which is converted into voice by a voice synthesis engine.

[0717] A "speech synthesis engine" is software or hardware for converting text data into narration voice.

[0718] The "terminal means" is a device used by a user, and has the function of playing back the narration audio and video transmitted from the server means in synchronization.

[0719] A "production line" is a system that shows the machines and work flow arranged to carry out production activities within a factory.

[0720] A "robot" is a mechanical device that performs tasks automatically or semi-automatically in a factory.

[0721] "Real-time" means that processing occurs immediately, with virtually no delay.

[0722] "Audio commentary" is the process of communicating analyzed information to the user in real time by voice.

[0723] "Progress" refers to the state and speed at which a production line or robot is working.

[0724] An "abnormality" refers to a state that deviates from normal operation, or a malfunction such as an error or breakdown.

[0725] The present invention provides a system that provides real-time audio commentary on the progress and abnormalities of manufacturing lines and robots. This system consists of three main elements: a server, a terminal, and a user.

[0726] server

[0727] 1. Data Collection and Learning

[0728] The server collects actual video and text data about manufacturing lines and robots collected from the Internet. This data is used as learning data to train machine learning models. Specific hardware used is a high-performance server.

[0729] 2. Data Preprocessing

[0730] The collected data is preprocessed by the server. For example, video data is broken down into frames and labeled. Text data is tokenized and converted into a format that is easy for machine learning models to understand. Open source software such as TensorFlow and OpenCV is used.

[0731] 3. Model training

[0732] Using the pre-processed data, the server trains machine learning models, built using advanced algorithms such as the Transformer model, capable of analyzing video footage of production lines and robots and generating live text.

[0733] 4. Model application and analysis

[0734] The server analyzes the video of the production line based on user requests. Video analysis includes real-time event detection. For example, it detects actions such as "the production line has stopped" or "a robot has encountered an error." The software used is TensorFlow and OpenCV.

[0735] 5. Live Text Generation and Speech Synthesis

[0736] Based on the analysis results, the server generates commentary text using a natural language generation model (such as GPT-3), which is then converted into narration audio by a speech synthesis engine (Amazon Polly, Google Text-to-Speech).

[0737] 6. Sending data to the user device

[0738] Once the narration voice data is generated, the server transmits it to the user's terminal in real time.

[0739] Terminal

[0740] 1. Data reception and synchronous playback

[0741] The user's device (such as smart glasses) receives the narration audio data sent from the server and plays it in sync with the video. This function allows the user to understand the status of the production line and robots in real time with audio commentary.

[0742] User

[0743] 1. Submit a request

[0744] Users can request live footage of a specific manufacturing line or robot from a terminal, which can include live video or streaming links.

[0745] Specific examples

[0746] For example, if a user sends a request from a terminal about the progress of a production line, the server generates a prompt sentence such as the following:

[0747] Example prompt sentence:

[0748] "The production line has stopped. What is the next action?"

[0749] In this way, by linking the server, terminal, and user, it becomes possible to provide real-time audio updates on the progress and abnormalities of the production line and robots, allowing workers to carry out their work efficiently.

[0750] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0751] Step 1: Data collection

[0752] The server collects large amounts of video and text data related to manufacturing lines and robots from the Internet. This data is used as a training dataset. The input is raw data related to the manufacturing process, and the output is the collected dataset. Specifically, the data is downloaded using a web scraping tool or API.

[0753] Step 2: Data Preprocessing

[0754] The server preprocesses the collected data. Specifically, the video data is decomposed into frames and important events are labeled. The text data is tokenized and formatted. The input is the collected video data and text data, and the output is the preprocessed data. Specifically, frame decomposition is performed using OpenCV and labeling is performed using TensorFlow.

[0755] Step 3: Model training

[0756] The server uses the preprocessed data to train a machine learning model. This model can then analyze video of production lines and robots and generate live text. The specific method used is a Transformer model. The input is the preprocessed data, and the output is the trained model. The specific operation is to train the Transformer model using the TensorFlow API.

[0757] Step 4: Submitting the request

[0758] A user requests a live report of a specific manufacturing line or robot from a terminal to a server. This request may include live video or a streaming link. The input is the user's request, and the output is a communication signal to the server. Specifically, the request is sent from the terminal's UI to the server.

[0759] Step 5: Video Analysis

[0760] The server analyzes the real-time video provided by the user frame by frame. This analysis recognizes important events and actions. For example, events such as "the production line has stopped" or "an error has occurred in a robot" can be detected. The input is the live video data, and the output is the analysis results. Specifically, the video analysis is performed using TensorFlow.

[0761] Step 6: Generate commentary text

[0762] The server generates commentary text using a natural language generation model based on the analysis results. The generated text has appropriate phrases and sentence structures depending on the situation. The input is the analysis results, and the output is the commentary text. Specifically, text generation is performed using GPT-3.

[0763] Step 7: Text-to-Speech

[0764] The server sends the generated commentary text to a speech synthesis engine to generate narration audio. This audio is high quality and fluent. The input is the commentary text, and the output is the narration audio data. Specifically, speech synthesis is performed using the Google Text-to-Speech API.

[0765] Step 8: Send narration audio

[0766] The server transmits the generated narration audio data to the user's device in real time. The input is the narration audio data, and the output is the audio data sent to the user's device. Specifically, the audio data is transmitted using a network protocol (e.g., WebSocket).

[0767] Step 9: Data reception and synchronous playback

[0768] The user's device receives the narration audio data sent from the server and plays it back in sync with the video. This function allows the user to understand the status of the production line or robot in real time with audio commentary. The input is the audio data sent from the server, and the output is synchronous playback of the video and audio. Specifically, the device's audio playback function is used to synchronize with the video data.

[0769] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0770] System Overview

[0771] The present invention is a system that utilizes AI to provide automatic commentary of sporting events. In addition, by combining it with an emotion engine that recognizes the user's emotions, it provides adaptive commentary that corresponds to the user's emotions. The server and terminals work together to provide commentary in the form of audio in real time. The specific processing of the program of the system of the present invention is explained below in natural language.

[0772] Learning Phase

[0773] server

[0774] 1. Data Collection

[0775] The server collects a large amount of video and live text from sporting events such as car races, horse races, and drone races from the Internet, which are used as large-scale training datasets for video analysis and natural language processing.

[0776] 2. Data Preprocessing

[0777] The server preprocesses the collected data: it labels important events and actions for each frame of video data, tokenizes text data and converts it into a format that machine learning models can easily understand, and collects and processes audio and facial expression data for the emotion engine.

[0778] 3. Model training

[0779] The server uses the pre-processed data to train machine learning models, which use advanced algorithms such as Transformer models, and train them to perform video analysis, live text generation, and emotion recognition.

[0780] 4. Save the model

[0781] The server will store the trained model and create a framework for periodic updates, enabling highly accurate analysis using the latest information.

[0782] Commentary Phase

[0783] User

[0784] 1. Submit a request

[0785] Users can use their devices to request live coverage of a particular sporting event from the server, providing live video or a streaming link, and voice and facial expression data is also collected to recognize the user's emotions.

[0786] Terminal

[0787] 1. Receiving a request

[0788] The terminal transmits the user's request, video data, and emotion data to the server.

[0789] server

[0790] 1. Request Processing

[0791] The server receives requests from users and loads and prepares the relevant trained models.

[0792] 2. Video Analysis

[0793] The server analyzes the provided video data frame by frame. This analysis allows it to recognize important events and actions. For example, in the case of a car race, events such as "the first lap is over" and "vehicle collision" can be detected.

[0794] 3. Emotion recognition

[0795] The server analyzes the voice and facial expression data sent by the user and uses an emotion engine to recognize the user's emotional state, including the user's tone of voice, facial expressions, and body movements.

[0796] 4. Live Text Generation

[0797] The server generates commentary text using a natural language generation model based on the video analysis results. The content and tone of the commentary text are adjusted according to the user's emotional state as recognized by the emotion engine. For example, if the user is excited, the server generates text that maintains the same sense of excitement.

[0798] 5. Speech Synthesis

[0799] The server sends the generated commentary text to a speech synthesis engine to create a narration voice, which is natural and easy to understand, and is provided in a tone that matches the user's emotions.

[0800] 6. Narration audio transmission

[0801] The server transmits the generated narration voice data to the user's terminal in real time.

[0802] Terminal

[0803] 1. Data reception and playback

[0804] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing users to enjoy real-time commentary based on their emotions along with the video.

[0805] Example: Car race commentary

[0806] Learning Phase

[0807] 1. Data Collection and Preprocessing

[0808] The server collects and preprocesses video footage from past F1 races and rallies, commentary text, and user emotion data, labeling video frames, tokenizing text, and processing audio and facial expression data for the emotion engine.

[0809] 2. Model training

[0810] The server trains a Transformer model on the preprocessed data, giving it high-precision video analysis, commentary generation, and emotion recognition capabilities.

[0811] 3. Save the model

[0812] The server stores the trained model and periodically updates it.

[0813] Commentary Phase

[0814] 1. User Request

[0815] The user requests a live report of the current F1 race from the server via their device, and data based on the user's emotional state is also collected at the same time.

[0816] 2. Sending and Receiving Devices

[0817] The device transmits the user's request, live video data, and emotion data to the server, and receives the live audio data sent from the server and plays it in sync with the video.

[0818] 3. Server Processing

[0819] The server receives the request, analyzes the video, and the emotion engine recognizes the emotional state. Events such as "First lap completed" and "Vehicle collision" are detected, and based on the user's emotional state, a live commentary text is generated, such as "The first lap has now completed. This is exciting! Two cars have collided!"

[0820] The server sends the commentary text to a speech synthesis engine to generate a narration voice that maintains an uplifting tone.

[0821] The server transmits the generated narration voice to the user's terminal.

[0822] 4. Playback of the device

[0823] The user's terminal plays back the narration audio in synchronization with the video, allowing the user to view a realistic commentary in real time.

[0824] This system allows viewers to watch sporting events in a way that adapts to their emotions, enabling real-time, adaptive commentary without the need for expert commentary, improving the sense of realism and entertainment, and making it easier to attract viewers' attention.

[0825] The processing flow will be explained below.

[0826] Learning Phase

[0827] server

[0828] Step 1:

[0829] The server collects videos, commentary text data, and user emotion data related to sporting events from the Internet. This includes videos of car races, horse races, drone races, and other events, along with commentary text and user voice and facial expression data.

[0830] Step 2:

[0831] The server preprocesses the collected data: it labels important events and actions for each frame of video data, tokenizes text data and converts it into a format that machine learning models can easily understand, and adds emotion labels from audio and facial expression data for the emotion engine.

[0832] Step 3:

[0833] The server uses the pre-processed data to train machine learning models, which are trained using advanced algorithms such as Transformer models to analyze video footage of sporting events, generate commentary, and recognize emotions.

[0834] Step 4:

[0835] The server stores the trained model and builds a framework to periodically update the model as needed, enabling highly accurate analysis using the latest information.

[0836] Commentary Phase

[0837] User

[0838] Step 1:

[0839] Users use their devices to request a live broadcast of a particular sporting event from the server, providing a live video or streaming link, and voice and facial expression data is also collected for emotion recognition.

[0840] Terminal

[0841] Step 2:

[0842] The terminal transmits the user's request, live video data, and emotion data to the server.

[0843] server

[0844] Step 3:

[0845] The server receives requests sent by users and loads the relevant trained models.

[0846] Step 4:

[0847] The server analyzes the received video data frame by frame to detect important events and actions (e.g., "first lap completed" or "vehicle collision").

[0848] Step 5:

[0849] The server analyzes the voice and facial expression data sent by the user and recognizes the user's emotional state (e.g., excitement, surprise) using an emotion engine.

[0850] Step 6:

[0851] The server generates commentary text using a natural language generation model based on the video analysis and emotion recognition results, and adjusts the content and tone of the commentary text to reflect the user's emotions.

[0852] Step 7:

[0853] The server sends the generated commentary text to a speech synthesis engine to create a narration voice, which is provided in a tone that matches the user's emotions.

[0854] Step 8:

[0855] The server transmits the generated narration voice data to the user's terminal in real time.

[0856] Terminal

[0857] Step 9:

[0858] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing users to enjoy real-time commentary based on their emotions along with the video.

[0859] Example: Car race commentary

[0860] Learning Phase

[0861] Step 1:

[0862] The server collects footage of past F1 races and rally competitions, commentary text, and user emotion data.

[0863] Step 2:

[0864] The server preprocesses the collected data, labels video frames, tokenizes text, and assigns emotion labels to audio and facial expression data for the emotion engine.

[0865] Step 3:

[0866] The server trains a Transformer model on the preprocessed data, giving it high-precision video analysis, commentary generation, and emotion recognition capabilities.

[0867] Step 4:

[0868] The server stores the trained model and periodically updates it.

[0869] Commentary Phase

[0870] Step 1:

[0871] The user requests a live report of the current F1 race from the server via their device, and data based on the user's emotional state is also collected at the same time.

[0872] Step 2:

[0873] The terminal transmits the user's request, live video data, and emotion data to the server.

[0874] Step 3:

[0875] The server receives the request and loads a model suitable for F1 racing.

[0876] Step 4:

[0877] The server analyzes the video and detects events such as "first lap completed" or "vehicle collision."

[0878] Step 5:

[0879] The server analyzes the user's voice and facial expression data, and the emotion engine recognizes the user's emotional state (e.g., the user is excited).

[0880] Step 6:

[0881] Based on the video analysis and emotion recognition results, the server generates live text such as, "The first lap is now complete. This is very exciting! Two cars have collided!"

[0882] Step 7:

[0883] The server sends the commentary text to a speech synthesis engine to generate a narration voice with a tone that reflects excitement.

[0884] Step 8:

[0885] The server transmits the generated narration voice to the user's terminal.

[0886] Step 9:

[0887] The terminal plays back the narration audio in synchronization with the video, allowing the user to view a realistic commentary in real time.

[0888] Example 2

[0889] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0890] In recent years, there has been a demand for real-time, realistic commentary when watching sporting events. However, professional commentators are not always available, so even general users lack the means to enjoy high-quality commentary. Furthermore, when it comes to realistic commentary that changes in real time based on the user's emotions, conventional systems have difficulty in handling this. Therefore, there is a need for a system that can adapt to the user's emotional state and provide high-quality commentary in real time.

[0891] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0892] In this invention, the server includes server means for collecting training data, server means for preprocessing the training data to train a machine learning model, server means for accepting requests and analyzing video of a sporting event, server means for generating commentary text based on the analysis results, server means for analyzing user emotion data to recognize the user's emotional state, server means for adjusting the tone of the commentary text based on the recognized emotional state, server means for converting the commentary text into narration using a speech synthesis engine, server means for transmitting the narration to the user's terminal, and terminal means for synchronously playing back the video and narration on the user's terminal, thereby enabling real-time, high-quality commentary that is adapted to the user's emotions.

[0893] "Training data" refers to data collected by the server and used to train machine learning models, including, among other things, footage and commentary about sporting events.

[0894] "Server means" means a server designed to perform a specific function, such as data collection, pre-processing, analysis, model training, text generation, speech synthesis, and emotion recognition.

[0895] "Terminal means" refers to a device used by a user, and has the function of receiving data from a server and playing back video and narration audio in sync.

[0896] A "machine learning model" is a collection of algorithms that can perform specific tasks by learning from data, and this system uses a Transformer model.

[0897] A "Transformer model" is a type of machine learning model with advanced pattern recognition and natural language generation capabilities.

[0898] "Emotion data" is data obtained from the user's voice, facial expressions, etc., and is used to analyze the user's emotional state.

[0899] "Emotional state" refers to the psychological state of the user as judged from their tone of voice, facial expressions, and physical movements.

[0900] A "play-by-play text" is a textual description of the situation at a sporting event in natural language, which is generated in a form that can be read or listened to by a user in real time.

[0901] A "speech synthesis engine" is a software or hardware system for converting text data into speech data.

[0902] "Narration voice" is a voice synthesized based on the generated commentary text, and is used to convey information to the user auditorily.

[0903] "Real-time" means that processing occurs and results are provided immediately, without delay.

[0904] MODE FOR CARRYING OUT THE INVENTION

[0905] The present invention is a system that uses AI to provide automatic commentary of sporting events. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides adaptive commentary that corresponds to the user's emotions. This system has a function that links a server and a terminal and provides commentary as audio in real time. Specific embodiments of the system of the present invention are described in detail below.

[0906] Server Roles

[0907] 1. Data Collection

[0908] The server collects a large amount of video and live commentary from the Internet about sporting events such as car races, horse races, and drone races, and uses these as a large-scale learning dataset for video analysis and natural language processing.

[0909] As a concrete example, the server collects F1 race footage and commentary text from YouTube and sports news sites.

[0910] 2. Data Preprocessing

[0911] The server labels the collected video data for important events and actions in each frame, and tokenizes the text data, converting it into a format that can be easily understood by machine learning models, and then stores it.

[0912] As a specific example, labels such as "first lap completed" or "vehicle collision" are added to video frames, and the text data is divided into words.

[0913] 3. Model training

[0914] The server uses the pre-processed data to train advanced algorithms such as Transformer models, which then have the capabilities of video analysis, live text generation, and emotion recognition.

[0915] As a concrete example, video analysis data and tokenized commentary text are input into the Transformer model to enable it to recognize "the scene from the start of a car race to the end of the first lap."

[0916] 4. Saving and updating the model

[0917] The server stores the trained model and periodically updates it to ensure that the latest information is always used for highly accurate analysis.

[0918] For example, new race data is added once a month to retrain the model and reflect the latest racing developments.

[0919] 5. Receiving and Processing Requests

[0920] The server receives requests from users and loads and prepares the relevant trained models.

[0921] As a concrete example, the server loads a "F1 race commentary model" and prepares it for analysis.

[0922] 6. Video Analysis

[0923] The server analyzes the provided video data frame by frame to recognize important events and actions.

[0924] As a specific example, events such as "first lap completed" and "vehicle collision" are detected in real time.

[0925] 7. Emotion recognition

[0926] The server analyzes the voice and facial expression data sent by the user and recognizes the user's emotional state via an emotion engine.

[0927] As a specific example, the emotional state of the user, such as "excited" or "relaxed," is analyzed from the tone of voice and facial expression.

[0928] 8. Live Text Generation

[0929] The server generates commentary text using a generative AI model based on the video analysis results, and adjusts the tone and content of the commentary text based on the emotion recognition results.

[0930] As a specific example, we generate a commentary such as "The first round is over. What an exciting development!"

[0931] 9. Speech Synthesis

[0932] The server sends the generated commentary text to a voice synthesis engine, which generates a narration voice in a tone that matches the user's emotions.

[0933] For example, the tone of the narration can be increased to further excite the user.

[0934] 10. Narration audio transmission

[0935] The server transmits the generated narration voice data to the user's terminal in real time.

[0936] As a specific example, narration audio data is sent to the user's smartphone in streaming format.

[0937] Device Role

[0938] 1. Submit a request

[0939] Users use their devices to request live coverage of a particular sporting event from the server, which in the process provides live video and streaming links, as well as voice and facial expression data for emotion recognition.

[0940] As a specific example, a user sends a request from a smartphone app saying, "Please give me a live report of the current F1 race."

[0941] 2. Data reception and playback

[0942] The terminal receives the narration audio data sent from the server and plays it in sync with the video.

[0943] As a specific example, a smartphone app receives narration audio and plays it simultaneously with the live video.

[0944] This system allows users to enjoy a real-time, emotionally adaptive commentary experience. Here are some example prompts:

[0945] "The first round has now finished. Please give us a running commentary on the next developments."

[0946] "Vehicle collision occurred. Please emphasize the tense situation."

[0947] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0948] Step 1: Data collection

[0949] The server collects a large amount of video and live text about sporting events such as car races, horse races, and drone races from the Internet. Specifically, the server uses a web crawler to download video and text data from video sharing sites and sports news sites. The input is a list of URLs, and the output is video files and text files stored in local storage.

[0950] Step 2: Preprocessing the data

[0951] The server splits the collected video data into frames and labels important events and actions. It tokenizes the text data, converts it into a format that is easy for machine learning models to understand, and stores it. Specifically, the server uses video analysis tools to extract video frames and uses a labeling algorithm to identify important events. The inputs are raw video data and live text data, and the output is labeled video frames and tokenized text data.

[0952] Step 3: Model training

[0953] The server uses the preprocessed data to train a machine learning model using advanced algorithms such as the Transformer model. Specifically, the server trains the model using a deep learning framework (e.g., TensorFlow) on a GPU cluster. The input is the preprocessed data, and the output is a trained machine learning model.

[0954] Step 4: Save and update the model

[0955] The server builds a framework to store the trained model and periodically update it. Specifically, the server saves the model in a database and retrains it whenever new data is added. The input is the trained model and the new data, and the output is the updated model.

[0956] Step 5: Submitting the request

[0957] A user uses a device to request a live broadcast of a specific sporting event from a server. In the process, live video and streaming links are provided, and voice and facial expression data for emotion recognition are also transmitted. Specifically, the user enters the necessary information into a request form in an application on the device and submits it. The input is the user's request and emotion data, and the output is the transmission of the request information.

[0958] Step 6: Receiving and Processing the Request

[0959] The server receives requests from users and loads and prepares the relevant trained models. Specifically, the server analyzes the content of the request and loads the appropriate model into memory. The input is the user request, and the output is the loaded model.

[0960] Step 7: Video Analysis

[0961] The server analyzes the provided video data frame by frame and recognizes important events and actions. Specifically, the server uses a video analysis algorithm to analyze the provided live video frame by frame in real time. The input is the live video data, and the output is recognized event information.

[0962] Step 8: Emotion Recognition

[0963] The server analyzes the voice and facial expression data sent by the user and recognizes the user's emotional state through an emotion engine. Specifically, the server analyzes the user's emotions using voice recognition software and facial recognition algorithms. The input is voice data and facial expression data, and the output is the recognized emotional state.

[0964] Step 9: Generate commentary text

[0965] The server generates commentary text using a generative AI model based on the video analysis results. The tone and content of the commentary text are adjusted based on the emotion recognition results. Specifically, the server sends input data to the generative AI model, which adapts the text content according to the emotional status. The input is event information and emotional state, and the output is commentary text.

[0966] Step 10: Text-to-Speech

[0967] The server sends the generated commentary text to a speech synthesis engine and generates a narration voice in a tone that matches the user's emotions. Specifically, the server creates a narration voice using a text-to-speech synthesis tool. The input is the commentary text, and the output is the narration voice.

[0968] Step 11: Send narration audio

[0969] The server transmits the generated narration audio data to the user's device in real time. Specifically, the server uses a protocol to transfer audio data in streaming format. The input is the narration audio, and the output is data transmission to the user's device.

[0970] Step 12: Data reception and playback

[0971] The device receives the narration audio data sent from the server and plays it in sync with the video. Specifically, the device stores the received audio data in a buffer and synchronizes it with the video playback. The input is the narration audio data, and the output is real-time playback.

[0972] (Application example 2)

[0973] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0974] Conventional production line monitoring systems require a human supervisor to constantly monitor the video feed, which requires a lot of effort and can delay the detection of abnormalities. Furthermore, in environments where a rapid response is required when an abnormality occurs, the supervisor's response may vary depending on their emotional state. If these issues are not resolved, the efficiency and safety of the production line may decrease.

[0975] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0976] In this invention, the server includes a computing device that collects learning data, a computing device that preprocesses the learning data and trains a machine learning model, a computing device that accepts requests and analyzes video of the production line, a computing device that generates alert text corresponding to anomaly detection and the manager's emotions based on the analysis results, a computing device that converts the alert text into narration using a speech synthesis engine, a computing device that sends the narration to the manager's terminal, and a terminal device that synchronously plays back the video and narration on the manager's terminal.This automates production line monitoring and enables adaptive alerts corresponding to the manager's emotions.

[0977] A "computing device" is a device that performs computational tasks such as collecting, analyzing, processing, and transmitting data.

[0978] "Training data" refers to input data for training a machine learning model, and can include video, text, and other forms of data.

[0979] "Preprocessing" is a series of operations that transform training data into a form suitable for analysis and model training.

[0980] A "machine learning model" is an algorithm that learns patterns from data and performs analysis and predictions.

[0981] "Video analytics" is the process of identifying information contained in images or video frames to detect specific events or anomalies.

[0982] "Anomaly detection" is the process of identifying unusual behavior or conditions on a manufacturing line.

[0983] "Emotion recognition" is a technology that determines the emotional state of an administrator from video and audio data.

[0984] An "alert text" is a text message generated to notify you of an abnormality or important event.

[0985] A "speech synthesis engine" is a system that converts text data into voice data.

[0986] "Narration voice" is a voice message generated by a voice synthesis engine.

[0987] "Terminal device" refers to a computer or smart device used by an administrator, and is a device for receiving and playing narration audio and video from the server.

[0988] Learning Phase

[0989] The system of the present invention functions through an initial training phase. During the training phase, a server collects video and text data related to manufacturing processes from the internet. The collected data is preprocessed into a format suitable for analysis and machine learning model training. This preprocessing includes frame-by-frame classification of video data and tokenization of text data. The server then uses the preprocessed data to train Transformer models and other machine learning algorithms to improve anomaly detection and emotion recognition capabilities. The trained models are stored on the server and periodically updated.

[0990] Execution Phase

[0991] Receiving a request

[0992] A user sends a request for monitoring the production line to the server through a terminal used by the manager. The request includes a streaming link of the live video and audio and video data for recognizing the manager's emotions. The terminal then sends this data to the server.

[0993] Video analysis and emotion recognition

[0994] The server analyzes the video data from the user frame by frame, allowing abnormal behavior and conditions to be detected in real time. For example, if the robot arm is not aligned correctly, the abnormality is immediately detected. At the same time, the server analyzes the administrator's voice and facial expression data, and uses an emotion engine to recognize the administrator's emotional state. This takes into account voice tone, facial expressions, body movements, and more.

[0995] Alert text generation and speech synthesis

[0996] The server generates adaptive alert text based on the results of video analysis and emotion recognition. The generated alert text is converted into a voice narration by a speech synthesis engine. This voice is natural and easy to understand, and the tone is adjusted to match the administrator's emotions.

[0997] Sending and playing narration audio

[0998] The server sends the generated narration audio data in real time to the administrator's terminal. The terminal receives the narration audio data sent from the server and plays it in sync with the video. This allows the administrator to respond to any abnormalities in real time while watching the video.

[0999] Specific examples

[1000] For example, if a manager is monitoring a production line and notices that a robot arm is not operating normally, the server will use video analysis to detect the abnormality. At the same time, if the manager is feeling nervous, the emotion engine will recognize this as well. As a result, an alert text is generated stating, "An abnormality has been detected in the robot arm's operation. Please be careful!", and an adaptive voice narration is created by the speech synthesis engine. The narration is sent to the manager's device in real time, allowing the manager to listen to the alert along with the video and respond quickly.

[1001] Prompt Sentence Examples

[1002] "Detect anomalies in the factory in real time and adaptively notify managers. Generate appropriate alerts based on the nature of the anomaly and the manager's sentiment."

[1003] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1004] Step 1: Collect training data

[1005] The server collects video and text data related to the manufacturing process from the internet. Specifically, the server obtains the data using web scraping or APIs. The collected data is used for subsequent analysis and training of machine learning models. The input is publicly available data on the internet, and the output is the collected dataset.

[1006] Step 2: Preprocessing the data

[1007] The server preprocesses the collected training data. Video data is classified by frame, and text data is tokenized. Data preprocessing also includes noise reduction and format conversion. Specific operations include segmenting video frames using OpenCV and tokenizing text using a natural language processing library. The input is the collected dataset, and the output is the preprocessed data.

[1008] Step 3: Train the machine learning model

[1009] The server uses the preprocessed data to train a machine learning model. Transformer models and image recognition models are typically used. Specifically, a multilayer neural network is trained using the dataset to create a model capable of anomaly detection and emotion recognition. The input is the preprocessed data, and the output is the trained model.

[1010] Step 4: Receiving the request

[1011] A user sends a request to monitor the production line to the server through a terminal used by the manager. The terminal then sends a request that includes a streaming link of the live video and audio and video data for recognizing the manager's emotions. Specifically, the user inputs the request using a dedicated application, and the terminal then sends it to the server. The input is the request data from the user, and the output is a confirmation that the request has been received.

[1012] Step 5: Video Analysis

[1013] The server analyzes the received video data frame by frame. Any abnormal behavior or state is detected. Specifically, video analysis is performed using OpenCV and anomaly detection is performed using a machine learning model. For example, if the robot arm is not aligned correctly, the anomaly is immediately detected. The input is the live video data, and the output is the anomaly detection result.

[1014] Step 6: Emotion Recognition

[1015] The server analyzes the received voice and facial expression data and recognizes the administrator's emotional state using an emotion engine. Specifically, voice and facial expression analysis are performed to determine the administrator's emotional state. The input is the user's voice and video data, and the output is the emotion recognition result.

[1016] Step 7: Generate the alert text

[1017] The server generates adaptive alert text based on the video analysis results and emotion recognition results. Specifically, the alert text is generated using a natural language generation model. For example, the text generated might read, "An abnormality has been detected in the robot arm's operation. Please be careful!" The input is the anomaly detection results and emotion recognition results, and the output is the alert text.

[1018] Step 8: Text-to-Speech

[1019] The server converts the generated alert text into a narration voice using a speech synthesis engine. Specifically, it converts the text into an audio file using gTTS. The audio is natural and easy to understand, and is provided in a tone that matches the administrator's emotions. The input is the alert text, and the output is the narration voice data.

[1020] Step 9: Send your narration audio

[1021] The server transmits the generated narration voice data to the administrator's terminal in real time. Specifically, the voice data is transmitted over the network. The input is the narration voice data, and the output is the transmitted voice data.

[1022] Step 10: Playing the narration audio

[1023] The device receives the narration audio data sent from the server and plays it in sync with the video. Specifically, the device receives the audio data and plays it back via an application for synchronous playback. The input is the narration audio data, and the output is the played back narration audio.

[1024] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1025] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1026] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1027] [Third embodiment]

[1028] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1029] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1031] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1032] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1033] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1035] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1036] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1038] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1039] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1040] System Overview

[1041] The present invention is a system that utilizes AI to provide automatic live commentary of sporting events. This system has a function that links a server and a terminal and provides users with real-time live commentary as audio. The specific processing of the program of the system of the present invention is explained below in natural language.

[1042] Learning Phase

[1043] server

[1044] 1. Data Collection

[1045] The server collects a large amount of video and live text from sporting events such as car races, horse races, and drone races from the Internet, which are used as large-scale training datasets for video analysis and natural language processing.

[1046] 2. Data Preprocessing

[1047] The server pre-processes the collected data: video data is labeled for each frame with important events and actions, and text data is tokenized and converted into a format that can be easily understood by machine learning models.

[1048] 3. Model training

[1049] The server uses the pre-processed data to train machine learning models, which are built using advanced algorithms such as Transformer models and trained to analyze video footage and generate commentary for sporting events.

[1050] 4. Save the model

[1051] The server stores the trained model and builds a framework for regularly updating it, enabling highly accurate analysis based on the latest information at all times.

[1052] Commentary Phase

[1053] User

[1054] 1. Submit a request

[1055] A user requests a live broadcast of a particular sporting event from their device, which sends a request for the live broadcast to the server.

[1056] Terminal

[1057] 1. Receiving a request

[1058] The device sends the user's request, which includes the live video or streaming link, to the server for analysis.

[1059] 2. Data reception and playback

[1060] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing the user to watch the video with commentary in real time.

[1061] server

[1062] 1. Request Processing

[1063] When the server receives a request from a user, it loads and prepares the appropriate trained model.

[1064] 2. Video Analysis

[1065] The server analyzes the real-time video provided by the user frame by frame. This analysis recognizes important events and actions. For example, in a car race, events such as "the first lap is over" and "vehicle collision" are detected.

[1066] 3. Live Text Generation

[1067] Based on the analysis results, the server uses a natural language generation model to generate live text, which is adjusted to include appropriate phrases and sentence structures depending on the situation.

[1068] 4. Speech Synthesis

[1069] The server sends the generated commentary text to a speech synthesis engine to generate a narration voice, which is tuned for high quality and fluency.

[1070] 5. Narration audio transmission

[1071] The server transmits the generated narration voice data to the user's terminal in real time.

[1072] Example: Car race commentary

[1073] Learning Phase

[1074] The server collects footage and commentary from past F1 races and rallies, preprocesses it, and then trains a machine learning model.

[1075] Commentary Phase

[1076] 1. User Request

[1077] The user requests a live broadcast of the F1 race currently underway from the terminal to the server.

[1078] 2. Server Processing

[1079] The server analyzes the live video data sent by the user and detects events such as "two cars collide" or "first lap completed."

[1080] Based on the detected events, the server generates commentary text such as "The first lap has now ended. Two cars have collided!" and generates narration audio using a speech synthesis engine.

[1081] The server transmits the generated voice to the user's terminal.

[1082] 3. Terminal Processing

[1083] The user's terminal plays back the live commentary audio data sent from the server in synchronization with the video, providing the user with a real-time commentary.

[1084] This system will enable viewers to enjoy sporting events such as car races, horse races, and drone races smoothly without the need for live commentary by experts. By providing specific commentary in real time, viewers can gain a deeper understanding of the action, providing a more immersive viewing experience.

[1085] The processing flow will be explained below.

[1086] Learning Phase

[1087] server

[1088] Step 1:

[1089] The server collects video and commentary data about sporting events from the Internet, including video footage of car races, horse races, drone races, and other events, along with commentary.

[1090] Step 2:

[1091] The server preprocesses the collected data, labeling the video data with important events and actions for each frame, and tokenizing the text data, converting it into a format that can be easily understood by machine learning models.

[1092] Step 3:

[1093] The server uses the pre-processed data to train machine learning models, which use advanced algorithms such as the Transformer architecture to improve video analysis and play-by-play text generation for sporting events.

[1094] Step 4:

[1095] The server stores the trained model and builds a framework to update the model as needed, enabling highly accurate analysis with the latest information.

[1096] Commentary Phase

[1097] User

[1098] Step 1:

[1099] A user uses his / her terminal to request a live broadcast of a particular sporting event from the server, providing a live video or streaming link.

[1100] Terminal

[1101] Step 2:

[1102] The terminal transmits the user's request and the video data to the server.

[1103] server

[1104] Step 3:

[1105] The server receives requests sent by users and loads the relevant trained models.

[1106] Step 4:

[1107] The server analyzes the provided video data frame by frame, detecting specific actions and events (e.g., "first lap completed," "vehicle collision").

[1108] Step 5:

[1109] The server uses a natural language generation model based on the analysis results to instantly generate live commentary text, such as "The first lap has now finished. Two cars have collided!"

[1110] Step 6:

[1111] The server sends the generated commentary text to a speech synthesis engine to create a narration voice that is natural and easy to understand.

[1112] Step 7:

[1113] The server transmits the generated narration voice data to the user's terminal in real time.

[1114] Terminal

[1115] Step 8:

[1116] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing users to enjoy the commentary in real time along with the video.

[1117] Example: Car race commentary

[1118] Learning Phase

[1119] Step 1:

[1120] The server collects footage and live commentary from past F1 races and rallies.

[1121] Step 2:

[1122] The server preprocesses the collected data, labels the video frames, and tokenizes the text.

[1123] Step 3:

[1124] The server trains a Transformer model with the preprocessed data, giving it high-precision video analysis and commentary generation capabilities.

[1125] Step 4:

[1126] The server stores the trained model and periodically updates it.

[1127] Commentary Phase

[1128] Step 1:

[1129] The user requests a live broadcast of the F1 race currently underway from the server via the terminal.

[1130] Step 2:

[1131] The terminal transmits the user's request and the live video data to the server.

[1132] Step 3:

[1133] The server receives the request and loads a model suitable for F1 racing.

[1134] Step 4:

[1135] The server analyzes the video and detects events such as "first lap completed" or "vehicle collision."

[1136] Step 5:

[1137] The server generates live text: "First lap now completed. Two cars have collided!"

[1138] Step 6:

[1139] The server sends the commentary text to a speech synthesis engine to generate the narration voice.

[1140] Step 7:

[1141] The server transmits the narration voice to the user's terminal.

[1142] Step 8:

[1143] The terminal plays back the narration audio in synchronization with the video, allowing the user to view the video with commentary in real time.

[1144] Example 1

[1145] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1146] Conventional live commentary of sporting events relied on human commentators, which had issues with real-timeness and accuracy. It was also difficult to secure commentators with specialized knowledge. Furthermore, there was a need to provide a realistic live commentary experience for users who could not attend the event in person.

[1147] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1148] In this invention, the server includes: means for collecting large amounts of video and commentary text related to sporting events such as car races, horse races, and drone races from the Internet; means for preprocessing the collected data, labeling important events and actions in each frame of the video data, and tokenizing the text data; means for training a machine learning model using the preprocessed data; means for periodically updating and saving the trained machine learning model; means for receiving requests from users and analyzing real-time sporting event video; means for generating commentary text based on the analysis results; means for converting the generated commentary text into narration using a speech synthesis engine; means for transmitting the narration text to a user's device; and means for synchronously playing back the video and narration text on the user's device. This enables automatic commentary with high real-time accuracy. This allows users without specialized knowledge to enjoy a realistic sports viewing experience, eliminating the need for a commentator.

[1149] "On the Internet" refers to a globally connected information and communications network and infrastructure for sending and receiving data.

[1150] "Car racing" is a competitive event involving automobiles traveling at high speeds, and is a sport in which multiple vehicles compete against each other for ranking.

[1151] "Horse racing" is a competitive event using horses, and is a sport in which multiple horses compete for rankings.

[1152] "Drone racing" is a competitive event using unmanned aerial vehicles (drones), and is a sport in which drones compete for rankings by driving around a course while avoiding obstacles.

[1153] "Footage" means real-time or recorded visual data of a sporting event provided in the form of video.

[1154] "Live commentary text" is a document that verbally explains the progress of a sporting event and important events, and is character data that serves as material for speech synthesis.

[1155] "Server means" refers to a computer system in general that has the function of managing and processing data via a network.

[1156] "Collecting" is the process of obtaining and storing data from the internet or other sources.

[1157] "Preprocessing" refers to the process of converting collected data into a form that is easier to analyze, and specifically includes labeling video frames and tokenizing text data.

[1158] A "machine learning model" is a set of algorithms or programs that can learn from collected data and perform specific tasks automatically.

[1159] A "Transformer model" is a deep learning model that demonstrates excellent performance in natural language processing and machine translation, and has the ability to generate output while taking into account the context of the input data.

[1160] "Analyzing" is the process of analyzing collected or received video or data to detect significant events or actions.

[1161] "Generating" refers to the process of creating new data or information based on the analysis results, and in this case refers to the creation of commentary text.

[1162] A "speech synthesis engine" is a set of software or hardware for converting text data into speech data.

[1163] "Narration voice" is voice data that sounds like a human being speaking, generated by a voice synthesis engine.

[1164] "Terminal means" refers to all electronic devices that are directly operated by users and used to input and output data.

[1165] MODE FOR CARRYING OUT THE INVENTION

[1166] The present invention is a system that uses AI to provide automatic commentary of sporting events. This system has a function that links a server and a terminal and provides users with real-time commentary as audio. Specific embodiments for carrying out the invention are described below.

[1167] server

[1168] The server collects a large amount of video and live text about sporting events from the Internet, using various sources such as YouTube and official sporting sites, and stores the collected data in a large database.

[1169] Next, we preprocess the collected data: for video data, we use a visual information analysis library (e.g., OpenCV) to label important events and actions for each frame, and for text data, we use a natural language processing (NLP) library (e.g., NLTK or SpaCy) to tokenize it.

[1170] Once the preprocessing is complete, the data is used to train a machine learning model. In this invention, advanced algorithms such as the Transformer model are used. This model is trained using machine learning frameworks such as TensorFlow and Scikit-learn. The trained model is periodically retrained with new data and updated to ensure the model's accuracy.

[1171] When a live video request is received from a user, the server loads the appropriate pre-trained model and analyzes the video in real time. Based on the analysis results, a natural language generation model is used to generate live commentary text. This text is then passed to a speech synthesis engine, which converts it into fluent narration audio in real time. For example, a text-to-speech (TTS) engine such as Amazon Polly is used for speech synthesis.

[1172] The narration audio is transmitted to the user's device in real time using protocols such as HTTP Live Streaming (HLS).

[1173] Terminal

[1174] The user's device receives the narration audio data sent from the server and plays it in sync with the live video. The user then watches the video with commentary in real time via a dedicated application or web browser.

[1175] User

[1176] Users request live coverage of a particular sporting event from their device. For example, if they want live coverage of a current F1 race, they click a request button in the application, enter the URL of the live video in the pop-up window, and send it to the server.

[1177] Specific examples

[1178] An example of such a prompt sentence is, "Please provide a live commentary of the F1 race currently being broadcast. Please give a detailed description of the events that occur from the start to the pit stop."

[1179] This system allows viewers to enjoy sporting events such as car races, horse races, and drone races smoothly without the need for live commentary by experts. By providing accurate commentary in real time, viewers can gain a deeper understanding of the action and enjoy a more immersive viewing experience.

[1180] keyword

[1181] Generative AI Models

[1182] Prompt statement

[1183] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1184] Step 1: Data collection

[1185] The server collects video and live commentary of sporting events such as car races, horse races, and drone races from the Internet. The input is a list of URLs on the Internet, and the output is the collected video and text data. Specifically, the server automatically downloads the video and text data from the specified URL list and stores them in a database.

[1186] Step 2: Data Preprocessing

[1187] The server preprocesses the collected data. For video data, it uses a visual information analysis library such as OpenCV to label important events and actions for each frame. For text data, it uses a natural language processing library such as NLTK or SpaCy to tokenize it. The input is the collected video and text data, and the output is preprocessed video frames and text tokens.

[1188] Step 3: Model training

[1189] The server uses the preprocessed data to train a machine learning model. Specifically, it uses a machine learning framework such as TensorFlow or Scikit-learn to train a Transformer model. The input is the preprocessed dataset, and the output is a trained machine learning model. The server uses video and text pairs to batch process the model. It updates the model parameters every epoch and minimizes the loss function.

[1190] Step 4: Save and update the model

[1191] The server saves the trained model in a database and periodically retrains it with new data. The input is the trained model and a new dataset, and the output is the updated model. Specifically, the server saves the parameters of the trained model and schedules periodic retraining with new data.

[1192] Step 5: Receiving the request

[1193] A user requests a live broadcast of a specific sporting event from their device. The input is the URL of the live video specified by the user, and the output is a request sent to the server. The user clicks the request button in the app, enters the URL in the pop-up window, and submits it.

[1194] Step 6: Request Processing

[1195] When the server receives a request from a user, it loads the appropriate trained model. The input is the user request and the URL of the live video, and the output is the loaded trained model. Specifically, the server analyzes the request and determines which model to load.

[1196] Step 7: Video Analysis

[1197] The server analyzes real-time video provided by the user frame by frame. The input is the live video stream, and the output is the analyzed event and action data. Specifically, the server receives the video stream and analyzes each frame using OpenCV.

[1198] Step 8: Live text generation

[1199] The server generates commentary text using a natural language generation model based on the analysis results. The input is the analyzed event data, and the output is the generated commentary text. Specifically, the server passes the analysis results as input to the natural language generation model to generate commentary text.

[1200] Step 9: Text-to-Speech

[1201] The server sends the generated commentary text to a speech synthesis engine to generate a narration voice. The input is the commentary text, and the output is the synthesized narration voice. Specifically, the server sends the generated text to the speech synthesis engine and converts it into voice data.

[1202] Step 10: Send narration audio

[1203] The server transmits the generated narration audio data to the user's terminal. The input is the narration audio, and the output is the transmission of audio data to the user's terminal. Specifically, the server streams the audio file to the terminal.

[1204] Step 11: Data reception and playback

[1205] The device receives the narration audio data sent from the server and plays it in sync with the live video. The input is the narration audio data and the live video, and the output is the synchronized playback of the live video. Specifically, the device receives an HTTP request and launches a player to play the streaming audio data.

[1206] (Application example 1)

[1207] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1208] Modern factories require real-time monitoring of the progress and abnormalities of production lines and robots. However, it is difficult for on-site workers to constantly monitor the situation, and they are unable to respond immediately when an abnormality occurs. Furthermore, visual information alone does not provide a sufficient understanding of the details of the production process, making it difficult to improve efficiency and control quality. To address these issues, the present invention aims to provide a system that provides real-time audio commentary on the progress and abnormalities of production lines and robots.

[1209] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1210] In this invention, the server includes means for collecting training data, means for preprocessing the training data and training a machine learning model, means for accepting requests and analyzing video of a sporting event or a production line, means for generating commentary text based on the analysis results, means for converting the commentary text into narration audio using a speech synthesis engine, means for transmitting the narration audio to a user's terminal, means for synchronously playing back the video and narration audio on the user's terminal, and means for providing real-time commentary on the progress and abnormalities of the production line or robot. This enables real-time audio commentary on the status of the manufacturing process in a factory, allowing workers to perform their work efficiently.

[1211] The "server means" is a computer system that controls the functions of data collection, preprocessing, analysis, generation, synthesis, and transmission, and communicates with the user's terminal.

[1212] "Training Data" is a data set collected by the Server Means and used to train a Machine Learning Model.

[1213] "Preprocessing" is the process of converting collected training data into a format that is easy for machine learning models to understand, using techniques such as tokenization and labeling.

[1214] A "machine learning model" is an algorithm, particularly a Transformer model, that is trained on training data to perform a specific task.

[1215] A "request" is a communication signal that a user uses from his / her own terminal to request a live report of a specific event from the server means.

[1216] "Video analytics" is the process of analyzing real-time video data to detect important events or actions.

[1217] "Live text" is explanatory text generated based on the results of video analysis, which is converted into voice by a voice synthesis engine.

[1218] A "speech synthesis engine" is software or hardware for converting text data into narration voice.

[1219] The "terminal means" is a device used by a user, and has the function of playing back the narration audio and video transmitted from the server means in synchronization.

[1220] A "production line" is a system that shows the machines and work flow arranged to carry out production activities within a factory.

[1221] A "robot" is a mechanical device that performs tasks automatically or semi-automatically in a factory.

[1222] "Real-time" means that processing occurs immediately, with virtually no delay.

[1223] "Audio commentary" is the process of communicating analyzed information to the user in real time by voice.

[1224] "Progress" refers to the state and speed at which a production line or robot is working.

[1225] An "abnormality" refers to a state that deviates from normal operation, or a malfunction such as an error or breakdown.

[1226] The present invention provides a system that provides real-time audio commentary on the progress and abnormalities of manufacturing lines and robots. This system consists of three main elements: a server, a terminal, and a user.

[1227] server

[1228] 1. Data Collection and Learning

[1229] The server collects actual video and text data about manufacturing lines and robots collected from the Internet. This data is used as learning data to train machine learning models. Specific hardware used is a high-performance server.

[1230] 2. Data Preprocessing

[1231] The collected data is preprocessed by the server. For example, video data is broken down into frames and labeled. Text data is tokenized and converted into a format that is easy for machine learning models to understand. Open source software such as TensorFlow and OpenCV is used.

[1232] 3. Model training

[1233] Using the pre-processed data, the server trains machine learning models, built using advanced algorithms such as the Transformer model, capable of analyzing video footage of production lines and robots and generating live text.

[1234] 4. Model application and analysis

[1235] The server analyzes the video of the production line based on user requests. Video analysis includes real-time event detection. For example, it detects actions such as "the production line has stopped" or "a robot has encountered an error." The software used is TensorFlow and OpenCV.

[1236] 5. Live Text Generation and Speech Synthesis

[1237] Based on the analysis results, the server generates commentary text using a natural language generation model (such as GPT-3), which is then converted into narration audio by a speech synthesis engine (Amazon Polly, Google Text-to-Speech).

[1238] 6. Sending data to the user device

[1239] Once the narration voice data is generated, the server transmits it to the user's terminal in real time.

[1240] Terminal

[1241] 1. Data reception and synchronous playback

[1242] The user's device (such as smart glasses) receives the narration audio data sent from the server and plays it in sync with the video. This function allows the user to understand the status of the production line and robots in real time with audio commentary.

[1243] User

[1244] 1. Submit a request

[1245] Users can request live footage of a specific manufacturing line or robot from a terminal, which can include live video or streaming links.

[1246] Specific examples

[1247] For example, if a user sends a request from a terminal about the progress of a production line, the server generates a prompt sentence such as the following:

[1248] Example prompt sentence:

[1249] "The production line has stopped. What is the next action?"

[1250] In this way, by linking the server, terminal, and user, it becomes possible to provide real-time audio updates on the progress and abnormalities of the production line and robots, allowing workers to carry out their work efficiently.

[1251] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1252] Step 1: Data collection

[1253] The server collects large amounts of video and text data related to manufacturing lines and robots from the Internet. This data is used as a training dataset. The input is raw data related to the manufacturing process, and the output is the collected dataset. Specifically, the data is downloaded using a web scraping tool or API.

[1254] Step 2: Data Preprocessing

[1255] The server preprocesses the collected data. Specifically, the video data is decomposed into frames and important events are labeled. The text data is tokenized and formatted. The input is the collected video data and text data, and the output is the preprocessed data. Specifically, frame decomposition is performed using OpenCV and labeling is performed using TensorFlow.

[1256] Step 3: Model training

[1257] The server uses the preprocessed data to train a machine learning model. This model can then analyze video of production lines and robots and generate live text. The specific method used is a Transformer model. The input is the preprocessed data, and the output is the trained model. The specific operation is to train the Transformer model using the TensorFlow API.

[1258] Step 4: Submitting the request

[1259] A user requests a live report of a specific manufacturing line or robot from a terminal to a server. This request may include live video or a streaming link. The input is the user's request, and the output is a communication signal to the server. Specifically, the request is sent from the terminal's UI to the server.

[1260] Step 5: Video Analysis

[1261] The server analyzes the real-time video provided by the user frame by frame. This analysis recognizes important events and actions. For example, events such as "the production line has stopped" or "an error has occurred in a robot" can be detected. The input is the live video data, and the output is the analysis results. Specifically, the video analysis is performed using TensorFlow.

[1262] Step 6: Generate commentary text

[1263] The server generates commentary text using a natural language generation model based on the analysis results. The generated text has appropriate phrases and sentence structures depending on the situation. The input is the analysis results, and the output is the commentary text. Specifically, text generation is performed using GPT-3.

[1264] Step 7: Text-to-Speech

[1265] The server sends the generated commentary text to a speech synthesis engine to generate narration audio. This audio is high quality and fluent. The input is the commentary text, and the output is the narration audio data. Specifically, speech synthesis is performed using the Google Text-to-Speech API.

[1266] Step 8: Send narration audio

[1267] The server transmits the generated narration audio data to the user's device in real time. The input is the narration audio data, and the output is the audio data sent to the user's device. Specifically, the audio data is transmitted using a network protocol (e.g., WebSocket).

[1268] Step 9: Data reception and synchronous playback

[1269] The user's device receives the narration audio data sent from the server and plays it back in sync with the video. This function allows the user to understand the status of the production line or robot in real time with audio commentary. The input is the audio data sent from the server, and the output is synchronous playback of the video and audio. Specifically, the device's audio playback function is used to synchronize with the video data.

[1270] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1271] System Overview

[1272] The present invention is a system that utilizes AI to provide automatic commentary of sporting events. In addition, by combining it with an emotion engine that recognizes the user's emotions, it provides adaptive commentary that corresponds to the user's emotions. The server and terminals work together to provide commentary in the form of audio in real time. The specific processing of the program of the system of the present invention is explained below in natural language.

[1273] Learning Phase

[1274] server

[1275] 1. Data Collection

[1276] The server collects a large amount of video and live text from sporting events such as car races, horse races, and drone races from the Internet, which are used as large-scale training datasets for video analysis and natural language processing.

[1277] 2. Data Preprocessing

[1278] The server preprocesses the collected data: it labels important events and actions for each frame of video data, tokenizes text data and converts it into a format that machine learning models can easily understand, and collects and processes audio and facial expression data for the emotion engine.

[1279] 3. Model training

[1280] The server uses the pre-processed data to train machine learning models, which use advanced algorithms such as Transformer models, and train them to perform video analysis, live text generation, and emotion recognition.

[1281] 4. Save the model

[1282] The server will store the trained model and create a framework for periodic updates, enabling highly accurate analysis using the latest information.

[1283] Commentary Phase

[1284] User

[1285] 1. Submit a request

[1286] Users can use their devices to request live coverage of a particular sporting event from the server, providing live video or a streaming link, and voice and facial expression data is also collected to recognize the user's emotions.

[1287] Terminal

[1288] 1. Receiving a request

[1289] The terminal transmits the user's request, video data, and emotion data to the server.

[1290] server

[1291] 1. Request Processing

[1292] The server receives requests from users and loads and prepares the relevant trained models.

[1293] 2. Video Analysis

[1294] The server analyzes the provided video data frame by frame. This analysis allows it to recognize important events and actions. For example, in the case of a car race, events such as "the first lap is over" and "vehicle collision" can be detected.

[1295] 3. Emotion recognition

[1296] The server analyzes the voice and facial expression data sent by the user and uses an emotion engine to recognize the user's emotional state, including the user's tone of voice, facial expressions, and body movements.

[1297] 4. Live Text Generation

[1298] The server generates commentary text using a natural language generation model based on the video analysis results. The content and tone of the commentary text are adjusted according to the user's emotional state as recognized by the emotion engine. For example, if the user is excited, the server generates text that maintains the same sense of excitement.

[1299] 5. Speech Synthesis

[1300] The server sends the generated commentary text to a speech synthesis engine to create a narration voice, which is natural and easy to understand, and is provided in a tone that matches the user's emotions.

[1301] 6. Narration audio transmission

[1302] The server transmits the generated narration voice data to the user's terminal in real time.

[1303] Terminal

[1304] 1. Data reception and playback

[1305] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing users to enjoy real-time commentary based on their emotions along with the video.

[1306] Example: Car race commentary

[1307] Learning Phase

[1308] 1. Data Collection and Preprocessing

[1309] The server collects and preprocesses video footage from past F1 races and rallies, commentary text, and user emotion data, labeling video frames, tokenizing text, and processing audio and facial expression data for the emotion engine.

[1310] 2. Model training

[1311] The server trains a Transformer model on the preprocessed data, giving it high-precision video analysis, commentary generation, and emotion recognition capabilities.

[1312] 3. Save the model

[1313] The server stores the trained model and periodically updates it.

[1314] Commentary Phase

[1315] 1. User Request

[1316] The user requests a live report of the current F1 race from the server via their device, and data based on the user's emotional state is also collected at the same time.

[1317] 2. Sending and Receiving Devices

[1318] The device transmits the user's request, live video data, and emotion data to the server, and receives the live audio data sent from the server and plays it in sync with the video.

[1319] 3. Server Processing

[1320] The server receives the request, analyzes the video, and the emotion engine recognizes the emotional state. Events such as "First lap completed" and "Vehicle collision" are detected, and based on the user's emotional state, a live commentary text is generated, such as "The first lap has now completed. This is exciting! Two cars have collided!"

[1321] The server sends the commentary text to a speech synthesis engine to generate a narration voice that maintains an uplifting tone.

[1322] The server transmits the generated narration voice to the user's terminal.

[1323] 4. Playback of the device

[1324] The user's terminal plays back the narration audio in synchronization with the video, allowing the user to view a realistic commentary in real time.

[1325] This system allows viewers to watch sporting events in a way that adapts to their emotions, enabling real-time, adaptive commentary without the need for expert commentary, improving the sense of realism and entertainment, and making it easier to attract viewers' attention.

[1326] The processing flow will be explained below.

[1327] Learning Phase

[1328] server

[1329] Step 1:

[1330] The server collects videos, commentary text data, and user emotion data related to sporting events from the Internet. This includes videos of car races, horse races, drone races, and other events, along with commentary text and user voice and facial expression data.

[1331] Step 2:

[1332] The server preprocesses the collected data: it labels important events and actions for each frame of video data, tokenizes text data and converts it into a format that machine learning models can easily understand, and adds emotion labels from audio and facial expression data for the emotion engine.

[1333] Step 3:

[1334] The server uses the pre-processed data to train machine learning models, which are trained using advanced algorithms such as Transformer models to analyze video footage of sporting events, generate commentary, and recognize emotions.

[1335] Step 4:

[1336] The server stores the trained model and builds a framework to periodically update the model as needed, enabling highly accurate analysis using the latest information.

[1337] Commentary Phase

[1338] User

[1339] Step 1:

[1340] Users use their devices to request a live broadcast of a particular sporting event from the server, providing a live video or streaming link, and voice and facial expression data is also collected for emotion recognition.

[1341] Terminal

[1342] Step 2:

[1343] The terminal transmits the user's request, live video data, and emotion data to the server.

[1344] server

[1345] Step 3:

[1346] The server receives requests sent by users and loads the relevant trained models.

[1347] Step 4:

[1348] The server analyzes the received video data frame by frame to detect important events and actions (e.g., "first lap completed" or "vehicle collision").

[1349] Step 5:

[1350] The server analyzes the voice and facial expression data sent by the user and recognizes the user's emotional state (e.g., excitement, surprise) using an emotion engine.

[1351] Step 6:

[1352] The server generates commentary text using a natural language generation model based on the video analysis and emotion recognition results, and adjusts the content and tone of the commentary text to reflect the user's emotions.

[1353] Step 7:

[1354] The server sends the generated commentary text to a speech synthesis engine to create a narration voice, which is provided in a tone that matches the user's emotions.

[1355] Step 8:

[1356] The server transmits the generated narration voice data to the user's terminal in real time.

[1357] Terminal

[1358] Step 9:

[1359] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing users to enjoy real-time commentary based on their emotions along with the video.

[1360] Example: Car race commentary

[1361] Learning Phase

[1362] Step 1:

[1363] The server collects footage of past F1 races and rally competitions, commentary text, and user emotion data.

[1364] Step 2:

[1365] The server preprocesses the collected data, labels video frames, tokenizes text, and assigns emotion labels to audio and facial expression data for the emotion engine.

[1366] Step 3:

[1367] The server trains a Transformer model on the preprocessed data, giving it high-precision video analysis, commentary generation, and emotion recognition capabilities.

[1368] Step 4:

[1369] The server stores the trained model and periodically updates it.

[1370] Commentary Phase

[1371] Step 1:

[1372] The user requests a live report of the current F1 race from the server via their device, and data based on the user's emotional state is also collected at the same time.

[1373] Step 2:

[1374] The terminal transmits the user's request, live video data, and emotion data to the server.

[1375] Step 3:

[1376] The server receives the request and loads a model suitable for F1 racing.

[1377] Step 4:

[1378] The server analyzes the video and detects events such as "first lap completed" or "vehicle collision."

[1379] Step 5:

[1380] The server analyzes the user's voice and facial expression data, and the emotion engine recognizes the user's emotional state (e.g., the user is excited).

[1381] Step 6:

[1382] Based on the video analysis and emotion recognition results, the server generates live text such as, "The first lap is now complete. This is very exciting! Two cars have collided!"

[1383] Step 7:

[1384] The server sends the commentary text to a speech synthesis engine to generate a narration voice with a tone that reflects excitement.

[1385] Step 8:

[1386] The server transmits the generated narration voice to the user's terminal.

[1387] Step 9:

[1388] The terminal plays back the narration audio in synchronization with the video, allowing the user to view a realistic commentary in real time.

[1389] Example 2

[1390] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1391] In recent years, there has been a demand for real-time, realistic commentary when watching sporting events. However, professional commentators are not always available, so even general users lack the means to enjoy high-quality commentary. Furthermore, when it comes to realistic commentary that changes in real time based on the user's emotions, conventional systems have difficulty in handling this. Therefore, there is a need for a system that can adapt to the user's emotional state and provide high-quality commentary in real time.

[1392] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1393] In this invention, the server includes server means for collecting training data, server means for preprocessing the training data to train a machine learning model, server means for accepting requests and analyzing video of a sporting event, server means for generating commentary text based on the analysis results, server means for analyzing user emotion data to recognize the user's emotional state, server means for adjusting the tone of the commentary text based on the recognized emotional state, server means for converting the commentary text into narration using a speech synthesis engine, server means for transmitting the narration to the user's terminal, and terminal means for synchronously playing back the video and narration on the user's terminal, thereby enabling real-time, high-quality commentary that is adapted to the user's emotions.

[1394] "Training data" refers to data collected by the server and used to train machine learning models, including, among other things, footage and commentary about sporting events.

[1395] "Server means" means a server designed to perform a specific function, such as data collection, pre-processing, analysis, model training, text generation, speech synthesis, and emotion recognition.

[1396] "Terminal means" refers to a device used by a user, and has the function of receiving data from a server and playing back video and narration audio in sync.

[1397] A "machine learning model" is a collection of algorithms that can perform specific tasks by learning from data, and this system uses a Transformer model.

[1398] A "Transformer model" is a type of machine learning model with advanced pattern recognition and natural language generation capabilities.

[1399] "Emotion data" is data obtained from the user's voice, facial expressions, etc., and is used to analyze the user's emotional state.

[1400] "Emotional state" refers to the psychological state of the user as judged from their tone of voice, facial expressions, and physical movements.

[1401] A "play-by-play text" is a textual description of the situation at a sporting event in natural language, which is generated in a form that can be read or listened to by a user in real time.

[1402] A "speech synthesis engine" is a software or hardware system for converting text data into speech data.

[1403] "Narration voice" is a voice synthesized based on the generated commentary text, and is used to convey information to the user auditorily.

[1404] "Real-time" means that processing occurs and results are provided immediately, without delay.

[1405] MODE FOR CARRYING OUT THE INVENTION

[1406] The present invention is a system that uses AI to provide automatic commentary of sporting events. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides adaptive commentary that corresponds to the user's emotions. This system has a function that links a server and a terminal and provides commentary as audio in real time. Specific embodiments of the system of the present invention are described in detail below.

[1407] Server Roles

[1408] 1. Data Collection

[1409] The server collects a large amount of video and live commentary from the Internet about sporting events such as car races, horse races, and drone races, and uses these as a large-scale learning dataset for video analysis and natural language processing.

[1410] As a concrete example, the server collects F1 race footage and commentary text from YouTube and sports news sites.

[1411] 2. Data Preprocessing

[1412] The server labels the collected video data for important events and actions in each frame, and tokenizes the text data, converting it into a format that can be easily understood by machine learning models, and then stores it.

[1413] As a specific example, labels such as "first lap completed" or "vehicle collision" are added to video frames, and the text data is divided into words.

[1414] 3. Model training

[1415] The server uses the pre-processed data to train advanced algorithms such as Transformer models, which then have the capabilities of video analysis, live text generation, and emotion recognition.

[1416] As a concrete example, video analysis data and tokenized commentary text are input into the Transformer model to enable it to recognize "the scene from the start of a car race to the end of the first lap."

[1417] 4. Saving and updating the model

[1418] The server stores the trained model and periodically updates it to ensure that the latest information is always used for highly accurate analysis.

[1419] For example, new race data is added once a month to retrain the model and reflect the latest racing developments.

[1420] 5. Receiving and Processing Requests

[1421] The server receives requests from users and loads and prepares the relevant trained models.

[1422] As a concrete example, the server loads a "F1 race commentary model" and prepares it for analysis.

[1423] 6. Video Analysis

[1424] The server analyzes the provided video data frame by frame to recognize important events and actions.

[1425] As a specific example, events such as "first lap completed" and "vehicle collision" are detected in real time.

[1426] 7. Emotion recognition

[1427] The server analyzes the voice and facial expression data sent by the user and recognizes the user's emotional state via an emotion engine.

[1428] As a specific example, the emotional state of the user, such as "excited" or "relaxed," is analyzed from the tone of voice and facial expression.

[1429] 8. Live Text Generation

[1430] The server generates commentary text using a generative AI model based on the video analysis results, and adjusts the tone and content of the commentary text based on the emotion recognition results.

[1431] As a specific example, we generate a commentary such as "The first round is over. What an exciting development!"

[1432] 9. Speech Synthesis

[1433] The server sends the generated commentary text to a voice synthesis engine, which generates a narration voice in a tone that matches the user's emotions.

[1434] For example, the tone of the narration can be increased to further excite the user.

[1435] 10. Narration audio transmission

[1436] The server transmits the generated narration voice data to the user's terminal in real time.

[1437] As a specific example, narration audio data is sent to the user's smartphone in streaming format.

[1438] Device Role

[1439] 1. Submit a request

[1440] Users use their devices to request live coverage of a particular sporting event from the server, which in the process provides live video and streaming links, as well as voice and facial expression data for emotion recognition.

[1441] As a specific example, a user sends a request from a smartphone app saying, "Please give me a live report of the current F1 race."

[1442] 2. Data reception and playback

[1443] The terminal receives the narration audio data sent from the server and plays it in sync with the video.

[1444] As a specific example, a smartphone app receives narration audio and plays it simultaneously with the live video.

[1445] This system allows users to enjoy a real-time, emotionally adaptive commentary experience. Here are some example prompts:

[1446] "The first round has now finished. Please give us a running commentary on the next developments."

[1447] "Vehicle collision occurred. Please emphasize the tense situation."

[1448] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1449] Step 1: Data collection

[1450] The server collects a large amount of video and live text about sporting events such as car races, horse races, and drone races from the Internet. Specifically, the server uses a web crawler to download video and text data from video sharing sites and sports news sites. The input is a list of URLs, and the output is video files and text files stored in local storage.

[1451] Step 2: Preprocessing the data

[1452] The server splits the collected video data into frames and labels important events and actions. It tokenizes the text data, converts it into a format that is easy for machine learning models to understand, and stores it. Specifically, the server uses video analysis tools to extract video frames and uses a labeling algorithm to identify important events. The inputs are raw video data and live text data, and the output is labeled video frames and tokenized text data.

[1453] Step 3: Model training

[1454] The server uses the preprocessed data to train a machine learning model using advanced algorithms such as the Transformer model. Specifically, the server trains the model using a deep learning framework (e.g., TensorFlow) on a GPU cluster. The input is the preprocessed data, and the output is a trained machine learning model.

[1455] Step 4: Save and update the model

[1456] The server builds a framework to store the trained model and periodically update it. Specifically, the server saves the model in a database and retrains it whenever new data is added. The input is the trained model and the new data, and the output is the updated model.

[1457] Step 5: Submitting the request

[1458] A user uses a device to request a live broadcast of a specific sporting event from a server. In the process, live video and streaming links are provided, and voice and facial expression data for emotion recognition are also transmitted. Specifically, the user enters the necessary information into a request form in an application on the device and submits it. The input is the user's request and emotion data, and the output is the transmission of the request information.

[1459] Step 6: Receiving and Processing the Request

[1460] The server receives requests from users and loads and prepares the relevant trained models. Specifically, the server analyzes the content of the request and loads the appropriate model into memory. The input is the user request, and the output is the loaded model.

[1461] Step 7: Video Analysis

[1462] The server analyzes the provided video data frame by frame and recognizes important events and actions. Specifically, the server uses a video analysis algorithm to analyze the provided live video frame by frame in real time. The input is the live video data, and the output is recognized event information.

[1463] Step 8: Emotion Recognition

[1464] The server analyzes the voice and facial expression data sent by the user and recognizes the user's emotional state through an emotion engine. Specifically, the server analyzes the user's emotions using voice recognition software and facial recognition algorithms. The input is voice data and facial expression data, and the output is the recognized emotional state.

[1465] Step 9: Generate commentary text

[1466] The server generates commentary text using a generative AI model based on the video analysis results. The tone and content of the commentary text are adjusted based on the emotion recognition results. Specifically, the server sends input data to the generative AI model, which adapts the text content according to the emotional status. The input is event information and emotional state, and the output is commentary text.

[1467] Step 10: Text-to-Speech

[1468] The server sends the generated commentary text to a speech synthesis engine and generates a narration voice in a tone that matches the user's emotions. Specifically, the server creates a narration voice using a text-to-speech synthesis tool. The input is the commentary text, and the output is the narration voice.

[1469] Step 11: Send narration audio

[1470] The server transmits the generated narration audio data to the user's device in real time. Specifically, the server uses a protocol to transfer audio data in streaming format. The input is the narration audio, and the output is data transmission to the user's device.

[1471] Step 12: Data reception and playback

[1472] The device receives the narration audio data sent from the server and plays it in sync with the video. Specifically, the device stores the received audio data in a buffer and synchronizes it with the video playback. The input is the narration audio data, and the output is real-time playback.

[1473] (Application example 2)

[1474] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1475] Conventional production line monitoring systems require a human supervisor to constantly monitor the video feed, which requires a lot of effort and can delay the detection of abnormalities. Furthermore, in environments where a rapid response is required when an abnormality occurs, the supervisor's response may vary depending on their emotional state. If these issues are not resolved, the efficiency and safety of the production line may decrease.

[1476] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1477] In this invention, the server includes a computing device that collects learning data, a computing device that preprocesses the learning data and trains a machine learning model, a computing device that accepts requests and analyzes video of the production line, a computing device that generates alert text corresponding to anomaly detection and the manager's emotions based on the analysis results, a computing device that converts the alert text into narration using a speech synthesis engine, a computing device that sends the narration to the manager's terminal, and a terminal device that synchronously plays back the video and narration on the manager's terminal.This automates production line monitoring and enables adaptive alerts corresponding to the manager's emotions.

[1478] A "computing device" is a device that performs computational tasks such as collecting, analyzing, processing, and transmitting data.

[1479] "Training data" refers to input data for training a machine learning model, and can include video, text, and other forms of data.

[1480] "Preprocessing" is a series of operations that transform training data into a form suitable for analysis and model training.

[1481] A "machine learning model" is an algorithm that learns patterns from data and performs analysis and predictions.

[1482] "Video analytics" is the process of identifying information contained in images or video frames to detect specific events or anomalies.

[1483] "Anomaly detection" is the process of identifying unusual behavior or conditions on a manufacturing line.

[1484] "Emotion recognition" is a technology that determines the emotional state of an administrator from video and audio data.

[1485] An "alert text" is a text message generated to notify you of an abnormality or important event.

[1486] A "speech synthesis engine" is a system that converts text data into voice data.

[1487] "Narration voice" is a voice message generated by a voice synthesis engine.

[1488] "Terminal device" refers to a computer or smart device used by an administrator, and is a device for receiving and playing narration audio and video from the server.

[1489] Learning Phase

[1490] The system of the present invention functions through an initial training phase. During the training phase, a server collects video and text data related to manufacturing processes from the internet. The collected data is preprocessed into a format suitable for analysis and machine learning model training. This preprocessing includes frame-by-frame classification of video data and tokenization of text data. The server then uses the preprocessed data to train Transformer models and other machine learning algorithms to improve anomaly detection and emotion recognition capabilities. The trained models are stored on the server and periodically updated.

[1491] Execution Phase

[1492] Receiving a request

[1493] A user sends a request for monitoring the production line to the server through a terminal used by the manager. The request includes a streaming link of the live video and audio and video data for recognizing the manager's emotions. The terminal then sends this data to the server.

[1494] Video analysis and emotion recognition

[1495] The server analyzes the video data from the user frame by frame, allowing abnormal behavior and conditions to be detected in real time. For example, if the robot arm is not aligned correctly, the abnormality is immediately detected. At the same time, the server analyzes the administrator's voice and facial expression data, and uses an emotion engine to recognize the administrator's emotional state. This takes into account voice tone, facial expressions, body movements, and more.

[1496] Alert text generation and speech synthesis

[1497] The server generates adaptive alert text based on the results of video analysis and emotion recognition. The generated alert text is converted into a voice narration by a speech synthesis engine. This voice is natural and easy to understand, and the tone is adjusted to match the administrator's emotions.

[1498] Sending and playing narration audio

[1499] The server sends the generated narration audio data in real time to the administrator's terminal. The terminal receives the narration audio data sent from the server and plays it in sync with the video. This allows the administrator to respond to any abnormalities in real time while watching the video.

[1500] Specific examples

[1501] For example, if a manager is monitoring a production line and notices that a robot arm is not operating normally, the server will use video analysis to detect the abnormality. At the same time, if the manager is feeling nervous, the emotion engine will recognize this as well. As a result, an alert text is generated stating, "An abnormality has been detected in the robot arm's operation. Please be careful!", and an adaptive voice narration is created by the speech synthesis engine. The narration is sent to the manager's device in real time, allowing the manager to listen to the alert along with the video and respond quickly.

[1502] Prompt Sentence Examples

[1503] "Detect anomalies in the factory in real time and adaptively notify managers. Generate appropriate alerts based on the nature of the anomaly and the manager's sentiment."

[1504] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1505] Step 1: Collect training data

[1506] The server collects video and text data related to the manufacturing process from the internet. Specifically, the server obtains the data using web scraping or APIs. The collected data is used for subsequent analysis and training of machine learning models. The input is publicly available data on the internet, and the output is the collected dataset.

[1507] Step 2: Preprocessing the data

[1508] The server preprocesses the collected training data. Video data is classified by frame, and text data is tokenized. Data preprocessing also includes noise reduction and format conversion. Specific operations include segmenting video frames using OpenCV and tokenizing text using a natural language processing library. The input is the collected dataset, and the output is the preprocessed data.

[1509] Step 3: Train the machine learning model

[1510] The server uses the preprocessed data to train a machine learning model. Transformer models and image recognition models are typically used. Specifically, a multilayer neural network is trained using the dataset to create a model capable of anomaly detection and emotion recognition. The input is the preprocessed data, and the output is the trained model.

[1511] Step 4: Receiving the request

[1512] A user sends a request to monitor the production line to the server through a terminal used by the manager. The terminal then sends a request that includes a streaming link of the live video and audio and video data for recognizing the manager's emotions. Specifically, the user inputs the request using a dedicated application, and the terminal then sends it to the server. The input is the request data from the user, and the output is a confirmation that the request has been received.

[1513] Step 5: Video Analysis

[1514] The server analyzes the received video data frame by frame. Any abnormal behavior or state is detected. Specifically, video analysis is performed using OpenCV and anomaly detection is performed using a machine learning model. For example, if the robot arm is not aligned correctly, the anomaly is immediately detected. The input is the live video data, and the output is the anomaly detection result.

[1515] Step 6: Emotion Recognition

[1516] The server analyzes the received voice and facial expression data and recognizes the administrator's emotional state using an emotion engine. Specifically, voice and facial expression analysis are performed to determine the administrator's emotional state. The input is the user's voice and video data, and the output is the emotion recognition result.

[1517] Step 7: Generate the alert text

[1518] The server generates adaptive alert text based on the video analysis results and emotion recognition results. Specifically, the alert text is generated using a natural language generation model. For example, the text generated might read, "An abnormality has been detected in the robot arm's operation. Please be careful!" The input is the anomaly detection results and emotion recognition results, and the output is the alert text.

[1519] Step 8: Text-to-Speech

[1520] The server converts the generated alert text into a narration voice using a speech synthesis engine. Specifically, it converts the text into an audio file using gTTS. The audio is natural and easy to understand, and is provided in a tone that matches the administrator's emotions. The input is the alert text, and the output is the narration voice data.

[1521] Step 9: Send your narration audio

[1522] The server transmits the generated narration voice data to the administrator's terminal in real time. Specifically, the voice data is transmitted over the network. The input is the narration voice data, and the output is the transmitted voice data.

[1523] Step 10: Playing the narration audio

[1524] The device receives the narration audio data sent from the server and plays it in sync with the video. Specifically, the device receives the audio data and plays it back via an application for synchronous playback. The input is the narration audio data, and the output is the played back narration audio.

[1525] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1526] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1527] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1528] [Fourth embodiment]

[1529] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1530] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1531] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1532] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1533] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1534] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1535] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1536] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1537] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1538] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1539] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1540] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1541] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1542] System Overview

[1543] The present invention is a system that utilizes AI to provide automatic live commentary of sporting events. This system has a function that links a server and a terminal and provides users with real-time live commentary as audio. The specific processing of the program of the system of the present invention is explained below in natural language.

[1544] Learning Phase

[1545] server

[1546] 1. Data Collection

[1547] The server collects a large amount of video and live text from sporting events such as car races, horse races, and drone races from the Internet, which are used as large-scale training datasets for video analysis and natural language processing.

[1548] 2. Data Preprocessing

[1549] The server pre-processes the collected data: video data is labeled for each frame with important events and actions, and text data is tokenized and converted into a format that can be easily understood by machine learning models.

[1550] 3. Model training

[1551] The server uses the pre-processed data to train machine learning models, which are built using advanced algorithms such as Transformer models and trained to analyze video footage and generate commentary for sporting events.

[1552] 4. Save the model

[1553] The server stores the trained model and builds a framework for regularly updating it, enabling highly accurate analysis based on the latest information at all times.

[1554] Commentary Phase

[1555] User

[1556] 1. Submit a request

[1557] A user requests a live broadcast of a particular sporting event from their device, which sends a request for the live broadcast to the server.

[1558] Terminal

[1559] 1. Receiving a request

[1560] The device sends the user's request, which includes the live video or streaming link, to the server for analysis.

[1561] 2. Data reception and playback

[1562] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing the user to watch the video with commentary in real time.

[1563] server

[1564] 1. Request Processing

[1565] When the server receives a request from a user, it loads and prepares the appropriate trained model.

[1566] 2. Video Analysis

[1567] The server analyzes the real-time video provided by the user frame by frame. This analysis recognizes important events and actions. For example, in a car race, events such as "the first lap is over" and "vehicle collision" are detected.

[1568] 3. Live Text Generation

[1569] Based on the analysis results, the server uses a natural language generation model to generate live text, which is adjusted to include appropriate phrases and sentence structures depending on the situation.

[1570] 4. Speech Synthesis

[1571] The server sends the generated commentary text to a speech synthesis engine to generate a narration voice, which is tuned for high quality and fluency.

[1572] 5. Narration audio transmission

[1573] The server transmits the generated narration voice data to the user's terminal in real time.

[1574] Example: Car race commentary

[1575] Learning Phase

[1576] The server collects footage and commentary from past F1 races and rallies, preprocesses it, and then trains a machine learning model.

[1577] Commentary Phase

[1578] 1. User Request

[1579] The user requests a live broadcast of the F1 race currently underway from the terminal to the server.

[1580] 2. Server Processing

[1581] The server analyzes the live video data sent by the user and detects events such as "two cars collide" or "first lap completed."

[1582] Based on the detected events, the server generates commentary text such as "The first lap has now ended. Two cars have collided!" and generates narration audio using a speech synthesis engine.

[1583] The server transmits the generated voice to the user's terminal.

[1584] 3. Terminal Processing

[1585] The user's terminal plays back the live commentary audio data sent from the server in synchronization with the video, providing the user with a real-time commentary.

[1586] This system will enable viewers to enjoy sporting events such as car races, horse races, and drone races smoothly without the need for live commentary by experts. By providing specific commentary in real time, viewers can gain a deeper understanding of the action, providing a more immersive viewing experience.

[1587] The processing flow will be explained below.

[1588] Learning Phase

[1589] server

[1590] Step 1:

[1591] The server collects video and commentary data about sporting events from the Internet, including video footage of car races, horse races, drone races, and other events, along with commentary.

[1592] Step 2:

[1593] The server preprocesses the collected data, labeling the video data with important events and actions for each frame, and tokenizing the text data, converting it into a format that can be easily understood by machine learning models.

[1594] Step 3:

[1595] The server uses the pre-processed data to train machine learning models, which use advanced algorithms such as the Transformer architecture to improve video analysis and play-by-play text generation for sporting events.

[1596] Step 4:

[1597] The server stores the trained model and builds a framework to update the model as needed, enabling highly accurate analysis with the latest information.

[1598] Commentary Phase

[1599] User

[1600] Step 1:

[1601] A user uses his / her terminal to request a live broadcast of a particular sporting event from the server, providing a live video or streaming link.

[1602] Terminal

[1603] Step 2:

[1604] The terminal transmits the user's request and the video data to the server.

[1605] server

[1606] Step 3:

[1607] The server receives requests sent by users and loads the relevant trained models.

[1608] Step 4:

[1609] The server analyzes the provided video data frame by frame, detecting specific actions and events (e.g., "first lap completed," "vehicle collision").

[1610] Step 5:

[1611] The server uses a natural language generation model based on the analysis results to instantly generate live commentary text, such as "The first lap has now finished. Two cars have collided!"

[1612] Step 6:

[1613] The server sends the generated commentary text to a speech synthesis engine to create a narration voice that is natural and easy to understand.

[1614] Step 7:

[1615] The server transmits the generated narration voice data to the user's terminal in real time.

[1616] Terminal

[1617] Step 8:

[1618] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing users to enjoy the commentary in real time along with the video.

[1619] Example: Car race commentary

[1620] Learning Phase

[1621] Step 1:

[1622] The server collects footage and live commentary from past F1 races and rallies.

[1623] Step 2:

[1624] The server preprocesses the collected data, labels the video frames, and tokenizes the text.

[1625] Step 3:

[1626] The server trains a Transformer model with the preprocessed data, giving it high-precision video analysis and commentary generation capabilities.

[1627] Step 4:

[1628] The server stores the trained model and periodically updates it.

[1629] Commentary Phase

[1630] Step 1:

[1631] The user requests a live broadcast of the F1 race currently underway from the server via the terminal.

[1632] Step 2:

[1633] The terminal transmits the user's request and the live video data to the server.

[1634] Step 3:

[1635] The server receives the request and loads a model suitable for F1 racing.

[1636] Step 4:

[1637] The server analyzes the video and detects events such as "first lap completed" or "vehicle collision."

[1638] Step 5:

[1639] The server generates live text: "First lap now completed. Two cars have collided!"

[1640] Step 6:

[1641] The server sends the commentary text to a speech synthesis engine to generate the narration voice.

[1642] Step 7:

[1643] The server transmits the narration voice to the user's terminal.

[1644] Step 8:

[1645] The terminal plays back the narration audio in synchronization with the video, allowing the user to view the video with commentary in real time.

[1646] Example 1

[1647] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1648] Conventional live commentary of sporting events relied on human commentators, which had issues with real-timeness and accuracy. It was also difficult to secure commentators with specialized knowledge. Furthermore, there was a need to provide a realistic live commentary experience for users who could not attend the event in person.

[1649] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1650] In this invention, the server includes: means for collecting large amounts of video and commentary text related to sporting events such as car races, horse races, and drone races from the Internet; means for preprocessing the collected data, labeling important events and actions in each frame of the video data, and tokenizing the text data; means for training a machine learning model using the preprocessed data; means for periodically updating and saving the trained machine learning model; means for receiving requests from users and analyzing real-time sporting event video; means for generating commentary text based on the analysis results; means for converting the generated commentary text into narration using a speech synthesis engine; means for transmitting the narration text to a user's device; and means for synchronously playing back the video and narration text on the user's device. This enables automatic commentary with high real-time accuracy. This allows users without specialized knowledge to enjoy a realistic sports viewing experience, eliminating the need for a commentator.

[1651] "On the Internet" refers to a globally connected information and communications network and infrastructure for sending and receiving data.

[1652] "Car racing" is a competitive event involving automobiles traveling at high speeds, and is a sport in which multiple vehicles compete against each other for ranking.

[1653] "Horse racing" is a competitive event using horses, and is a sport in which multiple horses compete for rankings.

[1654] "Drone racing" is a competitive event using unmanned aerial vehicles (drones), and is a sport in which drones compete for rankings by driving around a course while avoiding obstacles.

[1655] "Footage" means real-time or recorded visual data of a sporting event provided in the form of video.

[1656] "Live commentary text" is a document that verbally explains the progress of a sporting event and important events, and is character data that serves as material for speech synthesis.

[1657] "Server means" refers to a computer system in general that has the function of managing and processing data via a network.

[1658] "Collecting" is the process of obtaining and storing data from the internet or other sources.

[1659] "Preprocessing" refers to the process of converting collected data into a form that is easier to analyze, and specifically includes labeling video frames and tokenizing text data.

[1660] A "machine learning model" is a set of algorithms or programs that can learn from collected data and perform specific tasks automatically.

[1661] A "Transformer model" is a deep learning model that demonstrates excellent performance in natural language processing and machine translation, and has the ability to generate output while taking into account the context of the input data.

[1662] "Analyzing" is the process of analyzing collected or received video or data to detect significant events or actions.

[1663] "Generating" refers to the process of creating new data or information based on the analysis results, and in this case refers to the creation of commentary text.

[1664] A "speech synthesis engine" is a set of software or hardware for converting text data into speech data.

[1665] "Narration voice" is voice data that sounds like a human being speaking, generated by a voice synthesis engine.

[1666] "Terminal means" refers to all electronic devices that are directly operated by users and used to input and output data.

[1667] MODE FOR CARRYING OUT THE INVENTION

[1668] The present invention is a system that uses AI to provide automatic commentary of sporting events. This system has a function that links a server and a terminal and provides users with real-time commentary as audio. Specific embodiments for carrying out the invention are described below.

[1669] server

[1670] The server collects a large amount of video and live text about sporting events from the Internet, using various sources such as YouTube and official sporting sites, and stores the collected data in a large database.

[1671] Next, we preprocess the collected data: for video data, we use a visual information analysis library (e.g., OpenCV) to label important events and actions for each frame, and for text data, we use a natural language processing (NLP) library (e.g., NLTK or SpaCy) to tokenize it.

[1672] Once the preprocessing is complete, the data is used to train a machine learning model. In this invention, advanced algorithms such as the Transformer model are used. This model is trained using machine learning frameworks such as TensorFlow and Scikit-learn. The trained model is periodically retrained with new data and updated to ensure the model's accuracy.

[1673] When a live video request is received from a user, the server loads the appropriate pre-trained model and analyzes the video in real time. Based on the analysis results, a natural language generation model is used to generate live commentary text. This text is then passed to a speech synthesis engine, which converts it into fluent narration audio in real time. For example, a text-to-speech (TTS) engine such as Amazon Polly is used for speech synthesis.

[1674] The narration audio is transmitted to the user's device in real time using protocols such as HTTP Live Streaming (HLS).

[1675] Terminal

[1676] The user's device receives the narration audio data sent from the server and plays it in sync with the live video. The user then watches the video with commentary in real time via a dedicated application or web browser.

[1677] User

[1678] Users request live coverage of a particular sporting event from their device. For example, if they want live coverage of a current F1 race, they click a request button in the application, enter the URL of the live video in the pop-up window, and send it to the server.

[1679] Specific examples

[1680] An example of such a prompt sentence is, "Please provide a live commentary of the F1 race currently being broadcast. Please give a detailed description of the events that occur from the start to the pit stop."

[1681] This system allows viewers to enjoy sporting events such as car races, horse races, and drone races smoothly without the need for live commentary by experts. By providing accurate commentary in real time, viewers can gain a deeper understanding of the action and enjoy a more immersive viewing experience.

[1682] keyword

[1683] Generative AI Models

[1684] Prompt statement

[1685] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1686] Step 1: Data collection

[1687] The server collects video and live commentary of sporting events such as car races, horse races, and drone races from the Internet. The input is a list of URLs on the Internet, and the output is the collected video and text data. Specifically, the server automatically downloads the video and text data from the specified URL list and stores them in a database.

[1688] Step 2: Data Preprocessing

[1689] The server preprocesses the collected data. For video data, it uses a visual information analysis library such as OpenCV to label important events and actions for each frame. For text data, it uses a natural language processing library such as NLTK or SpaCy to tokenize it. The input is the collected video and text data, and the output is preprocessed video frames and text tokens.

[1690] Step 3: Model training

[1691] The server uses the preprocessed data to train a machine learning model. Specifically, it uses a machine learning framework such as TensorFlow or Scikit-learn to train a Transformer model. The input is the preprocessed dataset, and the output is a trained machine learning model. The server uses video and text pairs to batch process the model. It updates the model parameters every epoch and minimizes the loss function.

[1692] Step 4: Save and update the model

[1693] The server saves the trained model in a database and periodically retrains it with new data. The input is the trained model and a new dataset, and the output is the updated model. Specifically, the server saves the parameters of the trained model and schedules periodic retraining with new data.

[1694] Step 5: Receiving the request

[1695] A user requests a live broadcast of a specific sporting event from their device. The input is the URL of the live video specified by the user, and the output is a request sent to the server. The user clicks the request button in the app, enters the URL in the pop-up window, and submits it.

[1696] Step 6: Request Processing

[1697] When the server receives a request from a user, it loads the appropriate trained model. The input is the user request and the URL of the live video, and the output is the loaded trained model. Specifically, the server analyzes the request and determines which model to load.

[1698] Step 7: Video Analysis

[1699] The server analyzes real-time video provided by the user frame by frame. The input is the live video stream, and the output is the analyzed event and action data. Specifically, the server receives the video stream and analyzes each frame using OpenCV.

[1700] Step 8: Live text generation

[1701] The server generates commentary text using a natural language generation model based on the analysis results. The input is the analyzed event data, and the output is the generated commentary text. Specifically, the server passes the analysis results as input to the natural language generation model to generate commentary text.

[1702] Step 9: Text-to-Speech

[1703] The server sends the generated commentary text to a speech synthesis engine to generate a narration voice. The input is the commentary text, and the output is the synthesized narration voice. Specifically, the server sends the generated text to the speech synthesis engine and converts it into voice data.

[1704] Step 10: Send narration audio

[1705] The server transmits the generated narration audio data to the user's terminal. The input is the narration audio, and the output is the transmission of audio data to the user's terminal. Specifically, the server streams the audio file to the terminal.

[1706] Step 11: Data reception and playback

[1707] The device receives the narration audio data sent from the server and plays it in sync with the live video. The input is the narration audio data and the live video, and the output is the synchronized playback of the live video. Specifically, the device receives an HTTP request and launches a player to play the streaming audio data.

[1708] (Application example 1)

[1709] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1710] Modern factories require real-time monitoring of the progress and abnormalities of production lines and robots. However, it is difficult for on-site workers to constantly monitor the situation, and they are unable to respond immediately when an abnormality occurs. Furthermore, visual information alone does not provide a sufficient understanding of the details of the production process, making it difficult to improve efficiency and control quality. To address these issues, the present invention aims to provide a system that provides real-time audio commentary on the progress and abnormalities of production lines and robots.

[1711] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1712] In this invention, the server includes means for collecting training data, means for preprocessing the training data and training a machine learning model, means for accepting requests and analyzing video of a sporting event or a production line, means for generating commentary text based on the analysis results, means for converting the commentary text into narration audio using a speech synthesis engine, means for transmitting the narration audio to a user's terminal, means for synchronously playing back the video and narration audio on the user's terminal, and means for providing real-time commentary on the progress and abnormalities of the production line or robot. This enables real-time audio commentary on the status of the manufacturing process in a factory, allowing workers to perform their work efficiently.

[1713] The "server means" is a computer system that controls the functions of data collection, preprocessing, analysis, generation, synthesis, and transmission, and communicates with the user's terminal.

[1714] "Training Data" is a data set collected by the Server Means and used to train a Machine Learning Model.

[1715] "Preprocessing" is the process of converting collected training data into a format that is easy for machine learning models to understand, using techniques such as tokenization and labeling.

[1716] A "machine learning model" is an algorithm, particularly a Transformer model, that is trained on training data to perform a specific task.

[1717] A "request" is a communication signal that a user uses from his / her own terminal to request a live report of a specific event from the server means.

[1718] "Video analytics" is the process of analyzing real-time video data to detect important events or actions.

[1719] "Live text" is explanatory text generated based on the results of video analysis, which is converted into voice by a voice synthesis engine.

[1720] A "speech synthesis engine" is software or hardware for converting text data into narration voice.

[1721] The "terminal means" is a device used by a user, and has the function of playing back the narration audio and video transmitted from the server means in synchronization.

[1722] A "production line" is a system that shows the machines and work flow arranged to carry out production activities within a factory.

[1723] A "robot" is a mechanical device that performs tasks automatically or semi-automatically in a factory.

[1724] "Real-time" means that processing occurs immediately, with virtually no delay.

[1725] "Audio commentary" is the process of communicating analyzed information to the user in real time by voice.

[1726] "Progress" refers to the state and speed at which a production line or robot is working.

[1727] An "abnormality" refers to a state that deviates from normal operation, or a malfunction such as an error or breakdown.

[1728] The present invention provides a system that provides real-time audio commentary on the progress and abnormalities of manufacturing lines and robots. This system consists of three main elements: a server, a terminal, and a user.

[1729] server

[1730] 1. Data Collection and Learning

[1731] The server collects actual video and text data about manufacturing lines and robots collected from the Internet. This data is used as learning data to train machine learning models. Specific hardware used is a high-performance server.

[1732] 2. Data Preprocessing

[1733] The collected data is preprocessed by the server. For example, video data is broken down into frames and labeled. Text data is tokenized and converted into a format that is easy for machine learning models to understand. Open source software such as TensorFlow and OpenCV is used.

[1734] 3. Model training

[1735] Using the pre-processed data, the server trains machine learning models, built using advanced algorithms such as the Transformer model, capable of analyzing video footage of production lines and robots and generating live text.

[1736] 4. Model application and analysis

[1737] The server analyzes the video of the production line based on user requests. Video analysis includes real-time event detection. For example, it detects actions such as "the production line has stopped" or "a robot has encountered an error." The software used is TensorFlow and OpenCV.

[1738] 5. Live Text Generation and Speech Synthesis

[1739] Based on the analysis results, the server generates commentary text using a natural language generation model (such as GPT-3), which is then converted into narration audio by a speech synthesis engine (Amazon Polly, Google Text-to-Speech).

[1740] 6. Sending data to the user device

[1741] Once the narration voice data is generated, the server transmits it to the user's terminal in real time.

[1742] Terminal

[1743] 1. Data reception and synchronous playback

[1744] The user's device (such as smart glasses) receives the narration audio data sent from the server and plays it in sync with the video. This function allows the user to understand the status of the production line and robots in real time with audio commentary.

[1745] User

[1746] 1. Submit a request

[1747] Users can request live footage of a specific manufacturing line or robot from a terminal, which can include live video or streaming links.

[1748] Specific examples

[1749] For example, if a user sends a request from a terminal about the progress of a production line, the server generates a prompt sentence such as the following:

[1750] Example prompt sentence:

[1751] "The production line has stopped. What is the next action?"

[1752] In this way, by linking the server, terminal, and user, it becomes possible to provide real-time audio updates on the progress and abnormalities of the production line and robots, allowing workers to carry out their work efficiently.

[1753] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1754] Step 1: Data collection

[1755] The server collects large amounts of video and text data related to manufacturing lines and robots from the Internet. This data is used as a training dataset. The input is raw data related to the manufacturing process, and the output is the collected dataset. Specifically, the data is downloaded using a web scraping tool or API.

[1756] Step 2: Data Preprocessing

[1757] The server preprocesses the collected data. Specifically, the video data is decomposed into frames and important events are labeled. The text data is tokenized and formatted. The input is the collected video data and text data, and the output is the preprocessed data. Specifically, frame decomposition is performed using OpenCV and labeling is performed using TensorFlow.

[1758] Step 3: Model training

[1759] The server uses the preprocessed data to train a machine learning model. This model can then analyze video of production lines and robots and generate live text. The specific method used is a Transformer model. The input is the preprocessed data, and the output is the trained model. The specific operation is to train the Transformer model using the TensorFlow API.

[1760] Step 4: Submitting the request

[1761] A user requests a live report of a specific manufacturing line or robot from a terminal to a server. This request may include live video or a streaming link. The input is the user's request, and the output is a communication signal to the server. Specifically, the request is sent from the terminal's UI to the server.

[1762] Step 5: Video Analysis

[1763] The server analyzes the real-time video provided by the user frame by frame. This analysis recognizes important events and actions. For example, events such as "the production line has stopped" or "an error has occurred in a robot" can be detected. The input is the live video data, and the output is the analysis results. Specifically, the video analysis is performed using TensorFlow.

[1764] Step 6: Generate commentary text

[1765] The server generates commentary text using a natural language generation model based on the analysis results. The generated text has appropriate phrases and sentence structures depending on the situation. The input is the analysis results, and the output is the commentary text. Specifically, text generation is performed using GPT-3.

[1766] Step 7: Text-to-Speech

[1767] The server sends the generated commentary text to a speech synthesis engine to generate narration audio. This audio is high quality and fluent. The input is the commentary text, and the output is the narration audio data. Specifically, speech synthesis is performed using the Google Text-to-Speech API.

[1768] Step 8: Send narration audio

[1769] The server transmits the generated narration audio data to the user's device in real time. The input is the narration audio data, and the output is the audio data sent to the user's device. Specifically, the audio data is transmitted using a network protocol (e.g., WebSocket).

[1770] Step 9: Data reception and synchronous playback

[1771] The user's device receives the narration audio data sent from the server and plays it back in sync with the video. This function allows the user to understand the status of the production line or robot in real time with audio commentary. The input is the audio data sent from the server, and the output is synchronous playback of the video and audio. Specifically, the device's audio playback function is used to synchronize with the video data.

[1772] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1773] System Overview

[1774] The present invention is a system that utilizes AI to provide automatic commentary of sporting events. In addition, by combining it with an emotion engine that recognizes the user's emotions, it provides adaptive commentary that corresponds to the user's emotions. The server and terminals work together to provide commentary in the form of audio in real time. The specific processing of the program of the system of the present invention is explained below in natural language.

[1775] Learning Phase

[1776] server

[1777] 1. Data Collection

[1778] The server collects a large amount of video and live text from sporting events such as car races, horse races, and drone races from the Internet, which are used as large-scale training datasets for video analysis and natural language processing.

[1779] 2. Data Preprocessing

[1780] The server preprocesses the collected data: it labels important events and actions for each frame of video data, tokenizes text data and converts it into a format that machine learning models can easily understand, and collects and processes audio and facial expression data for the emotion engine.

[1781] 3. Model training

[1782] The server uses the pre-processed data to train machine learning models, which use advanced algorithms such as Transformer models, and train them to perform video analysis, live text generation, and emotion recognition.

[1783] 4. Save the model

[1784] The server will store the trained model and create a framework for periodic updates, enabling highly accurate analysis using the latest information.

[1785] Commentary Phase

[1786] User

[1787] 1. Submit a request

[1788] Users can use their devices to request live coverage of a particular sporting event from the server, providing live video or a streaming link, and voice and facial expression data is also collected to recognize the user's emotions.

[1789] Terminal

[1790] 1. Receiving a request

[1791] The terminal transmits the user's request, video data, and emotion data to the server.

[1792] server

[1793] 1. Request Processing

[1794] The server receives requests from users and loads and prepares the relevant trained models.

[1795] 2. Video Analysis

[1796] The server analyzes the provided video data frame by frame. This analysis allows it to recognize important events and actions. For example, in the case of a car race, events such as "the first lap is over" and "vehicle collision" can be detected.

[1797] 3. Emotion recognition

[1798] The server analyzes the voice and facial expression data sent by the user and uses an emotion engine to recognize the user's emotional state, including the user's tone of voice, facial expressions, and body movements.

[1799] 4. Live Text Generation

[1800] The server generates commentary text using a natural language generation model based on the video analysis results. The content and tone of the commentary text are adjusted according to the user's emotional state as recognized by the emotion engine. For example, if the user is excited, the server generates text that maintains the same sense of excitement.

[1801] 5. Speech Synthesis

[1802] The server sends the generated commentary text to a speech synthesis engine to create a narration voice, which is natural and easy to understand, and is provided in a tone that matches the user's emotions.

[1803] 6. Narration audio transmission

[1804] The server transmits the generated narration voice data to the user's terminal in real time.

[1805] Terminal

[1806] 1. Data reception and playback

[1807] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing users to enjoy real-time commentary based on their emotions along with the video.

[1808] Example: Car race commentary

[1809] Learning Phase

[1810] 1. Data Collection and Preprocessing

[1811] The server collects and preprocesses video footage from past F1 races and rallies, commentary text, and user emotion data, labeling video frames, tokenizing text, and processing audio and facial expression data for the emotion engine.

[1812] 2. Model training

[1813] The server trains a Transformer model on the preprocessed data, giving it high-precision video analysis, commentary generation, and emotion recognition capabilities.

[1814] 3. Save the model

[1815] The server stores the trained model and periodically updates it.

[1816] Commentary Phase

[1817] 1. User Request

[1818] The user requests a live report of the current F1 race from the server via their device, and data based on the user's emotional state is also collected at the same time.

[1819] 2. Sending and Receiving Devices

[1820] The device transmits the user's request, live video data, and emotion data to the server, and receives the live audio data sent from the server and plays it in sync with the video.

[1821] 3. Server Processing

[1822] The server receives the request, analyzes the video, and the emotion engine recognizes the emotional state. Events such as "First lap completed" and "Vehicle collision" are detected, and based on the user's emotional state, a live commentary text is generated, such as "The first lap has now completed. This is exciting! Two cars have collided!"

[1823] The server sends the commentary text to a speech synthesis engine to generate a narration voice that maintains an uplifting tone.

[1824] The server transmits the generated narration voice to the user's terminal.

[1825] 4. Playback of the device

[1826] The user's terminal plays back the narration audio in synchronization with the video, allowing the user to view a realistic commentary in real time.

[1827] This system allows viewers to watch sporting events in a way that adapts to their emotions, enabling real-time, adaptive commentary without the need for expert commentary, improving the sense of realism and entertainment, and making it easier to attract viewers' attention.

[1828] The processing flow will be explained below.

[1829] Learning Phase

[1830] server

[1831] Step 1:

[1832] The server collects videos, commentary text data, and user emotion data related to sporting events from the Internet. This includes videos of car races, horse races, drone races, and other events, along with commentary text and user voice and facial expression data.

[1833] Step 2:

[1834] The server preprocesses the collected data: it labels important events and actions for each frame of video data, tokenizes text data and converts it into a format that machine learning models can easily understand, and adds emotion labels from audio and facial expression data for the emotion engine.

[1835] Step 3:

[1836] The server uses the pre-processed data to train machine learning models, which are trained using advanced algorithms such as Transformer models to analyze video footage of sporting events, generate commentary, and recognize emotions.

[1837] Step 4:

[1838] The server stores the trained model and builds a framework to periodically update the model as needed, enabling highly accurate analysis using the latest information.

[1839] Commentary Phase

[1840] User

[1841] Step 1:

[1842] Users use their devices to request a live broadcast of a particular sporting event from the server, providing a live video or streaming link, and voice and facial expression data is also collected for emotion recognition.

[1843] Terminal

[1844] Step 2:

[1845] The terminal transmits the user's request, live video data, and emotion data to the server.

[1846] server

[1847] Step 3:

[1848] The server receives requests sent by users and loads the relevant trained models.

[1849] Step 4:

[1850] The server analyzes the received video data frame by frame to detect important events and actions (e.g., "first lap completed" or "vehicle collision").

[1851] Step 5:

[1852] The server analyzes the voice and facial expression data sent by the user and recognizes the user's emotional state (e.g., excitement, surprise) using an emotion engine.

[1853] Step 6:

[1854] The server generates commentary text using a natural language generation model based on the video analysis and emotion recognition results, and adjusts the content and tone of the commentary text to reflect the user's emotions.

[1855] Step 7:

[1856] The server sends the generated commentary text to a speech synthesis engine to create a narration voice, which is provided in a tone that matches the user's emotions.

[1857] Step 8:

[1858] The server transmits the generated narration voice data to the user's terminal in real time.

[1859] Terminal

[1860] Step 9:

[1861] The device receives the narration audio data sent from the server and plays it in sync with the video, allowing users to enjoy real-time commentary based on their emotions along with the video.

[1862] Example: Car race commentary

[1863] Learning Phase

[1864] Step 1:

[1865] The server collects footage of past F1 races and rally competitions, commentary text, and user emotion data.

[1866] Step 2:

[1867] The server preprocesses the collected data, labels video frames, tokenizes text, and assigns emotion labels to audio and facial expression data for the emotion engine.

[1868] Step 3:

[1869] The server trains a Transformer model on the preprocessed data, giving it high-precision video analysis, commentary generation, and emotion recognition capabilities.

[1870] Step 4:

[1871] The server stores the trained model and periodically updates it.

[1872] Commentary Phase

[1873] Step 1:

[1874] The user requests a live report of the current F1 race from the server via their device, and data based on the user's emotional state is also collected at the same time.

[1875] Step 2:

[1876] The terminal transmits the user's request, live video data, and emotion data to the server.

[1877] Step 3:

[1878] The server receives the request and loads a model suitable for F1 racing.

[1879] Step 4:

[1880] The server analyzes the video and detects events such as "first lap completed" or "vehicle collision."

[1881] Step 5:

[1882] The server analyzes the user's voice and facial expression data, and the emotion engine recognizes the user's emotional state (e.g., the user is excited).

[1883] Step 6:

[1884] Based on the video analysis and emotion recognition results, the server generates live text such as, "The first lap is now complete. This is very exciting! Two cars have collided!"

[1885] Step 7:

[1886] The server sends the commentary text to a speech synthesis engine to generate a narration voice with a tone that reflects excitement.

[1887] Step 8:

[1888] The server transmits the generated narration voice to the user's terminal.

[1889] Step 9:

[1890] The terminal plays back the narration audio in synchronization with the video, allowing the user to view a realistic commentary in real time.

[1891] Example 2

[1892] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1893] In recent years, there has been a demand for real-time, realistic commentary when watching sporting events. However, professional commentators are not always available, so even general users lack the means to enjoy high-quality commentary. Furthermore, when it comes to realistic commentary that changes in real time based on the user's emotions, conventional systems have difficulty in handling this. Therefore, there is a need for a system that can adapt to the user's emotional state and provide high-quality commentary in real time.

[1894] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1895] In this invention, the server includes server means for collecting training data, server means for preprocessing the training data to train a machine learning model, server means for accepting requests and analyzing video of a sporting event, server means for generating commentary text based on the analysis results, server means for analyzing user emotion data to recognize the user's emotional state, server means for adjusting the tone of the commentary text based on the recognized emotional state, server means for converting the commentary text into narration using a speech synthesis engine, server means for transmitting the narration to the user's terminal, and terminal means for synchronously playing back the video and narration on the user's terminal, thereby enabling real-time, high-quality commentary that is adapted to the user's emotions.

[1896] "Training data" refers to data collected by the server and used to train machine learning models, including, among other things, footage and commentary about sporting events.

[1897] "Server means" means a server designed to perform a specific function, such as data collection, pre-processing, analysis, model training, text generation, speech synthesis, and emotion recognition.

[1898] "Terminal means" refers to a device used by a user, and has the function of receiving data from a server and playing back video and narration audio in sync.

[1899] A "machine learning model" is a collection of algorithms that can perform specific tasks by learning from data, and this system uses a Transformer model.

[1900] A "Transformer model" is a type of machine learning model with advanced pattern recognition and natural language generation capabilities.

[1901] "Emotion data" is data obtained from the user's voice, facial expressions, etc., and is used to analyze the user's emotional state.

[1902] "Emotional state" refers to the psychological state of the user as judged from their tone of voice, facial expressions, and physical movements.

[1903] A "play-by-play text" is a textual description of the situation at a sporting event in natural language, which is generated in a form that can be read or listened to by a user in real time.

[1904] A "speech synthesis engine" is a software or hardware system for converting text data into speech data.

[1905] "Narration voice" is a voice synthesized based on the generated commentary text, and is used to convey information to the user auditorily.

[1906] "Real-time" means that processing occurs and results are provided immediately, without delay.

[1907] MODE FOR CARRYING OUT THE INVENTION

[1908] The present invention is a system that uses AI to provide automatic commentary of sporting events. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides adaptive commentary that corresponds to the user's emotions. This system has a function that links a server and a terminal and provides commentary as audio in real time. Specific embodiments of the system of the present invention are described in detail below.

[1909] Server Roles

[1910] 1. Data Collection

[1911] The server collects a large amount of video and live commentary from the Internet about sporting events such as car races, horse races, and drone races, and uses these as a large-scale learning dataset for video analysis and natural language processing.

[1912] As a concrete example, the server collects F1 race footage and commentary text from YouTube and sports news sites.

[1913] 2. Data Preprocessing

[1914] The server labels the collected video data for important events and actions in each frame, and tokenizes the text data, converting it into a format that can be easily understood by machine learning models, and then stores it.

[1915] As a specific example, labels such as "first lap completed" or "vehicle collision" are added to video frames, and the text data is divided into words.

[1916] 3. Model training

[1917] The server uses the pre-processed data to train advanced algorithms such as Transformer models, which then have the capabilities of video analysis, live text generation, and emotion recognition.

[1918] As a concrete example, video analysis data and tokenized commentary text are input into the Transformer model to enable it to recognize "the scene from the start of a car race to the end of the first lap."

[1919] 4. Saving and updating the model

[1920] The server stores the trained model and periodically updates it to ensure that the latest information is always used for highly accurate analysis.

[1921] For example, new race data is added once a month to retrain the model and reflect the latest racing developments.

[1922] 5. Receiving and Processing Requests

[1923] The server receives requests from users and loads and prepares the relevant trained models.

[1924] As a concrete example, the server loads a "F1 race commentary model" and prepares it for analysis.

[1925] 6. Video Analysis

[1926] The server analyzes the provided video data frame by frame to recognize important events and actions.

[1927] As a specific example, events such as "first lap completed" and "vehicle collision" are detected in real time.

[1928] 7. Emotion recognition

[1929] The server analyzes the voice and facial expression data sent by the user and recognizes the user's emotional state via an emotion engine.

[1930] As a specific example, the emotional state of the user, such as "excited" or "relaxed," is analyzed from the tone of voice and facial expression.

[1931] 8. Live Text Generation

[1932] The server generates commentary text using a generative AI model based on the video analysis results, and adjusts the tone and content of the commentary text based on the emotion recognition results.

[1933] As a specific example, we generate a commentary such as "The first round is over. What an exciting development!"

[1934] 9. Speech Synthesis

[1935] The server sends the generated commentary text to a voice synthesis engine, which generates a narration voice in a tone that matches the user's emotions.

[1936] For example, the tone of the narration can be increased to further excite the user.

[1937] 10. Narration audio transmission

[1938] The server transmits the generated narration voice data to the user's terminal in real time.

[1939] As a specific example, narration audio data is sent to the user's smartphone in streaming format.

[1940] Device Role

[1941] 1. Submit a request

[1942] Users use their devices to request live coverage of a particular sporting event from the server, which in the process provides live video and streaming links, as well as voice and facial expression data for emotion recognition.

[1943] As a specific example, a user sends a request from a smartphone app saying, "Please give me a live report of the current F1 race."

[1944] 2. Data reception and playback

[1945] The terminal receives the narration audio data sent from the server and plays it in sync with the video.

[1946] As a specific example, a smartphone app receives narration audio and plays it simultaneously with the live video.

[1947] This system allows users to enjoy a real-time, emotionally adaptive commentary experience. Here are some example prompts:

[1948] "The first round has now finished. Please give us a running commentary on the next developments."

[1949] "Vehicle collision occurred. Please emphasize the tense situation."

[1950] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1951] Step 1: Data collection

[1952] The server collects a large amount of video and live text about sporting events such as car races, horse races, and drone races from the Internet. Specifically, the server uses a web crawler to download video and text data from video sharing sites and sports news sites. The input is a list of URLs, and the output is video files and text files stored in local storage.

[1953] Step 2: Preprocessing the data

[1954] The server splits the collected video data into frames and labels important events and actions. It tokenizes the text data, converts it into a format that is easy for machine learning models to understand, and stores it. Specifically, the server uses video analysis tools to extract video frames and uses a labeling algorithm to identify important events. The inputs are raw video data and live text data, and the output is labeled video frames and tokenized text data.

[1955] Step 3: Model training

[1956] The server uses the preprocessed data to train a machine learning model using advanced algorithms such as the Transformer model. Specifically, the server trains the model using a deep learning framework (e.g., TensorFlow) on a GPU cluster. The input is the preprocessed data, and the output is a trained machine learning model.

[1957] Step 4: Save and update the model

[1958] The server builds a framework to store the trained model and periodically update it. Specifically, the server saves the model in a database and retrains it whenever new data is added. The input is the trained model and the new data, and the output is the updated model.

[1959] Step 5: Submitting the request

[1960] A user uses a device to request a live broadcast of a specific sporting event from a server. In the process, live video and streaming links are provided, and voice and facial expression data for emotion recognition are also transmitted. Specifically, the user enters the necessary information into a request form in an application on the device and submits it. The input is the user's request and emotion data, and the output is the transmission of the request information.

[1961] Step 6: Receiving and Processing the Request

[1962] The server receives requests from users and loads and prepares the relevant trained models. Specifically, the server analyzes the content of the request and loads the appropriate model into memory. The input is the user request, and the output is the loaded model.

[1963] Step 7: Video Analysis

[1964] The server analyzes the provided video data frame by frame and recognizes important events and actions. Specifically, the server uses a video analysis algorithm to analyze the provided live video frame by frame in real time. The input is the live video data, and the output is recognized event information.

[1965] Step 8: Emotion Recognition

[1966] The server analyzes the voice and facial expression data sent by the user and recognizes the user's emotional state through an emotion engine. Specifically, the server analyzes the user's emotions using voice recognition software and facial recognition algorithms. The input is voice data and facial expression data, and the output is the recognized emotional state.

[1967] Step 9: Generate commentary text

[1968] The server generates commentary text using a generative AI model based on the video analysis results. The tone and content of the commentary text are adjusted based on the emotion recognition results. Specifically, the server sends input data to the generative AI model, which adapts the text content according to the emotional status. The input is event information and emotional state, and the output is commentary text.

[1969] Step 10: Text-to-Speech

[1970] The server sends the generated commentary text to a speech synthesis engine and generates a narration voice in a tone that matches the user's emotions. Specifically, the server creates a narration voice using a text-to-speech synthesis tool. The input is the commentary text, and the output is the narration voice.

[1971] Step 11: Send narration audio

[1972] The server transmits the generated narration audio data to the user's device in real time. Specifically, the server uses a protocol to transfer audio data in streaming format. The input is the narration audio, and the output is data transmission to the user's device.

[1973] Step 12: Data reception and playback

[1974] The device receives the narration audio data sent from the server and plays it in sync with the video. Specifically, the device stores the received audio data in a buffer and synchronizes it with the video playback. The input is the narration audio data, and the output is real-time playback.

[1975] (Application example 2)

[1976] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1977] Conventional production line monitoring systems require a human supervisor to constantly monitor the video feed, which requires a lot of effort and can delay the detection of abnormalities. Furthermore, in environments where a rapid response is required when an abnormality occurs, the supervisor's response may vary depending on their emotional state. If these issues are not resolved, the efficiency and safety of the production line may decrease.

[1978] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1979] In this invention, the server includes a computing device that collects learning data, a computing device that preprocesses the learning data and trains a machine learning model, a computing device that accepts requests and analyzes video of the production line, a computing device that generates alert text corresponding to anomaly detection and the manager's emotions based on the analysis results, a computing device that converts the alert text into narration using a speech synthesis engine, a computing device that sends the narration to the manager's terminal, and a terminal device that synchronously plays back the video and narration on the manager's terminal.This automates production line monitoring and enables adaptive alerts corresponding to the manager's emotions.

[1980] A "computing device" is a device that performs computational tasks such as collecting, analyzing, processing, and transmitting data.

[1981] "Training data" refers to input data for training a machine learning model, and can include video, text, and other forms of data.

[1982] "Preprocessing" is a series of operations that transform training data into a form suitable for analysis and model training.

[1983] A "machine learning model" is an algorithm that learns patterns from data and performs analysis and predictions.

[1984] "Video analytics" is the process of identifying information contained in images or video frames to detect specific events or anomalies.

[1985] "Anomaly detection" is the process of identifying unusual behavior or conditions on a manufacturing line.

[1986] "Emotion recognition" is a technology that determines the emotional state of an administrator from video and audio data.

[1987] An "alert text" is a text message generated to notify you of an abnormality or important event.

[1988] A "speech synthesis engine" is a system that converts text data into voice data.

[1989] "Narration voice" is a voice message generated by a voice synthesis engine.

[1990] "Terminal device" refers to a computer or smart device used by an administrator, and is a device for receiving and playing narration audio and video from the server.

[1991] Learning Phase

[1992] The system of the present invention functions through an initial training phase. During the training phase, a server collects video and text data related to manufacturing processes from the internet. The collected data is preprocessed into a format suitable for analysis and machine learning model training. This preprocessing includes frame-by-frame classification of video data and tokenization of text data. The server then uses the preprocessed data to train Transformer models and other machine learning algorithms to improve anomaly detection and emotion recognition capabilities. The trained models are stored on the server and periodically updated.

[1993] Execution Phase

[1994] Receiving a request

[1995] A user sends a request for monitoring the production line to the server through a terminal used by the manager. The request includes a streaming link of the live video and audio and video data for recognizing the manager's emotions. The terminal then sends this data to the server.

[1996] Video analysis and emotion recognition

[1997] The server analyzes the video data from the user frame by frame, allowing abnormal behavior and conditions to be detected in real time. For example, if the robot arm is not aligned correctly, the abnormality is immediately detected. At the same time, the server analyzes the administrator's voice and facial expression data, and uses an emotion engine to recognize the administrator's emotional state. This takes into account voice tone, facial expressions, body movements, and more.

[1998] Alert text generation and speech synthesis

[1999] The server generates adaptive alert text based on the results of video analysis and emotion recognition. The generated alert text is converted into a voice narration by a speech synthesis engine. This voice is natural and easy to understand, and the tone is adjusted to match the administrator's emotions.

[2000] Sending and playing narration audio

[2001] The server sends the generated narration audio data in real time to the administrator's terminal. The terminal receives the narration audio data sent from the server and plays it in sync with the video. This allows the administrator to respond to any abnormalities in real time while watching the video.

[2002] Specific examples

[2003] For example, if a manager is monitoring a production line and notices that a robot arm is not operating normally, the server will use video analysis to detect the abnormality. At the same time, if the manager is feeling nervous, the emotion engine will recognize this as well. As a result, an alert text is generated stating, "An abnormality has been detected in the robot arm's operation. Please be careful!", and an adaptive voice narration is created by the speech synthesis engine. The narration is sent to the manager's device in real time, allowing the manager to listen to the alert along with the video and respond quickly.

[2004] Prompt Sentence Examples

[2005] "Detect anomalies in the factory in real time and adaptively notify managers. Generate appropriate alerts based on the nature of the anomaly and the manager's sentiment."

[2006] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2007] Step 1: Collect training data

[2008] The server collects video and text data related to the manufacturing process from the internet. Specifically, the server obtains the data using web scraping or APIs. The collected data is used for subsequent analysis and training of machine learning models. The input is publicly available data on the internet, and the output is the collected dataset.

[2009] Step 2: Preprocessing the data

[2010] The server preprocesses the collected training data. Video data is classified by frame, and text data is tokenized. Data preprocessing also includes noise reduction and format conversion. Specific operations include segmenting video frames using OpenCV and tokenizing text using a natural language processing library. The input is the collected dataset, and the output is the preprocessed data.

[2011] Step 3: Train the machine learning model

[2012] The server uses the preprocessed data to train a machine learning model. Transformer models and image recognition models are typically used. Specifical...

Claims

1. A server means for collecting learning data; a server means for preprocessing the learning data and training the machine learning model; a server means for accepting requests and analyzing footage of the sporting event; server means for generating commentary text based on the analysis results; a server means for converting the commentary text into a narration voice using a voice synthesis engine; a server means for transmitting narration voice to a user terminal; A system including a terminal means for synchronously playing back video and narration audio on a user's terminal.

2. The system according to claim 1 , wherein the machine learning model is a transformer model.

3. The system of claim 1, wherein a dataset on sporting events collected from the Internet is used for training.

Images