System

Abstract

A system is provided.SOLUTION: The system includes a means for receiving video data, a means for analyzing the received video data, a means for generating on-the-spot broadcasting voice on the basis of an analysis result, a means for integrating the generated on-the-spot broadcasting voice with the video data, and a means for distributing the integrated video data with the on-the-spot broadcasting voice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In video content, especially in sports broadcasts and game streaming, live commentary is an important element in attracting viewers' interest. However, it is difficult to secure commentators with the necessary expertise and experience, and cost and time are major issues. Furthermore, generating high-quality commentary in real time requires advanced technology and equipment. This has led to a demand for a method that allows anyone to easily create high-quality video content with live commentary. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for receiving video data, a means for analyzing the received video data, a means for generating commentary based on the analysis results, a means for integrating the generated commentary into video data, and a means for distributing the integrated video data with commentary. This system enables even users without specialized knowledge to easily create and distribute high-quality video with commentary. The analysis means uses artificial intelligence, and the commentary generation means utilizes voice synthesis technology, enabling highly accurate video analysis and commentary generation in real time.

[0006] "Video data" refers to all image information in digital form captured by a camera or other image capture device.

[0007] A "means" refers to a method, apparatus, or component of a system used to accomplish a particular purpose.

[0008] "Analysis" refers to the computational methods and algorithms applied to recognize and identify objects and actions within video data.

[0009] "Artificial intelligence" refers to the technology of analyzing data and making decisions using techniques such as machine learning and deep learning.

[0010] "Live audio" refers to audio data generated to describe events or situations occurring in real time.

[0011] "Speech synthesis" refers to the technology of artificially generating voice data based on text information.

[0012] "Consolidation" refers to the process of combining multiple pieces of data into one format or file.

[0013] "Distribution" refers to the act of providing users with generated and integrated video data with commentary via the Internet, etc. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention relates to a system that receives video data, analyzes it, generates live audio, integrates it with the video, and distributes it. The program of this system and its processing are described in detail below.

[0036] Processing on the user terminal

[0037] The user device first acquires video data in real time using a camera or screen capture software. This video data consists of a series of frames, each captured at a fixed interval. Then, a socket connection is established to transmit the acquired video data to the AI ​​server in real time. Through this connection, the frame data is sent to the AI ​​server.

[0038] In a specific example, a user captures video using the camera on their smartphone or PC, and the video is sent to a server via a network.

[0039] Processing on the AI ​​server

[0040] The AI ​​server receives video data sent from the user's device. The received data is temporarily stored in a buffer and then input into a model for video analysis. The AI ​​model analyzes objects and actions in the digital video and identifies events in real time based on the results. Appropriate commentary text is generated based on the recognized events. This commentary text is then converted into audio data using speech synthesis technology. This audio data is synchronized with the original video data to generate video data with commentary audio.

[0041] For example, real-time video of a sporting event is sent to an AI server, where the AI ​​identifies players' movements and scoring scenes, and generates commentary such as, "The player has just scored a goal!" This commentary is converted into audio and combined with the video.

[0042] Processing on the distribution server

[0043] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as by filtering or encoding. Finally, the processed video data with commentary is distributed to viewers live or on demand. Distribution is carried out over the internet, and viewers can watch it via a website or app.

[0044] In a specific example, users can watch live sports broadcasts on their smartphones or PCs, and AI-powered commentary will be provided in real time, improving the viewing experience even without the need for a professional commentator.

[0045] This system automatically analyzes video data, generates appropriate commentary, and delivers it to viewers, allowing even users without specialized knowledge to easily create and distribute high-quality video content with commentary.

[0046] The processing flow will be explained below.

[0047] Step 1:

[0048] The user device acquires video data in real time through a camera or screen capture software. The video data consists of a series of frames, each captured at a fixed interval.

[0049] Step 2:

[0050] The user device establishes a socket connection to transmit the acquired video data to the AI ​​server in real time. Through this connection, frame data is transferred to the AI ​​server.

[0051] Step 3:

[0052] The server receives video data sent from the user terminal, temporarily stores the received data in a buffer, and prepares it for analysis.

[0053] Step 4:

[0054] The server inputs the received data into an AI model for video analysis, which analyzes objects and actions in the video in real time to identify specific events and scenes.

[0055] Step 5:

[0056] The server generates appropriate commentary text based on the analysis results. Commentary content is defined for recognized events (e.g., goal shots, goals scored, transitions between offense and defense, etc.).

[0057] Step 6:

[0058] The server converts the generated commentary text into audio data using speech synthesis technology. In this process, the commentary text is converted into audio data that has the speaker's voice and intonation.

[0059] Step 7:

[0060] The server synchronizes the generated audio data with the original video data, creating video data with live audio. During this integration process, processing is performed to align the timing of the video and audio.

[0061] Step 8:

[0062] The server converts the video data with live audio into a format for distribution, which may include filtering and encoding as needed.

[0063] Step 9:

[0064] The server then distributes the final video data with commentary to viewers over the Internet, who can then watch the video in real time or on demand via websites or applications.

[0065] The above processing steps enable users to easily create high-quality video content with commentary and distribute it in real time, even without specialized knowledge.

[0066] Example 1

[0067] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0068] In response to the increasing diversity of modern video content and the demand for a high-quality viewer experience, there is a lack of technology to automatically generate and distribute video with appropriate commentary in real time. In particular, there is a demand for fast, high-quality commentary generation for live streaming and event video. Furthermore, there is a need for a method to easily create and distribute high-quality video with commentary, even without specialized knowledge.

[0069] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0070] In this invention, the server includes means for a user terminal to acquire video data in real time using a camera or screen capture software and transmit the video data via socket communication, means for the server to receive the transmitted video data and store the received data in a buffer, means for inputting the data stored in the buffer into an AI model to analyze objects and actions in the video and generate commentary text based on event identification, means for converting the generated commentary text into audio data and generating video data with commentary audio in synchronization with the original video data, and means for the distribution server to receive the video data with commentary audio, filter and encode it, and distribute the data to viewers. This makes it possible to automatically generate high-quality video content with commentary in real time and quickly distribute it to viewers.

[0071] A "user terminal" is a device used by a user, which acquires video data in real time using a camera or screen capture software, and transmits the data to a server via socket communication.

[0072] A "camera" is a device for taking still or moving images, and is used in the present invention to capture video data in real time.

[0073] "Screen capture software" is a program that captures the contents of a computer or device screen in real time and saves and transfers it as video data.

[0074] "Socket communication" is a communication method based on a protocol for transmitting and receiving data over a network, and in the present invention is a technique used for transmitting video data from a user terminal to a server.

[0075] A "server" is a computer system that plays a central role in receiving video data sent from user terminals and analyzing, processing, and distributing it.

[0076] A "buffer" is a memory area for temporarily storing data, and serves to temporarily store received video data as a preliminary step to analysis.

[0077] An "AI model" is an algorithm or machine learning model that uses artificial intelligence technology to analyze objects and actions in video and is used to identify events and generate commentary text.

[0078] An "object" is a specific element such as a person, object, or animal that exists in a video and is the subject of analysis.

[0079] "Movement" refers to the actions and behavior of objects in a video, and is an element that can be identified through analysis.

[0080] An "event" is a notable occurrence triggered by a particular object or action, and is the basis for generating live text.

[0081] "Live text" is a sentence generated based on the analyzed events, and is character information for explaining the situation of the video.

[0082] "Speech synthesis technology" is a technology for converting text data into voice data, and is used when converting generated commentary text into voice.

[0083] "Video data" refers to all video information captured by a camera or screen capture software, and is subject to analysis and distribution.

[0084] "Filtering" is a process performed during data processing to remove noise and improve quality, and is used to optimize data before distribution.

[0085] "Encoding" is the process of converting digital data into a format, and is used to convert video data into a format that can be distributed.

[0086] "Distribution" is the process of transmitting video data over a network and making it accessible to viewers, either live or on-demand.

[0087] The present invention relates to a system for acquiring video data in real time, analyzing the data, generating appropriate commentary audio, integrating the audio with the video data, and distributing the audio. Detailed embodiments of this system will be described below.

[0088] Processing on the user terminal

[0089] The user terminal acquires video data in real time using a camera or screen capture software. This video data consists of successive frames. The user captures live video using, for example, the camera on their smartphone or PC. The captured video data is sent to the server via socket communication.

[0090] Specific examples

[0091] Imagine a scenario where a user uses a smartphone camera to capture live footage of cityscapes or events and sends the data to a server via the Internet.

[0092] Prompt Sentence Examples

[0093] "Please send this video data to the AI ​​server."

[0094] Processing on the AI ​​server

[0095] The AI ​​server receives video data sent from the user device via socket communication. The received video data is temporarily stored in a buffer. The buffered video data is then input into the AI ​​model. The AI ​​model analyzes objects and actions in the video and identifies events in real time based on the results. Commentary text is generated in response to the identified events. This commentary text is converted into audio data using speech synthesis technology. The audio data is then synchronized with the original video data to generate video data with commentary audio.

[0096] Specific examples

[0097] Live footage of a sporting event is sent to an AI server, and the AI ​​identifies player movements and goal-scoring scenes to generate commentary. The generated commentary is converted into audio data and properly synchronized with the video. Commentary such as "Player A has scored a goal!" is generated.

[0098] Prompt Sentence Examples

[0099] "Analyze the player's movements in the footage and generate appropriate commentary."

[0100] Processing on the distribution server

[0101] The distribution server receives the video data with commentary sent from the AI ​​server. The received video data is filtered and encoded as needed to convert it into the optimal format. Finally, the distribution server distributes this data to viewers via the Internet. Viewers can watch this video data with commentary in real time via a website or dedicated application.

[0102] Specific examples

[0103] Consider a situation where viewers are watching live sports broadcasts on their PCs or smartphones at home. Real-time commentary by AI provides an immersive viewing experience.

[0104] Prompt Sentence Examples

[0105] "Encode and distribute the video data with commentary integrated into it to users."

[0106] This system enables users without specialized knowledge to easily create high-quality video content with live commentary using advanced technology and quickly distribute it to viewers.

[0107] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0108] Step 1: Acquiring and transmitting video data

[0109] User terminal processing

[0110] The user device acquires video data in real time using a camera or screen capture software. The input is the video data from the camera or capture software. Specifically, each frame of video data is captured at regular intervals and created as a continuous data stream. This video data establishes a socket connection and is sent to the AI ​​server. The output is the video data sent to the server via the network.

[0111] Specific actions

[0112] A user picks up their smartphone, launches the camera app and captures live footage of the event, which is then sent in real time via a socket to the server.

[0113] Step 2: Receiving and buffering video data

[0114] Server Processing

[0115] The server receives video data sent from the user terminal. The input is video data received over the network. The received data is temporarily stored in a buffer. The buffering is for maintaining data consistency and preparing for the next analysis step. The output is the video data stored in the buffer.

[0116] Specific actions

[0117] The server receives real-time video data coming from the user terminal and stores it in a buffer in its internal memory.

[0118] Step 3: Analyzing the video data

[0119] Server Processing

[0120] The server inputs the buffered video data into the AI ​​model. The input is the buffered video data. The AI ​​model analyzes the objects and actions in the video to identify events. The analysis identifies objects in the video and determines their actions and relationships. The output is the identified event information.

[0121] Specific actions

[0122] Footage of sporting events is input into the AI ​​model, and the system analyzes and identifies player movements, goal scenes, etc.

[0123] Step 4: Generate commentary text

[0124] Server Processing

[0125] The server generates commentary text based on the identified event information. The input is the event information output from the AI ​​model. Specific text appropriate for the event is generated, and the generated commentary text becomes the commentary content to be provided to the user. The output is the generated commentary text.

[0126] Specific actions

[0127] The text "Player A scores a goal!" is generated based on the recognized event.

[0128] Step 5: Generate audio data

[0129] Server Processing

[0130] The server converts the generated commentary text into audio data using speech synthesis technology. The input is the commentary text. Using speech synthesis technology, the text is converted into audio data, and the audio is synchronized with the original video data. The output is audio data.

[0131] Specific actions

[0132] The text "Player A scores a goal!" is converted into audio and integrated into the video.

[0133] Step 6: Generate video data with commentary

[0134] Server Processing

[0135] The server synchronizes the audio data with the original video data to generate video data with live audio commentary. The input is the audio data and the original video data. These data are synchronized based on timestamps and other factors and combined into a single integrated data. The output is video data with live audio commentary.

[0136] Specific actions

[0137] The video and audio are appropriately combined to create video content with commentary.

[0138] Step 7: Streaming video data

[0139] Distribution server processing

[0140] The distribution server receives the video data with commentary sent from the AI ​​server. The input is the video data with commentary. The received video data is filtered, encoded, and optimized as needed. Finally, the data is distributed to viewers live or on demand. The output is the optimized video data distributed over the internet.

[0141] Specific actions

[0142] Viewers can watch live sports broadcasts on their PCs or smartphones at home, and the AI-powered automated commentary provided in real time creates a truly immersive experience.

[0143] (Application example 1)

[0144] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0145] Providing live commentary of live events or sports over the Internet typically requires a professional commentator. However, providing high-quality commentary in real time is difficult when there are no professional commentators or when the system relies on the commentator's schedule. Furthermore, it is technically difficult to consistently provide timely and accurate commentary for a wide variety of content at large-scale events. This can reduce viewer satisfaction and make the event less appealing. Therefore, there is a growing demand for a system that can automatically provide real-time commentary without the need for a professional commentator.

[0146] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0147] In this invention, the server includes a means for receiving video data, a means for analyzing the video data and using a generative algorithm to identify events, and a means for using generative synthesis technology to automatically generate commentary based on the identified events and output it as synchronized audio. This makes it possible to provide commentary and live commentary of events in real time without the need for a professional commentator. Viewers can always enjoy high-quality video with commentary, maximizing the appeal of the event.

[0148] "Video Data" means visual information captured from a camera or other image capture device.

[0149] The "receiving means" is a mechanism for receiving video data transmitted from a user terminal.

[0150] "Means for analyzing" means methods for processing received video data to detect and analyze images, motion, and other elements.

[0151] A "knowledge model" is an artificial intelligence model that has been trained in advance to identify events in video.

[0152] "Generative synthesis technology" is a technology that generates audio from text and integrates it with video.

[0153] The "means of distribution" refers to a mechanism for transmitting the integrated video data with commentary via the Internet or other means and delivering it to viewers.

[0154] The "transmitting device" is a device for transmitting video data from a user terminal to a server.

[0155] A "generation algorithm" is an algorithm for identifying events from video data and generating appropriate commentary content based on that information.

[0156] A "prompt algorithm" is an algorithm for converting generated speech text into a more natural format.

[0157] This invention is a system that automatically generates commentary audio for video data shot by a user in real time and distributes the video with commentary audio to viewers. This system is realized mainly by three main components: a user terminal, an AI server, and a distribution server.

[0158] Processing on the user terminal

[0159] The user device first acquires video data in real time using a camera or screen capture software. The acquired video data is then sent to the AI ​​server in real time. To do this, the user device establishes a socket connection and sequentially sends frame data to the AI ​​server.

[0160] Processing on the AI ​​server

[0161] The AI ​​server receives video data sent from the user's device. The received data is then temporarily stored in a buffer and input into a generation algorithm for video analysis. This generation algorithm analyzes objects and actions in the video and identifies events in real time based on the results. Appropriate commentary text is automatically generated based on the identified events.

[0162] This commentary text is then converted into audio data using speech synthesis technology. Generative synthesis technologies such as Google® Text-to-Speech (gTTS) are used for this purpose. The audio data is synchronized with the original video data to generate video data with commentary audio. During this process, it is important to utilize a prompting algorithm to convert the generated text into a more natural format. An example of a prompt sentence is, "Generate a commentary of a life event. This is a live video of a soccer match. Generate commentary when a player scores a goal."

[0163] Processing on the distribution server

[0164] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as filtered and encoded, before finally being distributed to viewers live or on demand. This distribution is done via the Internet, and viewers can watch the video through a dedicated website or application.

[0165] Specific examples

[0166] When a user films a soccer match using a smartphone, the video data is sent to an AI server in real time. The AI ​​server identifies player movements and goal-scoring scenes, and generates commentary such as "The player has scored a goal!" The generated commentary is converted into audio using voice synthesis technology, and this audio is integrated into the video. Viewers can enjoy the video with this commentary audio via live streaming. This system makes it possible to provide high-quality video with commentary without the need for a professional commentator.

[0167] In this way, collaboration between user devices, AI servers, and distribution servers can automatically provide real-time commentary, improving the viewer experience.

[0168] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0169] Step 1:

[0170] The user terminal captures video data in real time using a camera or screen capture software. The camera device or screen capture software is used as input. The output is the captured video data. This video data consists of a series of frames, each captured at a fixed interval.

[0171] Step 2:

[0172] The user terminal establishes a socket communication to send the acquired video data to the AI ​​server in real time. The acquired video data is used as input. The output is the transmitted frame data. This frame data is sent to the AI ​​server sequentially.

[0173] Step 3:

[0174] The AI ​​server receives video data sent from the user terminal. The frame data sent from the user terminal is used as input. The output is video data temporarily stored in a buffer.

[0175] Step 4:

[0176] The AI ​​server inputs the video data stored in the buffer into the generative algorithm. The video data is used as input. The output is the analysis result. The generative AI model analyzes the objects and actions in the video and generates the results.

[0177] Step 5:

[0178] The AI ​​server identifies events based on the analysis results. The analysis results of the generative AI model are used as input. The output is the identified event information. This information indicates a specific event (e.g., a goal is scored).

[0179] Step 6:

[0180] The AI ​​server automatically generates commentary text based on the identified events. The identified event information is used as input. The output is the generated commentary text. An example prompt sentence is "Generate a commentary of a life event. This is a live video of a soccer match. Generate a commentary when a player scores a goal."

[0181] Step 7:

[0182] The AI ​​server converts the generated commentary text into audio data. The commentary text is used as input. The output is synthesized audio data. This conversion uses voice synthesis technology such as Google Text-to-Speech (gTTS).

[0183] Step 8:

[0184] The AI ​​server synchronizes the audio data with the original video data. The synthesized audio data and the original video data are used as input. The output is video data with commentary. This data is played back with the audio and video synchronized.

[0185] Step 9:

[0186] The distribution server receives the video data with commentary. As input, it uses the video data with commentary sent from the AI ​​server. The output is video data ready for distribution.

[0187] Step 10:

[0188] The distribution server processes the data as necessary, such as filtering and encoding. The input is video data with commentary. The output is the processed video data.

[0189] Step 11:

[0190] The distribution server delivers the final video data to the viewer. The processed video data is used as input. The output is live or on-demand video delivered to the viewer's device. The viewer can then watch this video through a website or application.

[0191] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0192] This invention combines a system that receives video data, analyzes it, generates live audio, integrates it with the video, and distributes it, with an emotion engine that recognizes the user's emotions. The program of this system and its processing are described in detail below.

[0193] Processing on the user terminal

[0194] The user device first acquires video data in real time using a camera or screen capture software. This video data consists of a series of frames, each captured at a fixed interval. Then, a socket connection is established to transmit the acquired video data to the AI ​​server in real time. Through this connection, the frame data is sent to the AI ​​server.

[0195] In a specific example, a user captures video using the camera on their smartphone or PC, and the video is sent to a server via a network. At the same time, the user's facial expressions and voice data are also acquired and analyzed by the emotion engine.

[0196] Processing on the AI ​​server

[0197] The AI ​​server receives video data and the user's voice and facial expression data sent from the user's device. First, it temporarily stores the received video data and user emotion data in a buffer. The data is then input into a video analysis model, which analyzes objects and actions in the digital video in real time and identifies specific events and scenes. Meanwhile, the emotion engine analyzes emotions from the user's facial expressions and voice and takes the results into account.

[0198] Based on the analysis results, appropriate commentary text is generated. This commentary text includes the recognized events as well as content corresponding to the user's emotions. Next, this commentary text is converted into audio data using speech synthesis technology. The commentary audio is synchronized with the original video data to generate video data with commentary audio corresponding to the user's emotions.

[0199] As a concrete example, real-time video of a sporting event is sent to an AI server, and the AI ​​identifies the players' movements and scoring scenes, generating commentary such as, "The player has just scored a goal!" When this commentary is converted into audio, rich emotional voice is generated according to the user's feelings of excitement and joy.

[0200] Processing on the distribution server

[0201] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as by filtering or encoding. Finally, the processed video data with commentary is distributed to viewers live or on demand. Distribution is carried out over the internet, and viewers can watch it via a website or app.

[0202] For example, users can watch live sports broadcasts on their smartphones or PCs, and AI-powered commentary is provided in real time, improving the viewing experience even without the need for a professional commentator. Furthermore, commentary that responds to the user's emotions creates a greater sense of immersion.

[0203] This system automatically analyzes video data, generates appropriate commentary, and delivers it to viewers. This allows even users without specialized knowledge to easily create and distribute high-quality video content with commentary. Furthermore, commentary that reflects the user's emotions further enhances the viewing experience.

[0204] The processing flow will be explained below.

[0205] Step 1:

[0206] The user device captures video data in real time through a camera or screen capture software. This video data consists of a series of frames, each captured at regular intervals. At the same time, data is also captured to record the user's facial expressions and voice.

[0207] Step 2:

[0208] The user device establishes a socket connection to transmit the captured video data and the user's facial expression and voice data to the AI ​​server in real time. Through this connection, frame data and emotion data are transferred to the AI ​​server.

[0209] Step 3:

[0210] The server receives the video data and the user's facial expression and voice data sent from the user's device. The received data is temporarily stored in a buffer and prepared for analysis.

[0211] Step 4:

[0212] The server inputs the received video data into an AI model for video analysis, which analyzes objects and actions in the digital video in real time to identify specific events and scenes.

[0213] Step 5:

[0214] The server inputs the user's facial and voice data into the emotion engine, which analyzes this data and identifies the user's emotional state (e.g., joy, surprise, excitement, etc.) in real time.

[0215] Step 6:

[0216] The server generates appropriate commentary text based on the video analysis results and the emotion analysis results from the emotion engine. For example, when a goal is scored in a sporting event, the server generates the text "A goal has been scored!" and emphasizes the text according to the user's excitement level.

[0217] Step 7:

[0218] The server converts the generated commentary text into voice data using speech synthesis technology, and in the process adds the speaker's tone and intonation according to the user's emotions.

[0219] Step 8:

[0220] The server synchronizes the generated audio data with the original video data, creating video data with live audio. During this integration process, processing is performed to align the timing of the video and audio.

[0221] Step 9:

[0222] The server converts the video data with live audio into a format for distribution, which may include filtering and encoding as needed.

[0223] Step 10:

[0224] The server then distributes the final video data with commentary to viewers over the Internet, who can then watch the video in real time or on demand via websites or applications.

[0225] The above processing steps enable users to easily create high-quality video content with commentary and distribute it in real time, even without specialized knowledge. Furthermore, commentary that reflects the user's emotions makes the viewing experience even more engaging.

[0226] Example 2

[0227] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0228] Conventional video distribution systems require manual analysis of video and generation of commentary, which requires a great deal of time and effort. Furthermore, it is difficult to provide commentary that reflects the user's real-time emotions, and there is a demand for an improved viewing experience. Furthermore, it is difficult for users without specialized knowledge to easily create and distribute high-quality video content with commentary.

[0229] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0230] In this invention, the server includes means for acquiring video data, means for transmitting the acquired video data in real time, means for receiving and analyzing the video data and user emotion data, means for generating commentary text based on the analysis results, means for converting the generated commentary text into audio, means for integrating the generated commentary audio into the video data, and means for distributing the integrated video data with the commentary audio. This makes it possible to automatically analyze the video data and generate and distribute commentary corresponding to the user's emotions in real time.

[0231] "Video data" is a collection of visual information captured using a camera or capture device.

[0232] "Means of acquisition" refers to the method of collecting video data using a camera or capture device, or software that controls them.

[0233] A "transmitting means" is a method or protocol for transmitting data over a network to another device or server.

[0234] "Means for receiving and analyzing" refers to methods for receiving transmitted data and using artificial intelligence or other technologies to decipher and analyze its contents.

[0235] "Live text" is text that explains events and situations in the video.

[0236] A "means of generation" is a method of creating new data or information using a specific algorithm or program.

[0237] "Means for converting to voice" refers to technology or software for converting text data into voice data.

[0238] "Means of integration" is the process of bringing together multiple pieces of data or information into one.

[0239] "Means of distribution" refers to the communication method or technology used to deliver the final data to users or viewers.

[0240] A "generative AI model" is an algorithm or tool that uses artificial intelligence techniques to generate new data or content.

[0241] "Emotion data" is information about the emotional state of the user collected from facial expressions, voice, etc.

[0242] "Real-time" means that data is processed and transmitted in a very short time, providing near-immediate results.

[0243] This invention combines a system that receives video data, analyzes it, generates live audio, integrates it with the video, and distributes it, with an emotion engine that recognizes the user's emotions. The program of this system and its processing are described in detail below.

[0244] The user device first acquires video data in real time using a camera or screen capture software. This video data consists of a series of frames, each captured at a fixed interval. Then, a socket connection is established to transmit the acquired video data to the AI ​​server in real time. Through this connection, the frame data is sent to the AI ​​server.

[0245] In a specific example, a user captures video using the camera on their smartphone or PC, and the video is sent to a server via a network. At the same time, the user's facial expressions and voice data are also acquired and analyzed by the emotion engine.

[0246] The AI ​​server receives video data and the user's voice and facial expression data sent from the user's device. First, it temporarily stores the received video data and the user's emotional data in a buffer. The data is then input into a video analysis model, which analyzes objects and actions in the digital video in real time and identifies specific events and scenes. Image processing libraries such as OpenCV are used for video analysis.

[0247] Meanwhile, the emotion engine analyzes the user's emotions from their facial expressions and voice and stores the results in a buffer. For emotion analysis, a service such as Microsoft® Azure®'s Emotion API is used.

[0248] Based on the analysis results, appropriate commentary text is generated. This commentary text includes the recognized events as well as content corresponding to the user's emotions. For example, a natural language generation model (such as GPT-3 (registered trademark)) is used to generate the text. Next, the generated commentary text is converted into audio data using speech synthesis technology. For speech synthesis, Google Text-to-Speech API or similar is used. The commentary audio is synchronized with the original video data, and video data with commentary audio corresponding to the user's emotions is generated.

[0249] As a concrete example, real-time video of a sporting event is sent to an AI server, and the AI ​​identifies the players' movements and scoring scenes, generating commentary such as, "The player has just scored a goal!" When this commentary is converted into audio, rich emotional voice is generated according to the user's feelings of excitement and joy.

[0250] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as by filtering or encoding. FFmpeg is used for filtering and encoding, for example. Finally, the processed video data with commentary is distributed to viewers live or on demand. Distribution is carried out over the internet, and viewers can watch it via a website or app.

[0251] For example, users can watch live sports broadcasts on their smartphones or PCs, and AI-powered commentary is provided in real time, improving the viewing experience even without the need for a professional commentator. Furthermore, commentary that responds to the user's emotions creates a greater sense of immersion.

[0252] Example prompt sentence:

[0253] "Analyze video data from a soccer match in real time, recognize player movements and goal-scoring scenes, and generate commentary. If the user's emotion is classified as 'excited,' include an emotionally rich expression in the commentary. For example, say, 'The player scored a goal! What a great shot!'"

[0254] This system automatically analyzes video data, generates appropriate commentary, and delivers it to viewers. This allows even users without specialized knowledge to easily create and distribute high-quality video content with commentary. Furthermore, commentary that reflects the user's emotions further enhances the viewing experience.

[0255] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0256] Step 1:

[0257] A user device acquires video data using a camera or screen capture software (e.g., OBS Studio). The acquired video data consists of a series of frames. The input is camera video or screen capture, and the output is video data (frame data). For example, a user might launch a camera app on their smartphone and capture live video.

[0258] Step 2:

[0259] The user device establishes a socket connection to send the acquired video data to the AI ​​server in real time. Specifically, the Python socket library is used to periodically send video frame data to the server. The input is the video data, and the output is the frame data sent to the server. During this process, the user's facial expressions and voice data are also collected at the same time.

[0260] Step 3:

[0261] The server receives the video data and the user's voice and facial expression data sent from the user's device and temporarily stores them in a buffer. The input is the video data and emotion data from the user's device, and the output is the data stored in the buffer. For example, when the video data and voice data arrive at the server, they are temporarily saved.

[0262] Step 4:

[0263] The server analyzes the received video data using the OpenCV library. Specifically, it identifies objects and actions in the digital video in real time. The input is the video data stored in the buffer, and the output is the analyzed video data (event identification results). For example, it analyzes video footage of a sporting event to identify player movements and goal scenes.

[0264] Step 5:

[0265] The server uses an emotion engine (e.g., Emotion API) to analyze emotions from the user's facial expressions and voice. The input is the user's facial expression and voice data, and the output is the analyzed emotion data. For example, the emotion engine analyzes the user's camera footage and determines that the user is "excited."

[0266] Step 6:

[0267] The server generates commentary text based on the video data analysis results and emotion data. In this process, a generative AI model (e.g., GPT-3) is used to generate the text. The input is the analyzed event data and emotion data, and the output is the commentary text. For example, a commentary such as "The player scored a goal!" is generated.

[0268] Step 7:

[0269] The server converts the generated commentary text into audio data using speech synthesis technology (e.g., Google Text-to-Speech API). The input is the commentary text, and the output is audio data. For example, the text "The player scored a goal!" is converted into audio.

[0270] Step 8:

[0271] The server synchronizes the generated commentary with the original video data. The input is audio data and video data, and the output is video data with commentary. For example, the generated commentary is adjusted so that it is played at the appropriate timing within the video.

[0272] Step 9:

[0273] The server sends the video data with commentary to the distribution server. The input is the video data with commentary, and the output is the data sent to the distribution server. For example, the audio and video are integrated using FFmpeg, and then the data is sent to the distribution server.

[0274] (Application example 2)

[0275] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0276] While conventional video commentary systems can provide commentary based on visual information or specific events, it is difficult to provide personalized commentary that reflects the viewer's emotions in real time. Furthermore, there are limited ways to enhance the viewer's sense of realism and immersion. This makes it impossible for the viewing experience to be uniform and reflect the individual excitement and emotions.

[0277] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0278] In this invention, the server includes means for receiving video data, means for analyzing the received video data, means for recognizing a user's emotion, means for generating commentary audio based on the analysis result and the user's emotion data, means for integrating the generated commentary audio into the video data, and means for delivering the integrated video data with commentary audio. This enables personalized commentary that reflects the viewer's emotion, thereby improving the viewing experience.

[0279] "Video data" is a collection of visual information captured by a camera or other image capture device.

[0280] "Means for analyzing" refers to a processing device or algorithm that analyzes received video data or other information and recognizes certain patterns or events.

[0281] "Means for recognizing user emotions" refers to technology that analyzes a user's facial expressions, voice, and other biometric data to determine their emotional state.

[0282] The "means for generating live audio" is a technology for generating audio data that verbally explains or provides commentary to viewers based on video data and emotional data.

[0283] The "means of integration" refers to the means of integrating the generated live audio with the original video data to create a single visual and audio content.

[0284] "Means of distribution" refers to the technology and infrastructure for delivering the integrated video data with commentary to viewers over the Internet.

[0285] "Artificial intelligence" refers to computer systems or algorithms that mimic human intellectual processes by analyzing and learning from large amounts of data.

[0286] "Speech synthesis technology" is a technology for analyzing text information and converting it into natural speech.

[0287] This invention is a system that receives and analyzes video data in real time, recognizes the user's emotions, and generates and distributes live commentary that integrates this information. The detailed configuration and specific operation of this system are described below.

[0288] Hardware Configuration

[0289] User device:

[0290] Smart glasses: Equipped with a camera for video capture and acquisition of user facial expression data.

[0291] Microphone: Used to capture the user's voice data.

[0292] Speaker: Used to provide the generated commentary to the user.

[0293] server:

[0294] AI Server: Used to analyze video data and emotional data. Specifically, it is equipped with an AI model for analyzing video data and an emotion engine for analyzing user emotions.

[0295] Distribution server: A server that distributes integrated video data with live audio to viewers in real time.

[0296] Software Configuration

[0297] On the user's device:

[0298] The user device first acquires video data in real time using the camera installed in the smart glasses. This video data consists of frames captured at regular intervals. The user's voice data is also acquired through the microphone. This data is then sent to the AI ​​server via a socket connection.

[0299] Processing on the AI ​​server:

[0300] The AI ​​server receives video data and audio / facial expression data sent from the user's device. The received video data and emotional data are temporarily stored in a buffer and then input into an AI model for video analysis and an emotion engine that analyzes emotions from facial expressions and audio. This allows specific events and scenes in the video to be identified, while also analyzing the user's emotional state. Based on the analysis results, appropriate commentary text is generated, and this commentary text is converted into audio data using speech synthesis technology.

[0301] Processing on the distribution server:

[0302] The distribution server receives the integrated video data with live commentary from the AI ​​server, encodes and filters it in real time, and then distributes it to viewers, who can view the data on their smartphones, PCs, or other devices.

[0303] Specific examples

[0304] As a specific example of use, consider a user watching a soccer match. When the user wears smart glasses and watches a soccer match in real time, the smart glasses' camera captures video data, and at the same time, the user's facial expression data is captured. This data is sent to the AI ​​server, which recognizes important events such as goal scenes and further analyzes the user's emotions. If a goal scene is recognized and the user is excited, a live commentary voice saying "Goal! The user is excited too!" is generated and delivered to the user in real time through the smart glasses' speaker.

[0305] Prompt Sentence Examples

[0306] Examples of prompts include:

[0307] While the user is watching a sporting event, generate the most compelling commentary based on real-time visual and emotional data that meets the following criteria:

[0308] Users' attention is focused on the players

[0309] The user's facial expression is excited (recognized by the emotion engine)

[0310] Video analysis engine detects goal scenes

[0311] Example of expected output:

[0312] "Goal! The users are excited!" or "Great play! The users look very happy!"

[0313] Thus, the present invention combines video analysis and emotion recognition to provide viewers with an immersive, personalized, and live experience.

[0314] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0315] Step 1:

[0316] The device uses the smart glasses' camera to capture video data in real time, acquiring video frames at specific time intervals, and also uses the microphone to capture the user's voice data.

[0317] Input: Video frames from camera, audio data from microphone

[0318] Output: Captured video and audio data

[0319] Specific operation: Activate the camera of the smart glasses, capture frames at regular intervals, and simultaneously capture audio through the microphone.

[0320] Step 2:

[0321] The device establishes a socket communication to transmit the captured video and audio data to the AI ​​server in real time.

[0322] Input: Captured video and audio data

[0323] Output: Data sent to the AI ​​server

[0324] Specific operation: Establish socket communication and send acquired data to the AI ​​server.

[0325] Step 3:

[0326] The server receives the video data and audio data sent from the terminal and temporarily stores them in a buffer.

[0327] Input: Video and audio data sent from the device

[0328] Output: Buffered data

[0329] Specific operation: Received data is stored in a buffer and prepared for analysis.

[0330] Step 4:

[0331] The server analyzes the received video data using an AI model to identify specific events or scenes (e.g., goal scenes), and uses an emotion engine to analyze the user's voice and facial expression data to determine their emotional state.

[0332] Input: Video and audio data stored in the buffer

[0333] Output: Parsed event data and emotion data

[0334] Specific operation: Video data is input into the AI ​​model to identify specific events, and voice and facial expression data is input into the emotion engine to determine emotions.

[0335] Step 5:

[0336] The server generates appropriate commentary text based on the parsed event data and emotion data.

[0337] Input: Parsed event data and emotion data

[0338] Output: Generated commentary text

[0339] Specific operation: Create text based on event data and adjust commentary content to reflect emotional data.

[0340] Step 6:

[0341] The server converts the generated commentary text into voice data using voice synthesis technology.

[0342] Input: Generated commentary text

[0343] Output: Generated live audio data

[0344] What it does: Inputs text into a speech synthesis engine and generates a natural-sounding audio file.

[0345] Step 7:

[0346] The server integrates the generated commentary audio data with the original video data to generate video data with commentary audio.

[0347] Input: Original video data, generated live audio data

[0348] Output: Video data with integrated commentary

[0349] Specific operation: Synchronizes audio data and video data and adds audio to video.

[0350] Step 8:

[0351] The server delivers the integrated video data with live audio in real time to viewers, who can view the data via their smartphones or PC devices.

[0352] Input: Video data with integrated commentary

[0353] Output: Video data delivered to viewers

[0354] Specific operation: Encodes video data and distributes it in real time over the Internet.

[0355] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0356] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0357] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0358] [Second embodiment]

[0359] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0360] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0361] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0362] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0363] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0364] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0365] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0366] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0367] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0368] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0369] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0370] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0371] This invention relates to a system that receives video data, analyzes it, generates live audio, integrates it with the video, and distributes it. The program of this system and its processing are described in detail below.

[0372] Processing on the user terminal

[0373] The user device first acquires video data in real time using a camera or screen capture software. This video data consists of a series of frames, each captured at a fixed interval. Then, a socket connection is established to transmit the acquired video data to the AI ​​server in real time. Through this connection, the frame data is sent to the AI ​​server.

[0374] In a specific example, a user captures video using the camera on their smartphone or PC, and the video is sent to a server via a network.

[0375] Processing on the AI ​​server

[0376] The AI ​​server receives video data sent from the user's device. The received data is temporarily stored in a buffer and then input into a model for video analysis. The AI ​​model analyzes objects and actions in the digital video and identifies events in real time based on the results. Appropriate commentary text is generated based on the recognized events. This commentary text is then converted into audio data using speech synthesis technology. This audio data is synchronized with the original video data to generate video data with commentary audio.

[0377] For example, real-time video of a sporting event is sent to an AI server, where the AI ​​identifies players' movements and scoring scenes, and generates commentary such as, "The player has just scored a goal!" This commentary is converted into audio and combined with the video.

[0378] Processing on the distribution server

[0379] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as by filtering or encoding. Finally, the processed video data with commentary is distributed to viewers live or on demand. Distribution is carried out over the internet, and viewers can watch it via a website or app.

[0380] In a specific example, users can watch live sports broadcasts on their smartphones or PCs, and AI-powered commentary will be provided in real time, improving the viewing experience even without the need for a professional commentator.

[0381] This system automatically analyzes video data, generates appropriate commentary, and delivers it to viewers, allowing even users without specialized knowledge to easily create and distribute high-quality video content with commentary.

[0382] The processing flow will be explained below.

[0383] Step 1:

[0384] The user device acquires video data in real time through a camera or screen capture software. The video data consists of a series of frames, each captured at a fixed interval.

[0385] Step 2:

[0386] The user device establishes a socket connection to transmit the acquired video data to the AI ​​server in real time. Through this connection, frame data is transferred to the AI ​​server.

[0387] Step 3:

[0388] The server receives video data sent from the user terminal, temporarily stores the received data in a buffer, and prepares it for analysis.

[0389] Step 4:

[0390] The server inputs the received data into an AI model for video analysis, which analyzes objects and actions in the video in real time to identify specific events and scenes.

[0391] Step 5:

[0392] The server generates appropriate commentary text based on the analysis results. Commentary content is defined for recognized events (e.g., goal shots, goals scored, transitions between offense and defense, etc.).

[0393] Step 6:

[0394] The server converts the generated commentary text into audio data using speech synthesis technology. In this process, the commentary text is converted into audio data that has the speaker's voice and intonation.

[0395] Step 7:

[0396] The server synchronizes the generated audio data with the original video data, creating video data with live audio. During this integration process, processing is performed to align the timing of the video and audio.

[0397] Step 8:

[0398] The server converts the video data with live audio into a format for distribution, which may include filtering and encoding as needed.

[0399] Step 9:

[0400] The server then distributes the final video data with commentary to viewers over the Internet, who can then watch the video in real time or on demand via websites or applications.

[0401] The above processing steps enable users to easily create high-quality video content with commentary and distribute it in real time, even without specialized knowledge.

[0402] Example 1

[0403] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0404] In response to the increasing diversity of modern video content and the demand for a high-quality viewer experience, there is a lack of technology to automatically generate and distribute video with appropriate commentary in real time. In particular, there is a demand for fast, high-quality commentary generation for live streaming and event video. Furthermore, there is a need for a method to easily create and distribute high-quality video with commentary, even without specialized knowledge.

[0405] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0406] In this invention, the server includes means for a user terminal to acquire video data in real time using a camera or screen capture software and transmit the video data via socket communication, means for the server to receive the transmitted video data and store the received data in a buffer, means for inputting the data stored in the buffer into an AI model to analyze objects and actions in the video and generate commentary text based on event identification, means for converting the generated commentary text into audio data and generating video data with commentary audio in synchronization with the original video data, and means for the distribution server to receive the video data with commentary audio, filter and encode it, and distribute the data to viewers. This makes it possible to automatically generate high-quality video content with commentary in real time and quickly distribute it to viewers.

[0407] A "user terminal" is a device used by a user, which acquires video data in real time using a camera or screen capture software, and transmits the data to a server via socket communication.

[0408] A "camera" is a device for taking still or moving images, and is used in the present invention to capture video data in real time.

[0409] "Screen capture software" is a program that captures the contents of a computer or device screen in real time and saves and transfers it as video data.

[0410] "Socket communication" is a communication method based on a protocol for transmitting and receiving data over a network, and in the present invention is a technique used for transmitting video data from a user terminal to a server.

[0411] A "server" is a computer system that plays a central role in receiving video data sent from user terminals and analyzing, processing, and distributing it.

[0412] A "buffer" is a memory area for temporarily storing data, and serves to temporarily store received video data as a preliminary step to analysis.

[0413] An "AI model" is an algorithm or machine learning model that uses artificial intelligence technology to analyze objects and actions in video and is used to identify events and generate commentary text.

[0414] An "object" is a specific element such as a person, object, or animal that exists in a video and is the subject of analysis.

[0415] "Movement" refers to the actions and behavior of objects in a video, and is an element that can be identified through analysis.

[0416] An "event" is a notable occurrence triggered by a particular object or action, and is the basis for generating live text.

[0417] "Live text" is a sentence generated based on the analyzed events, and is character information for explaining the situation of the video.

[0418] "Speech synthesis technology" is a technology for converting text data into voice data, and is used when converting generated commentary text into voice.

[0419] "Video data" refers to all video information captured by a camera or screen capture software, and is subject to analysis and distribution.

[0420] "Filtering" is a process performed during data processing to remove noise and improve quality, and is used to optimize data before distribution.

[0421] "Encoding" is the process of converting digital data into a format, and is used to convert video data into a format that can be distributed.

[0422] "Distribution" is the process of transmitting video data over a network and making it accessible to viewers, either live or on-demand.

[0423] The present invention relates to a system for acquiring video data in real time, analyzing the data, generating appropriate commentary audio, integrating the audio with the video data, and distributing the audio. Detailed embodiments of this system will be described below.

[0424] Processing on the user terminal

[0425] The user terminal acquires video data in real time using a camera or screen capture software. This video data consists of successive frames. The user captures live video using, for example, the camera on their smartphone or PC. The captured video data is sent to the server via socket communication.

[0426] Specific examples

[0427] Imagine a scenario where a user uses a smartphone camera to capture live footage of cityscapes or events and sends the data to a server via the Internet.

[0428] Prompt Sentence Examples

[0429] "Please send this video data to the AI ​​server."

[0430] Processing on the AI ​​server

[0431] The AI ​​server receives video data sent from the user device via socket communication. The received video data is temporarily stored in a buffer. The buffered video data is then input into the AI ​​model. The AI ​​model analyzes objects and actions in the video and identifies events in real time based on the results. Commentary text is generated in response to the identified events. This commentary text is converted into audio data using speech synthesis technology. The audio data is then synchronized with the original video data to generate video data with commentary audio.

[0432] Specific examples

[0433] Live footage of a sporting event is sent to an AI server, and the AI ​​identifies player movements and goal-scoring scenes to generate commentary. The generated commentary is converted into audio data and properly synchronized with the video. Commentary such as "Player A has scored a goal!" is generated.

[0434] Prompt Sentence Examples

[0435] "Analyze the player's movements in the footage and generate appropriate commentary."

[0436] Processing on the distribution server

[0437] The distribution server receives the video data with commentary sent from the AI ​​server. The received video data is filtered and encoded as needed to convert it into the optimal format. Finally, the distribution server distributes this data to viewers via the Internet. Viewers can watch this video data with commentary in real time via a website or dedicated application.

[0438] Specific examples

[0439] Consider a situation where viewers are watching live sports broadcasts on their PCs or smartphones at home. Real-time commentary by AI provides an immersive viewing experience.

[0440] Prompt Sentence Examples

[0441] "Encode and distribute the video data with commentary integrated into it to users."

[0442] This system enables users without specialized knowledge to easily create high-quality video content with live commentary using advanced technology and quickly distribute it to viewers.

[0443] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0444] Step 1: Acquiring and transmitting video data

[0445] User terminal processing

[0446] The user device acquires video data in real time using a camera or screen capture software. The input is the video data from the camera or capture software. Specifically, each frame of video data is captured at regular intervals and created as a continuous data stream. This video data establishes a socket connection and is sent to the AI ​​server. The output is the video data sent to the server via the network.

[0447] Specific actions

[0448] A user picks up their smartphone, launches the camera app and captures live footage of the event, which is then sent in real time via a socket to the server.

[0449] Step 2: Receiving and buffering video data

[0450] Server Processing

[0451] The server receives video data sent from the user terminal. The input is video data received over the network. The received data is temporarily stored in a buffer. The buffering is for maintaining data consistency and preparing for the next analysis step. The output is the video data stored in the buffer.

[0452] Specific actions

[0453] The server receives real-time video data coming from the user terminal and stores it in a buffer in its internal memory.

[0454] Step 3: Analyzing the video data

[0455] Server Processing

[0456] The server inputs the buffered video data into the AI ​​model. The input is the buffered video data. The AI ​​model analyzes the objects and actions in the video to identify events. The analysis identifies objects in the video and determines their actions and relationships. The output is the identified event information.

[0457] Specific actions

[0458] Footage of sporting events is input into the AI ​​model, and the system analyzes and identifies player movements, goal scenes, etc.

[0459] Step 4: Generate commentary text

[0460] Server Processing

[0461] The server generates commentary text based on the identified event information. The input is the event information output from the AI ​​model. Specific text appropriate for the event is generated, and the generated commentary text becomes the commentary content to be provided to the user. The output is the generated commentary text.

[0462] Specific actions

[0463] The text "Player A scores a goal!" is generated based on the recognized event.

[0464] Step 5: Generate audio data

[0465] Server Processing

[0466] The server converts the generated commentary text into audio data using speech synthesis technology. The input is the commentary text. Using speech synthesis technology, the text is converted into audio data, and the audio is synchronized with the original video data. The output is audio data.

[0467] Specific actions

[0468] The text "Player A scores a goal!" is converted into audio and integrated into the video.

[0469] Step 6: Generate video data with commentary

[0470] Server Processing

[0471] The server synchronizes the audio data with the original video data to generate video data with live audio commentary. The input is the audio data and the original video data. These data are synchronized based on timestamps and other factors and combined into a single integrated data. The output is video data with live audio commentary.

[0472] Specific actions

[0473] The video and audio are appropriately combined to create video content with commentary.

[0474] Step 7: Streaming video data

[0475] Distribution server processing

[0476] The distribution server receives the video data with commentary sent from the AI ​​server. The input is the video data with commentary. The received video data is filtered, encoded, and optimized as needed. Finally, the data is distributed to viewers live or on demand. The output is the optimized video data distributed over the internet.

[0477] Specific actions

[0478] Viewers can watch live sports broadcasts on their PCs or smartphones at home, and the AI-powered automated commentary provided in real time creates a truly immersive experience.

[0479] (Application example 1)

[0480] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0481] Providing live commentary of live events or sports over the Internet typically requires a professional commentator. However, providing high-quality commentary in real time is difficult when there are no professional commentators or when the system relies on the commentator's schedule. Furthermore, it is technically difficult to consistently provide timely and accurate commentary for a wide variety of content at large-scale events. This can reduce viewer satisfaction and make the event less appealing. Therefore, there is a growing demand for a system that can automatically provide real-time commentary without the need for a professional commentator.

[0482] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0483] In this invention, the server includes a means for receiving video data, a means for analyzing the video data and using a generative algorithm to identify events, and a means for using generative synthesis technology to automatically generate commentary based on the identified events and output it as synchronized audio. This makes it possible to provide commentary and live commentary of events in real time without the need for a professional commentator. Viewers can always enjoy high-quality video with commentary, maximizing the appeal of the event.

[0484] "Video Data" means visual information captured from a camera or other image capture device.

[0485] The "receiving means" is a mechanism for receiving video data transmitted from a user terminal.

[0486] "Means for analyzing" means methods for processing received video data to detect and analyze images, motion, and other elements.

[0487] A "knowledge model" is an artificial intelligence model that has been trained in advance to identify events in video.

[0488] "Generative synthesis technology" is a technology that generates audio from text and integrates it with video.

[0489] The "means of distribution" refers to a mechanism for transmitting the integrated video data with commentary via the Internet or other means and delivering it to viewers.

[0490] The "transmitting device" is a device for transmitting video data from a user terminal to a server.

[0491] A "generation algorithm" is an algorithm for identifying events from video data and generating appropriate commentary content based on that information.

[0492] A "prompt algorithm" is an algorithm for converting generated speech text into a more natural format.

[0493] This invention is a system that automatically generates commentary audio for video data shot by a user in real time and distributes the video with commentary audio to viewers. This system is realized mainly by three main components: a user terminal, an AI server, and a distribution server.

[0494] Processing on the user terminal

[0495] The user device first acquires video data in real time using a camera or screen capture software. The acquired video data is then sent to the AI ​​server in real time. To do this, the user device establishes a socket connection and sequentially sends frame data to the AI ​​server.

[0496] Processing on the AI ​​server

[0497] The AI ​​server receives video data sent from the user's device. The received data is then temporarily stored in a buffer and input into a generation algorithm for video analysis. This generation algorithm analyzes objects and actions in the video and identifies events in real time based on the results. Appropriate commentary text is automatically generated based on the identified events.

[0498] This commentary text is then converted into audio data using speech synthesis technology. Generative synthesis technologies such as Google Text-to-Speech (gTTS) are used for this purpose. The audio data is synchronized with the original video data to generate video data with commentary audio. During this process, it is important to utilize a prompting algorithm to convert the generated text into a more natural format. An example of a prompt sentence is, "Generate a commentary of a life event. This is a live video of a soccer match. Generate commentary when a player scores a goal."

[0499] Processing on the distribution server

[0500] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as filtered and encoded, before finally being distributed to viewers live or on demand. This distribution is done via the Internet, and viewers can watch the video through a dedicated website or application.

[0501] Specific examples

[0502] When a user films a soccer match using a smartphone, the video data is sent to an AI server in real time. The AI ​​server identifies player movements and goal-scoring scenes, and generates commentary such as "The player has scored a goal!" The generated commentary is converted into audio using voice synthesis technology, and this audio is integrated into the video. Viewers can enjoy the video with this commentary audio via live streaming. This system makes it possible to provide high-quality video with commentary without the need for a professional commentator.

[0503] In this way, collaboration between user devices, AI servers, and distribution servers can automatically provide real-time commentary, improving the viewer experience.

[0504] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0505] Step 1:

[0506] The user terminal captures video data in real time using a camera or screen capture software. The camera device or screen capture software is used as input. The output is the captured video data. This video data consists of a series of frames, each captured at a fixed interval.

[0507] Step 2:

[0508] The user terminal establishes a socket communication to send the acquired video data to the AI ​​server in real time. The acquired video data is used as input. The output is the transmitted frame data. This frame data is sent to the AI ​​server sequentially.

[0509] Step 3:

[0510] The AI ​​server receives video data sent from the user terminal. The frame data sent from the user terminal is used as input. The output is video data temporarily stored in a buffer.

[0511] Step 4:

[0512] The AI ​​server inputs the video data stored in the buffer into the generative algorithm. The video data is used as input. The output is the analysis result. The generative AI model analyzes the objects and actions in the video and generates the results.

[0513] Step 5:

[0514] The AI ​​server identifies events based on the analysis results. The analysis results of the generative AI model are used as input. The output is the identified event information. This information indicates a specific event (e.g., a goal is scored).

[0515] Step 6:

[0516] The AI ​​server automatically generates commentary text based on the identified events. The identified event information is used as input. The output is the generated commentary text. An example prompt sentence is "Generate a commentary of a life event. This is a live video of a soccer match. Generate a commentary when a player scores a goal."

[0517] Step 7:

[0518] The AI ​​server converts the generated commentary text into audio data. The commentary text is used as input. The output is synthesized audio data. This conversion uses voice synthesis technology such as Google Text-to-Speech (gTTS).

[0519] Step 8:

[0520] The AI ​​server synchronizes the audio data with the original video data. The synthesized audio data and the original video data are used as input. The output is video data with commentary. This data is played back with the audio and video synchronized.

[0521] Step 9:

[0522] The distribution server receives the video data with commentary. As input, it uses the video data with commentary sent from the AI ​​server. The output is video data ready for distribution.

[0523] Step 10:

[0524] The distribution server processes the data as necessary, such as filtering and encoding. The input is video data with commentary. The output is the processed video data.

[0525] Step 11:

[0526] The distribution server delivers the final video data to the viewer. The processed video data is used as input. The output is live or on-demand video delivered to the viewer's device. The viewer can then watch this video through a website or application.

[0527] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0528] This invention combines a system that receives video data, analyzes it, generates live audio, integrates it with the video, and distributes it, with an emotion engine that recognizes the user's emotions. The program of this system and its processing are described in detail below.

[0529] Processing on the user terminal

[0530] The user device first acquires video data in real time using a camera or screen capture software. This video data consists of a series of frames, each captured at a fixed interval. Then, a socket connection is established to transmit the acquired video data to the AI ​​server in real time. Through this connection, the frame data is sent to the AI ​​server.

[0531] In a specific example, a user captures video using the camera on their smartphone or PC, and the video is sent to a server via a network. At the same time, the user's facial expressions and voice data are also acquired and analyzed by the emotion engine.

[0532] Processing on the AI ​​server

[0533] The AI ​​server receives video data and the user's voice and facial expression data sent from the user's device. First, it temporarily stores the received video data and user emotion data in a buffer. The data is then input into a video analysis model, which analyzes objects and actions in the digital video in real time and identifies specific events and scenes. Meanwhile, the emotion engine analyzes emotions from the user's facial expressions and voice and takes the results into account.

[0534] Based on the analysis results, appropriate commentary text is generated. This commentary text includes the recognized events as well as content corresponding to the user's emotions. Next, this commentary text is converted into audio data using speech synthesis technology. The commentary audio is synchronized with the original video data to generate video data with commentary audio corresponding to the user's emotions.

[0535] As a concrete example, real-time video of a sporting event is sent to an AI server, and the AI ​​identifies the players' movements and scoring scenes, generating commentary such as, "The player has just scored a goal!" When this commentary is converted into audio, rich emotional voice is generated according to the user's feelings of excitement and joy.

[0536] Processing on the distribution server

[0537] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as by filtering or encoding. Finally, the processed video data with commentary is distributed to viewers live or on demand. Distribution is carried out over the internet, and viewers can watch it via a website or app.

[0538] For example, users can watch live sports broadcasts on their smartphones or PCs, and AI-powered commentary is provided in real time, improving the viewing experience even without the need for a professional commentator. Furthermore, commentary that responds to the user's emotions creates a greater sense of immersion.

[0539] This system automatically analyzes video data, generates appropriate commentary, and delivers it to viewers. This allows even users without specialized knowledge to easily create and distribute high-quality video content with commentary. Furthermore, commentary that reflects the user's emotions further enhances the viewing experience.

[0540] The processing flow will be explained below.

[0541] Step 1:

[0542] The user device captures video data in real time through a camera or screen capture software. This video data consists of a series of frames, each captured at regular intervals. At the same time, data is also captured to record the user's facial expressions and voice.

[0543] Step 2:

[0544] The user device establishes a socket connection to transmit the captured video data and the user's facial expression and voice data to the AI ​​server in real time. Through this connection, frame data and emotion data are transferred to the AI ​​server.

[0545] Step 3:

[0546] The server receives the video data and the user's facial expression and voice data sent from the user's device. The received data is temporarily stored in a buffer and prepared for analysis.

[0547] Step 4:

[0548] The server inputs the received video data into an AI model for video analysis, which analyzes objects and actions in the digital video in real time to identify specific events and scenes.

[0549] Step 5:

[0550] The server inputs the user's facial and voice data into the emotion engine, which analyzes this data and identifies the user's emotional state (e.g., joy, surprise, excitement, etc.) in real time.

[0551] Step 6:

[0552] The server generates appropriate commentary text based on the video analysis results and the emotion analysis results from the emotion engine. For example, when a goal is scored in a sporting event, the server generates the text "A goal has been scored!" and emphasizes the text according to the user's excitement level.

[0553] Step 7:

[0554] The server converts the generated commentary text into voice data using speech synthesis technology, and in the process adds the speaker's tone and intonation according to the user's emotions.

[0555] Step 8:

[0556] The server synchronizes the generated audio data with the original video data, creating video data with live audio. During this integration process, processing is performed to align the timing of the video and audio.

[0557] Step 9:

[0558] The server converts the video data with live audio into a format for distribution, which may include filtering and encoding as needed.

[0559] Step 10:

[0560] The server then distributes the final video data with commentary to viewers over the Internet, who can then watch the video in real time or on demand via websites or applications.

[0561] The above processing steps enable users to easily create high-quality video content with commentary and distribute it in real time, even without specialized knowledge. Furthermore, commentary that reflects the user's emotions makes the viewing experience even more engaging.

[0562] Example 2

[0563] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0564] Conventional video distribution systems require manual analysis of video and generation of commentary, which requires a great deal of time and effort. Furthermore, it is difficult to provide commentary that reflects the user's real-time emotions, and there is a demand for an improved viewing experience. Furthermore, it is difficult for users without specialized knowledge to easily create and distribute high-quality video content with commentary.

[0565] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0566] In this invention, the server includes means for acquiring video data, means for transmitting the acquired video data in real time, means for receiving and analyzing the video data and user emotion data, means for generating commentary text based on the analysis results, means for converting the generated commentary text into audio, means for integrating the generated commentary audio into the video data, and means for distributing the integrated video data with the commentary audio. This makes it possible to automatically analyze the video data and generate and distribute commentary corresponding to the user's emotions in real time.

[0567] "Video data" is a collection of visual information captured using a camera or capture device.

[0568] "Means of acquisition" refers to the method of collecting video data using a camera or capture device, or software that controls them.

[0569] A "transmitting means" is a method or protocol for transmitting data over a network to another device or server.

[0570] "Means for receiving and analyzing" refers to methods for receiving transmitted data and using artificial intelligence or other technologies to decipher and analyze its contents.

[0571] "Live text" is text that explains events and situations in the video.

[0572] A "means of generation" is a method of creating new data or information using a specific algorithm or program.

[0573] "Means for converting to voice" refers to technology or software for converting text data into voice data.

[0574] "Means of integration" is the process of bringing together multiple pieces of data or information into one.

[0575] "Means of distribution" refers to the communication method or technology used to deliver the final data to users or viewers.

[0576] A "generative AI model" is an algorithm or tool that uses artificial intelligence techniques to generate new data or content.

[0577] "Emotion data" is information about the emotional state of the user collected from facial expressions, voice, etc.

[0578] "Real-time" means that data is processed and transmitted in a very short time, providing near-immediate results.

[0579] This invention combines a system that receives video data, analyzes it, generates live audio, integrates it with the video, and distributes it, with an emotion engine that recognizes the user's emotions. The program of this system and its processing are described in detail below.

[0580] The user device first acquires video data in real time using a camera or screen capture software. This video data consists of a series of frames, each captured at a fixed interval. Then, a socket connection is established to transmit the acquired video data to the AI ​​server in real time. Through this connection, the frame data is sent to the AI ​​server.

[0581] In a specific example, a user captures video using the camera on their smartphone or PC, and the video is sent to a server via a network. At the same time, the user's facial expressions and voice data are also acquired and analyzed by the emotion engine.

[0582] The AI ​​server receives video data and the user's voice and facial expression data sent from the user's device. First, it temporarily stores the received video data and the user's emotional data in a buffer. The data is then input into a video analysis model, which analyzes objects and actions in the digital video in real time and identifies specific events and scenes. Image processing libraries such as OpenCV are used for video analysis.

[0583] On the other hand, the emotion engine analyzes the user's emotions from their facial expressions and voice and stores the results in a buffer. For emotion analysis, a service such as Microsoft Azure's Emotion API is used.

[0584] Based on the analysis results, appropriate commentary text is generated. This commentary text includes the recognized events as well as content corresponding to the user's emotions. A natural language generation model (such as GPT-3) is used to generate the text. The generated commentary text is then converted into audio data using speech synthesis technology. The Google Text-to-Speech API or similar is used for speech synthesis. The commentary audio is synchronized with the original video data, and video data with commentary audio corresponding to the user's emotions is generated.

[0585] As a concrete example, real-time video of a sporting event is sent to an AI server, and the AI ​​identifies the players' movements and scoring scenes, generating commentary such as, "The player has just scored a goal!" When this commentary is converted into audio, rich emotional voice is generated according to the user's feelings of excitement and joy.

[0586] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as by filtering or encoding. FFmpeg is used for filtering and encoding, for example. Finally, the processed video data with commentary is distributed to viewers live or on demand. Distribution is carried out over the internet, and viewers can watch it via a website or app.

[0587] For example, users can watch live sports broadcasts on their smartphones or PCs, and AI-powered commentary is provided in real time, improving the viewing experience even without the need for a professional commentator. Furthermore, commentary that responds to the user's emotions creates a greater sense of immersion.

[0588] Example prompt sentence:

[0589] "Analyze video data from a soccer match in real time, recognize player movements and goal-scoring scenes, and generate commentary. If the user's emotion is classified as 'excited,' include an emotionally rich expression in the commentary. For example, say, 'The player scored a goal! What a great shot!'"

[0590] This system automatically analyzes video data, generates appropriate commentary, and delivers it to viewers. This allows even users without specialized knowledge to easily create and distribute high-quality video content with commentary. Furthermore, commentary that reflects the user's emotions further enhances the viewing experience.

[0591] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0592] Step 1:

[0593] A user device acquires video data using a camera or screen capture software (e.g., OBS Studio). The acquired video data consists of a series of frames. The input is camera video or screen capture, and the output is video data (frame data). For example, a user might launch a camera app on their smartphone and capture live video.

[0594] Step 2:

[0595] The user device establishes a socket connection to send the acquired video data to the AI ​​server in real time. Specifically, the Python socket library is used to periodically send video frame data to the server. The input is the video data, and the output is the frame data sent to the server. During this process, the user's facial expressions and voice data are also collected at the same time.

[0596] Step 3:

[0597] The server receives the video data and the user's voice and facial expression data sent from the user's device and temporarily stores them in a buffer. The input is the video data and emotion data from the user's device, and the output is the data stored in the buffer. For example, when the video data and voice data arrive at the server, they are temporarily saved.

[0598] Step 4:

[0599] The server analyzes the received video data using the OpenCV library. Specifically, it identifies objects and actions in the digital video in real time. The input is the video data stored in the buffer, and the output is the analyzed video data (event identification results). For example, it analyzes video footage of a sporting event to identify player movements and goal scenes.

[0600] Step 5:

[0601] The server uses an emotion engine (e.g., Emotion API) to analyze emotions from the user's facial expressions and voice. The input is the user's facial expression and voice data, and the output is the analyzed emotion data. For example, the emotion engine analyzes the user's camera footage and determines that the user is "excited."

[0602] Step 6:

[0603] The server generates commentary text based on the video data analysis results and emotion data. In this process, a generative AI model (e.g., GPT-3) is used to generate the text. The input is the analyzed event data and emotion data, and the output is the commentary text. For example, a commentary such as "The player scored a goal!" is generated.

[0604] Step 7:

[0605] The server converts the generated commentary text into audio data using speech synthesis technology (e.g., Google Text-to-Speech API). The input is the commentary text, and the output is audio data. For example, the text "The player scored a goal!" is converted into audio.

[0606] Step 8:

[0607] The server synchronizes the generated commentary with the original video data. The input is audio data and video data, and the output is video data with commentary. For example, the generated commentary is adjusted so that it is played at the appropriate timing within the video.

[0608] Step 9:

[0609] The server sends the video data with commentary to the distribution server. The input is the video data with commentary, and the output is the data sent to the distribution server. For example, the audio and video are integrated using FFmpeg, and then the data is sent to the distribution server.

[0610] (Application example 2)

[0611] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0612] While conventional video commentary systems can provide commentary based on visual information or specific events, it is difficult to provide personalized commentary that reflects the viewer's emotions in real time. Furthermore, there are limited ways to enhance the viewer's sense of realism and immersion. This makes it impossible for the viewing experience to be uniform and reflect the individual excitement and emotions.

[0613] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0614] In this invention, the server includes means for receiving video data, means for analyzing the received video data, means for recognizing a user's emotion, means for generating commentary audio based on the analysis result and the user's emotion data, means for integrating the generated commentary audio into the video data, and means for delivering the integrated video data with commentary audio. This enables personalized commentary that reflects the viewer's emotion, thereby improving the viewing experience.

[0615] "Video data" is a collection of visual information captured by a camera or other image capture device.

[0616] "Means for analyzing" refers to a processing device or algorithm that analyzes received video data or other information and recognizes certain patterns or events.

[0617] "Means for recognizing user emotions" refers to technology that analyzes a user's facial expressions, voice, and other biometric data to determine their emotional state.

[0618] The "means for generating live audio" is a technology for generating audio data that verbally explains or provides commentary to viewers based on video data and emotional data.

[0619] The "means of integration" refers to the means of integrating the generated live audio with the original video data to create a single visual and audio content.

[0620] "Means of distribution" refers to the technology and infrastructure for delivering the integrated video data with commentary to viewers over the Internet.

[0621] "Artificial intelligence" refers to computer systems or algorithms that mimic human intellectual processes by analyzing and learning from large amounts of data.

[0622] "Speech synthesis technology" is a technology for analyzing text information and converting it into natural speech.

[0623] This invention is a system that receives and analyzes video data in real time, recognizes the user's emotions, and generates and distributes live commentary that integrates this information. The detailed configuration and specific operation of this system are described below.

[0624] Hardware Configuration

[0625] User device:

[0626] Smart glasses: Equipped with a camera for video capture and acquisition of user facial expression data.

[0627] Microphone: Used to capture the user's voice data.

[0628] Speaker: Used to provide the generated commentary to the user.

[0629] server:

[0630] AI Server: Used to analyze video data and emotional data. Specifically, it is equipped with an AI model for analyzing video data and an emotion engine for analyzing user emotions.

[0631] Distribution server: A server that distributes integrated video data with live audio to viewers in real time.

[0632] Software Configuration

[0633] On the user's device:

[0634] The user device first acquires video data in real time using the camera installed in the smart glasses. This video data consists of frames captured at regular intervals. The user's voice data is also acquired through the microphone. This data is then sent to the AI ​​server via a socket connection.

[0635] Processing on the AI ​​server:

[0636] The AI ​​server receives video data and audio / facial expression data sent from the user's device. The received video data and emotional data are temporarily stored in a buffer and then input into an AI model for video analysis and an emotion engine that analyzes emotions from facial expressions and audio. This allows specific events and scenes in the video to be identified, while also analyzing the user's emotional state. Based on the analysis results, appropriate commentary text is generated, and this commentary text is converted into audio data using speech synthesis technology.

[0637] Processing on the distribution server:

[0638] The distribution server receives the integrated video data with live commentary from the AI ​​server, encodes and filters it in real time, and then distributes it to viewers, who can view the data on their smartphones, PCs, or other devices.

[0639] Specific examples

[0640] As a specific example of use, consider a user watching a soccer match. When the user wears smart glasses and watches a soccer match in real time, the smart glasses' camera captures video data, and at the same time, the user's facial expression data is captured. This data is sent to the AI ​​server, which recognizes important events such as goal scenes and further analyzes the user's emotions. If a goal scene is recognized and the user is excited, a live commentary voice saying "Goal! The user is excited too!" is generated and delivered to the user in real time through the smart glasses' speaker.

[0641] Prompt Sentence Examples

[0642] Examples of prompts include:

[0643] While the user is watching a sporting event, generate the most compelling commentary based on real-time visual and emotional data that meets the following criteria:

[0644] Users' attention is focused on the players

[0645] The user's facial expression is excited (recognized by the emotion engine)

[0646] Video analysis engine detects goal scenes

[0647] Example of expected output:

[0648] "Goal! The users are excited!" or "Great play! The users look very happy!"

[0649] Thus, the present invention combines video analysis and emotion recognition to provide viewers with an immersive, personalized, and live experience.

[0650] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0651] Step 1:

[0652] The device uses the smart glasses' camera to capture video data in real time, acquiring video frames at specific time intervals, and also uses the microphone to capture the user's voice data.

[0653] Input: Video frames from camera, audio data from microphone

[0654] Output: Captured video and audio data

[0655] Specific operation: Activate the camera of the smart glasses, capture frames at regular intervals, and simultaneously capture audio through the microphone.

[0656] Step 2:

[0657] The device establishes a socket communication to transmit the captured video and audio data to the AI ​​server in real time.

[0658] Input: Captured video and audio data

[0659] Output: Data sent to the AI ​​server

[0660] Specific operation: Establish socket communication and send acquired data to the AI ​​server.

[0661] Step 3:

[0662] The server receives the video data and audio data sent from the terminal and temporarily stores them in a buffer.

[0663] Input: Video and audio data sent from the device

[0664] Output: Buffered data

[0665] Specific operation: Received data is stored in a buffer and prepared for analysis.

[0666] Step 4:

[0667] The server analyzes the received video data using an AI model to identify specific events or scenes (e.g., goal scenes), and uses an emotion engine to analyze the user's voice and facial expression data to determine their emotional state.

[0668] Input: Video and audio data stored in the buffer

[0669] Output: Parsed event data and emotion data

[0670] Specific operation: Video data is input into the AI ​​model to identify specific events, and voice and facial expression data is input into the emotion engine to determine emotions.

[0671] Step 5:

[0672] The server generates appropriate commentary text based on the parsed event data and emotion data.

[0673] Input: Parsed event data and emotion data

[0674] Output: Generated commentary text

[0675] Specific operation: Create text based on event data and adjust commentary content to reflect emotional data.

[0676] Step 6:

[0677] The server converts the generated commentary text into voice data using voice synthesis technology.

[0678] Input: Generated commentary text

[0679] Output: Generated live audio data

[0680] What it does: Inputs text into a speech synthesis engine and generates a natural-sounding audio file.

[0681] Step 7:

[0682] The server integrates the generated commentary audio data with the original video data to generate video data with commentary audio.

[0683] Input: Original video data, generated live audio data

[0684] Output: Video data with integrated commentary

[0685] Specific operation: Synchronizes audio data and video data and adds audio to video.

[0686] Step 8:

[0687] The server delivers the integrated video data with live audio in real time to viewers, who can view the data via their smartphones or PC devices.

[0688] Input: Video data with integrated commentary

[0689] Output: Video data delivered to viewers

[0690] Specific operation: Encodes video data and distributes it in real time over the Internet.

[0691] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0692] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0693] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0694] [Third embodiment]

[0695] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0696] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0697] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0698] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0699] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0700] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0701] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0702] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0703] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0704] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0705] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0706] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0707] This invention relates to a system that receives video data, analyzes it, generates live audio, integrates it with the video, and distributes it. The program of this system and its processing are described in detail below.

[0708] Processing on the user terminal

[0709] The user device first acquires video data in real time using a camera or screen capture software. This video data consists of a series of frames, each captured at a fixed interval. Then, a socket connection is established to transmit the acquired video data to the AI ​​server in real time. Through this connection, the frame data is sent to the AI ​​server.

[0710] In a specific example, a user captures video using the camera on their smartphone or PC, and the video is sent to a server via a network.

[0711] Processing on the AI ​​server

[0712] The AI ​​server receives video data sent from the user's device. The received data is temporarily stored in a buffer and then input into a model for video analysis. The AI ​​model analyzes objects and actions in the digital video and identifies events in real time based on the results. Appropriate commentary text is generated based on the recognized events. This commentary text is then converted into audio data using speech synthesis technology. This audio data is synchronized with the original video data to generate video data with commentary audio.

[0713] For example, real-time video of a sporting event is sent to an AI server, where the AI ​​identifies players' movements and scoring scenes, and generates commentary such as, "The player has just scored a goal!" This commentary is converted into audio and combined with the video.

[0714] Processing on the distribution server

[0715] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as by filtering or encoding. Finally, the processed video data with commentary is distributed to viewers live or on demand. Distribution is carried out over the internet, and viewers can watch it via a website or app.

[0716] In a specific example, users can watch live sports broadcasts on their smartphones or PCs, and AI-powered commentary will be provided in real time, improving the viewing experience even without the need for a professional commentator.

[0717] This system automatically analyzes video data, generates appropriate commentary, and delivers it to viewers, allowing even users without specialized knowledge to easily create and distribute high-quality video content with commentary.

[0718] The processing flow will be explained below.

[0719] Step 1:

[0720] The user device acquires video data in real time through a camera or screen capture software. The video data consists of a series of frames, each captured at a fixed interval.

[0721] Step 2:

[0722] The user device establishes a socket connection to transmit the acquired video data to the AI ​​server in real time. Through this connection, frame data is transferred to the AI ​​server.

[0723] Step 3:

[0724] The server receives video data sent from the user terminal, temporarily stores the received data in a buffer, and prepares it for analysis.

[0725] Step 4:

[0726] The server inputs the received data into an AI model for video analysis, which analyzes objects and actions in the video in real time to identify specific events and scenes.

[0727] Step 5:

[0728] The server generates appropriate commentary text based on the analysis results. Commentary content is defined for recognized events (e.g., goal shots, goals scored, transitions between offense and defense, etc.).

[0729] Step 6:

[0730] The server converts the generated commentary text into audio data using speech synthesis technology. In this process, the commentary text is converted into audio data that has the speaker's voice and intonation.

[0731] Step 7:

[0732] The server synchronizes the generated audio data with the original video data, creating video data with live audio. During this integration process, processing is performed to align the timing of the video and audio.

[0733] Step 8:

[0734] The server converts the video data with live audio into a format for distribution, which may include filtering and encoding as needed.

[0735] Step 9:

[0736] The server then distributes the final video data with commentary to viewers over the Internet, who can then watch the video in real time or on demand via websites or applications.

[0737] The above processing steps enable users to easily create high-quality video content with commentary and distribute it in real time, even without specialized knowledge.

[0738] Example 1

[0739] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0740] In response to the increasing diversity of modern video content and the demand for a high-quality viewer experience, there is a lack of technology to automatically generate and distribute video with appropriate commentary in real time. In particular, there is a demand for fast, high-quality commentary generation for live streaming and event video. Furthermore, there is a need for a method to easily create and distribute high-quality video with commentary, even without specialized knowledge.

[0741] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0742] In this invention, the server includes means for a user terminal to acquire video data in real time using a camera or screen capture software and transmit the video data via socket communication, means for the server to receive the transmitted video data and store the received data in a buffer, means for inputting the data stored in the buffer into an AI model to analyze objects and actions in the video and generate commentary text based on event identification, means for converting the generated commentary text into audio data and generating video data with commentary audio in synchronization with the original video data, and means for the distribution server to receive the video data with commentary audio, filter and encode it, and distribute the data to viewers. This makes it possible to automatically generate high-quality video content with commentary in real time and quickly distribute it to viewers.

[0743] A "user terminal" is a device used by a user, which acquires video data in real time using a camera or screen capture software, and transmits the data to a server via socket communication.

[0744] A "camera" is a device for taking still or moving images, and is used in the present invention to capture video data in real time.

[0745] "Screen capture software" is a program that captures the contents of a computer or device screen in real time and saves and transfers it as video data.

[0746] "Socket communication" is a communication method based on a protocol for transmitting and receiving data over a network, and in the present invention is a technique used for transmitting video data from a user terminal to a server.

[0747] A "server" is a computer system that plays a central role in receiving video data sent from user terminals and analyzing, processing, and distributing it.

[0748] A "buffer" is a memory area for temporarily storing data, and serves to temporarily store received video data as a preliminary step to analysis.

[0749] An "AI model" is an algorithm or machine learning model that uses artificial intelligence technology to analyze objects and actions in video and is used to identify events and generate commentary text.

[0750] An "object" is a specific element such as a person, object, or animal that exists in a video and is the subject of analysis.

[0751] "Movement" refers to the actions and behavior of objects in a video, and is an element that can be identified through analysis.

[0752] An "event" is a notable occurrence triggered by a particular object or action, and is the basis for generating live text.

[0753] "Live text" is a sentence generated based on the analyzed events, and is character information for explaining the situation of the video.

[0754] "Speech synthesis technology" is a technology for converting text data into voice data, and is used when converting generated commentary text into voice.

[0755] "Video data" refers to all video information captured by a camera or screen capture software, and is subject to analysis and distribution.

[0756] "Filtering" is a process performed during data processing to remove noise and improve quality, and is used to optimize data before distribution.

[0757] "Encoding" is the process of converting digital data into a format, and is used to convert video data into a format that can be distributed.

[0758] "Distribution" is the process of transmitting video data over a network and making it accessible to viewers, either live or on-demand.

[0759] The present invention relates to a system for acquiring video data in real time, analyzing the data, generating appropriate commentary audio, integrating the audio with the video data, and distributing the audio. Detailed embodiments of this system will be described below.

[0760] Processing on the user terminal

[0761] The user terminal acquires video data in real time using a camera or screen capture software. This video data consists of successive frames. The user captures live video using, for example, the camera on their smartphone or PC. The captured video data is sent to the server via socket communication.

[0762] Specific examples

[0763] Imagine a scenario where a user uses a smartphone camera to capture live footage of cityscapes or events and sends the data to a server via the Internet.

[0764] Prompt Sentence Examples

[0765] "Please send this video data to the AI ​​server."

[0766] Processing on the AI ​​server

[0767] The AI ​​server receives video data sent from the user device via socket communication. The received video data is temporarily stored in a buffer. The buffered video data is then input into the AI ​​model. The AI ​​model analyzes objects and actions in the video and identifies events in real time based on the results. Commentary text is generated in response to the identified events. This commentary text is converted into audio data using speech synthesis technology. The audio data is then synchronized with the original video data to generate video data with commentary audio.

[0768] Specific examples

[0769] Live footage of a sporting event is sent to an AI server, and the AI ​​identifies player movements and goal-scoring scenes to generate commentary. The generated commentary is converted into audio data and properly synchronized with the video. Commentary such as "Player A has scored a goal!" is generated.

[0770] Prompt Sentence Examples

[0771] "Analyze the player's movements in the footage and generate appropriate commentary."

[0772] Processing on the distribution server

[0773] The distribution server receives the video data with commentary sent from the AI ​​server. The received video data is filtered and encoded as needed to convert it into the optimal format. Finally, the distribution server distributes this data to viewers via the Internet. Viewers can watch this video data with commentary in real time via a website or dedicated application.

[0774] Specific examples

[0775] Consider a situation where viewers are watching live sports broadcasts on their PCs or smartphones at home. Real-time commentary by AI provides an immersive viewing experience.

[0776] Prompt Sentence Examples

[0777] "Encode and distribute the video data with commentary integrated into it to users."

[0778] This system enables users without specialized knowledge to easily create high-quality video content with live commentary using advanced technology and quickly distribute it to viewers.

[0779] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0780] Step 1: Acquiring and transmitting video data

[0781] User terminal processing

[0782] The user device acquires video data in real time using a camera or screen capture software. The input is the video data from the camera or capture software. Specifically, each frame of video data is captured at regular intervals and created as a continuous data stream. This video data establishes a socket connection and is sent to the AI ​​server. The output is the video data sent to the server via the network.

[0783] Specific actions

[0784] A user picks up their smartphone, launches the camera app and captures live footage of the event, which is then sent in real time via a socket to the server.

[0785] Step 2: Receiving and buffering video data

[0786] Server Processing

[0787] The server receives video data sent from the user terminal. The input is video data received over the network. The received data is temporarily stored in a buffer. The buffering is for maintaining data consistency and preparing for the next analysis step. The output is the video data stored in the buffer.

[0788] Specific actions

[0789] The server receives real-time video data coming from the user terminal and stores it in a buffer in its internal memory.

[0790] Step 3: Analyzing the video data

[0791] Server Processing

[0792] The server inputs the buffered video data into the AI ​​model. The input is the buffered video data. The AI ​​model analyzes the objects and actions in the video to identify events. The analysis identifies objects in the video and determines their actions and relationships. The output is the identified event information.

[0793] Specific actions

[0794] Footage of sporting events is input into the AI ​​model, and the system analyzes and identifies player movements, goal scenes, etc.

[0795] Step 4: Generate commentary text

[0796] Server Processing

[0797] The server generates commentary text based on the identified event information. The input is the event information output from the AI ​​model. Specific text appropriate for the event is generated, and the generated commentary text becomes the commentary content to be provided to the user. The output is the generated commentary text.

[0798] Specific actions

[0799] The text "Player A scores a goal!" is generated based on the recognized event.

[0800] Step 5: Generate audio data

[0801] Server Processing

[0802] The server converts the generated commentary text into audio data using speech synthesis technology. The input is the commentary text. Using speech synthesis technology, the text is converted into audio data, and the audio is synchronized with the original video data. The output is audio data.

[0803] Specific actions

[0804] The text "Player A scores a goal!" is converted into audio and integrated into the video.

[0805] Step 6: Generate video data with commentary

[0806] Server Processing

[0807] The server synchronizes the audio data with the original video data to generate video data with live audio commentary. The input is the audio data and the original video data. These data are synchronized based on timestamps and other factors and combined into a single integrated data. The output is video data with live audio commentary.

[0808] Specific actions

[0809] The video and audio are appropriately combined to create video content with commentary.

[0810] Step 7: Streaming video data

[0811] Distribution server processing

[0812] The distribution server receives the video data with commentary sent from the AI ​​server. The input is the video data with commentary. The received video data is filtered, encoded, and optimized as needed. Finally, the data is distributed to viewers live or on demand. The output is the optimized video data distributed over the internet.

[0813] Specific actions

[0814] Viewers can watch live sports broadcasts on their PCs or smartphones at home, and the AI-powered automated commentary provided in real time creates a truly immersive experience.

[0815] (Application example 1)

[0816] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0817] Providing live commentary of live events or sports over the Internet typically requires a professional commentator. However, providing high-quality commentary in real time is difficult when there are no professional commentators or when the system relies on the commentator's schedule. Furthermore, it is technically difficult to consistently provide timely and accurate commentary for a wide variety of content at large-scale events. This can reduce viewer satisfaction and make the event less appealing. Therefore, there is a growing demand for a system that can automatically provide real-time commentary without the need for a professional commentator.

[0818] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0819] In this invention, the server includes a means for receiving video data, a means for analyzing the video data and using a generative algorithm to identify events, and a means for using generative synthesis technology to automatically generate commentary based on the identified events and output it as synchronized audio. This makes it possible to provide commentary and live commentary of events in real time without the need for a professional commentator. Viewers can always enjoy high-quality video with commentary, maximizing the appeal of the event.

[0820] "Video Data" means visual information captured from a camera or other image capture device.

[0821] The "receiving means" is a mechanism for receiving video data transmitted from a user terminal.

[0822] "Means for analyzing" means methods for processing received video data to detect and analyze images, motion, and other elements.

[0823] A "knowledge model" is an artificial intelligence model that has been trained in advance to identify events in video.

[0824] "Generative synthesis technology" is a technology that generates audio from text and integrates it with video.

[0825] The "means of distribution" refers to a mechanism for transmitting the integrated video data with commentary via the Internet or other means and delivering it to viewers.

[0826] The "transmitting device" is a device for transmitting video data from a user terminal to a server.

[0827] A "generation algorithm" is an algorithm for identifying events from video data and generating appropriate commentary content based on that information.

[0828] A "prompt algorithm" is an algorithm for converting generated speech text into a more natural format.

[0829] This invention is a system that automatically generates commentary audio for video data shot by a user in real time and distributes the video with commentary audio to viewers. This system is realized mainly by three main components: a user terminal, an AI server, and a distribution server.

[0830] Processing on the user terminal

[0831] The user device first acquires video data in real time using a camera or screen capture software. The acquired video data is then sent to the AI ​​server in real time. To do this, the user device establishes a socket connection and sequentially sends frame data to the AI ​​server.

[0832] Processing on the AI ​​server

[0833] The AI ​​server receives video data sent from the user's device. The received data is then temporarily stored in a buffer and input into a generation algorithm for video analysis. This generation algorithm analyzes objects and actions in the video and identifies events in real time based on the results. Appropriate commentary text is automatically generated based on the identified events.

[0834] This commentary text is then converted into audio data using speech synthesis technology. Generative synthesis technologies such as Google Text-to-Speech (gTTS) are used for this purpose. The audio data is synchronized with the original video data to generate video data with commentary audio. During this process, it is important to utilize a prompting algorithm to convert the generated text into a more natural format. An example of a prompt sentence is, "Generate a commentary of a life event. This is a live video of a soccer match. Generate commentary when a player scores a goal."

[0835] Processing on the distribution server

[0836] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as filtered and encoded, before finally being distributed to viewers live or on demand. This distribution is done via the Internet, and viewers can watch the video through a dedicated website or application.

[0837] Specific examples

[0838] When a user films a soccer match using a smartphone, the video data is sent to an AI server in real time. The AI ​​server identifies player movements and goal-scoring scenes, and generates commentary such as "The player has scored a goal!" The generated commentary is converted into audio using voice synthesis technology, and this audio is integrated into the video. Viewers can enjoy the video with this commentary audio via live streaming. This system makes it possible to provide high-quality video with commentary without the need for a professional commentator.

[0839] In this way, collaboration between user devices, AI servers, and distribution servers can automatically provide real-time commentary, improving the viewer experience.

[0840] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0841] Step 1:

[0842] The user terminal captures video data in real time using a camera or screen capture software. The camera device or screen capture software is used as input. The output is the captured video data. This video data consists of a series of frames, each captured at a fixed interval.

[0843] Step 2:

[0844] The user terminal establishes a socket communication to send the acquired video data to the AI ​​server in real time. The acquired video data is used as input. The output is the transmitted frame data. This frame data is sent to the AI ​​server sequentially.

[0845] Step 3:

[0846] The AI ​​server receives video data sent from the user terminal. The frame data sent from the user terminal is used as input. The output is video data temporarily stored in a buffer.

[0847] Step 4:

[0848] The AI ​​server inputs the video data stored in the buffer into the generative algorithm. The video data is used as input. The output is the analysis result. The generative AI model analyzes the objects and actions in the video and generates the results.

[0849] Step 5:

[0850] The AI ​​server identifies events based on the analysis results. The analysis results of the generative AI model are used as input. The output is the identified event information. This information indicates a specific event (e.g., a goal is scored).

[0851] Step 6:

[0852] The AI ​​server automatically generates commentary text based on the identified events. The identified event information is used as input. The output is the generated commentary text. An example prompt sentence is "Generate a commentary of a life event. This is a live video of a soccer match. Generate a commentary when a player scores a goal."

[0853] Step 7:

[0854] The AI ​​server converts the generated commentary text into audio data. The commentary text is used as input. The output is synthesized audio data. This conversion uses voice synthesis technology such as Google Text-to-Speech (gTTS).

[0855] Step 8:

[0856] The AI ​​server synchronizes the audio data with the original video data. The synthesized audio data and the original video data are used as input. The output is video data with commentary. This data is played back with the audio and video synchronized.

[0857] Step 9:

[0858] The distribution server receives the video data with commentary. As input, it uses the video data with commentary sent from the AI ​​server. The output is video data ready for distribution.

[0859] Step 10:

[0860] The distribution server processes the data as necessary, such as filtering and encoding. The input is video data with commentary. The output is the processed video data.

[0861] Step 11:

[0862] The distribution server delivers the final video data to the viewer. The processed video data is used as input. The output is live or on-demand video delivered to the viewer's device. The viewer can then watch this video through a website or application.

[0863] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0864] This invention combines a system that receives video data, analyzes it, generates live audio, integrates it with the video, and distributes it, with an emotion engine that recognizes the user's emotions. The program of this system and its processing are described in detail below.

[0865] Processing on the user terminal

[0866] The user device first acquires video data in real time using a camera or screen capture software. This video data consists of a series of frames, each captured at a fixed interval. Then, a socket connection is established to transmit the acquired video data to the AI ​​server in real time. Through this connection, the frame data is sent to the AI ​​server.

[0867] In a specific example, a user captures video using the camera on their smartphone or PC, and the video is sent to a server via a network. At the same time, the user's facial expressions and voice data are also acquired and analyzed by the emotion engine.

[0868] Processing on the AI ​​server

[0869] The AI ​​server receives video data and the user's voice and facial expression data sent from the user's device. First, it temporarily stores the received video data and user emotion data in a buffer. The data is then input into a video analysis model, which analyzes objects and actions in the digital video in real time and identifies specific events and scenes. Meanwhile, the emotion engine analyzes emotions from the user's facial expressions and voice and takes the results into account.

[0870] Based on the analysis results, appropriate commentary text is generated. This commentary text includes the recognized events as well as content corresponding to the user's emotions. Next, this commentary text is converted into audio data using speech synthesis technology. The commentary audio is synchronized with the original video data to generate video data with commentary audio corresponding to the user's emotions.

[0871] As a concrete example, real-time video of a sporting event is sent to an AI server, and the AI ​​identifies the players' movements and scoring scenes, generating commentary such as, "The player has just scored a goal!" When this commentary is converted into audio, rich emotional voice is generated according to the user's feelings of excitement and joy.

[0872] Processing on the distribution server

[0873] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as by filtering or encoding. Finally, the processed video data with commentary is distributed to viewers live or on demand. Distribution is carried out over the internet, and viewers can watch it via a website or app.

[0874] For example, users can watch live sports broadcasts on their smartphones or PCs, and AI-powered commentary is provided in real time, improving the viewing experience even without the need for a professional commentator. Furthermore, commentary that responds to the user's emotions creates a greater sense of immersion.

[0875] This system automatically analyzes video data, generates appropriate commentary, and delivers it to viewers. This allows even users without specialized knowledge to easily create and distribute high-quality video content with commentary. Furthermore, commentary that reflects the user's emotions further enhances the viewing experience.

[0876] The processing flow will be explained below.

[0877] Step 1:

[0878] The user device captures video data in real time through a camera or screen capture software. This video data consists of a series of frames, each captured at regular intervals. At the same time, data is also captured to record the user's facial expressions and voice.

[0879] Step 2:

[0880] The user device establishes a socket connection to transmit the captured video data and the user's facial expression and voice data to the AI ​​server in real time. Through this connection, frame data and emotion data are transferred to the AI ​​server.

[0881] Step 3:

[0882] The server receives the video data and the user's facial expression and voice data sent from the user's device. The received data is temporarily stored in a buffer and prepared for analysis.

[0883] Step 4:

[0884] The server inputs the received video data into an AI model for video analysis, which analyzes objects and actions in the digital video in real time to identify specific events and scenes.

[0885] Step 5:

[0886] The server inputs the user's facial and voice data into the emotion engine, which analyzes this data and identifies the user's emotional state (e.g., joy, surprise, excitement, etc.) in real time.

[0887] Step 6:

[0888] The server generates appropriate commentary text based on the video analysis results and the emotion analysis results from the emotion engine. For example, when a goal is scored in a sporting event, the server generates the text "A goal has been scored!" and emphasizes the text according to the user's excitement level.

[0889] Step 7:

[0890] The server converts the generated commentary text into voice data using speech synthesis technology, and in the process adds the speaker's tone and intonation according to the user's emotions.

[0891] Step 8:

[0892] The server synchronizes the generated audio data with the original video data, creating video data with live audio. During this integration process, processing is performed to align the timing of the video and audio.

[0893] Step 9:

[0894] The server converts the video data with live audio into a format for distribution, which may include filtering and encoding as needed.

[0895] Step 10:

[0896] The server then distributes the final video data with commentary to viewers over the Internet, who can then watch the video in real time or on demand via websites or applications.

[0897] The above processing steps enable users to easily create high-quality video content with commentary and distribute it in real time, even without specialized knowledge. Furthermore, commentary that reflects the user's emotions makes the viewing experience even more engaging.

[0898] Example 2

[0899] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0900] Conventional video distribution systems require manual analysis of video and generation of commentary, which requires a great deal of time and effort. Furthermore, it is difficult to provide commentary that reflects the user's real-time emotions, and there is a demand for an improved viewing experience. Furthermore, it is difficult for users without specialized knowledge to easily create and distribute high-quality video content with commentary.

[0901] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0902] In this invention, the server includes means for acquiring video data, means for transmitting the acquired video data in real time, means for receiving and analyzing the video data and user emotion data, means for generating commentary text based on the analysis results, means for converting the generated commentary text into audio, means for integrating the generated commentary audio into the video data, and means for distributing the integrated video data with the commentary audio. This makes it possible to automatically analyze the video data and generate and distribute commentary corresponding to the user's emotions in real time.

[0903] "Video data" is a collection of visual information captured using a camera or capture device.

[0904] "Means of acquisition" refers to the method of collecting video data using a camera or capture device, or software that controls them.

[0905] A "transmitting means" is a method or protocol for transmitting data over a network to another device or server.

[0906] "Means for receiving and analyzing" refers to methods for receiving transmitted data and using artificial intelligence or other technologies to decipher and analyze its contents.

[0907] "Live text" is text that explains events and situations in the video.

[0908] A "means of generation" is a method of creating new data or information using a specific algorithm or program.

[0909] "Means for converting to voice" refers to technology or software for converting text data into voice data.

[0910] "Means of integration" is the process of bringing together multiple pieces of data or information into one.

[0911] "Means of distribution" refers to the communication method or technology used to deliver the final data to users or viewers.

[0912] A "generative AI model" is an algorithm or tool that uses artificial intelligence techniques to generate new data or content.

[0913] "Emotion data" is information about the emotional state of the user collected from facial expressions, voice, etc.

[0914] "Real-time" means that data is processed and transmitted in a very short time, providing near-immediate results.

[0915] This invention combines a system that receives video data, analyzes it, generates live audio, integrates it with the video, and distributes it, with an emotion engine that recognizes the user's emotions. The program of this system and its processing are described in detail below.

[0916] The user device first acquires video data in real time using a camera or screen capture software. This video data consists of a series of frames, each captured at a fixed interval. Then, a socket connection is established to transmit the acquired video data to the AI ​​server in real time. Through this connection, the frame data is sent to the AI ​​server.

[0917] In a specific example, a user captures video using the camera on their smartphone or PC, and the video is sent to a server via a network. At the same time, the user's facial expressions and voice data are also acquired and analyzed by the emotion engine.

[0918] The AI ​​server receives video data and the user's voice and facial expression data sent from the user's device. First, it temporarily stores the received video data and the user's emotional data in a buffer. The data is then input into a video analysis model, which analyzes objects and actions in the digital video in real time and identifies specific events and scenes. Image processing libraries such as OpenCV are used for video analysis.

[0919] On the other hand, the emotion engine analyzes the user's emotions from their facial expressions and voice and stores the results in a buffer. For emotion analysis, a service such as Microsoft Azure's Emotion API is used.

[0920] Based on the analysis results, appropriate commentary text is generated. This commentary text includes the recognized events as well as content corresponding to the user's emotions. A natural language generation model (such as GPT-3) is used to generate the text. The generated commentary text is then converted into audio data using speech synthesis technology. The Google Text-to-Speech API or similar is used for speech synthesis. The commentary audio is synchronized with the original video data, and video data with commentary audio corresponding to the user's emotions is generated.

[0921] As a concrete example, real-time video of a sporting event is sent to an AI server, and the AI ​​identifies the players' movements and scoring scenes, generating commentary such as, "The player has just scored a goal!" When this commentary is converted into audio, rich emotional voice is generated according to the user's feelings of excitement and joy.

[0922] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as by filtering or encoding. FFmpeg is used for filtering and encoding, for example. Finally, the processed video data with commentary is distributed to viewers live or on demand. Distribution is carried out over the internet, and viewers can watch it via a website or app.

[0923] For example, users can watch live sports broadcasts on their smartphones or PCs, and AI-powered commentary is provided in real time, improving the viewing experience even without the need for a professional commentator. Furthermore, commentary that responds to the user's emotions creates a greater sense of immersion.

[0924] Example prompt sentence:

[0925] "Analyze video data from a soccer match in real time, recognize player movements and goal-scoring scenes, and generate commentary. If the user's emotion is classified as 'excited,' include an emotionally rich expression in the commentary. For example, say, 'The player scored a goal! What a great shot!'"

[0926] This system automatically analyzes video data, generates appropriate commentary, and delivers it to viewers. This allows even users without specialized knowledge to easily create and distribute high-quality video content with commentary. Furthermore, commentary that reflects the user's emotions further enhances the viewing experience.

[0927] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0928] Step 1:

[0929] A user device acquires video data using a camera or screen capture software (e.g., OBS Studio). The acquired video data consists of a series of frames. The input is camera video or screen capture, and the output is video data (frame data). For example, a user might launch a camera app on their smartphone and capture live video.

[0930] Step 2:

[0931] The user device establishes a socket connection to send the acquired video data to the AI ​​server in real time. Specifically, the Python socket library is used to periodically send video frame data to the server. The input is the video data, and the output is the frame data sent to the server. During this process, the user's facial expressions and voice data are also collected at the same time.

[0932] Step 3:

[0933] The server receives the video data and the user's voice and facial expression data sent from the user's device and temporarily stores them in a buffer. The input is the video data and emotion data from the user's device, and the output is the data stored in the buffer. For example, when the video data and voice data arrive at the server, they are temporarily saved.

[0934] Step 4:

[0935] The server analyzes the received video data using the OpenCV library. Specifically, it identifies objects and actions in the digital video in real time. The input is the video data stored in the buffer, and the output is the analyzed video data (event identification results). For example, it analyzes video footage of a sporting event to identify player movements and goal scenes.

[0936] Step 5:

[0937] The server uses an emotion engine (e.g., Emotion API) to analyze emotions from the user's facial expressions and voice. The input is the user's facial expression and voice data, and the output is the analyzed emotion data. For example, the emotion engine analyzes the user's camera footage and determines that the user is "excited."

[0938] Step 6:

[0939] The server generates commentary text based on the video data analysis results and emotion data. In this process, a generative AI model (e.g., GPT-3) is used to generate the text. The input is the analyzed event data and emotion data, and the output is the commentary text. For example, a commentary such as "The player scored a goal!" is generated.

[0940] Step 7:

[0941] The server converts the generated commentary text into audio data using speech synthesis technology (e.g., Google Text-to-Speech API). The input is the commentary text, and the output is audio data. For example, the text "The player scored a goal!" is converted into audio.

[0942] Step 8:

[0943] The server synchronizes the generated commentary with the original video data. The input is audio data and video data, and the output is video data with commentary. For example, the generated commentary is adjusted so that it is played at the appropriate timing within the video.

[0944] Step 9:

[0945] The server sends the video data with commentary to the distribution server. The input is the video data with commentary, and the output is the data sent to the distribution server. For example, the audio and video are integrated using FFmpeg, and then the data is sent to the distribution server.

[0946] (Application example 2)

[0947] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0948] While conventional video commentary systems can provide commentary based on visual information or specific events, it is difficult to provide personalized commentary that reflects the viewer's emotions in real time. Furthermore, there are limited ways to enhance the viewer's sense of realism and immersion. This makes it impossible for the viewing experience to be uniform and reflect the individual excitement and emotions.

[0949] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0950] In this invention, the server includes means for receiving video data, means for analyzing the received video data, means for recognizing a user's emotion, means for generating commentary audio based on the analysis result and the user's emotion data, means for integrating the generated commentary audio into the video data, and means for delivering the integrated video data with commentary audio. This enables personalized commentary that reflects the viewer's emotion, thereby improving the viewing experience.

[0951] "Video data" is a collection of visual information captured by a camera or other image capture device.

[0952] "Means for analyzing" refers to a processing device or algorithm that analyzes received video data or other information and recognizes certain patterns or events.

[0953] "Means for recognizing user emotions" refers to technology that analyzes a user's facial expressions, voice, and other biometric data to determine their emotional state.

[0954] The "means for generating live audio" is a technology for generating audio data that verbally explains or provides commentary to viewers based on video data and emotional data.

[0955] The "means of integration" refers to the means of integrating the generated live audio with the original video data to create a single visual and audio content.

[0956] "Means of distribution" refers to the technology and infrastructure for delivering the integrated video data with commentary to viewers over the Internet.

[0957] "Artificial intelligence" refers to computer systems or algorithms that mimic human intellectual processes by analyzing and learning from large amounts of data.

[0958] "Speech synthesis technology" is a technology for analyzing text information and converting it into natural speech.

[0959] This invention is a system that receives and analyzes video data in real time, recognizes the user's emotions, and generates and distributes live commentary that integrates this information. The detailed configuration and specific operation of this system are described below.

[0960] Hardware Configuration

[0961] User device:

[0962] Smart glasses: Equipped with a camera for video capture and acquisition of user facial expression data.

[0963] Microphone: Used to capture the user's voice data.

[0964] Speaker: Used to provide the generated commentary to the user.

[0965] server:

[0966] AI Server: Used to analyze video data and emotional data. Specifically, it is equipped with an AI model for analyzing video data and an emotion engine for analyzing user emotions.

[0967] Distribution server: A server that distributes integrated video data with live audio to viewers in real time.

[0968] Software Configuration

[0969] On the user's device:

[0970] The user device first acquires video data in real time using the camera installed in the smart glasses. This video data consists of frames captured at regular intervals. The user's voice data is also acquired through the microphone. This data is then sent to the AI ​​server via a socket connection.

[0971] Processing on the AI ​​server:

[0972] The AI ​​server receives video data and audio / facial expression data sent from the user's device. The received video data and emotional data are temporarily stored in a buffer and then input into an AI model for video analysis and an emotion engine that analyzes emotions from facial expressions and audio. This allows specific events and scenes in the video to be identified, while also analyzing the user's emotional state. Based on the analysis results, appropriate commentary text is generated, and this commentary text is converted into audio data using speech synthesis technology.

[0973] Processing on the distribution server:

[0974] The distribution server receives the integrated video data with live commentary from the AI ​​server, encodes and filters it in real time, and then distributes it to viewers, who can view the data on their smartphones, PCs, or other devices.

[0975] Specific examples

[0976] As a specific example of use, consider a user watching a soccer match. When the user wears smart glasses and watches a soccer match in real time, the smart glasses' camera captures video data, and at the same time, the user's facial expression data is captured. This data is sent to the AI ​​server, which recognizes important events such as goal scenes and further analyzes the user's emotions. If a goal scene is recognized and the user is excited, a live commentary voice saying "Goal! The user is excited too!" is generated and delivered to the user in real time through the smart glasses' speaker.

[0977] Prompt Sentence Examples

[0978] Examples of prompts include:

[0979] While the user is watching a sporting event, generate the most compelling commentary based on real-time visual and emotional data that meets the following criteria:

[0980] Users' attention is focused on the players

[0981] The user's facial expression is excited (recognized by the emotion engine)

[0982] Video analysis engine detects goal scenes

[0983] Example of expected output:

[0984] "Goal! The users are excited!" or "Great play! The users look very happy!"

[0985] Thus, the present invention combines video analysis and emotion recognition to provide viewers with an immersive, personalized, and live experience.

[0986] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0987] Step 1:

[0988] The device uses the smart glasses' camera to capture video data in real time, acquiring video frames at specific time intervals, and also uses the microphone to capture the user's voice data.

[0989] Input: Video frames from camera, audio data from microphone

[0990] Output: Captured video and audio data

[0991] Specific operation: Activate the camera of the smart glasses, capture frames at regular intervals, and simultaneously capture audio through the microphone.

[0992] Step 2:

[0993] The device establishes a socket communication to transmit the captured video and audio data to the AI ​​server in real time.

[0994] Input: Captured video and audio data

[0995] Output: Data sent to the AI ​​server

[0996] Specific operation: Establish socket communication and send acquired data to the AI ​​server.

[0997] Step 3:

[0998] The server receives the video data and audio data sent from the terminal and temporarily stores them in a buffer.

[0999] Input: Video and audio data sent from the device

[1000] Output: Buffered data

[1001] Specific operation: Received data is stored in a buffer and prepared for analysis.

[1002] Step 4:

[1003] The server analyzes the received video data using an AI model to identify specific events or scenes (e.g., goal scenes), and uses an emotion engine to analyze the user's voice and facial expression data to determine their emotional state.

[1004] Input: Video and audio data stored in the buffer

[1005] Output: Parsed event data and emotion data

[1006] Specific operation: Video data is input into the AI ​​model to identify specific events, and voice and facial expression data is input into the emotion engine to determine emotions.

[1007] Step 5:

[1008] The server generates appropriate commentary text based on the parsed event data and emotion data.

[1009] Input: Parsed event data and emotion data

[1010] Output: Generated commentary text

[1011] Specific operation: Create text based on event data and adjust commentary content to reflect emotional data.

[1012] Step 6:

[1013] The server converts the generated commentary text into voice data using voice synthesis technology.

[1014] Input: Generated commentary text

[1015] Output: Generated live audio data

[1016] What it does: Inputs text into a speech synthesis engine and generates a natural-sounding audio file.

[1017] Step 7:

[1018] The server integrates the generated commentary audio data with the original video data to generate video data with commentary audio.

[1019] Input: Original video data, generated live audio data

[1020] Output: Video data with integrated commentary

[1021] Specific operation: Synchronizes audio data and video data and adds audio to video.

[1022] Step 8:

[1023] The server delivers the integrated video data with live audio in real time to viewers, who can view the data via their smartphones or PC devices.

[1024] Input: Video data with integrated commentary

[1025] Output: Video data delivered to viewers

[1026] Specific operation: Encodes video data and distributes it in real time over the Internet.

[1027] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1028] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1029] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1030] [Fourth embodiment]

[1031] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1032] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1034] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1035] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1036] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1038] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1039] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1040] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1042] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1044] This invention relates to a system that receives video data, analyzes it, generates live audio, integrates it with the video, and distributes it. The program of this system and its processing are described in detail below.

[1045] Processing on the user terminal

[1046] The user device first acquires video data in real time using a camera or screen capture software. This video data consists of a series of frames, each captured at a fixed interval. Then, a socket connection is established to transmit the acquired video data to the AI ​​server in real time. Through this connection, the frame data is sent to the AI ​​server.

[1047] In a specific example, a user captures video using the camera on their smartphone or PC, and the video is sent to a server via a network.

[1048] Processing on the AI ​​server

[1049] The AI ​​server receives video data sent from the user's device. The received data is temporarily stored in a buffer and then input into a model for video analysis. The AI ​​model analyzes objects and actions in the digital video and identifies events in real time based on the results. Appropriate commentary text is generated based on the recognized events. This commentary text is then converted into audio data using speech synthesis technology. This audio data is synchronized with the original video data to generate video data with commentary audio.

[1050] For example, real-time video of a sporting event is sent to an AI server, where the AI ​​identifies players' movements and scoring scenes, and generates commentary such as, "The player has just scored a goal!" This commentary is converted into audio and combined with the video.

[1051] Processing on the distribution server

[1052] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as by filtering or encoding. Finally, the processed video data with commentary is distributed to viewers live or on demand. Distribution is carried out over the internet, and viewers can watch it via a website or app.

[1053] In a specific example, users can watch live sports broadcasts on their smartphones or PCs, and AI-powered commentary will be provided in real time, improving the viewing experience even without the need for a professional commentator.

[1054] This system automatically analyzes video data, generates appropriate commentary, and delivers it to viewers, allowing even users without specialized knowledge to easily create and distribute high-quality video content with commentary.

[1055] The processing flow will be explained below.

[1056] Step 1:

[1057] The user device acquires video data in real time through a camera or screen capture software. The video data consists of a series of frames, each captured at a fixed interval.

[1058] Step 2:

[1059] The user device establishes a socket connection to transmit the acquired video data to the AI ​​server in real time. Through this connection, frame data is transferred to the AI ​​server.

[1060] Step 3:

[1061] The server receives video data sent from the user terminal, temporarily stores the received data in a buffer, and prepares it for analysis.

[1062] Step 4:

[1063] The server inputs the received data into an AI model for video analysis, which analyzes objects and actions in the video in real time to identify specific events and scenes.

[1064] Step 5:

[1065] The server generates appropriate commentary text based on the analysis results. Commentary content is defined for recognized events (e.g., goal shots, goals scored, transitions between offense and defense, etc.).

[1066] Step 6:

[1067] The server converts the generated commentary text into audio data using speech synthesis technology. In this process, the commentary text is converted into audio data that has the speaker's voice and intonation.

[1068] Step 7:

[1069] The server synchronizes the generated audio data with the original video data, creating video data with live audio. During this integration process, processing is performed to align the timing of the video and audio.

[1070] Step 8:

[1071] The server converts the video data with live audio into a format for distribution, which may include filtering and encoding as needed.

[1072] Step 9:

[1073] The server then distributes the final video data with commentary to viewers over the Internet, who can then watch the video in real time or on demand via websites or applications.

[1074] The above processing steps enable users to easily create high-quality video content with commentary and distribute it in real time, even without specialized knowledge.

[1075] Example 1

[1076] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1077] In response to the increasing diversity of modern video content and the demand for a high-quality viewer experience, there is a lack of technology to automatically generate and distribute video with appropriate commentary in real time. In particular, there is a demand for fast, high-quality commentary generation for live streaming and event video. Furthermore, there is a need for a method to easily create and distribute high-quality video with commentary, even without specialized knowledge.

[1078] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1079] In this invention, the server includes means for a user terminal to acquire video data in real time using a camera or screen capture software and transmit the video data via socket communication, means for the server to receive the transmitted video data and store the received data in a buffer, means for inputting the data stored in the buffer into an AI model to analyze objects and actions in the video and generate commentary text based on event identification, means for converting the generated commentary text into audio data and generating video data with commentary audio in synchronization with the original video data, and means for the distribution server to receive the video data with commentary audio, filter and encode it, and distribute the data to viewers. This makes it possible to automatically generate high-quality video content with commentary in real time and quickly distribute it to viewers.

[1080] A "user terminal" is a device used by a user, which acquires video data in real time using a camera or screen capture software, and transmits the data to a server via socket communication.

[1081] A "camera" is a device for taking still or moving images, and is used in the present invention to capture video data in real time.

[1082] "Screen capture software" is a program that captures the contents of a computer or device screen in real time and saves and transfers it as video data.

[1083] "Socket communication" is a communication method based on a protocol for transmitting and receiving data over a network, and in the present invention is a technique used for transmitting video data from a user terminal to a server.

[1084] A "server" is a computer system that plays a central role in receiving video data sent from user terminals and analyzing, processing, and distributing it.

[1085] A "buffer" is a memory area for temporarily storing data, and serves to temporarily store received video data as a preliminary step to analysis.

[1086] An "AI model" is an algorithm or machine learning model that uses artificial intelligence technology to analyze objects and actions in video and is used to identify events and generate commentary text.

[1087] An "object" is a specific element such as a person, object, or animal that exists in a video and is the subject of analysis.

[1088] "Movement" refers to the actions and behavior of objects in a video, and is an element that can be identified through analysis.

[1089] An "event" is a notable occurrence triggered by a particular object or action, and is the basis for generating live text.

[1090] "Live text" is a sentence generated based on the analyzed events, and is character information for explaining the situation of the video.

[1091] "Speech synthesis technology" is a technology for converting text data into voice data, and is used when converting generated commentary text into voice.

[1092] "Video data" refers to all video information captured by a camera or screen capture software, and is subject to analysis and distribution.

[1093] "Filtering" is a process performed during data processing to remove noise and improve quality, and is used to optimize data before distribution.

[1094] "Encoding" is the process of converting digital data into a format, and is used to convert video data into a format that can be distributed.

[1095] "Distribution" is the process of transmitting video data over a network and making it accessible to viewers, either live or on-demand.

[1096] The present invention relates to a system for acquiring video data in real time, analyzing the data, generating appropriate commentary audio, integrating the audio with the video data, and distributing the audio. Detailed embodiments of this system will be described below.

[1097] Processing on the user terminal

[1098] The user terminal acquires video data in real time using a camera or screen capture software. This video data consists of successive frames. The user captures live video using, for example, the camera on their smartphone or PC. The captured video data is sent to the server via socket communication.

[1099] Specific examples

[1100] Imagine a scenario where a user uses a smartphone camera to capture live footage of cityscapes or events and sends the data to a server via the Internet.

[1101] Prompt Sentence Examples

[1102] "Please send this video data to the AI ​​server."

[1103] Processing on the AI ​​server

[1104] The AI ​​server receives video data sent from the user device via socket communication. The received video data is temporarily stored in a buffer. The buffered video data is then input into the AI ​​model. The AI ​​model analyzes objects and actions in the video and identifies events in real time based on the results. Commentary text is generated in response to the identified events. This commentary text is converted into audio data using speech synthesis technology. The audio data is then synchronized with the original video data to generate video data with commentary audio.

[1105] Specific examples

[1106] Live footage of a sporting event is sent to an AI server, and the AI ​​identifies player movements and goal-scoring scenes to generate commentary. The generated commentary is converted into audio data and properly synchronized with the video. Commentary such as "Player A has scored a goal!" is generated.

[1107] Prompt Sentence Examples

[1108] "Analyze the player's movements in the footage and generate appropriate commentary."

[1109] Processing on the distribution server

[1110] The distribution server receives the video data with commentary sent from the AI ​​server. The received video data is filtered and encoded as needed to convert it into the optimal format. Finally, the distribution server distributes this data to viewers via the Internet. Viewers can watch this video data with commentary in real time via a website or dedicated application.

[1111] Specific examples

[1112] Consider a situation where viewers are watching live sports broadcasts on their PCs or smartphones at home. Real-time commentary by AI provides an immersive viewing experience.

[1113] Prompt Sentence Examples

[1114] "Encode and distribute the video data with commentary integrated into it to users."

[1115] This system enables users without specialized knowledge to easily create high-quality video content with live commentary using advanced technology and quickly distribute it to viewers.

[1116] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1117] Step 1: Acquiring and transmitting video data

[1118] User terminal processing

[1119] The user device acquires video data in real time using a camera or screen capture software. The input is the video data from the camera or capture software. Specifically, each frame of video data is captured at regular intervals and created as a continuous data stream. This video data establishes a socket connection and is sent to the AI ​​server. The output is the video data sent to the server via the network.

[1120] Specific actions

[1121] A user picks up their smartphone, launches the camera app and captures live footage of the event, which is then sent in real time via a socket to the server.

[1122] Step 2: Receiving and buffering video data

[1123] Server Processing

[1124] The server receives video data sent from the user terminal. The input is video data received over the network. The received data is temporarily stored in a buffer. The buffering is for maintaining data consistency and preparing for the next analysis step. The output is the video data stored in the buffer.

[1125] Specific actions

[1126] The server receives real-time video data coming from the user terminal and stores it in a buffer in its internal memory.

[1127] Step 3: Analyzing the video data

[1128] Server Processing

[1129] The server inputs the buffered video data into the AI ​​model. The input is the buffered video data. The AI ​​model analyzes the objects and actions in the video to identify events. The analysis identifies objects in the video and determines their actions and relationships. The output is the identified event information.

[1130] Specific actions

[1131] Footage of sporting events is input into the AI ​​model, and the system analyzes and identifies player movements, goal scenes, etc.

[1132] Step 4: Generate commentary text

[1133] Server Processing

[1134] The server generates commentary text based on the identified event information. The input is the event information output from the AI ​​model. Specific text appropriate for the event is generated, and the generated commentary text becomes the commentary content to be provided to the user. The output is the generated commentary text.

[1135] Specific actions

[1136] The text "Player A scores a goal!" is generated based on the recognized event.

[1137] Step 5: Generate audio data

[1138] Server Processing

[1139] The server converts the generated commentary text into audio data using speech synthesis technology. The input is the commentary text. Using speech synthesis technology, the text is converted into audio data, and the audio is synchronized with the original video data. The output is audio data.

[1140] Specific actions

[1141] The text "Player A scores a goal!" is converted into audio and integrated into the video.

[1142] Step 6: Generate video data with commentary

[1143] Server Processing

[1144] The server synchronizes the audio data with the original video data to generate video data with live audio commentary. The input is the audio data and the original video data. These data are synchronized based on timestamps and other factors and combined into a single integrated data. The output is video data with live audio commentary.

[1145] Specific actions

[1146] The video and audio are appropriately combined to create video content with commentary.

[1147] Step 7: Streaming video data

[1148] Distribution server processing

[1149] The distribution server receives the video data with commentary sent from the AI ​​server. The input is the video data with commentary. The received video data is filtered, encoded, and optimized as needed. Finally, the data is distributed to viewers live or on demand. The output is the optimized video data distributed over the internet.

[1150] Specific actions

[1151] Viewers can watch live sports broadcasts on their PCs or smartphones at home, and the AI-powered automated commentary provided in real time creates a truly immersive experience.

[1152] (Application example 1)

[1153] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1154] Providing live commentary of live events or sports over the Internet typically requires a professional commentator. However, providing high-quality commentary in real time is difficult when there are no professional commentators or when the system relies on the commentator's schedule. Furthermore, it is technically difficult to consistently provide timely and accurate commentary for a wide variety of content at large-scale events. This can reduce viewer satisfaction and make the event less appealing. Therefore, there is a growing demand for a system that can automatically provide real-time commentary without the need for a professional commentator.

[1155] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1156] In this invention, the server includes a means for receiving video data, a means for analyzing the video data and using a generative algorithm to identify events, and a means for using generative synthesis technology to automatically generate commentary based on the identified events and output it as synchronized audio. This makes it possible to provide commentary and live commentary of events in real time without the need for a professional commentator. Viewers can always enjoy high-quality video with commentary, maximizing the appeal of the event.

[1157] "Video Data" means visual information captured from a camera or other image capture device.

[1158] The "receiving means" is a mechanism for receiving video data transmitted from a user terminal.

[1159] "Means for analyzing" means methods for processing received video data to detect and analyze images, motion, and other elements.

[1160] A "knowledge model" is an artificial intelligence model that has been trained in advance to identify events in video.

[1161] "Generative synthesis technology" is a technology that generates audio from text and integrates it with video.

[1162] The "means of distribution" refers to a mechanism for transmitting the integrated video data with commentary via the Internet or other means and delivering it to viewers.

[1163] The "transmitting device" is a device for transmitting video data from a user terminal to a server.

[1164] A "generation algorithm" is an algorithm for identifying events from video data and generating appropriate commentary content based on that information.

[1165] A "prompt algorithm" is an algorithm for converting generated speech text into a more natural format.

[1166] This invention is a system that automatically generates commentary audio for video data shot by a user in real time and distributes the video with commentary audio to viewers. This system is realized mainly by three main components: a user terminal, an AI server, and a distribution server.

[1167] Processing on the user terminal

[1168] The user device first acquires video data in real time using a camera or screen capture software. The acquired video data is then sent to the AI ​​server in real time. To do this, the user device establishes a socket connection and sequentially sends frame data to the AI ​​server.

[1169] Processing on the AI ​​server

[1170] The AI ​​server receives video data sent from the user's device. The received data is then temporarily stored in a buffer and input into a generation algorithm for video analysis. This generation algorithm analyzes objects and actions in the video and identifies events in real time based on the results. Appropriate commentary text is automatically generated based on the identified events.

[1171] This commentary text is then converted into audio data using speech synthesis technology. Generative synthesis technologies such as Google Text-to-Speech (gTTS) are used for this purpose. The audio data is synchronized with the original video data to generate video data with commentary audio. During this process, it is important to utilize a prompting algorithm to convert the generated text into a more natural format. An example of a prompt sentence is, "Generate a commentary of a life event. This is a live video of a soccer match. Generate commentary when a player scores a goal."

[1172] Processing on the distribution server

[1173] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as filtered and encoded, before finally being distributed to viewers live or on demand. This distribution is done via the Internet, and viewers can watch the video through a dedicated website or application.

[1174] Specific examples

[1175] When a user films a soccer match using a smartphone, the video data is sent to an AI server in real time. The AI ​​server identifies player movements and goal-scoring scenes, and generates commentary such as "The player has scored a goal!" The generated commentary is converted into audio using voice synthesis technology, and this audio is integrated into the video. Viewers can enjoy the video with this commentary audio via live streaming. This system makes it possible to provide high-quality video with commentary without the need for a professional commentator.

[1176] In this way, collaboration between user devices, AI servers, and distribution servers can automatically provide real-time commentary, improving the viewer experience.

[1177] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1178] Step 1:

[1179] The user terminal captures video data in real time using a camera or screen capture software. The camera device or screen capture software is used as input. The output is the captured video data. This video data consists of a series of frames, each captured at a fixed interval.

[1180] Step 2:

[1181] The user terminal establishes a socket communication to send the acquired video data to the AI ​​server in real time. The acquired video data is used as input. The output is the transmitted frame data. This frame data is sent to the AI ​​server sequentially.

[1182] Step 3:

[1183] The AI ​​server receives video data sent from the user terminal. The frame data sent from the user terminal is used as input. The output is video data temporarily stored in a buffer.

[1184] Step 4:

[1185] The AI ​​server inputs the video data stored in the buffer into the generative algorithm. The video data is used as input. The output is the analysis result. The generative AI model analyzes the objects and actions in the video and generates the results.

[1186] Step 5:

[1187] The AI ​​server identifies events based on the analysis results. The analysis results of the generative AI model are used as input. The output is the identified event information. This information indicates a specific event (e.g., a goal is scored).

[1188] Step 6:

[1189] The AI ​​server automatically generates commentary text based on the identified events. The identified event information is used as input. The output is the generated commentary text. An example prompt sentence is "Generate a commentary of a life event. This is a live video of a soccer match. Generate a commentary when a player scores a goal."

[1190] Step 7:

[1191] The AI ​​server converts the generated commentary text into audio data. The commentary text is used as input. The output is synthesized audio data. This conversion uses voice synthesis technology such as Google Text-to-Speech (gTTS).

[1192] Step 8:

[1193] The AI ​​server synchronizes the audio data with the original video data. The synthesized audio data and the original video data are used as input. The output is video data with commentary. This data is played back with the audio and video synchronized.

[1194] Step 9:

[1195] The distribution server receives the video data with commentary. As input, it uses the video data with commentary sent from the AI ​​server. The output is video data ready for distribution.

[1196] Step 10:

[1197] The distribution server processes the data as necessary, such as filtering and encoding. The input is video data with commentary. The output is the processed video data.

[1198] Step 11:

[1199] The distribution server delivers the final video data to the viewer. The processed video data is used as input. The output is live or on-demand video delivered to the viewer's device. The viewer can then watch this video through a website or application.

[1200] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1201] This invention combines a system that receives video data, analyzes it, generates live audio, integrates it with the video, and distributes it, with an emotion engine that recognizes the user's emotions. The program of this system and its processing are described in detail below.

[1202] Processing on the user terminal

[1203] The user device first acquires video data in real time using a camera or screen capture software. This video data consists of a series of frames, each captured at a fixed interval. Then, a socket connection is established to transmit the acquired video data to the AI ​​server in real time. Through this connection, the frame data is sent to the AI ​​server.

[1204] In a specific example, a user captures video using the camera on their smartphone or PC, and the video is sent to a server via a network. At the same time, the user's facial expressions and voice data are also acquired and analyzed by the emotion engine.

[1205] Processing on the AI ​​server

[1206] The AI ​​server receives video data and the user's voice and facial expression data sent from the user's device. First, it temporarily stores the received video data and user emotion data in a buffer. The data is then input into a video analysis model, which analyzes objects and actions in the digital video in real time and identifies specific events and scenes. Meanwhile, the emotion engine analyzes emotions from the user's facial expressions and voice and takes the results into account.

[1207] Based on the analysis results, appropriate commentary text is generated. This commentary text includes the recognized events as well as content corresponding to the user's emotions. Next, this commentary text is converted into audio data using speech synthesis technology. The commentary audio is synchronized with the original video data to generate video data with commentary audio corresponding to the user's emotions.

[1208] As a concrete example, real-time video of a sporting event is sent to an AI server, and the AI ​​identifies the players' movements and scoring scenes, generating commentary such as, "The player has just scored a goal!" When this commentary is converted into audio, rich emotional voice is generated according to the user's feelings of excitement and joy.

[1209] Processing on the distribution server

[1210] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as by filtering or encoding. Finally, the processed video data with commentary is distributed to viewers live or on demand. Distribution is carried out over the internet, and viewers can watch it via a website or app.

[1211] For example, users can watch live sports broadcasts on their smartphones or PCs, and AI-powered commentary is provided in real time, improving the viewing experience even without the need for a professional commentator. Furthermore, commentary that responds to the user's emotions creates a greater sense of immersion.

[1212] This system automatically analyzes video data, generates appropriate commentary, and delivers it to viewers. This allows even users without specialized knowledge to easily create and distribute high-quality video content with commentary. Furthermore, commentary that reflects the user's emotions further enhances the viewing experience.

[1213] The processing flow will be explained below.

[1214] Step 1:

[1215] The user device captures video data in real time through a camera or screen capture software. This video data consists of a series of frames, each captured at regular intervals. At the same time, data is also captured to record the user's facial expressions and voice.

[1216] Step 2:

[1217] The user device establishes a socket connection to transmit the captured video data and the user's facial expression and voice data to the AI ​​server in real time. Through this connection, frame data and emotion data are transferred to the AI ​​server.

[1218] Step 3:

[1219] The server receives the video data and the user's facial expression and voice data sent from the user's device. The received data is temporarily stored in a buffer and prepared for analysis.

[1220] Step 4:

[1221] The server inputs the received video data into an AI model for video analysis, which analyzes objects and actions in the digital video in real time to identify specific events and scenes.

[1222] Step 5:

[1223] The server inputs the user's facial and voice data into the emotion engine, which analyzes this data and identifies the user's emotional state (e.g., joy, surprise, excitement, etc.) in real time.

[1224] Step 6:

[1225] The server generates appropriate commentary text based on the video analysis results and the emotion analysis results from the emotion engine. For example, when a goal is scored in a sporting event, the server generates the text "A goal has been scored!" and emphasizes the text according to the user's excitement level.

[1226] Step 7:

[1227] The server converts the generated commentary text into voice data using speech synthesis technology, and in the process adds the speaker's tone and intonation according to the user's emotions.

[1228] Step 8:

[1229] The server synchronizes the generated audio data with the original video data, creating video data with live audio. During this integration process, processing is performed to align the timing of the video and audio.

[1230] Step 9:

[1231] The server converts the video data with live audio into a format for distribution, which may include filtering and encoding as needed.

[1232] Step 10:

[1233] The server then distributes the final video data with commentary to viewers over the Internet, who can then watch the video in real time or on demand via websites or applications.

[1234] The above processing steps enable users to easily create high-quality video content with commentary and distribute it in real time, even without specialized knowledge. Furthermore, commentary that reflects the user's emotions makes the viewing experience even more engaging.

[1235] Example 2

[1236] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1237] Conventional video distribution systems require manual analysis of video and generation of commentary, which requires a great deal of time and effort. Furthermore, it is difficult to provide commentary that reflects the user's real-time emotions, and there is a demand for an improved viewing experience. Furthermore, it is difficult for users without specialized knowledge to easily create and distribute high-quality video content with commentary.

[1238] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1239] In this invention, the server includes means for acquiring video data, means for transmitting the acquired video data in real time, means for receiving and analyzing the video data and user emotion data, means for generating commentary text based on the analysis results, means for converting the generated commentary text into audio, means for integrating the generated commentary audio into the video data, and means for distributing the integrated video data with the commentary audio. This makes it possible to automatically analyze the video data and generate and distribute commentary corresponding to the user's emotions in real time.

[1240] "Video data" is a collection of visual information captured using a camera or capture device.

[1241] "Means of acquisition" refers to the method of collecting video data using a camera or capture device, or software that controls them.

[1242] A "transmitting means" is a method or protocol for transmitting data over a network to another device or server.

[1243] "Means for receiving and analyzing" refers to methods for receiving transmitted data and using artificial intelligence or other technologies to decipher and analyze its contents.

[1244] "Live text" is text that explains events and situations in the video.

[1245] A "means of generation" is a method of creating new data or information using a specific algorithm or program.

[1246] "Means for converting to voice" refers to technology or software for converting text data into voice data.

[1247] "Means of integration" is the process of bringing together multiple pieces of data or information into one.

[1248] "Means of distribution" refers to the communication method or technology used to deliver the final data to users or viewers.

[1249] A "generative AI model" is an algorithm or tool that uses artificial intelligence techniques to generate new data or content.

[1250] "Emotion data" is information about the emotional state of the user collected from facial expressions, voice, etc.

[1251] "Real-time" means that data is processed and transmitted in a very short time, providing near-immediate results.

[1252] This invention combines a system that receives video data, analyzes it, generates live audio, integrates it with the video, and distributes it, with an emotion engine that recognizes the user's emotions. The program of this system and its processing are described in detail below.

[1253] The user device first acquires video data in real time using a camera or screen capture software. This video data consists of a series of frames, each captured at a fixed interval. Then, a socket connection is established to transmit the acquired video data to the AI ​​server in real time. Through this connection, the frame data is sent to the AI ​​server.

[1254] In a specific example, a user captures video using the camera on their smartphone or PC, and the video is sent to a server via a network. At the same time, the user's facial expressions and voice data are also acquired and analyzed by the emotion engine.

[1255] The AI ​​server receives video data and the user's voice and facial expression data sent from the user's device. First, it temporarily stores the received video data and the user's emotional data in a buffer. The data is then input into a video analysis model, which analyzes objects and actions in the digital video in real time and identifies specific events and scenes. Image processing libraries such as OpenCV are used for video analysis.

[1256] On the other hand, the emotion engine analyzes the user's emotions from their facial expressions and voice and stores the results in a buffer. For emotion analysis, a service such as Microsoft Azure's Emotion API is used.

[1257] Based on the analysis results, appropriate commentary text is generated. This commentary text includes the recognized events as well as content corresponding to the user's emotions. A natural language generation model (such as GPT-3) is used to generate the text. The generated commentary text is then converted into audio data using speech synthesis technology. The Google Text-to-Speech API or similar is used for speech synthesis. The commentary audio is synchronized with the original video data, and video data with commentary audio corresponding to the user's emotions is generated.

[1258] As a concrete example, real-time video of a sporting event is sent to an AI server, and the AI ​​identifies the players' movements and scoring scenes, generating commentary such as, "The player has just scored a goal!" When this commentary is converted into audio, rich emotional voice is generated according to the user's feelings of excitement and joy.

[1259] The distribution server receives the integrated video data with commentary from the AI ​​server. The received data is then processed as needed, such as by filtering or encoding. FFmpeg is used for filtering and encoding, for example. Finally, the processed video data with commentary is distributed to viewers live or on demand. Distribution is carried out over the internet, and viewers can watch it via a website or app.

[1260] For example, users can watch live sports broadcasts on their smartphones or PCs, and AI-powered commentary is provided in real time, improving the viewing experience even without the need for a professional commentator. Furthermore, commentary that responds to the user's emotions creates a greater sense of immersion.

[1261] Example prompt sentence:

[1262] "Analyze video data from a soccer match in real time, recognize player movements and goal-scoring scenes, and generate commentary. If the user's emotion is classified as 'excited,' include an emotionally rich expression in the commentary. For example, say, 'The player scored a goal! What a great shot!'"

[1263] This system automatically analyzes video data, generates appropriate commentary, and delivers it to viewers. This allows even users without specialized knowledge to easily create and distribute high-quality video content with commentary. Furthermore, commentary that reflects the user's emotions further enhances the viewing experience.

[1264] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1265] Step 1:

[1266] A user device acquires video data using a camera or screen capture software (e.g., OBS Studio). The acquired video data consists of a series of frames. The input is camera video or screen capture, and the output is video data (frame data). For example, a user might launch a camera app on their smartphone and capture live video.

[1267] Step 2:

[1268] The user device establishes a socket connection to send the acquired video data to the AI ​​server in real time. Specifically, the Python socket library is used to periodically send video frame data to the server. The input is the video data, and the output is the frame data sent to the server. During this process, the user's facial expressions and voice data are also collected at the same time.

[1269] Step 3:

[1270] The server receives the video data and the user's voice and facial expression data sent from the user's device and temporarily stores them in a buffer. The input is the video data and emotion data from the user's device, and the output is the data stored in the buffer. For example, when the video data and voice data arrive at the server, they are temporarily saved.

[1271] Step 4:

[1272] The server analyzes the received video data using the OpenCV library. Specifically, it identifies objects and actions in the digital video in real time. The input is the video data stored in the buffer, and the output is the analyzed video data (event identification results). For example, it analyzes video footage of a sporting event to identify player movements and goal scenes.

[1273] Step 5:

[1274] The server uses an emotion engine (e.g., Emotion API) to analyze emotions from the user's facial expressions and voice. The input is the user's facial expression and voice data, and the output is the analyzed emotion data. For example, the emotion engine analyzes the user's camera footage and determines that the user is "excited."

[1275] Step 6:

[1276] The server generates commentary text based on the video data analysis results and emotion data. In this process, a generative AI model (e.g., GPT-3) is used to generate the text. The input is the analyzed event data and emotion data, and the output is the commentary text. For example, a commentary such as "The player scored a goal!" is generated.

[1277] Step 7:

[1278] The server converts the generated commentary text into audio data using speech synthesis technology (e.g., Google Text-to-Speech API). The input is the commentary text, and the output is audio data. For example, the text "The player scored a goal!" is converted into audio.

[1279] Step 8:

[1280] The server synchronizes the generated commentary with the original video data. The input is audio data and video data, and the output is video data with commentary. For example, the generated commentary is adjusted so that it is played at the appropriate timing within the video.

[1281] Step 9:

[1282] The server sends the video data with commentary to the distribution server. The input is the video data with commentary, and the output is the data sent to the distribution server. For example, the audio and video are integrated using FFmpeg, and then the data is sent to the distribution server.

[1283] (Application example 2)

[1284] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1285] While conventional video commentary systems can provide commentary based on visual information or specific events, it is difficult to provide personalized commentary that reflects the viewer's emotions in real time. Furthermore, there are limited ways to enhance the viewer's sense of realism and immersion. This makes it impossible for the viewing experience to be uniform and reflect the individual excitement and emotions.

[1286] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1287] In this invention, the server includes means for receiving video data, means for analyzing the received video data, means for recognizing a user's emotion, means for generating commentary audio based on the analysis result and the user's emotion data, means for integrating the generated commentary audio into the video data, and means for delivering the integrated video data with commentary audio. This enables personalized commentary that reflects the viewer's emotion, thereby improving the viewing experience.

[1288] "Video data" is a collection of visual information captured by a camera or other image capture device.

[1289] "Means for analyzing" refers to a processing device or algorithm that analyzes received video data or other information and recognizes certain patterns or events.

[1290] "Means for recognizing user emotions" refers to technology that analyzes a user's facial expressions, voice, and other biometric data to determine their emotional state.

[1291] The "means for generating live audio" is a technology for generating audio data that verbally explains or provides commentary to viewers based on video data and emotional data.

[1292] The "means of integration" refers to the means of integrating the generated live audio with the original video data to create a single visual and audio content.

[1293] "Means of distribution" refers to the technology and infrastructure for delivering the integrated video data with commentary to viewers over the Internet.

[1294] "Artificial intelligence" refers to computer systems or algorithms that mimic human intellectual processes by analyzing and learning from large amounts of data.

[1295] "Speech synthesis technology" is a technology for analyzing text information and converting it into natural speech.

[1296] This invention is a system that receives and analyzes video data in real time, recognizes the user's emotions, and generates and distributes live commentary that integrates this information. The detailed configuration and specific operation of this system are described below.

[1297] Hardware Configuration

[1298] User device:

[1299] Smart glasses: Equipped with a camera for video capture and acquisition of user facial expression data.

[1300] Microphone: Used to capture the user's voice data.

[1301] Speaker: Used to provide the generated commentary to the user.

[1302] server:

[1303] AI Server: Used to analyze video data and emotional data. Specifically, it is equipped with an AI model for analyzing video data and an emotion engine for analyzing user emotions.

[1304] Distribution server: A server that distributes integrated video data with live audio to viewers in real time.

[1305] Software Configuration

[1306] On the user's device:

[1307] The user device first acquires video data in real time using the camera installed in the smart glasses. This video data consists of frames captured at regular intervals. The user's voice data is also acquired through the microphone. This data is then sent to the AI ​​server via a socket connection.

[1308] Processing on the AI ​​server:

[1309] The AI ​​server receives video data and audio / facial expression data sent from the user's device. The received video data and emotional data are temporarily stored in a buffer and then input into an AI model for video analysis and an emotion engine that analyzes emotions from facial expressions and audio. This allows specific events and scenes in the video to be identified, while also analyzing the user's emotional state. Based on the analysis results, appropriate commentary text is generated, and this commentary text is converted into audio data using speech synthesis technology.

[1310] Processing on the distribution server:

[1311] The distribution server receives the integrated video data with live commentary from the AI ​​server, encodes and filters it in real time, and then distributes it to viewers, who can view the data on their smartphones, PCs, or other devices.

[1312] Specific examples

[1313] As a specific example of use, consider a user watching a soccer match. When the user wears smart glasses and watches a soccer match in real time, the smart glasses' camera captures video data, and at the same time, the user's facial expression data is captured. This data is sent to the AI ​​server, which recognizes important events such as goal scenes and further analyzes the user's emotions. If a goal scene is recognized and the user is excited, a live commentary voice saying "Goal! The user is excited too!" is generated and delivered to the user in real time through the smart glasses' speaker.

[1314] Prompt Sentence Examples

[1315] Examples of prompts include:

[1316] While the user is watching a sporting event, generate the most compelling commentary based on real-time visual and emotional data that meets the following criteria:

[1317] Users' attention is focused on the players

[1318] The user's facial expression is excited (recognized by the emotion engine)

[1319] Video analysis engine detects goal scenes

[1320] Example of expected output:

[1321] "Goal! The users are excited!" or "Great play! The users look very happy!"

[1322] Thus, the present invention combines video analysis and emotion recognition to provide viewers with an immersive, personalized, and live experience.

[1323] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1324] Step 1:

[1325] The device uses the smart glasses' camera to capture video data in real time, acquiring video frames at specific time intervals, and also uses the microphone to capture the user's voice data.

[1326] Input: Video frames from camera, audio data from microphone

[1327] Output: Captured video and audio data

[1328] Specific operation: Activate the camera of the smart glasses, capture frames at regular intervals, and simultaneously capture audio through the microphone.

[1329] Step 2:

[1330] The device establishes a socket communication to transmit the captured video and audio data to the AI ​​server in real time.

[1331] Input: Captured video and audio data

[1332] Output: Data sent to the AI ​​server

[1333] Specific operation: Establish socket communication and send acquired data to the AI ​​server.

[1334] Step 3:

[1335] The server receives the video data and audio data sent from the terminal and temporarily stores them in a buffer.

[1336] Input: Video and audio data sent from the device

[1337] Output: Buffered data

[1338] Specific operation: Received data is stored in a buffer and prepared for analysis.

[1339] Step 4:

[1340] The server analyzes the received video data using an AI model to identify specific events or scenes (e.g., goal scenes), and uses an emotion engine to analyze the user's voice and facial expression data to determine their emotional state.

[1341] Input: Video and audio data stored in the buffer

[1342] Output: Parsed event data and emotion data

[1343] Specific operation: Video data is input into the AI ​​model to identify specific events, and voice and facial expression data is input into the emotion engine to determine emotions.

[1344] Step 5:

[1345] The server generates appropriate commentary text based on the parsed event data and emotion data.

[1346] Input: Parsed event data and emotion data

[1347] Output: Generated commentary text

[1348] Specific operation: Create text based on event data and adjust commentary content to reflect emotional data.

[1349] Step 6:

[1350] The server converts the generated commentary text into voice data using voice synthesis technology.

[1351] Input: Generated commentary text

[1352] Output: Generated live audio data

[1353] What it does: Inputs text into a speech synthesis engine and generates a natural-sounding audio file.

[1354] Step 7:

[1355] The server integrates the generated commentary audio data with the original video data to generate video data with commentary audio.

[1356] Input: Original video data, generated live audio data

[1357] Output: Video data with integrated commentary

[1358] Specific operation: Synchronizes audio data and video data and adds audio to video.

[1359] Step 8:

[1360] The server delivers the integrated video data with live audio in real time to viewers, who can view the data via their smartphones or PC devices.

[1361] Input: Video data with integrated commentary

[1362] Output: Video data delivered to viewers

[1363] Specific operation: Encodes video data and distributes it in real time over the Internet.

[1364] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1365] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1366] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1367] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1368] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1369] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1370] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1371] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1372] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1373] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1374] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1375] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1376] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1377] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1378] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1379] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1380] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1381] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1382] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1383] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1384] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1385] The following is further disclosed regarding the above embodiment.

[1386] (Claim 1)

[1387] means for receiving video data;

[1388] means for analyzing the received video data;

[1389] means for generating commentary audio based on the analysis results;

[1390] means for integrating the generated commentary audio into video data;

[1391] A system including a means for distributing integrated video data with live audio commentary.

[1392] (Claim 2)

[1393] 10. The system of claim 1, wherein the means for analyzing the video data utilizes artificial intelligence.

[1394] (Claim 3)

[1395] 10. The system of claim 1, wherein the means for generating commentary utilizes voice synthesis technology.

[1396] "Example 1"

[1397] (Claim 1)

[1398] A means for a user terminal to acquire video data in real time using a camera or screen capture software and transmit the video data via socket communication;

[1399] a means for the server to receive the transmitted video data and store the received data in a buffer;

[1400] a means for inputting the buffered data into an AI model to analyze objects and actions in the video and generate commentary text based on event identification;

[1401] means for converting the generated commentary text into audio data and generating video data with commentary audio in synchronization with the original video data;

[1402] A system including a means for a distribution server to receive video data with live audio, filter and encode it, and distribute the data to viewers.

[1403] (Claim 2)

[1404] 10. The system of claim 1, wherein the means for analyzing the video data utilizes a generative AI model.

[1405] (Claim 3)

[1406] 10. The system of claim 1, wherein the means for generating commentary utilizes voice synthesis technology.

[1407] "Application Example 1"

[1408] (Claim 1)

[1409] means for receiving video data;

[1410] means for analyzing the received video data;

[1411] means for generating commentary audio based on the analysis results;

[1412] means for integrating the generated commentary audio into video data;

[1413] a means for delivering the integrated video data with commentary;

[1414] a device for transmitting video data;

[1415] means for analyzing the transmitted video data and using a knowledge model to identify events within the image;

[1416] a means for utilizing generative synthesis technology to automatically generate commentary based on the identified events and output it as synchronized audio;

[1417] A system including:

[1418] (Claim 2)

[1419] 10. The system of claim 1, wherein the system utilizes a generative algorithm to analyze the received video data and identify the events.

[1420] (Claim 3)

[1421] 10. The system of claim 1, wherein the means for generating commentary utilizes speech synthesis technology and prompting algorithms.

[1422] "Example 2: Combining Emotion Engines"

[1423] (Claim 1)

[1424] a means for acquiring video data;

[1425] means for transmitting the acquired video data in real time;

[1426] means for receiving and analyzing video data and user emotion data;

[1427] means for generating commentary text based on the analysis results;

[1428] A means for converting the generated commentary text into audio;

[1429] means for integrating the generated commentary audio into video data;

[1430] A system including means for delivering integrated live audio and video data.

[1431] (Claim 2)

[1432] 10. The system of claim 1, wherein the means for analyzing the video data and emotion data utilizes artificial intelligence.

[1433] (Claim 3)

[1434] 10. The system of claim 1, wherein the means for generating the commentary text utilizes a generative AI model.

[1435] (Claim 4)

[1436] 2. The system according to claim 1, wherein the live audio integrated into the video data utilizes voice synthesis technology.

[1437] "Application example 2 when combining emotion engines"

[1438] (Claim 1)

[1439] means for receiving video data;

[1440] means for analyzing the received video data;

[1441] means for recognizing a user's emotion;

[1442] means for generating commentary voice based on the analysis result and the user's emotion data;

[1443] means for integrating the generated commentary audio into video data;

[1444] A system including a means for distributing integrated video data with live audio commentary.

[1445] (Claim 2)

[1446] 10. The system of claim 1, wherein the system utilizes artificial intelligence to analyze the video data and the user's emotional data.

[1447] (Claim 3)

[1448] 2. The system according to claim 1, wherein the commentary voice generated using voice synthesis technology includes content corresponding to the user's emotions. [Explanation of symbols]

[1449] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving video data; means for analyzing the received video data; means for generating commentary audio based on the analysis results; means for integrating the generated commentary audio into video data; A system including a means for distributing integrated video data with live audio commentary.

2. 10. The system of claim 1, wherein the means for analyzing the video data utilizes artificial intelligence.

3. 10. The system of claim 1, wherein the means for generating commentary utilizes voice synthesis technology.

Images