System

Abstract

To provide a system.SOLUTION: A system includes: means for inputting video data of a variety of sports and entertainments; means including generative AI generating live broadcast / subtitles on the basis of the video data; means for adjusting output of the generative AI according to a user's taste; means including an emotion engine that recognizes the user's emotion and adjusts output of generative AI on the basis of the emotion; means for using a camera and a microphone of a smart phone and inputting the user's expressions and tone of voice as information showing the user's emotion; and means for using generative AI to generate live broadcast and subtitles in accordance with the user's emotion in real time.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Current live broadcasts and subtitles for sports and entertainment are one-way and do not allow viewers to choose. In particular, major sports broadcasts are limited in the number of languages ​​available and may not support certain languages. [Means for solving the problem]

[0005] As a means of solving this problem, we provide a system that includes a generation AI that inputs video data of various sports and entertainment and generates live broadcasts and subtitles based on the video data. In addition, we include a means for adjusting the output of the generation AI to suit the user's preferences. This allows viewers to enjoy live broadcasts and subtitles in their preferred voices and writing styles. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16]FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. [Figure 19] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system according to the embodiment 2 of the second embodiment in the case where an emotion engine is combined. [Figure 20] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when combined with an emotion engine. [Figure 21] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the embodiment 3 of the form example 3 in the case where an emotion engine is combined. [Figure 22] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One embodiment of the present invention is a system that inputs video data of various sports and entertainment. This system includes a generation AI that generates live broadcasts and subtitles based on the video data. The generation AI automatically generates live broadcasts and subtitles from the video data using AI technology such as deep learning.

[0029] "Example 2"

[0030] Furthermore, it includes a means for adjusting the output of the generating AI to suit the user's preferences. Specifically, it allows the user to set the tone of voice, speed, writing style, etc. that they prefer. For example, the user can select a "calm male voice" or an "exciting female voice," as well as a writing style such as "polite expression" or "casual expression."

[0031] "Example 3"

[0032] The AI ​​generator also has the ability to convert the content into the user's native language according to their preferences. For example, if the user is Japanese, the AI ​​generator will convert the generated live broadcast and subtitles into Japanese. This allows users to enjoy live broadcasts and subtitles in a language they can easily understand.

[0033] The process flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The user inputs the sports or entertainment video data they want to watch into the system.

[0036] Step 2: The system's generation AI analyzes the video data and generates live commentary and subtitles. At this time, the AI ​​understands the content of the video data and generates appropriate live commentary and subtitles.

[0037] Step 3: The generated commentary and subtitles are displayed to the user.

[0038] "Example 2"

[0039] Step 1: The user provides the system with their preferred voice tone, speed and writing style.

[0040] Step 2: The AI ​​generator adjusts the tone, speed, and writing style of the commentary and subtitles it generates based on the user's settings.

[0041] Step 3: The adjusted commentary and subtitles are then displayed to the user.

[0042] "Example 3"

[0043] Step 1: The user sets the preferred language on the system.

[0044] Step 2: The generation AI converts the live commentary and subtitles into the specified language based on the user's settings.

[0045] Step 3: The converted live commentary and subtitles will be displayed to users.

[0046] Example 1

[0047] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0048] Conventional live broadcast and subtitle generation systems based on video data require manual editing, making it difficult to adjust in real time and to meet user preferences. In addition, the analysis of video data and the quality of the generated content often lacked consistency. This resulted in a poor user experience and made it difficult to generate content efficiently.

[0049] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0050] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means for dividing the video data into frames, a means for inputting the divided frames to a generative AI model, a means for detecting important events and actions from the video data using the generative AI model and generating live coverage and subtitles, a means for overlaying the generated live coverage and subtitles on the video data, and a means for adjusting the output of the generative AI to suit the user's preferences. This enables automatic generation of high-quality live coverage and subtitles in real time, and allows easy adjustment according to the user's preferences.

[0051] "Video data" is data in digital format that contains visual information such as sports and entertainment.

[0052] A "frame" is an individual still image that makes up video data.

[0053] A "generative AI model" is an artificial intelligence algorithm that uses deep learning technology to detect important events and actions from video data and generate live commentary and subtitles.

[0054] "Significant Events and Actions" refers to occurrences or movements that are of particular interest within sports or entertainment video data.

[0055] "Live coverage" refers to commentary and comments given in real time to video data.

[0056] "Subtitles" are text information that is displayed overlaid on video data.

[0057] "User preferences" refers to individual settings such as the user's desired voice adjustments or writing style.

[0058] "Means of adjustment" refers to functions or methods for modifying the output of a generative AI model to suit the user's preferences.

[0059] The "server" is a computer system that performs processes such as analyzing video data, running generative AI models, and overlaying generated live coverage and subtitles.

[0060] This invention is a system that automatically generates live commentary and subtitles using video data of various sports and entertainment. The system includes a generative AI model and uses deep learning technology to generate live commentary and subtitles from the video data.

[0061] Hardware and Software Used

[0062] Server: A computer system that performs processes such as analyzing video data, running generative AI models, and overlaying generated live coverage and subtitles.

[0063] Terminal: A device on which a user uploads video data and views the processed video. Specifically, this applies to computers and smartphones.

[0064] Generative AI models: Built using deep learning frameworks such as TENSORFLOW(R) and PyTorch.

[0065] Data processing and calculation

[0066] 1. Input of video data

[0067] Users upload sports and entertainment video data to the system.

[0068] The terminal transmits the uploaded video data to the server.

[0069] 2. Analysis of video data

[0070] The server divides the received video data into frames. To do this, it uses a video processing library such as FFmpeg.

[0071] The server inputs the segmented frames into a generative AI model.

[0072] 3. Generative AI Processing

[0073] The generative AI model uses deep learning techniques to detect important events and actions in video data.

[0074] The generative AI model generates appropriate commentary and subtitles based on the detected events.

[0075] 4. Output of results

[0076] The server overlays the generated commentary and subtitles onto the video data, specifically by using FFmpeg to overlay the text onto the video.

[0077] The server transmits the processed video data to the terminal.

[0078] Users can watch live broadcasts and videos with added subtitles through their terminals.

[0079] Examples

[0080] For example, consider the case where a user uploads a video of a soccer match to the system. The server divides the video data into frames and inputs them into the generative AI model. The generative AI model detects goal and foul scenes, and generates commentary and subtitles such as "Goal!" and "A foul has been committed" based on the detected scenes. The server overlays the generated commentary and subtitles onto the video and sends it to the device. The user can then watch the soccer match video with the commentary and subtitles added via the device.

[0081] Examples of prompt statements

[0082] "Upload a video of a soccer game. Our generative AI will detect goals and fouls, and automatically generate commentary and subtitles."

[0083] This system allows users to enjoy high-quality live broadcasts and videos with added subtitles in real time.

[0084] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0085] Step 1:

[0086] Users upload sports and entertainment video data to the system.

[0087] Input: A video file selected by the user.

[0088] Specific actions: A user opens the system's web interface, selects a video file, and clicks the upload button.

[0089] Output: Video data is sent from the device to the server.

[0090] Step 2:

[0091] The terminal transmits the uploaded video data to the server.

[0092] Input: Video data uploaded by the user.

[0093] Specific operation: The device uploads video data to the server using an HTTP POST request.

[0094] Output: The server receives the video data.

[0095] Step 3:

[0096] The server divides the received video data into frames.

[0097] Input: Video data received by the server.

[0098] Specific operation: The server uses a video processing library such as FFmpeg to divide the video data into frames. Specifically, it executes FFmpeg commands to divide the video into frames.

[0099] ffmpeg -i input.mp4 -vf "fps=30" frame_%04d.png

[0100] Output: Segmented frame images.

[0101] Step 4:

[0102] The server inputs the segmented frames into a generative AI model.

[0103] Input: Segmented frame images.

[0104] Specific operation: The server converts the frame images into an appropriate format and inputs them into the generative AI model.

[0105] Output: The frame data fed into the generative AI model.

[0106] Step 5:

[0107] The generative AI model uses deep learning techniques to detect important events and actions in video data.

[0108] Input: The frame data fed into the generative AI model.

[0109] How it works: The generative AI model analyzes frame data to detect important events and actions, for example, goals and fouls in a soccer game.

[0110] Output: Information about the detected event and / or action.

[0111] Step 6:

[0112] The generative AI model generates appropriate commentary and subtitles based on the detected events.

[0113] Input: Information about the detected event or action.

[0114] How it works: Based on the detected events, the generative AI model generates appropriate text, such as commentary or subtitles like "Goal!" or "A foul has been committed."

[0115] Output: The generated play-by-play and subtitle text.

[0116] Step 7:

[0117] The server overlays the generated live coverage and subtitles onto the video data.

[0118] Input: The generated play-by-play or subtitle text.

[0119] What it does: The server uses FFmpeg to overlay the generated text onto the video.

[0120] ffmpeg -i input.mp4 -vf "drawtext=text='Goal!':x=10:y=10:fontsize=24:fontcolor=white" output.mp4

[0121] Output: Video data with live coverage and subtitles overlaid.

[0122] Step 8:

[0123] The server transmits the processed video data to the terminal.

[0124] Input: Video data with commentary and subtitles overlaid.

[0125] Specific operation: The server sends the processed video file to the terminal as an HTTP response.

[0126] Output: The device receives the processed video data.

[0127] Step 9:

[0128] Users can watch live broadcasts and videos with added subtitles through their terminals.

[0129] Input: The processed video data received by the device.

[0130] Specific behavior: The user uses the device's media player to play live broadcast or video with added subtitles.

[0131] Output: The video the user sees.

[0132] (Application example 1)

[0133] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] In conventional sports and entertainment video distribution systems, real-time live broadcasts and subtitle generation were often done manually, making it difficult to provide information quickly and accurately. In addition, it was difficult to customize to meet user preferences, and support for multiple languages ​​was limited. This led to problems such as reduced viewer satisfaction.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0136] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means for displaying the generated live coverage and subtitles in real time, and a means for providing live coverage and subtitles in multiple languages. This enables accurate generation of live coverage and subtitles in real time, and allows customization to suit the user's preferences and support in multiple languages.

[0137] "Various sports and entertainment video data" refers to video information of sporting events and entertainment content, including live and recorded footage.

[0138] "Input means" refers to a method or device for inputting video data into the system.

[0139] "Generative AI" refers to algorithms and models that use artificial intelligence techniques such as deep learning to automatically generate live commentary and subtitles from video data.

[0140] "Means for tailoring to user preferences" refers to a method or apparatus for customizing the content or format of the generated commentary or subtitles based on a user's settings or preferences.

[0141] "Means for displaying in real time" refers to a method or apparatus for displaying the generated commentary or subtitles on a user's device immediately and without delay.

[0142] "Means for providing commentary and subtitles in multiple languages" refers to a method or apparatus for providing the generated commentary and subtitles in multiple different languages.

[0143] The system for carrying out the present invention inputs video data of various sports and entertainment, and automatically generates live broadcasts and subtitles using a generation AI. A specific embodiment of this system will be described below.

[0144] System Configuration

[0145] The system mainly uses the following hardware and software:

[0146] Hardware: Smartphone (iOS or ANDROID (registered trademark)), server

[0147] Software: OpenCV, TensorFlow, Transformers (Hugging Face)

[0148] Data processing and calculation

[0149] Video data input

[0150] The server receives video data of various sports and entertainment events, either live or recorded.

[0151] Video data pre-processing

[0152] The server uses OpenCV to split the video data into frames and perform pre-processing, which includes frame resizing and noise removal.

[0153] Analysis by generative AI

[0154] The server analyzes the preprocessed frames using TensorFlow and Transformers. Specifically, a generative AI model (e.g., GPT-2) is used to generate commentary and subtitles from the video data. The generative AI model takes the prompt as input and outputs appropriate commentary and subtitles.

[0155] Tailored to your preferences

[0156] The server then tailors the generated commentary and subtitles to the user's preferences, including customizing the tone of voice and writing style.

[0157] Real-time display

[0158] The device (smartphone) displays the generated live coverage and subtitles in real time, allowing users to receive information without delay.

[0159] Multiple Languages

[0160] The server provides the generated commentary and subtitles in multiple languages, allowing users who speak different languages ​​to enjoy the same content.

[0161] Examples

[0162] For example, when a goal is scored during a soccer game, the server inputs the prompt sentence "Describe the current sports event: A player is running towards the goal with the ball" into the generative AI model. Based on this, the generative AI model generates a live report and subtitles such as "Goal! Player A scores a great shot!" and displays them on the device in real time.

[0163] In this way, a system can be realized that generates live commentary and subtitles for sporting events in real time.

[0164] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0165] Step 1:

[0166] The server receives video data of various sports and entertainment events. The video data is provided in the form of live or recorded video. The input video data is prepared for processing within the server.

[0167] Step 2:

[0168] The server splits the video data into frames using OpenCV and performs preprocessing. Preprocessing includes frame resizing and noise removal. Specifically, the video data is captured at a constant frame rate and each frame is resized to a standardized size. The input is the video data and the output is the preprocessed frames.

[0169] Step 3:

[0170] The server analyzes the preprocessed frames using TensorFlow and Transformers. Specifically, a generative AI model (e.g., GPT-2) is used to generate commentary and subtitles from the video data. The generative AI model takes a prompt as input and outputs appropriate commentary and subtitles. The input is the preprocessed frames and the prompt, and the output is the generated commentary and subtitles.

[0171] Step 4:

[0172] The server adjusts the generated commentary and subtitles to suit the user's preferences. The adjustments include customizing the tone of voice and sentence expressions. Specifically, the server refers to the user's settings information and converts the generated text appropriately. The input is the generated commentary and subtitles, and the output is the adjusted commentary and subtitles.

[0173] Step 5:

[0174] The device (smartphone) displays the generated live broadcast and subtitles in real time, allowing the user to receive information without delay. Specifically, text is overlaid on the device's display. The input is the adjusted live broadcast and subtitles, and the output is the text displayed on the device's display.

[0175] Step 6:

[0176] The server provides the generated commentary and subtitles in multiple languages, allowing users who speak different languages ​​to enjoy the same content. Specifically, a generative AI model is used to translate the original text into other languages. The input is the generated commentary and subtitles, and the output is the commentary and subtitles in multiple languages.

[0177] Example 2

[0178] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0179] Conventional generative AI systems have had difficulty providing output that meets individual user preferences. In particular, they were unable to accommodate detailed settings that differ from user to user, such as tone of voice, speed, and writing style. As a result, the user experience was uniform and insufficient to customize to meet individual needs.

[0180] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0181] In this invention, the server includes a means for inputting various data, a means including a generative AI that generates information based on the data, a means for adjusting the output of the generative AI based on the user's preferences, a means for acquiring the user's settings, a means for generating a prompt sentence to be input to the generative AI model, a means for executing the generative AI model, a means for adjusting the output obtained from the generative AI model, and a means for transmitting the adjusted output to the user's terminal, thereby enabling customized output according to the individual preferences of the user.

[0182] "Various types of data" refers to data in various formats, including information entered by the user and information obtained by the system.

[0183] "Generative AI" refers to artificial intelligence that generates information based on input data.

[0184] "User preferences" refers to individual settings such as tone of voice, speed, writing style, etc. set by the User.

[0185] "Means for acquiring user settings" refers to a function that enables the system to acquire preference information that the user has set in advance.

[0186] A "prompt sentence" is an instruction sentence to be input into a generative AI model.

[0187] "Means for executing a generative AI model" refers to a function for inputting a prompt sentence into the generative AI model and generating information.

[0188] "Means for adjusting the output obtained from the generative AI model" refers to a function for adjusting the information obtained from the generative AI model based on user preferences.

[0189] A "means for transmitting the conditioned output to a user's terminal" is a function for transmitting the conditioned information to a user's terminal.

[0190] The present invention relates to a generative AI system that provides customized output according to user preferences. A specific embodiment of the system will be described below.

[0191] System Configuration

[0192] Hardware and Software

[0193] The server runs the system using the following hardware and software:

[0194] Hardware: High-performance server computer

[0195] Software: Generative AI models (e.g., GPT-3®), text-to-speech software (e.g., Amazon Polly or Google® Text-to-Speech)

[0196] Program Processing

[0197] Retrieving user settings

[0198] The server retrieves information about the user's predefined preferences for tone of voice, speed, writing style, etc. This information is loaded from the user's profile when they log into the system.

[0199] Example: If a user selects "calm male voice" and "polite speech," the server retrieves these preferences from a database.

[0200] Generate prompt statement

[0201] The server generates prompts based on the user's requests to feed into the generative AI model, which is then tailored to obtain an output that is tailored to the user's preferences.

[0202] Example: If a user requests, "Please explain how the generative AI model works," the server generates a prompt that reads, "Please explain the following in a calm, male voice using polite language: How the generative AI model works."

[0203] Running generative AI models

[0204] The server inputs a prompt sentence using a generative AI model (e.g., GPT-3) and generates an output according to the user's request.

[0205] Example: The server sends the generated prompt text to GPT-3, which generates an explanation of "how the generative AI model works."

[0206] Adjusting the Output

[0207] The server then adjusts the output of the generative AI model based on the user's settings, using voice synthesis software (e.g., Amazon Polly or Google Text-to-Speech) to generate speech at the user's selected tone and speed.

[0208] Example: The server sends the text obtained from GPT-3 to Amazon Polly, which generates a speech in a "calm male voice."

[0209] Send to device

[0210] The server then sends the adjusted output to the user's device, where the user can view the audio and text.

[0211] Example: The server sends the generated audio file to the user's smartphone, allowing the user to play it.

[0212] Adding concrete examples

[0213] Here is a concrete example of what happens when a user selects "exciting female voice" and "casual expression."

[0214] Example prompt: "In an exciting female voice, use casual language to explain: How does a generative AI model work?"

[0215] Example output from a generative AI model: "Hey guys! Today we're going to talk about generative AI models. They're super cool!"

[0216] Speech synthesis example: Using Google Text-to-Speech, convert the above text into an "exciting female voice."

[0217] In this way, the system provides generative AI output tailored to the user's preferences.

[0218] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0219] Step 1: Get user settings

[0220] The server retrieves information such as the user's predefined preferred tone of voice, speed, writing style, etc. When a user logs into the system, the server reads the user's profile information from the database and retrieves the user's preferred settings.

[0221] Input: User login information

[0222] Data processing: Search for user settings information from the database

[0223] Output: User preferences

[0224] Step 2: Generate a prompt

[0225] The server generates a prompt sentence to be input to the generative AI model based on the user's request. The server combines the user's request content and the acquired settings to generate a prompt sentence to be input to the generative AI model.

[0226] Input: User request, user preference settings

[0227] Data processing: Generate prompts by combining the request content and configuration information

[0228] Output: The generated prompt statement

[0229] Step 3: Running the generative AI model

[0230] The server inputs a prompt sentence using a generative AI model (e.g., GPT-3) and generates an output according to the user's request. It sends the generated prompt sentence to the generative AI model and receives the generated text.

[0231] Input: Generated prompt text

[0232] Data calculation: Enter a prompt into the generative AI model to generate text

[0233] Output: The generated text

[0234] Step 4: Adjust the output

[0235] The server then adjusts the output of the generative AI model based on the user's settings, using voice synthesis software (e.g., Amazon Polly or Google Text-to-Speech) to generate speech at the user's selected tone and speed.

[0236] Input: Generated text, user preferences and settings

[0237] Data processing: Input text into speech synthesis software to generate speech

[0238] Output: Adjusted audio file

[0239] Step 5: Send to device

[0240] The server sends the adjusted output to the user's device, and the generated audio file is sent to the user's smartphone so that the user can play it.

[0241] Input: Adjusted audio file

[0242] Data processing: Send audio files to the user's device

[0243] Output: An audio file that can be played on the user's device.

[0244] In this way, the system provides generative AI output tailored to the user's preferences.

[0245] (Application example 2)

[0246] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0247] In conventional content delivery systems, it was difficult to adjust voice tone, speed, and writing style according to individual user preferences. This resulted in problems in which users could not watch or listen to content more comfortably. In particular, there is a demand for audio output that matches the user's preferences for content such as news articles, audiobooks, and podcasts, but it was difficult to achieve this with current technology.

[0248] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0249] In this invention, the server includes a means for inputting various video data, a means including a generation AI for generating voice output based on the video data, a means for adjusting the output of the generation AI according to the user's preferences, a means for allowing the user to set the tone of voice, speed, and style of writing that they prefer, a means for inputting a prompt sentence into the generation AI model, and a means for outputting the generated text by voice, which makes it possible to adjust the tone of voice, speed, and style of writing according to the individual preferences of the user.

[0250] "Video data" refers to digital data that contains visual information, such as videos and images.

[0251] "Generative AI" is a system that uses artificial intelligence technology to generate new output from input data.

[0252] "User Preferences" refers to the individual preferences of each individual User, including the particular tone of voice, speed, writing style, etc.

[0253] "Tone of voice" refers to the pitch and texture of the voice, and includes characteristics such as a calm voice or an exciting voice.

[0254] "Speed" refers to the speed at which audio is played back, usually expressed as the number of words per unit time.

[0255] "Writing style" refers to the way a sentence is written and the characteristics of the language used, and includes polite and casual expressions.

[0256] A "prompt sentence" is text data input into a generative AI and refers to the sentence that forms the basis of the generated output.

[0257] "Voice output" refers to playing back the text generated by the generation AI as audio.

[0258] A system for implementing this invention includes means for receiving various types of video data as input, means including a generation AI that generates audio output based on the video data, means for adjusting the output of the generation AI to suit the user's preferences, means for allowing the user to set the tone, speed, and style of voice expression preferred by the user, means for inputting a prompt sentence into the generation AI model, and means for audio output of the generated text.

[0259] Explanation of system program processing

[0260] The server first receives various video data as input. This video data includes content such as news articles, audiobooks, and podcasts. It then uses a generative AI to generate audio output based on this video data. For example, the generative AI uses the "pipeline" function from the "transformers" library.

[0261] Users can set their preferred voice tone, speed, and writing style through their smartphones or other devices. For this, the "pyttsx3" library is used as the voice engine. The output of the generative AI is adjusted based on the user's settings. Specifically, users can select a "calm male voice" or an "exciting female voice," as well as a writing style such as "polite expression" or "casual expression."

[0262] As a means of inputting prompt sentences into the generative AI model, the text data entered by the user is passed to the generative AI. For example, "News article: The weather is sunny today" is input as a prompt sentence for a news article. Based on this prompt sentence, the generative AI generates sentences that match the user's preferences.

[0263] The generated text is output as a voice using a voice engine. The voice engine reads the generated text aloud, adjusting the tone and speed of the voice based on the user's settings. This allows users to enjoy content in a voice that matches their preferences.

[0264] Examples

[0265] For example, if a user prefers a "calm male voice" and "polite expressions," the following prompt sentence can be input into the generative AI model.

[0266] Prompt: "News article: The weather is sunny today."

[0267] Based on this prompt, the generative AI model generates sentences that match the user's preferences, such as, "It's sunny today and the temperature is very comfortable. I hope you all have a wonderful day."

[0268] In this way, it is possible to adjust the tone, speed and writing style of speech to suit the user's individual preferences.

[0269] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0270] Step 1:

[0271] The server receives various types of video data as input, including content such as news articles, audiobooks, podcasts, etc. The input video data is stored in a database.

[0272] Step 2:

[0273] The server uses a generative AI to generate audio output based on the input video data. Specifically, it inputs a prompt sentence into a generative AI model (e.g., GPT-3) and obtains the generated text. This prompt sentence summarizes the content of the video data.

[0274] Step 3:

[0275] Users can set their preferred voice tone, speed, and writing style through the device. For example, they can use a smartphone application to select a "calm male voice" or an "exciting female voice," or "polite expressions" or "casual expressions."

[0276] Step 4:

[0277] The server adjusts the output of the generation AI based on the user's settings. Specifically, it uses a voice engine (e.g., pyttsx3) to generate voice output that matches the voice tone and speed set by the user.

[0278] Step 5:

[0279] The server inputs a prompt sentence into the generative AI model. For example, it inputs "News article: Today's weather is sunny" as a prompt sentence for a news article. Based on this prompt sentence, the generative AI generates a sentence that matches the user's preferences.

[0280] Step 6:

[0281] The server outputs the generated text using a voice engine. The voice engine reads the generated text aloud, adjusting the tone and speed of the voice based on the user's settings. This allows the user to enjoy the content in a voice that matches their preferences.

[0282] Example 3

[0283] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0284] Conventional live broadcast and subtitle generation systems have difficulty converting the language into the user's native language in real time, which means that users cannot enjoy content in a language that is easy to understand. In addition, they do not adequately adjust to the user's preferences, making it impossible to provide an optimal experience for each individual user.

[0285] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0286] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means for receiving a prompt text input by the user, a means for acquiring information on the user's native language, a means for converting the generated text into the user's native language, and a means for transmitting the converted text to the user's terminal. This allows the user to enjoy live coverage and subtitles in real time in a language that is easy for the user to understand.

[0287] "Video data" is data in digital format that contains visual information such as sports and entertainment.

[0288] "Generative AI" is an artificial intelligence model that generates text or speech based on input data.

[0289] A "prompt" is text that contains instructions or requests that a user enters into a system.

[0290] "User's native language information" is information about the language the user is most comfortable understanding.

[0291] "Text conversion means" means a function or process for converting the generated text into the user's native language.

[0292] A "terminal" is a device used by a user, such as a computer, smartphone, or tablet.

[0293] An "HTTP request" is a communication protocol for a client to request data from a server.

[0294] An "HTTP response" is a communication protocol that enables a server to return data to a client.

[0295] "Translation API" means an application programming interface for automatically translating text into different languages.

[0296] This invention relates to a system that allows users to enjoy live broadcasts and subtitles in real time in a language that is easy for users to understand. The system takes video data of various sports and entertainment as input, generates live broadcasts and subtitles using a generative AI model, and has the ability to adjust them to suit the user's preferences.

[0297] Hardware and Software Used

[0298] Hardware

[0299] Server: A central processing unit for running the generative AI model, obtaining the user’s native language information, and converting the generated text.

[0300] Terminal: A device on which a user enters prompts and displays the converted text. Examples include computers, smartphones, and tablets.

[0301] software

[0302] Generative AI models: For example, using advanced natural language processing models such as GPT-4 (registered trademark).

[0303] Translation API: For example, use the Google Translate API to translate the generated text into the user's native language.

[0304] Data processing and calculation

[0305] 1. Receiving a prompt: The user enters a prompt into the input field of the terminal. For example, the user enters "Please provide a commentary of this game in Japanese."

[0306] 2. Sending the prompt: The terminal sends the entered prompt to the server as an HTTP request.

[0307] 3. Get native language information: The server gets the user's profile information from the database and checks what the user's native language is.

[0308] 4. Applying the generative AI model: The server uses a generative AI model (GPT-4) to generate commentary and subtitle text based on the prompt text.

[0309] 5. Text conversion: The server converts the generated text into the user's native language using a translation API (Google Translate API).

[0310] 6. Sending the converted text: The server sends the converted text to the terminal as an HTTP response.

[0311] 7. Displaying text: The device will display the received converted text on the screen, allowing users to enjoy live commentary and subtitles in Japanese.

[0312] Examples

[0313] For example, suppose a user is watching a soccer game. The user enters a prompt sentence such as "Please provide commentary on this game in Japanese." The device sends this prompt sentence to the server, and the server uses a generative AI model to generate commentary text. The server then converts the generated text into Japanese using a translation API and sends the converted text to the device. Finally, the device displays the converted commentary on the screen, allowing the user to enjoy the commentary in Japanese.

[0314] In this way, the user can enjoy live coverage and subtitles in real time in a language that the user can easily understand. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0315] Step 1:

[0316] The user enters a prompt sentence.

[0317] The user inputs a prompt sentence into the input field of the terminal, such as "Please provide a commentary of this game in Japanese." The input prompt sentence is temporarily stored in the memory of the terminal.

[0318] Step 2:

[0319] The terminal sends a prompt to the server.

[0320] The terminal sends the prompt text entered by the user to the server as an HTTP request. Specifically, a data packet containing the prompt text is sent to the server via the Internet. The input is the prompt text, and the output is the request sent to the server.

[0321] Step 3:

[0322] The server obtains the user's native language information.

[0323] The server identifies the user's ID based on the received prompt sentence, and retrieves the user's profile information from the database. Specifically, it retrieves the user's native language information using an SQL query. The input is the user's ID, and the output is the user's native language information.

[0324] Step 4:

[0325] The server uses a generative AI model to generate text based on the prompt.

[0326] The server uses a generative AI model (e.g., GPT-4) to generate commentary and subtitle text based on the prompt sentence. The generative AI model parses the prompt sentence and generates appropriate text. The input is the prompt sentence and the output is the generated text.

[0327] Step 5:

[0328] The server translates the generated text into the user's native language.

[0329] The server translates the generated text into the user's native language using a translation API (e.g., Google translation API). Specifically, the server sends the generated text to the translation API and receives the translated text. The input is the generated text and the output is the translated text.

[0330] Step 6:

[0331] The server sends the converted text to the terminal.

[0332] The server sends the translated text to the terminal as an HTTP response. Specifically, a data packet containing the translated text is sent to the terminal over the Internet. The input is the translated text, and the output is the response sent to the terminal.

[0333] Step 7:

[0334] The terminal displays the translated text.

[0335] The terminal receives the translated text from the server and displays it on the screen, in particular by rendering the received text in a user interface for visual confirmation by the user. The input is the translated text and the output is the text displayed on the screen.

[0336] (Application example 3)

[0337] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0338] In conventional live streaming and video distribution services, users have had difficulty understanding live coverage and subtitles provided in different languages. In addition, users have had limited access to content translated in real time into their native language, limiting their viewing experience. This has created an information gap between users who speak different languages.

[0339] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0340] In this invention, the server includes a means for receiving video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means for translating the generated live coverage and subtitles into the user's native language in real time, a means for displaying the translated live coverage and subtitles, and a means for providing an interface that allows the user to select a language of their choice. This enables the user to watch live coverage and subtitles in real time in a language that they can easily understand.

[0341] "Various sports and entertainment video data" refers to video information of sporting events and entertainment content, including live streaming and recorded videos.

[0342] "Generative AI" refers to artificial intelligence technology that automatically generates live commentary and subtitles based on input video data.

[0343] "Means for tailoring to user preferences" refers to the ability to customize the content and format of the generated commentary and / or subtitles to a user's individual preferences.

[0344] "Means of translating into the user's native language in real time" refers to the function of instantly translating the generated live broadcast and subtitles into a language specified by the user.

[0345] "Means for displaying translated commentary and / or subtitles" refers to the functionality for visually displaying the translated content on a User's device.

[0346] "An interface that allows users to select their preferred language" refers to a function that provides an operation screen or menu that allows users to select their desired language.

[0347] The system for implementing this invention is configured as follows. First, the server receives video data of various sports and entertainment as input. Next, the server uses a generation AI to generate live coverage and subtitles based on the input video data. This generation AI can adjust the voice and writing style to suit the user's preferences.

[0348] The generated live commentary and subtitles are translated into the user's native language in real time. Translation services such as Google Translate API are used for this translation process. The translated live commentary and subtitles are displayed on the user's device. The user can visually check the translated content using a device such as a smartphone or tablet.

[0349] Additionally, users can specify their preferred language through a language selection interface, which is designed to be easy for users to navigate and provides language selection menus and settings screens.

[0350] As a specific example, consider the case where a user is watching a live stream in English. In this case, the server receives the English video data and uses generation AI to generate English commentary and subtitles. The generated English commentary and subtitles are then translated into Japanese in real time and displayed on the user's smartphone. The user can watch the translated Japanese commentary and subtitles by selecting Japanese through the interface.

[0351] An example of a prompt is "Please translate the English live streaming subtitles into Japanese in real time and display them." By using this prompt, users can enjoy live streaming in a language they can easily understand.

[0352] This system will eliminate the information gap between users who speak different languages, allowing users to watch live broadcasts and subtitles in their own native language in real time.

[0353] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0354] Step 1:

[0355] The server receives video data of various sports and entertainment as input. The input video data is provided in the form of live streaming, recorded video, etc. The server analyzes the format of the data and converts it into an appropriate format for processing.

[0356] Step 2:

[0357] The server uses a generation AI to generate live commentary and subtitles based on the input video data. The generation AI extracts audio and text information from the video data, and creates live commentary and subtitles in real time based on that information. Voice recognition and natural language processing technologies are used in this process. The generated live commentary and subtitles are output as text data.

[0358] Step 3:

[0359] The server adjusts the generated live broadcast and subtitles to suit the user's preferences. Specifically, it customizes the tone of voice and sentence expression based on the user's settings. This adjustment is achieved by changing the parameters of the generation AI. After adjustment, the live broadcast and subtitles are output again as text data.

[0360] Step 4:

[0361] The server translates the adjusted live broadcast and subtitles into the user's native language in real time. This translation process uses a translation service such as Google Translate API. The server sends the adjusted text data to the translation API and receives the translated text data. The translated live broadcast and subtitles are output in the language specified by the user.

[0362] Step 5:

[0363] The server sends the translated live broadcast and subtitles to the user's device. The user's device displays the received text data on the screen. The display is done on a device such as a smartphone or tablet. The user can visually check the translated content.

[0364] Step 6:

[0365] Users specify their preferred language through an interface that allows them to select their preferred language. The interface is designed to be easy for users to operate, and provides a language selection menu and a settings screen. The user's selection is sent to the server and reflected in the next translation process.

[0366] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0367] "Example 1"

[0368] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes the user's emotions and adjusts the output of the generation AI based on those emotions. Specifically, the system estimates the user's emotions from facial expressions, tone of voice, eye movements, etc. while watching a video. Then, the system adjusts the output of the generation AI according to those emotions. For example, if the user shows an emotion of joy, the generation AI generates a more exciting live broadcast and subtitles. Conversely, if the user shows an emotion of confusion or discomfort, the generation AI generates a more understandable and polite live broadcast and subtitles.

[0369] "Example 2"

[0370] The emotion engine also adjusts the voice based on the user's emotions. Specifically, if the user is happy, the AI ​​will generate commentary and subtitles in a brighter, more cheerful voice. Conversely, if the user is sad, the AI ​​will generate commentary and subtitles in a calmer voice. This enables more natural communication that matches the user's emotions.

[0371] "Example 3"

[0372] In addition, the emotion engine adjusts the text expression based on the user's emotion. Specifically, if the user expresses surprise, the generation AI generates commentary and subtitles with more detailed explanations. Conversely, if the user expresses relief, the generation AI generates simpler commentary and subtitles. This makes it possible to provide optimal information according to the user's emotion.

[0373] The process flow of each embodiment will be described below.

[0374] "Example 1"

[0375] Step 1: The user watches the video.

[0376] Step 2: The emotion engine infers emotions from the user's facial expressions, tone of voice, eye movements, etc.

[0377] Step 3: Based on the estimated emotions, the generation AI adjusts the output and generates commentary and subtitles.

[0378] "Example 2"

[0379] Step 1: The user watches the video.

[0380] Step 2: The emotion engine estimates the user's emotion.

[0381] Step 3: Based on the estimated emotions, the generation AI adjusts the voice and generates commentary and subtitles.

[0382] "Example 3"

[0383] Step 1: The user watches the video.

[0384] Step 2: The emotion engine estimates the user's emotion.

[0385] Step 3: Based on the estimated emotions, the generation AI adjusts the sentence expression and generates live commentary and subtitles.

[0386] Example 1

[0387] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0388] Conventional live broadcast and subtitle generation systems do not adjust according to the user's emotions, and there is a problem that the user experience is not sufficiently improved. In particular, there is a demand for real-time adjustment according to the user's emotions while watching the video.

[0389] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for inputting various video data, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for collecting user emotion data, and a means for adjusting the output of the generation AI based on the emotion data. This enables the generation of real-time live coverage and subtitles according to the user's emotions.

[0390] "Video data" is data in digital format that contains visual information such as sports and entertainment.

[0391] "Generative AI" is artificial intelligence that uses deep learning technology to automatically generate live commentary and subtitles from video data.

[0392] "User emotion data" refers to information about emotions obtained from the user's facial expressions, tone of voice, eye movements, etc.

[0393] An "emotion engine" is a system that includes machine learning algorithms for analyzing a user's emotional data and inferring their emotions.

[0394] "Means for adjusting output" refers to a function that changes the output content and tone of the generation AI based on feedback from the emotion engine.

[0395] "Live coverage" refers to information conveying the progress of an event taking place in real time through audio and text.

[0396] "Subtitles" are text information added to video data, providing supplementary explanations and dialogue to the viewer.

[0397] The present invention relates to a system that takes video data of sports or entertainment as input and generates live coverage and subtitles according to the user's emotions. This system is composed of three main elements: a server, a terminal, and a user.

[0398] The server first receives various types of video data via the Internet. This video data is in digital format containing visual information such as sports matches and entertainment events. The server inputs the received video data into the generative AI. The generative AI uses deep learning frameworks such as TensorFlow and PyTorch to analyze the video data and identify important events (e.g., goal scenes, goal-scoring scenes, interviews, etc.).

[0399] While the user is watching the video, the device uses the built-in camera and microphone to collect the user's facial expressions and tone of voice. This is done using face recognition technology and voice analysis technology. For example, if the user smiles, the facial expression data is captured. The collected emotion data is sent from the device to the emotion engine. The emotion engine uses libraries such as OpenCV and Dlib to perform face recognition and estimate the user's emotion. For example, if the user is smiling, it is determined to be "joy."

[0400] The server receives feedback from the emotion engine and adjusts the output of the generation AI. Specifically, it changes the tone and content of the play-by-play and subtitles depending on the user's emotions. For example, if the user is happy, the generation AI generates an exciting play-by-play such as, "Great goal! The crowd is excited!" Conversely, if the user is confused, it provides an easy-to-understand explanation such as, "An offside was called. This is because the attacking player got ahead of the opposing defender."

[0401] The device then displays the live broadcast and subtitles to the user in real time, allowing the user to enjoy content that best suits their emotions.

[0402] As a concrete example, consider a user watching a soccer game. If the user shows a look of joy at a goal, the AI ​​will generate an exciting live broadcast such as, "What a great goal! The crowd is so excited!" On the other hand, if the user is confused by an offside call, the AI ​​will provide a clear explanation such as, "Offside was called. This is because the attacking player got ahead of the opposing defender."

[0403] An example prompt would be:

[0404] "Using video data of a soccer match as input, generate live commentary and subtitles according to the user's emotions. If the user is happy, provide an exciting commentary, and if the user is confused, provide an easy-to-understand explanation."

[0405] In this way, the present invention can provide optimal live coverage and subtitles according to the user's emotions.

[0406] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0407] Step 1: Receiving video data

[0408] The server receives various video data via the Internet. This video data is in digital format containing visual information such as sports games and entertainment events. The input is video data, and the output is video data ready to be input to the generative AI.

[0409] Step 2: Analyze the video data

[0410] The server inputs the received video data into the generation AI. The generation AI uses deep learning frameworks such as TensorFlow and PyTorch to analyze the video data and identify important events (e.g., goal scenes, scoring scenes, interviews, etc.). The input is the video data, and the output is the identified important event information.

[0411] Step 3: Collect user emotion data

[0412] The device uses a built-in camera and microphone to collect the user's facial expressions and tone of voice while the user is watching a video. This is done using facial recognition and voice analysis technologies. The input is the video and audio data the user is watching, and the output is the collected emotion data.

[0413] Step 4: Analyze the sentiment data

[0414] The terminal sends the collected emotion data to the emotion engine. The emotion engine uses libraries such as OpenCV and Dlib to perform face recognition and estimate the user's emotion. The input is the collected emotion data, and the output is the estimated user's emotion information.

[0415] Step 5: Adjusting the output of the generation AI

[0416] The server receives feedback from the emotion engine and adjusts the output of the generative AI. Specifically, it changes the tone and content of the live broadcast and subtitles according to the user's emotions. The input is the estimated user's emotion information, and the output is the adjusted content of the live broadcast and subtitles.

[0417] Step 6: Generate and display live commentary and subtitles

[0418] The server generates live coverage and subtitles based on the adjusted output of the generation AI. The terminal displays the generated live coverage and subtitles to the user in real time. The input is the adjusted live coverage and subtitle content, and the output is the live coverage and subtitles displayed to the user.

[0419] In this way, the system can provide optimal live coverage and subtitles according to the user's emotions.

[0420] (Application example 1)

[0421] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0422] Conventional sports and entertainment video distribution systems have the problem that they are unable to provide live coverage or subtitles that respond to the user's emotions, resulting in a uniform viewing experience. In addition, there are limited adjustments to suit user preferences, making it difficult to provide optimal content for each individual viewer.

[0423] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0424] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means including an emotion engine for recognizing the user's emotions and adjusting the output of the generation AI based on the emotions, a means for inputting the user's facial expressions and tone of voice using a smartphone camera and microphone, and a means for the generation AI to generate live coverage and subtitles in real time according to the user's emotions. This makes it possible to provide live coverage and subtitles that are individually optimized according to the user's emotions.

[0425] "Various sports and entertainment video data" refers to video information of sporting events and entertainment content, including live and recorded footage.

[0426] "Generative AI" refers to a system that uses artificial intelligence technologies such as deep learning to automatically generate live commentary and subtitles from input video data.

[0427] "Means of adjusting to user preferences" refers to the function of customizing the output of the generative AI based on the user's viewing history and settings information.

[0428] An "emotion engine" is a system that recognizes emotions from the user's facial expressions, tone of voice, etc., and adjusts the output of the generative AI based on those emotions.

[0429] "Means of inputting a user's facial expressions and tone of voice using a smartphone's camera and microphone" refers to a function that uses a smartphone's built-in camera and microphone to capture a user's facial expressions and tone of voice in real time.

[0430] "Means of generation in real time" refers to the ability to instantly generate live commentary and subtitles based on user emotions and the context of video data, and provide them without delay.

[0431] The system for implementing this invention is configured as follows. First, the server receives video data of various sports and entertainment as input. Next, the server uses a generation AI to automatically generate live coverage and subtitles based on the input video data. This generation AI utilizes artificial intelligence technology such as deep learning.

[0432] In addition, the server is equipped with an emotion engine that recognizes the user's emotions. This emotion engine captures the user's facial expressions and tone of voice in real time using the smartphone's camera and microphone, and analyzes them. Based on the analysis results, the output of the generation AI is adjusted according to the user's emotions.

[0433] Specifically, if the user expresses joy, the AI ​​will generate more exciting commentary and subtitles. On the other hand, if the user expresses confusion or displeasure, the AI ​​will generate more understandable and detailed commentary and subtitles.

[0434] The hardware required is a smartphone camera and microphone, the software uses OpenCV to process camera input, DeepFace to recognize emotions, and the Transformers library to implement a generative AI model.

[0435] As a concrete example, consider a case where a user is watching a soccer game and expressing joy. In this case, the following prompt sentence is input to the generation AI.

[0436] Example prompt:

[0437] If the user indicates the emotion of joy, generate a play-by-play based on the following context: "The moment when team A scores a goal in a soccer game."

[0438] Based on this prompt, the generative AI will generate an exciting live broadcast and provide it to the user, allowing the user to have a more emotionally relevant viewing experience.

[0439] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0440] Step 1:

[0441] The server receives video data from various sports and entertainment sources as input. The video data can be real-time or pre-recorded. The server analyzes the data and extracts the necessary information. Specifically, it analyzes frames of the video data to detect important events (e.g., goals scored or highlights).

[0442] Step 2:

[0443] The server uses a generative AI to automatically generate commentary and subtitles based on the input video data. The generative AI uses a deep learning model to generate text from video data. The input includes frames of video data and their analysis results. The output is the generated commentary and subtitle text.

[0444] Step 3:

[0445] The server captures the user's facial expressions and tone of voice in real time using the smartphone's camera and microphone. The device sends input data from the camera and microphone to the server. The input includes video of the user's face and audio data. The output is sent to the server.

[0446] Step 4:

[0447] The server uses an emotion engine to recognize emotions from the user's facial expressions and tone of voice. The emotion engine analyzes input video and audio data using emotion recognition models such as DeepFace. Inputs include video of the user's face and audio data. Outputs include the user's emotions (e.g., happy, confused, displeased).

[0448] Step 5:

[0449] The server adjusts the output of the generative AI based on the recognized user emotions. Specifically, if the user expresses joy, the generative AI generates a more exciting commentary and subtitles. Conversely, if the user expresses confusion or displeasure, the generative AI generates a more understandable and polite commentary and subtitles. The input includes the user's emotions and the context of the video data. The output is the adjusted commentary and subtitle text.

[0450] Step 6:

[0451] The server provides the generated commentary and subtitles to the user in real time. The device displays the commentary and subtitles sent from the server. The input includes the adjusted commentary and subtitle text. The output is the commentary and subtitles displayed on the user's smartphone screen.

[0452] This allows users to have a viewing experience that is individually optimized to fit their emotions.

[0453] Example 2

[0454] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0455] Conventional generative AI systems have limited ability to adjust output to suit user preferences, making it difficult to communicate naturally according to the user's emotions. In addition, it is difficult to generate voice and written expressions that reflect the user's emotions, and there is a demand for improved user experience.

[0456] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0457] In this invention, the server includes a means for inputting various video data, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI according to the user's preferences, a means for detecting the user's emotions, and a means for adjusting the output of the generation AI based on the user's emotions. This makes it possible to generate natural and appropriate voice and text expressions according to the user's preferences and emotions.

[0458] "Various types of video data" refers to various types of video information such as sports and entertainment.

[0459] "Generative AI" refers to artificial intelligence technology that generates text or speech based on input data.

[0460] "User preferences" refers to individual preferences, such as tone of voice, speed, and writing style, set by the User.

[0461] "Means of adjustment" refers to functions and methods for changing the output of the generative AI to suit the user's preferences and emotions.

[0462] "User emotion" refers to an emotional state, such as happiness, sadness, excitement, etc., exhibited by a user.

[0463] "Means for emotion detection" refers to technologies or methods for analyzing and identifying emotions from user input data.

[0464] A "prompt" is an instruction given to a generation AI to generate a specific output.

[0465] "Speech synthesis software" refers to software technology for converting text data into speech data.

[0466] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0467] The present invention relates to a system for adjusting the output of a generation AI according to the preferences and emotions of a user. A specific embodiment of this system will be described below.

[0468] Hardware and Software Used

[0469] Hardware: Servers, terminals (PCs, smartphones, tablets, etc.)

[0470] Software: Generative AI models (e.g. GPT-4), emotion engines, text-to-speech software (e.g. Amazon Polly, Google Text-to-Speech)

[0471] System Configuration

[0472] 1. Getting user settings

[0473] A user logs into the system using a terminal.

[0474] On the settings screen, users can select their preferred voice tone (e.g., calm male voice, exciting female voice), speed, and writing style (e.g., polite expression, casual expression).

[0475] The terminal transmits the user's setting information to the server.

[0476] 2. Emotion Detection

[0477] The user inputs text or voice data and transmits it to the server via the terminal.

[0478] The server passes the received data to the emotion engine and analyzes the user's emotions.

[0479] Specifically, the emotion engine uses text analysis algorithms to identify emotions from user input, for example detecting the emotion "joy" from the text "Today was such a fun day!"

[0480] 3. Prompt generation for generative AI models

[0481] Based on the user's settings and the analysis results of the emotion engine, the server generates a prompt sentence to be input into the generative AI model.

[0482] As a specific operation, the server creates a prompt sentence saying, "If the user is expressing an emotion of joy, please generate text in a bright, cheerful voice and with casual expressions."

[0483] 4. Text Generation

[0484] The server inputs the prompt text into a generative AI model (e.g., GPT-4) and generates text that matches the user's preferences and emotions.

[0485] Specifically, the generative AI model generates the text "Today was absolutely amazing!" based on the prompt sentence.

[0486] 5. Speech synthesis

[0487] The server inputs the generated text into speech synthesis software, which generates speech in a tone and speed that matches the user's settings.

[0488] Specifically, the speech synthesis software (e.g., Amazon Polly) generates voice data in accordance with the instructions, "Generate an exciting female voice with casual expressions."

[0489] 6. Providing Output

[0490] The server transmits the generated voice data to the terminal.

[0491] The terminal reproduces the received voice data and provides it to the user.

[0492] In concrete terms, the user can play back the audio generated on the terminal and enjoy audio output tailored to his or her preferences.

[0493] Examples

[0494] Examples of prompt statements

[0495] "If the user is expressing the emotion of happiness, generate text in a cheerful, upbeat voice and with casual expressions."

[0496] "If the user indicates sadness, generate text in a calm, polite voice."

[0497] Real-world examples

[0498] A user logs into the system and selects "exciting female voice" and "casual expression."

[0499] The user types, "Today was such a fun day!"

[0500] The server's emotion engine detects "joy" and inputs the prompt sentence "Generate text in a bright, cheerful voice and casual expressions" into the generative AI model.

[0501] The server's generative AI model generates the text "Today was amazing!", and the speech synthesis software generates the audio in an exciting female voice.

[0502] The user can play back and enjoy the audio generated on the terminal.

[0503] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0504] Step 1:

[0505] A user logs into the system using a terminal. After logging in, the user selects the tone of voice, speed, and writing style he or she prefers on the settings screen. This setting information is sent from the terminal to the server.

[0506] Input: User login information, tone of voice, speed, writing style

[0507] Output: User configuration information sent to the server.

[0508] Step 2:

[0509] Users input text or voice data and send it to the server via their device. The server passes the received data to the emotion engine, which analyzes the user's emotions. The emotion engine uses text analysis algorithms to identify emotions from the user's input.

[0510] Input: User text or voice data

[0511] Output: Sentiment analysis results by the emotion engine (e.g., happy, sad)

[0512] Step 3:

[0513] The server generates a prompt sentence to be input to the generative AI model based on the user's settings and the analysis results of the emotion engine. Specifically, the server creates a prompt sentence that reads, "If the user is expressing joy, please generate text in a bright, cheerful voice and with casual expressions."

[0514] Input: User settings information, sentiment analysis results

[0515] Output: Prompt sentence to be input to the generative AI model

[0516] Step 4:

[0517] The server inputs the prompt sentence into a generative AI model (e.g. GPT-4) and generates text that matches the user's settings and emotions. The generative AI model generates text based on the prompt sentence.

[0518] Input: prompt statement

[0519] Output: The generated text (e.g. "Today was absolutely amazing!")

[0520] Step 5:

[0521] The server inputs the generated text into the speech synthesis software, which generates speech at a tone and speed that matches the user's settings. The speech synthesis software generates speech data according to the instruction, "Please generate a voice with an exciting female voice and casual expressions."

[0522] Input: Generated text, user settings information

[0523] Output: The generated audio data

[0524] Step 6:

[0525] The server transmits the generated voice data to the terminal. The terminal plays back the received voice data and provides it to the user. The user can play back the generated voice on the terminal and enjoy voice output tailored to their preferences.

[0526] Input: Generated audio data

[0527] Output: Audio played on the device

[0528] (Application example 2)

[0529] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0530] Conventional generative AI systems have limited ability to adjust voice and written expressions to suit user preferences, making it difficult to achieve natural communication based on the user's emotions. In addition, they cannot fine-tune voice properties based on user settings, making it difficult to provide an optimal experience for each individual user.

[0531] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0532] In this invention, the server includes a means for inputting various video data, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means for adjusting the voice based on the user's emotions, a means for setting voice properties based on the user's settings, and a means for recognizing the user's emotions using an emotion recognition engine. This enables natural and personalized voice output according to the user's preferences and emotions.

[0533] "Various types of video data" refers to video information in various fields such as sports, entertainment, and education.

[0534] "Generative AI" refers to artificial intelligence technology that automatically generates voice or text based on input data.

[0535] "User Preferences" refers to the individual settings, such as tone of voice, speed, writing style, etc., selected by the User.

[0536] An "emotion recognition engine" refers to technology that analyzes and recognizes emotions from a user's facial expressions, tone of voice, text, etc.

[0537] "Speech properties" refers to characteristics of speech such as tone, speed, and pitch.

[0538] "Live coverage and subtitles" refers to audio commentary and text display generated in real time for video data.

[0539] "User emotion" refers to an emotional state, such as happiness, sadness, anger, etc., expressed by a user.

[0540] "Natural communication" refers to seamless dialogue using voice and text that is tailored to the user's emotions and preferences.

[0541] The system for carrying out this invention is configured as follows: The server includes means for receiving various video data as input, means including a generation AI for generating live coverage and subtitles based on the video data, means for adjusting the output of the generation AI to suit the user's preferences, means for adjusting the voice based on the user's emotions, means for setting voice properties based on the user's settings, and means for recognizing the user's emotions using an emotion recognition engine.

[0542] Program processing explanation

[0543] The server first receives various video data as input. This video data includes video information from various fields such as sports, entertainment, and education. Next, it uses generative AI to generate live coverage and subtitles based on the input video data. This generative AI is an artificial intelligence technology that automatically generates audio and text based on the input data.

[0544] The user's device adjusts the output of the generative AI to suit the user's preferences. Specifically, it sets voice properties based on individual settings selected by the user, such as tone of voice, speed, and writing style. It also recognizes the user's emotions using an emotion recognition engine and adjusts the voice based on the user's emotions. This emotion recognition engine is a technology that analyzes and recognizes emotions from the user's facial expressions, tone of voice, text, etc.

[0545] Examples

[0546] For example, if a user selects a "calm male voice" with "polite expression" and the emotion is "joy," the following prompt sentence is input into the generative AI model.

[0547] Example prompt:

[0548] User Settings:

[0549] Tone of voice: Calm male voice

[0550] Speed: 150

[0551] Writing style: Polite

[0552] Emotion: Joy

[0553] Text:This is a sample audiobook text.

[0554] These prompts are then fed into a generative AI model that can generate voice output tailored to the user's preferences and emotions, providing a natural and personalized voice experience.

[0555] The hardware used includes servers and user devices (smartphones, tablets, PCs, etc.). The software used includes generative AI models, emotion recognition engines, and voice property setting modules. Specific software examples include GPT-3 for generative AI models, EmotionRecognizer for emotion recognition engines, and pyttsx3 for voice property setting.

[0556] The flow of the identification process in the application example 2 will be described with reference to FIG.

[0557] Step 1:

[0558] The server receives various video data as input. This video data includes video information from various fields such as sports, entertainment, and education. The input video data is converted into an appropriate format within the server and passed to the generative AI model.

[0559] Step 2:

[0560] The server uses a generative AI model to generate live commentary and subtitles based on the input video data. The generative AI model extracts important information from the video data and generates audio and text based on that information. The generated audio and text are then sent to the user's device.

[0561] Step 3:

[0562] The user sets his / her preferences through the terminal. Specifically, the user selects the tone of voice, speed, writing style, etc. This setting information is sent from the terminal to the server.

[0563] Step 4:

[0564] The server sets the voice properties based on the user's settings, specifically, by using a voice property setting module such as pyttsx3 to adjust the tone and speed of the generated voice, and the adjusted voice is sent to the user's device.

[0565] Step 5:

[0566] The user expresses his / her emotions through the terminal. The emotions are recognized from the user's facial expressions, tone of voice, text, etc. The terminal analyzes the user's emotions using an emotion recognition engine and transmits the results to the server.

[0567] Step 6:

[0568] The server adjusts the generated voice based on the emotion information received from the emotion recognition engine. Specifically, if the user is expressing a happy emotion, the tone of the voice is made brighter and the speed is made faster. Conversely, if the user is expressing a sad emotion, the tone of the voice is made calmer and the speed is made slower. This adjusted voice is sent to the user's terminal.

[0569] Step 7:

[0570] Users receive tailored audio through their devices, providing a natural and personalized audio experience, with audio output tailored to the user's preferences and emotions, resulting in a more satisfying experience.

[0571] Example 3

[0572] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0573] Conventional live broadcast and subtitle generation systems have difficulty in flexibly adjusting to the user's preferences and emotions, and they are insufficient in providing information in a language that is easy for the user to understand. In addition, since information is not provided based on the user's emotions, the user experience is limited. As a result, it is not possible to provide the optimal information that users desire.

[0574] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0575] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means including a generative AI model for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generative AI model to suit the user's preferences, a means for translating into the user's native language, a means for detecting the user's emotions, and a means for adjusting the text expression based on the user's emotions. This makes it possible to provide optimal information according to the user's preferences and emotions.

[0576] "Various sports and entertainment video data" refers to digital data that includes visual information of sporting events and entertainment content.

[0577] A "generative AI model" is an artificial intelligence algorithm that automatically generates text or speech based on input data.

[0578] "Means for tailoring to user preferences" refers to functionality that allows users to customize the generated content based on their individual tastes and requirements.

[0579] A "means of translation into the user's native language" is a feature that translates generated content into a language the user is most comfortable with.

[0580] "Means for detecting user emotions" refers to technology that analyzes the user's facial expressions, tone of voice, etc. to identify their emotional state at that time.

[0581] The "means for adjusting the expression of text based on the user's emotions" is a function for changing the expression method of generated content according to the detected emotions of the user.

[0582] This invention is a system that generates live coverage and subtitles according to the user's preferences and emotions. The system operates in cooperation with a server, a terminal, and a user.

[0583] Hardware and Software Used

[0584] Hardware: Servers, terminals (PCs, smartphones, tablets, etc.)

[0585] Software: generative AI models (e.g. GPT-4), emotion engines, translation software (e.g. Google Translate API)

[0586] System Operation Overview

[0587] 1. Receiving user input

[0588] A user uses the device to request live commentary or subtitle generation by entering text into the device's input field and pressing the send button.

[0589] Example: A user types, "Please give me a live broadcast of the soccer game in Japanese."

[0590] 2. Content generation using generative AI models

[0591] The server receives the user's request and sends a prompt to a generative AI model (e.g. GPT-4), which generates appropriate content based on the prompt.

[0592] Example: The server sends GPT-4 the prompt "Generate a live broadcast of a soccer match," and GPT-4 generates the sentence "The soccer match has started. The players are on the field, and the whistle has blown."

[0593] 3. Translation tailored to user preferences

[0594] The server translates the generated content into the user's native language. Translation software (e.g. Google Translate API) is used to translate the generated English text into Japanese.

[0595] Example: The server translates "The soccer match has started. The players are on the field, and the whistle has blown." to "The soccer match has started. The players are on the field, and the whistle has blown."

[0596] 4. Detect user emotions

[0597] To detect the user's emotions, the device uses the camera and microphone to analyze the user's facial expressions and tone of voice. The emotion engine analyzes this data to identify the user's emotions.

[0598] Example: The device captures a user's facial expressions with its camera, and the emotion engine detects that the user has a surprised expression.

[0599] 5. Adjusting sentence expression based on emotions

[0600] The server receives the results of the emotion engine and adjusts the generated content based on the user's emotion, for example, making the content more detailed if the user expresses surprise, and simpler if the user expresses relief.

[0601] Example: The server adjusts "The soccer game has started. The players are on the field and the whistle has blown to start the game." to "A player scores a goal! The crowd is cheering!"

[0602] 6. Delivery of Final Content

[0603] The server sends the tailored content to the user's device, where it is viewed by the user. The device displays the received content, and the user confirms it.

[0604] Example: A detailed live broadcast in Japanese is displayed on the user's device.

[0605] Examples of concrete examples and prompts

[0606] Example: A user requests a live soccer game broadcast in Japanese and expresses surprise.

[0607] User input: "Please provide live commentary of soccer games in Japanese."

[0608] Output of the generative AI model: "The soccer match has started. The players are on the field, and the whistle has blown."

[0609] Translation software translates: "The soccer game has begun. The players are on the field and the whistle has blown to start the game."

[0610] Emotion engine tuning: "The player scores a goal! The crowd is cheering!"

[0611] Example prompt:

[0612] "Generate a live broadcast of a soccer game in Japanese. The user is showing surprise emotion."

[0613] In this way, the system can provide optimal live coverage and subtitles according to the user's preferences and feelings. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0614] Step 1:

[0615] A user uses a terminal to request live broadcasting and subtitle generation. Specifically, the user enters text into the input field of the terminal and presses the send button. For example, the user enters "Please broadcast the live soccer game in Japanese." The terminal then sends the user's request to the server.

[0616] Step 2:

[0617] The server receives the user's request and sends a prompt to the generative AI model. The input contains the user's request. The server sends the prompt "Please generate a live broadcast of a soccer match" to the generative AI model. The generative AI model generates appropriate content based on the prompt, and generates the sentence "The soccer match has started. The players are on the field, and the whistle has blown." as the output.

[0618] Step 3:

[0619] The server translates the generated content into the user's native language. The input includes an English sentence that is the output from the generative AI model. The server uses translation software (e.g. Google Translate API) to translate "The soccer match has started. The players are on the field, and the whistle has blown." to "The soccer match has started. The players are on the field, and the whistle has blown.". The output is the translated Japanese sentence.

[0620] Step 4:

[0621] To detect the user's emotions, the device uses a camera and microphone to analyze the user's facial expressions and tone of voice. Inputs include real-time video and audio data of the user. The emotion engine analyzes these data to identify the user's emotions. The output is that the user is showing an emotion of surprise.

[0622] Step 5:

[0623] The server receives the results of the emotion engine and adjusts the generated content based on the user's emotions. The input includes the translated Japanese text and the emotion engine results. The server adjusts the text to include more details if the user indicates surprise, and to a simpler text if the user indicates relief. The output is the adjusted text: "The player scored a goal! The crowd is cheering!"

[0624] Step 6:

[0625] The server sends the adapted content to the user's device, where the user views it. The input includes adapted Japanese text. The device displays the received content, which the user confirms. The output is a detailed live report in Japanese on the user's device.

[0626] (Application example 3)

[0627] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0628] Conventional live broadcasting and subtitle generation systems do not support the user's native language, which can make it difficult for users to understand. In addition, they are unable to provide information according to the user's emotions, resulting in a suboptimal user experience. This can result in a decrease in the user's understanding of and satisfaction with the content they are watching.

[0629] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0630] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means for converting into the user's native language, and a means for adjusting the text expression based on the user's emotions. This allows the user to receive live coverage and subtitles in real time that are easy to understand in his or her native language, and further allows the provision of optimal information according to the user's emotions.

[0631] "Video data" is data in digital format that contains visual information such as sports and entertainment.

[0632] "Generative AI" is an artificial intelligence technology that automatically generates live commentary and subtitles based on input data.

[0633] "User Preferences" refers to the individual tastes and settings that a User has regarding the Content they view.

[0634] "Native language" refers to the language that a user understands most naturally.

[0635] "Emotion" refers to the psychological response or state a user exhibits while watching.

[0636] "Writing" refers to the content and style of the generated text.

[0637] A system for implementing the present invention is configured as follows.

[0638] System Configuration

[0639] Hardware

[0640] Servers: Servers with powerful computing power are required, including GPUs and CPUs to run generative AI models.

[0641] Devices: Includes mobile devices such as smartphones and tablets used by users.

[0642] software

[0643] Generative AI models: For example, using advanced natural language generation models such as GPT-4.

[0644] Emotion recognition engine: As an example, use the Microsoft® Azure® Emotion API.

[0645] Speech Recognition API: As an example, we will use the Google Speech-to-Text API.

[0646] Translation API: As an example, we will use the Google Translate API.

[0647] Data processing and calculation

[0648] Server Processing

[0649] 1. Getting voice input:

[0650] The server receives the audio of the video data sent from the device in real time and converts the audio into text using a speech recognition API.

[0651] 2. Emotion recognition:

[0652] The server uses an emotion recognition engine to analyze the user's facial expressions and voice data sent from the device and identify the user's emotions.

[0653] 3. Text Generation:

[0654] The server uses a generative AI model to generate commentary and subtitles based on the acquired text, and adjusts the level of detail and expression of the generated text based on the results of emotion recognition.

[0655] 4. Translation:

[0656] The server uses a translation API to translate the generated text into the user's native language.

[0657] Terminal handling

[0658] 1.Display:

[0659] The terminal displays the translated text sent from the server on the screen in real time.

[0660] Examples

[0661] If the user is Japanese and is watching an English live stream, the system will work as follows:

[0662] 1. Capture the audio of the live stream and convert it to text using a speech recognition API.

[0663] 2. The emotion recognition engine detects the emotion of "surprise" from the user's facial expressions and voice.

[0664] 3. Generative AI models generate detailed commentary and subtitles.

[0665] 4. The translation API translates the generated text into Japanese.

[0666] 5. The translated Japanese text will appear on your smartphone screen.

[0667] Examples of prompt statements

[0668] If the user expresses surprise emotion, generate a detailed play-by-play. Translate the following English text into Japanese and generate a play-by-play with a detailed explanation:

[0669] In this way, users can receive the most relevant information in real time, in their native language, and according to their emotions.

[0670] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0671] Step 1:

[0672] Acquiring voice input

[0673] The server captures the audio of the video data sent from the device in real time. Specifically, it uses the device's microphone to capture the audio of the live stream and sends the audio data to the server. The input is the audio data, and the output is the audio data passed to the speech recognition API.

[0674] Step 2:

[0675] Speech to text conversion

[0676] The server uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the acquired voice data into text. The input is voice data, and the output is text data. Specifically, the server sends voice data to the speech recognition API and receives the returned text data.

[0677] Step 3:

[0678] emotion recognition

[0679] The server analyzes the user's facial and voice data sent from the device using an emotion recognition engine (e.g. Microsoft Azure Emotion API) to identify the user's emotions. The input is the user's facial and voice data, and the output is emotion data. Specifically, the server sends data to the emotion recognition engine and receives the returned emotion data.

[0680] Step 4:

[0681] Text Generation

[0682] The server uses a generative AI model (e.g., GPT-4) to generate live commentary and subtitles based on the acquired text data. Based on the results of emotion recognition, it adjusts the level of detail and expression of the generated text. The input is text data and emotion data, and the output is the generated live commentary and subtitle text data. Specifically, it sends a prompt sentence to the generative AI model and receives the returned text data.

[0683] Step 5:

[0684] translation

[0685] The server uses a translation API (e.g., Google Translate API) to translate the generated text data into the user's native language. The input is the generated text data, and the output is the translated text data. Specifically, the server sends text data to the translation API and receives the translated text in return.

[0686] Step 6:

[0687] The display terminal displays the translated text data sent from the server on the screen in real time. The input is the translated text data, and the output is the text displayed on the terminal screen. Specifically, a UI component is used to display the text data on the terminal display.

[0688] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0689] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0690] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0691] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0692] [Second embodiment]

[0693] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0694] .

[0695] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0696] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0697] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0698] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0699] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0700] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0701] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0702] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0703] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0704] In the smart glasses 214, the processor 46 performs the reception and output process.

[0705] A reception output program 60 is stored in the storage 50. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0706] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0707] "Example 1"

[0708] One embodiment of the present invention is a system that inputs video data of various sports and entertainment. This system includes a generation AI that generates live broadcasts and subtitles based on the video data. The generation AI automatically generates live broadcasts and subtitles from the video data using AI technology such as deep learning.

[0709] "Example 2"

[0710] Furthermore, it includes a means for adjusting the output of the generating AI to suit the user's preferences. Specifically, it allows the user to set the tone of voice, speed, writing style, etc. that they prefer. For example, the user can select a "calm male voice" or an "exciting female voice," as well as a writing style such as "polite expression" or "casual expression."

[0711] "Example 3"

[0712] The AI ​​generator also has the ability to convert the content into the user's native language according to their preferences. For example, if the user is Japanese, the AI ​​generator will convert the generated live broadcast and subtitles into Japanese. This allows users to enjoy live broadcasts and subtitles in a language they can easily understand.

[0713] The process flow of each embodiment will be described below.

[0714] "Example 1"

[0715] Step 1: Users input the sports or entertainment video data they want to watch into the system.

[0716] Step 2: The system's generation AI analyzes the video data and generates live commentary and subtitles. At this time, the AI ​​understands the content of the video data and generates appropriate live commentary and subtitles.

[0717] Step 3: The generated commentary and subtitles are displayed to the user.

[0718] "Example 2"

[0719] Step 1: The user provides the system with their preferred voice tone, speed and writing style.

[0720] Step 2: The AI ​​generator adjusts the tone, speed, and writing style of the commentary and subtitles it generates based on the user's settings.

[0721] Step 3: The adjusted commentary and subtitles are then displayed to the user.

[0722] "Example 3"

[0723] Step 1: The user sets the preferred language on the system.

[0724] Step 2: The generation AI converts the live commentary and subtitles into the specified language based on the user's settings.

[0725] Step 3: The converted live commentary and subtitles will be displayed to users.

[0726] Example 1

[0727] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0728] Conventional live broadcast and subtitle generation systems based on video data require manual editing, making it difficult to adjust in real time and to meet user preferences. In addition, the analysis of video data and the quality of the generated content often lacked consistency. This resulted in a poor user experience and made it difficult to generate content efficiently.

[0729] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0730] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means for dividing the video data into frames, a means for inputting the divided frames to a generative AI model, a means for detecting important events and actions from the video data using the generative AI model and generating live coverage and subtitles, a means for overlaying the generated live coverage and subtitles on the video data, and a means for adjusting the output of the generative AI to suit the user's preferences. This enables automatic generation of high-quality live coverage and subtitles in real time, and allows easy adjustment according to the user's preferences.

[0731] "Video data" is data in digital format that contains visual information such as sports and entertainment.

[0732] A "frame" is an individual still image that makes up video data.

[0733] A "generative AI model" is an artificial intelligence algorithm that uses deep learning technology to detect important events and actions from video data and generate live commentary and subtitles.

[0734] "Significant Events and Actions" refers to occurrences or movements that are of particular interest within sports or entertainment video data.

[0735] "Live coverage" refers to commentary and comments given in real time to video data.

[0736] "Subtitles" are text information that is displayed overlaid on video data.

[0737] "User preferences" refers to individual settings such as the user's desired voice adjustments or writing style.

[0738] "Means of adjustment" refers to functions or methods for modifying the output of a generative AI model to suit the user's preferences.

[0739] The "server" is a computer system that performs processes such as analyzing video data, running generative AI models, and overlaying generated live coverage and subtitles.

[0740] This invention is a system that automatically generates live commentary and subtitles using video data of various sports and entertainment. The system includes a generative AI model and uses deep learning technology to generate live commentary and subtitles from the video data.

[0741] Hardware and Software Used

[0742] Server: A computer system that performs processes such as analyzing video data, running generative AI models, and overlaying generated live coverage and subtitles.

[0743] Terminal: A device on which a user uploads video data and views the processed video. Specifically, this applies to computers and smartphones.

[0744] Generative AI models: These are built using deep learning frameworks such as TensorFlow and PyTorch.

[0745] Data processing and calculation

[0746] 1. Input of video data

[0747] Users upload sports and entertainment video data to the system.

[0748] The terminal transmits the uploaded video data to the server.

[0749] 2. Analysis of video data

[0750] The server divides the received video data into frames. To do this, it uses a video processing library such as FFmpeg.

[0751] The server inputs the segmented frames into a generative AI model.

[0752] 3. Generative AI Processing

[0753] The generative AI model uses deep learning techniques to detect important events and actions in video data.

[0754] The generative AI model generates appropriate commentary and subtitles based on the detected events.

[0755] 4. Output of results

[0756] The server overlays the generated commentary and subtitles onto the video data, specifically by using FFmpeg to overlay the text onto the video.

[0757] The server transmits the processed video data to the terminal.

[0758] Users can watch live broadcasts and videos with added subtitles through their terminals.

[0759] Examples

[0760] For example, consider the case where a user uploads a video of a soccer match to the system. The server divides the video data into frames and inputs them into the generative AI model. The generative AI model detects goal and foul scenes, and generates commentary and subtitles such as "Goal!" and "A foul has been committed" based on the detected scenes. The server overlays the generated commentary and subtitles onto the video and sends it to the device. The user can then watch the soccer match video with the commentary and subtitles added via the device.

[0761] Examples of prompt statements

[0762] "Upload a video of a soccer game. Our generative AI will detect goals and fouls, and automatically generate commentary and subtitles."

[0763] This system allows users to enjoy high-quality live broadcasts and videos with added subtitles in real time.

[0764] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0765] Step 1:

[0766] Users upload sports and entertainment video data to the system.

[0767] Input: A video file selected by the user.

[0768] Specific actions: A user opens the system's web interface, selects a video file, and clicks the upload button.

[0769] Output: Video data is sent from the device to the server.

[0770] Step 2:

[0771] The terminal transmits the uploaded video data to the server.

[0772] Input: Video data uploaded by the user.

[0773] Specific operation: The device uploads video data to the server using an HTTP POST request.

[0774] Output: The server receives the video data.

[0775] Step 3:

[0776] The server divides the received video data into frames.

[0777] Input: Video data received by the server.

[0778] Specific operation: The server uses a video processing library such as FFmpeg to divide the video data into frames. Specifically, it executes FFmpeg commands to divide the video into frames.

[0779] ffmpeg -i input.mp4 -vf "fps=30" frame_%04d.png

[0780] Output: Segmented frame images.

[0781] Step 4:

[0782] The server inputs the segmented frames into a generative AI model.

[0783] Input: Segmented frame images.

[0784] Specific operation: The server converts the frame images into an appropriate format and inputs them into the generative AI model.

[0785] Output: The frame data fed into the generative AI model.

[0786] Step 5:

[0787] The generative AI model uses deep learning techniques to detect important events and actions in video data.

[0788] Input: The frame data fed into the generative AI model.

[0789] How it works: The generative AI model analyzes frame data to detect important events and actions, for example, goals and fouls in a soccer game.

[0790] Output: Information about the detected event and / or action.

[0791] Step 6:

[0792] The generative AI model generates appropriate commentary and subtitles based on the detected events.

[0793] Input: Information about the detected event or action.

[0794] How it works: Based on the detected events, the generative AI model generates appropriate text, such as commentary or subtitles like "Goal!" or "A foul has been committed."

[0795] Output: The generated play-by-play and subtitle text.

[0796] Step 7:

[0797] The server overlays the generated live coverage and subtitles onto the video data.

[0798] Input: The generated play-by-play or subtitle text.

[0799] What it does: The server uses FFmpeg to overlay the generated text onto the video.

[0800] ffmpeg -i input.mp4 -vf "drawtext=text='Goal!':x=10:y=10:fontsize=24:fontcolor=white" output.mp4

[0801] Output: Video data with live coverage and subtitles overlaid.

[0802] Step 8:

[0803] The server transmits the processed video data to the terminal.

[0804] Input: Video data with commentary and subtitles overlaid.

[0805] Specific operation: The server sends the processed video file to the terminal as an HTTP response.

[0806] Output: The device receives the processed video data.

[0807] Step 9:

[0808] Users can watch live broadcasts and videos with added subtitles through their terminals.

[0809] Input: The processed video data received by the device.

[0810] Specific behavior: The user uses the device's media player to play live broadcast or video with added subtitles.

[0811] Output: The video the user sees.

[0812] (Application example 1)

[0813] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0814] In conventional sports and entertainment video distribution systems, real-time live broadcasts and subtitle generation were often done manually, making it difficult to provide information quickly and accurately. In addition, it was difficult to customize to meet user preferences, and support for multiple languages ​​was limited. This led to problems such as reduced viewer satisfaction.

[0815] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0816] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means for displaying the generated live coverage and subtitles in real time, and a means for providing live coverage and subtitles in multiple languages. This enables accurate generation of live coverage and subtitles in real time, and allows customization to suit the user's preferences and support in multiple languages.

[0817] "Various sports and entertainment video data" refers to video information of sporting events and entertainment content, including live and recorded footage.

[0818] "Input means" refers to a method or device for inputting video data into the system.

[0819] "Generative AI" refers to algorithms and models that use artificial intelligence techniques such as deep learning to automatically generate live commentary and subtitles from video data.

[0820] "Means for tailoring to user preferences" refers to a method or apparatus for customizing the content or format of the generated commentary or subtitles based on a user's settings or preferences.

[0821] "Means for displaying in real time" refers to a method or apparatus for displaying the generated commentary or subtitles on a user's device immediately and without delay.

[0822] "Means for providing commentary and subtitles in multiple languages" refers to a method or apparatus for providing the generated commentary and subtitles in multiple different languages.

[0823] The system for carrying out the present invention inputs video data of various sports and entertainment, and automatically generates live broadcasts and subtitles using a generation AI. A specific embodiment of this system will be described below.

[0824] System Configuration

[0825] The system mainly uses the following hardware and software:

[0826] Hardware: Smartphone (iOS or Android), server

[0827] Software: OpenCV, TensorFlow, Transformers (Hugging Face)

[0828] Data processing and calculation

[0829] Video data input

[0830] The server receives video data of various sports and entertainment events, either live or recorded.

[0831] Video data pre-processing

[0832] The server uses OpenCV to split the video data into frames and perform pre-processing, which includes frame resizing and noise removal.

[0833] Analysis by generative AI

[0834] The server analyzes the preprocessed frames using TensorFlow and Transformers. Specifically, a generative AI model (e.g., GPT-2) is used to generate commentary and subtitles from the video data. The generative AI model takes the prompt as input and outputs appropriate commentary and subtitles.

[0835] Tailored to your preferences

[0836] The server then tailors the generated commentary and subtitles to the user's preferences, including customizing the tone of voice and writing style.

[0837] Real-time display

[0838] The device (smartphone) displays the generated live coverage and subtitles in real time, allowing users to receive information without delay.

[0839] Multiple Languages

[0840] The server provides the generated commentary and subtitles in multiple languages, allowing users who speak different languages ​​to enjoy the same content.

[0841] Examples

[0842] For example, when a goal is scored during a soccer game, the server inputs the prompt sentence "Describe the current sports event: A player is running towards the goal with the ball" into the generative AI model. Based on this, the generative AI model generates a live report and subtitles such as "Goal! Player A scores a great shot!" and displays them on the device in real time.

[0843] In this way, a system can be realized that generates live commentary and subtitles for sporting events in real time.

[0844] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0845] Step 1:

[0846] The server receives video data of various sports and entertainment events. The video data is provided in the form of live or recorded video. The input video data is prepared for processing within the server.

[0847] Step 2:

[0848] The server splits the video data into frames using OpenCV and performs preprocessing. Preprocessing includes frame resizing and noise removal. Specifically, the video data is captured at a constant frame rate and each frame is resized to a standardized size. The input is the video data and the output is the preprocessed frames.

[0849] Step 3:

[0850] The server analyzes the preprocessed frames using TensorFlow and Transformers. Specifically, a generative AI model (e.g., GPT-2) is used to generate commentary and subtitles from the video data. The generative AI model takes a prompt as input and outputs appropriate commentary and subtitles. The input is the preprocessed frames and the prompt, and the output is the generated commentary and subtitles.

[0851] Step 4:

[0852] The server adjusts the generated commentary and subtitles to suit the user's preferences. The adjustments include customizing the tone of voice and sentence expressions. Specifically, the server refers to the user's settings information and converts the generated text appropriately. The input is the generated commentary and subtitles, and the output is the adjusted commentary and subtitles.

[0853] Step 5:

[0854] The device (smartphone) displays the generated live broadcast and subtitles in real time, allowing the user to receive information without delay. Specifically, text is overlaid on the device's display. The input is the adjusted live broadcast and subtitles, and the output is the text displayed on the device's display.

[0855] Step 6:

[0856] The server provides the generated commentary and subtitles in multiple languages, allowing users who speak different languages ​​to enjoy the same content. Specifically, a generative AI model is used to translate the original text into other languages. The input is the generated commentary and subtitles, and the output is the commentary and subtitles in multiple languages.

[0857] Example 2

[0858] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0859] Conventional generative AI systems have had difficulty providing output that meets individual user preferences. In particular, they were unable to accommodate detailed settings that differ from user to user, such as tone of voice, speed, and writing style. As a result, the user experience was uniform and insufficient to customize to meet individual needs.

[0860] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0861] In this invention, the server includes a means for inputting various data, a means including a generative AI that generates information based on the data, a means for adjusting the output of the generative AI based on the user's preferences, a means for acquiring the user's settings, a means for generating a prompt sentence to be input to the generative AI model, a means for executing the generative AI model, a means for adjusting the output obtained from the generative AI model, and a means for transmitting the adjusted output to the user's terminal, thereby enabling customized output according to the individual preferences of the user.

[0862] "Various types of data" refers to data in various formats, including information entered by the user and information obtained by the system.

[0863] "Generative AI" refers to artificial intelligence that generates information based on input data.

[0864] "User preferences" refers to individual settings such as tone of voice, speed, writing style, etc. set by the User.

[0865] "Means for acquiring user settings" refers to a function that enables the system to acquire preference information that the user has set in advance.

[0866] A "prompt sentence" is an instruction sentence to be input into a generative AI model.

[0867] "Means for executing a generative AI model" refers to a function for inputting a prompt sentence into the generative AI model and generating information.

[0868] "Means for adjusting the output obtained from the generative AI model" refers to a function for adjusting the information obtained from the generative AI model based on user preferences.

[0869] A "means for transmitting the conditioned output to a user's terminal" is a function for transmitting the conditioned information to a user's terminal.

[0870] The present invention relates to a generative AI system that provides customized output according to user preferences. A specific embodiment of the system will be described below.

[0871] System Configuration

[0872] Hardware and Software

[0873] The server runs the system using the following hardware and software:

[0874] Hardware: High-performance server computer

[0875] Software: Generative AI models (e.g., GPT-3), text-to-speech software (e.g., Amazon Polly or Google Text-to-Speech)

[0876] Program Processing

[0877] Retrieving user settings

[0878] The server retrieves information about the user's predefined preferences for tone of voice, speed, writing style, etc. This information is loaded from the user's profile when they log into the system.

[0879] Example: If a user selects "calm male voice" and "polite speech," the server retrieves these preferences from a database.

[0880] Generate prompt statement

[0881] The server generates prompts based on the user's requests to feed into the generative AI model, which is then tailored to obtain an output that is tailored to the user's preferences.

[0882] Example: If a user requests, "Please explain how the generative AI model works," the server generates a prompt that reads, "Please explain the following in a calm, male voice using polite language: How the generative AI model works."

[0883] Running generative AI models

[0884] The server inputs a prompt sentence using a generative AI model (e.g., GPT-3) and generates an output according to the user's request.

[0885] Example: The server sends the generated prompt text to GPT-3, which generates an explanation of "how the generative AI model works."

[0886] Adjusting the Output

[0887] The server then adjusts the output of the generative AI model based on the user's settings, using voice synthesis software (e.g., Amazon Polly or Google Text-to-Speech) to generate speech at the user's selected tone and speed.

[0888] Example: The server sends the text obtained from GPT-3 to Amazon Polly, which generates a speech in a "calm male voice."

[0889] Send to device

[0890] The server then sends the adjusted output to the user's device, where the user can view the audio and text.

[0891] Example: The server sends the generated audio file to the user's smartphone, allowing the user to play it.

[0892] Adding concrete examples

[0893] Here is a concrete example of what happens when a user selects "exciting female voice" and "casual expression."

[0894] Example prompt: "In an exciting female voice, use casual language to explain: How does a generative AI model work?"

[0895] Example output from a generative AI model: "Hey guys! Today we're going to talk about generative AI models. They're super cool!"

[0896] Speech synthesis example: Using Google Text-to-Speech, convert the above text into an "exciting female voice."

[0897] In this way, the system provides generative AI output tailored to the user's preferences.

[0898] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0899] Step 1: Get user settings

[0900] The server retrieves information such as the user's predefined preferred tone of voice, speed, writing style, etc. When a user logs into the system, the server reads the user's profile information from the database and retrieves the user's preferred settings.

[0901] Input: User login information

[0902] Data processing: Search for user settings information from the database

[0903] Output: User preferences

[0904] Step 2: Generate a prompt

[0905] The server generates a prompt sentence to be input to the generative AI model based on the user's request. The server combines the user's request content and the acquired settings to generate a prompt sentence to be input to the generative AI model.

[0906] Input: User request, user preference settings

[0907] Data processing: Generate prompts by combining the request content and configuration information

[0908] Output: The generated prompt statement

[0909] Step 3: Running the generative AI model

[0910] The server inputs a prompt sentence using a generative AI model (e.g., GPT-3) and generates an output according to the user's request. It sends the generated prompt sentence to the generative AI model and receives the generated text.

[0911] Input: Generated prompt text

[0912] Data calculation: Enter a prompt into the generative AI model to generate text

[0913] Output: The generated text

[0914] Step 4: Adjust the output

[0915] The server then adjusts the output of the generative AI model based on the user's settings, using voice synthesis software (e.g., Amazon Polly or Google Text-to-Speech) to generate speech at the user's selected tone and speed.

[0916] Input: Generated text, user preferences and settings

[0917] Data processing: Input text into speech synthesis software to generate speech

[0918] Output: Adjusted audio file

[0919] Step 5: Send to device

[0920] The server sends the adjusted output to the user's device, and the generated audio file is sent to the user's smartphone so that the user can play it.

[0921] Input: Adjusted audio file

[0922] Data processing: Send audio files to the user's device

[0923] Output: An audio file that can be played on the user's device.

[0924] In this way, the system provides generative AI output tailored to the user's preferences.

[0925] (Application example 2)

[0926] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0927] In conventional content delivery systems, it was difficult to adjust voice tone, speed, and writing style according to individual user preferences. This resulted in problems in which users could not watch or listen to content more comfortably. In particular, there is a demand for audio output that matches the user's preferences for content such as news articles, audiobooks, and podcasts, but it was difficult to achieve this with current technology.

[0928] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0929] In this invention, the server includes a means for inputting various video data, a means including a generation AI for generating voice output based on the video data, a means for adjusting the output of the generation AI according to the user's preferences, a means for allowing the user to set the tone of voice, speed, and style of writing that they prefer, a means for inputting a prompt sentence into the generation AI model, and a means for outputting the generated text by voice, which makes it possible to adjust the tone of voice, speed, and style of writing according to the individual preferences of the user.

[0930] "Video data" refers to digital data that contains visual information, such as videos and images.

[0931] "Generative AI" is a system that uses artificial intelligence technology to generate new output from input data.

[0932] "User Preferences" refers to the individual preferences of each individual User, including the particular tone of voice, speed, writing style, etc.

[0933] "Tone of voice" refers to the pitch and texture of the voice, and includes characteristics such as a calm voice or an exciting voice.

[0934] "Speed" refers to the speed at which audio is played back, usually expressed as the number of words per unit time.

[0935] "Writing style" refers to the way a sentence is written and the characteristics of the language used, and includes polite and casual expressions.

[0936] A "prompt sentence" is text data input into a generative AI and refers to the sentence that forms the basis of the generated output.

[0937] "Voice output" refers to playing back the text generated by the generation AI as audio.

[0938] A system for implementing this invention includes means for receiving various types of video data as input, means including a generation AI that generates audio output based on the video data, means for adjusting the output of the generation AI to suit the user's preferences, means for allowing the user to set the tone, speed, and style of voice expression preferred by the user, means for inputting a prompt sentence into the generation AI model, and means for audio output of the generated text.

[0939] Explanation of system program processing

[0940] The server first receives various video data as input. This video data includes content such as news articles, audiobooks, and podcasts. It then uses a generative AI to generate audio output based on this video data. For example, the generative AI uses the "pipeline" function from the "transformers" library.

[0941] Users can set their preferred voice tone, speed, and writing style through their smartphones or other devices. For this, the "pyttsx3" library is used as the voice engine. The output of the generative AI is adjusted based on the user's settings. Specifically, users can select a "calm male voice" or an "exciting female voice," as well as a writing style such as "polite expression" or "casual expression."

[0942] As a means of inputting prompt sentences into the generative AI model, the text data entered by the user is passed to the generative AI. For example, "News article: The weather is sunny today" is input as a prompt sentence for a news article. Based on this prompt sentence, the generative AI generates sentences that match the user's preferences.

[0943] The generated text is output as a voice using a voice engine. The voice engine reads the generated text aloud, adjusting the tone and speed of the voice based on the user's settings. This allows users to enjoy content in a voice that matches their preferences.

[0944] Examples

[0945] For example, if a user prefers a "calm male voice" and "polite expressions," the following prompt sentence can be input into the generative AI model.

[0946] Prompt: "News article: The weather is sunny today."

[0947] Based on this prompt, the generative AI model generates sentences that match the user's preferences, such as, "It's sunny today and the temperature is very comfortable. I hope you all have a wonderful day."

[0948] In this way, it is possible to adjust the tone, speed and writing style of speech to suit the user's individual preferences.

[0949] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0950] Step 1:

[0951] The server receives various types of video data as input, including content such as news articles, audiobooks, podcasts, etc. The input video data is stored in a database.

[0952] Step 2:

[0953] The server uses a generative AI to generate audio output based on the input video data. Specifically, it inputs a prompt sentence into a generative AI model (e.g., GPT-3) and obtains the generated text. This prompt sentence summarizes the content of the video data.

[0954] Step 3:

[0955] Users can set their preferred voice tone, speed, and writing style through the device. For example, they can use a smartphone application to select a "calm male voice" or an "exciting female voice," or "polite expressions" or "casual expressions."

[0956] Step 4:

[0957] The server adjusts the output of the generation AI based on the user's settings. Specifically, it uses a voice engine (e.g., pyttsx3) to generate voice output that matches the voice tone and speed set by the user.

[0958] Step 5:

[0959] The server inputs a prompt sentence into the generative AI model. For example, it inputs "News article: Today's weather is sunny" as a prompt sentence for a news article. Based on this prompt sentence, the generative AI generates a sentence that matches the user's preferences.

[0960] Step 6:

[0961] The server outputs the generated text using a voice engine. The voice engine reads the generated text aloud, adjusting the tone and speed of the voice based on the user's settings. This allows the user to enjoy the content in a voice that matches their preferences.

[0962] Example 3

[0963] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0964] Conventional live broadcast and subtitle generation systems have difficulty converting the language into the user's native language in real time, which means that users cannot enjoy content in a language that is easy to understand. In addition, they do not adequately adjust to the user's preferences, making it impossible to provide an optimal experience for each individual user.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0966] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means for receiving a prompt text input by the user, a means for acquiring information on the user's native language, a means for converting the generated text into the user's native language, and a means for transmitting the converted text to the user's terminal. This allows the user to enjoy live coverage and subtitles in real time in a language that is easy for the user to understand.

[0967] "Video data" is data in digital format that contains visual information such as sports and entertainment.

[0968] "Generative AI" is an artificial intelligence model that generates text or speech based on input data.

[0969] A "prompt" is text that contains instructions or requests that a user enters into a system.

[0970] "User's native language information" is information about the language the user is most comfortable understanding.

[0971] "Text conversion means" means a function or process for converting the generated text into the user's native language.

[0972] A "terminal" is a device used by a user, such as a computer, smartphone, or tablet.

[0973] An "HTTP request" is a communication protocol for a client to request data from a server.

[0974] An "HTTP response" is a communication protocol that enables a server to return data to a client.

[0975] "Translation API" means an application programming interface for automatically translating text into different languages.

[0976] This invention relates to a system that allows users to enjoy live broadcasts and subtitles in real time in a language that is easy for users to understand. The system takes video data of various sports and entertainment as input, generates live broadcasts and subtitles using a generative AI model, and has the ability to adjust them to suit the user's preferences.

[0977] Hardware and Software Used

[0978] Hardware

[0979] Server: A central processing unit for running the generative AI model, obtaining the user’s native language information, and converting the generated text.

[0980] Terminal: A device on which a user enters prompts and displays the converted text. Examples include computers, smartphones, and tablets.

[0981] software

[0982] Generative AI models: For example, using advanced natural language processing models such as GPT-4.

[0983] Translation API: For example, use the Google Translate API to translate the generated text into the user's native language.

[0984] Data processing and calculation

[0985] 1. Receiving a prompt: The user enters a prompt into the input field of the terminal. For example, the user enters "Please provide a commentary of this game in Japanese."

[0986] 2. Sending the prompt: The terminal sends the entered prompt to the server as an HTTP request.

[0987] 3. Get native language information: The server gets the user's profile information from the database and checks what the user's native language is.

[0988] 4. Applying the generative AI model: The server uses a generative AI model (GPT-4) to generate commentary and subtitle text based on the prompt text.

[0989] 5. Text conversion: The server converts the generated text into the user's native language using a translation API (Google Translate API).

[0990] 6. Sending the converted text: The server sends the converted text to the terminal as an HTTP response.

[0991] 7. Displaying text: The device will display the received converted text on the screen, allowing users to enjoy live commentary and subtitles in Japanese.

[0992] Examples

[0993] For example, suppose a user is watching a soccer game. The user enters a prompt sentence such as "Please provide commentary on this game in Japanese." The device sends this prompt sentence to the server, and the server uses a generative AI model to generate commentary text. The server then converts the generated text into Japanese using a translation API and sends the converted text to the device. Finally, the device displays the converted commentary on the screen, allowing the user to enjoy the commentary in Japanese.

[0994] In this way, the user can enjoy live coverage and subtitles in real time in a language that the user can easily understand. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0995] Step 1:

[0996] The user enters a prompt sentence.

[0997] The user inputs a prompt sentence into the input field of the terminal, such as "Please provide a commentary of this game in Japanese." The input prompt sentence is temporarily stored in the memory of the terminal.

[0998] Step 2:

[0999] The terminal sends a prompt to the server.

[1000] The terminal sends the prompt text entered by the user to the server as an HTTP request. Specifically, a data packet containing the prompt text is sent to the server via the Internet. The input is the prompt text, and the output is the request sent to the server.

[1001] Step 3:

[1002] The server obtains the user's native language information.

[1003] The server identifies the user's ID based on the received prompt sentence, and retrieves the user's profile information from the database. Specifically, it retrieves the user's native language information using an SQL query. The input is the user's ID, and the output is the user's native language information.

[1004] Step 4:

[1005] The server uses a generative AI model to generate text based on the prompt.

[1006] The server uses a generative AI model (e.g., GPT-4) to generate commentary and subtitle text based on the prompt sentence. The generative AI model parses the prompt sentence and generates appropriate text. The input is the prompt sentence and the output is the generated text.

[1007] Step 5:

[1008] The server translates the generated text into the user's native language.

[1009] The server translates the generated text into the user's native language using a translation API (e.g., Google translation API). Specifically, the server sends the generated text to the translation API and receives the translated text. The input is the generated text and the output is the translated text.

[1010] Step 6:

[1011] The server sends the converted text to the terminal.

[1012] The server sends the translated text to the terminal as an HTTP response. Specifically, a data packet containing the translated text is sent to the terminal over the Internet. The input is the translated text, and the output is the response sent to the terminal.

[1013] Step 7:

[1014] The terminal displays the translated text.

[1015] The terminal receives the translated text from the server and displays it on the screen, in particular by rendering the received text in a user interface for visual confirmation by the user. The input is the translated text and the output is the text displayed on the screen.

[1016] (Application example 3)

[1017] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1018] In conventional live streaming and video distribution services, users have had difficulty understanding live coverage and subtitles provided in different languages. In addition, users have had limited access to content translated in real time into their native language, limiting their viewing experience. This has created an information gap between users who speak different languages.

[1019] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1020] In this invention, the server includes a means for receiving video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means for translating the generated live coverage and subtitles into the user's native language in real time, a means for displaying the translated live coverage and subtitles, and a means for providing an interface that allows the user to select a language of their choice. This enables the user to watch live coverage and subtitles in real time in a language that they can easily understand.

[1021] "Various sports and entertainment video data" refers to video information of sporting events and entertainment content, including live streaming and recorded videos.

[1022] "Generative AI" refers to artificial intelligence technology that automatically generates live commentary and subtitles based on input video data.

[1023] "Means for tailoring to user preferences" refers to the ability to customize the content and format of the generated commentary and / or subtitles to a user's individual preferences.

[1024] "Means of translating into the user's native language in real time" refers to the function of instantly translating the generated live broadcast and subtitles into a language specified by the user.

[1025] "Means for displaying translated commentary and / or subtitles" refers to the functionality for visually displaying the translated content on a User's device.

[1026] "An interface that allows users to select their preferred language" refers to a function that provides an operation screen or menu that allows users to select their desired language.

[1027] The system for implementing this invention is configured as follows. First, the server receives video data of various sports and entertainment as input. Next, the server uses a generation AI to generate live coverage and subtitles based on the input video data. This generation AI can adjust the voice and writing style to suit the user's preferences.

[1028] The generated live commentary and subtitles are translated into the user's native language in real time. Translation services such as Google Translate API are used for this translation process. The translated live commentary and subtitles are displayed on the user's device. The user can visually check the translated content using a device such as a smartphone or tablet.

[1029] Additionally, users can specify their preferred language through a language selection interface, which is designed to be easy for users to navigate and provides language selection menus and settings screens.

[1030] As a specific example, consider the case where a user is watching a live stream in English. In this case, the server receives the English video data and uses generation AI to generate English commentary and subtitles. The generated English commentary and subtitles are then translated into Japanese in real time and displayed on the user's smartphone. The user can watch the translated Japanese commentary and subtitles by selecting Japanese through the interface.

[1031] An example of a prompt is "Please translate the English live streaming subtitles into Japanese in real time and display them." By using this prompt, users can enjoy live streaming in a language they can easily understand.

[1032] This system will eliminate the information gap between users who speak different languages, allowing users to watch live broadcasts and subtitles in their own native language in real time.

[1033] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1034] Step 1:

[1035] The server receives video data of various sports and entertainment as input. The input video data is provided in the form of live streaming, recorded video, etc. The server analyzes the format of the data and converts it into an appropriate format for processing.

[1036] Step 2:

[1037] The server uses a generation AI to generate live commentary and subtitles based on the input video data. The generation AI extracts audio and text information from the video data, and creates live commentary and subtitles in real time based on that information. Voice recognition and natural language processing technologies are used in this process. The generated live commentary and subtitles are output as text data.

[1038] Step 3:

[1039] The server adjusts the generated live broadcast and subtitles to suit the user's preferences. Specifically, it customizes the tone of voice and sentence expression based on the user's settings. This adjustment is achieved by changing the parameters of the generation AI. After adjustment, the live broadcast and subtitles are output again as text data.

[1040] Step 4:

[1041] The server translates the adjusted live broadcast and subtitles into the user's native language in real time. This translation process uses a translation service such as Google Translate API. The server sends the adjusted text data to the translation API and receives the translated text data. The translated live broadcast and subtitles are output in the language specified by the user.

[1042] Step 5:

[1043] The server sends the translated live broadcast and subtitles to the user's device. The user's device displays the received text data on the screen. The display is done on a device such as a smartphone or tablet. The user can visually check the translated content.

[1044] Step 6:

[1045] Users specify their preferred language through an interface that allows them to select their preferred language. The interface is designed to be easy for users to operate, and provides a language selection menu and a settings screen. The user's selection is sent to the server and is reflected in the next translation process.

[1046] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1047] "Example 1"

[1048] In one embodiment of the present invention, a system incorporating an emotion engine is provided. The system recognizes the emotions of a user and adjusts the output of the generative AI based on the emotions.

[1049] In essence, the system estimates emotions from facial expressions, tone of voice, eye movements, etc. while the user is watching the video. The system then adjusts the output of the generation AI according to the emotion. For example, if the user is happy, the generation AI will generate a more exciting live commentary and subtitles. Conversely, if the user is confused or uncomfortable, the generation AI will generate a more understandable and polite live commentary and subtitles.

[1050] "Example 2"

[1051] The emotion engine also adjusts the voice based on the user's emotions. Specifically, if the user is happy, the AI ​​will generate commentary and subtitles in a brighter, more cheerful voice. Conversely, if the user is sad, the AI ​​will generate commentary and subtitles in a calmer voice. This enables more natural communication that matches the user's emotions.

[1052] "Example 3"

[1053] In addition, the emotion engine adjusts the text expression based on the user's emotion. Specifically, if the user expresses surprise, the generation AI generates commentary and subtitles with more detailed explanations. Conversely, if the user expresses relief, the generation AI generates simpler commentary and subtitles. This makes it possible to provide optimal information according to the user's emotion.

[1054] The process flow of each embodiment will be described below.

[1055] "Example 1"

[1056] Step 1: The user watches the video.

[1057] Step 2: The emotion engine infers emotions from the user's facial expressions, tone of voice, eye movements, etc.

[1058] Step 3: Based on the estimated emotions, the generation AI adjusts the output and generates commentary and subtitles.

[1059] "Example 2"

[1060] Step 1: The user watches the video.

[1061] Step 2: The emotion engine estimates the user's emotion.

[1062] Step 3: Based on the estimated emotions, the generation AI adjusts the voice and generates commentary and subtitles.

[1063] "Example 3"

[1064] Step 1: The user watches the video.

[1065] Step 2: The emotion engine estimates the user's emotion.

[1066] Step 3: Based on the estimated emotions, the generation AI adjusts the sentence expression and generates live commentary and subtitles.

[1067] Example 1

[1068] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1069] Conventional live broadcast and subtitle generation systems do not adjust according to the user's emotions, and there is a problem that the user experience is not sufficiently improved. In particular, there is a demand for real-time adjustment according to the user's emotions while watching the video.

[1070] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for inputting various video data, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for collecting user emotion data, and a means for adjusting the output of the generation AI based on the emotion data. This enables the generation of real-time live coverage and subtitles according to the user's emotions.

[1071] "Video data" is data in digital format that contains visual information such as sports and entertainment.

[1072] "Generative AI" is artificial intelligence that uses deep learning technology to automatically generate live commentary and subtitles from video data.

[1073] "User emotion data" refers to information about emotions obtained from the user's facial expressions, tone of voice, eye movements, etc.

[1074] An "emotion engine" is a system that includes machine learning algorithms for analyzing a user's emotional data and inferring their emotions.

[1075] "Means for adjusting output" refers to a function that changes the output content and tone of the generation AI based on feedback from the emotion engine.

[1076] "Live coverage" refers to information conveying the progress of an event taking place in real time through audio and text.

[1077] "Subtitles" are text information added to video data, providing supplementary explanations and dialogue to the viewer.

[1078] The present invention relates to a system that takes video data of sports or entertainment as input and generates live coverage and subtitles according to the user's emotions. This system is composed of three main elements: a server, a terminal, and a user.

[1079] The server first receives various types of video data via the Internet. This video data is in digital format containing visual information such as sports matches and entertainment events. The server inputs the received video data into the generative AI. The generative AI uses deep learning frameworks such as TensorFlow and PyTorch to analyze the video data and identify important events (e.g., goal scenes, goal-scoring scenes, interviews, etc.).

[1080] While the user is watching the video, the device uses the built-in camera and microphone to collect the user's facial expressions and tone of voice. This is done using face recognition technology and voice analysis technology. For example, if the user smiles, the facial expression data is captured. The collected emotion data is sent from the device to the emotion engine. The emotion engine uses libraries such as OpenCV and Dlib to perform face recognition and estimate the user's emotion. For example, if the user is smiling, it is determined to be "joy."

[1081] The server receives feedback from the emotion engine and adjusts the output of the generation AI. Specifically, it changes the tone and content of the play-by-play and subtitles depending on the user's emotions. For example, if the user is happy, the generation AI generates an exciting play-by-play such as, "Great goal! The crowd is excited!" Conversely, if the user is confused, it provides an easy-to-understand explanation such as, "An offside was called. This is because the attacking player got ahead of the opposing defender."

[1082] The device then displays the live broadcast and subtitles to the user in real time, allowing the user to enjoy content that best suits their emotions.

[1083] As a concrete example, consider a user watching a soccer game. If the user shows a look of joy at a goal, the AI ​​will generate an exciting live broadcast such as, "What a great goal! The crowd is so excited!" On the other hand, if the user is confused by an offside call, the AI ​​will provide a clear explanation such as, "Offside was called. This is because the attacking player got ahead of the opposing defender."

[1084] An example prompt would be:

[1085] "Using video data of a soccer match as input, generate live commentary and subtitles according to the user's emotions. If the user is happy, provide an exciting commentary, and if the user is confused, provide an easy-to-understand explanation."

[1086] In this way, the present invention can provide optimal live coverage and subtitles according to the user's emotions.

[1087] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1088] Step 1: Receiving video data

[1089] The server receives various video data via the Internet. This video data is in digital format containing visual information such as sports games and entertainment events. The input is video data, and the output is video data ready to be input to the generative AI.

[1090] Step 2: Analyze the video data

[1091] The server inputs the received video data into the generation AI. The generation AI uses deep learning frameworks such as TensorFlow and PyTorch to analyze the video data and identify important events (e.g., goal scenes, scoring scenes, interviews, etc.). The input is the video data, and the output is the identified important event information.

[1092] Step 3: Collect user emotion data

[1093] The device uses a built-in camera and microphone to collect the user's facial expressions and tone of voice while the user is watching a video. This is done using facial recognition and voice analysis technologies. The input is the video and audio data the user is watching, and the output is the collected emotion data.

[1094] Step 4: Analyze the sentiment data

[1095] The terminal sends the collected emotion data to the emotion engine. The emotion engine uses libraries such as OpenCV and Dlib to perform face recognition and estimate the user's emotion. The input is the collected emotion data, and the output is the estimated user's emotion information.

[1096] Step 5: Adjusting the output of the generation AI

[1097] The server receives feedback from the emotion engine and adjusts the output of the generative AI. Specifically, it changes the tone and content of the live broadcast and subtitles according to the user's emotions. The input is the estimated user's emotion information, and the output is the adjusted content of the live broadcast and subtitles.

[1098] Step 6: Generate and display live commentary and subtitles

[1099] The server generates live coverage and subtitles based on the adjusted output of the generation AI. The terminal displays the generated live coverage and subtitles to the user in real time. The input is the adjusted live coverage and subtitle content, and the output is the live coverage and subtitles displayed to the user.

[1100] In this way, the system can provide optimal live coverage and subtitles according to the user's emotions.

[1101] (Application example 1)

[1102] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1103] Conventional sports and entertainment video distribution systems have the problem that they are unable to provide live coverage or subtitles that respond to the user's emotions, resulting in a uniform viewing experience. In addition, there are limited adjustments to suit user preferences, making it difficult to provide optimal content for each individual viewer.

[1104] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1105] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means including an emotion engine for recognizing the user's emotions and adjusting the output of the generation AI based on the emotions, a means for inputting the user's facial expressions and tone of voice using a smartphone camera and microphone, and a means for the generation AI to generate live coverage and subtitles in real time according to the user's emotions. This makes it possible to provide live coverage and subtitles that are individually optimized according to the user's emotions.

[1106] "Various sports and entertainment video data" refers to video information of sporting events and entertainment content, including live and recorded footage.

[1107] "Generative AI" refers to a system that uses artificial intelligence technologies such as deep learning to automatically generate live commentary and subtitles from input video data.

[1108] "Means of adjusting to user preferences" refers to the function of customizing the output of the generative AI based on the user's viewing history and settings information.

[1109] An "emotion engine" is a system that recognizes emotions from the user's facial expressions, tone of voice, etc., and adjusts the output of the generative AI based on those emotions.

[1110] "Means of inputting a user's facial expressions and tone of voice using a smartphone's camera and microphone" refers to a function that uses a smartphone's built-in camera and microphone to capture a user's facial expressions and tone of voice in real time.

[1111] "Means of generation in real time" refers to the ability to instantly generate live commentary and subtitles based on user emotions and the context of video data, and provide them without delay.

[1112] The system for implementing this invention is configured as follows. First, the server receives video data of various sports and entertainment as input. Next, the server uses a generation AI to automatically generate live coverage and subtitles based on the input video data. This generation AI utilizes artificial intelligence technology such as deep learning.

[1113] In addition, the server is equipped with an emotion engine that recognizes the user's emotions. This emotion engine captures the user's facial expressions and tone of voice in real time using the smartphone's camera and microphone, and analyzes them. Based on the analysis results, the output of the generation AI is adjusted according to the user's emotions.

[1114] Specifically, if the user expresses joy, the AI ​​will generate more exciting commentary and subtitles. On the other hand, if the user expresses confusion or displeasure, the AI ​​will generate more understandable and detailed commentary and subtitles.

[1115] The hardware required is a smartphone camera and microphone, the software uses OpenCV to process camera input, DeepFace to recognize emotions, and the Transformers library to implement a generative AI model.

[1116] As a concrete example, consider a case where a user is watching a soccer game and expressing joy. In this case, the following prompt sentence is input to the generation AI.

[1117] Example prompt:

[1118] If the user indicates the emotion of joy, generate a play-by-play based on the following context: "The moment when team A scores a goal in a soccer game."

[1119] Based on this prompt, the generative AI will generate an exciting live broadcast and provide it to the user, allowing the user to have a more emotionally relevant viewing experience.

[1120] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1121] Step 1:

[1122] The server receives video data from various sports and entertainment sources as input. The video data can be real-time or pre-recorded. The server analyzes the data and extracts the necessary information. Specifically, it analyzes frames of the video data to detect important events (e.g., goals scored or highlights).

[1123] Step 2:

[1124] The server uses a generative AI to automatically generate commentary and subtitles based on the input video data. The generative AI uses a deep learning model to generate text from video data. The input includes frames of video data and their analysis results. The output is the generated commentary and subtitle text.

[1125] Step 3:

[1126] The server captures the user's facial expressions and tone of voice in real time using the smartphone's camera and microphone. The device sends input data from the camera and microphone to the server. The input includes video of the user's face and audio data. The output is sent to the server.

[1127] Step 4:

[1128] The server uses an emotion engine to recognize emotions from the user's facial expressions and tone of voice. The emotion engine analyzes input video and audio data using emotion recognition models such as DeepFace. Inputs include video of the user's face and audio data. Outputs include the user's emotions (e.g., happy, confused, displeased).

[1129] Step 5:

[1130] The server adjusts the output of the generative AI based on the recognized user emotions. Specifically, if the user expresses joy, the generative AI generates a more exciting commentary and subtitles. Conversely, if the user expresses confusion or displeasure, the generative AI generates a more understandable and polite commentary and subtitles. The input includes the user's emotions and the context of the video data. The output is the adjusted commentary and subtitle text.

[1131] Step 6:

[1132] The server provides the generated commentary and subtitles to the user in real time. The device displays the commentary and subtitles sent from the server. The input includes the adjusted commentary and subtitle text. The output is the commentary and subtitles displayed on the user's smartphone screen.

[1133] This allows users to have a viewing experience that is individually optimized to fit their emotions.

[1134] Example 2

[1135] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[1136] Conventional generative AI systems have limited ability to adjust output to suit user preferences, making it difficult to communicate naturally according to the user's emotions. In addition, it is difficult to generate voice and written expressions that reflect the user's emotions, and there is a demand for improved user experience.

[1137] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1138] In this invention, the server includes a means for inputting various video data, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI according to the user's preferences, a means for detecting the user's emotions, and a means for adjusting the output of the generation AI based on the user's emotions. This makes it possible to generate natural and appropriate voice and text expressions according to the user's preferences and emotions.

[1139] "Various types of video data" refers to various types of video information such as sports and entertainment.

[1140] "Generative AI" refers to artificial intelligence technology that generates text or speech based on input data.

[1141] "User preferences" refers to individual preferences, such as tone of voice, speed, and writing style, set by the User.

[1142] "Means of adjustment" refers to functions and methods for changing the output of the generative AI to suit the user's preferences and emotions.

[1143] "User emotion" refers to an emotional state, such as happiness, sadness, excitement, etc., exhibited by a user.

[1144] "Means for emotion detection" refers to technologies or methods for analyzing and identifying emotions from user input data.

[1145] A "prompt" is an instruction given to a generation AI to generate a specific output.

[1146] "Speech synthesis software" refers to software technology for converting text data into speech data.

[1147] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1148] The present invention relates to a system for adjusting the output of a generation AI according to the preferences and emotions of a user. A specific embodiment of this system will be described below.

[1149] Hardware and Software Used

[1150] Hardware: Servers, terminals (PCs, smartphones, tablets, etc.)

[1151] Software: Generative AI models (e.g. GPT-4), emotion engines, text-to-speech software (e.g. Amazon Polly, Google Text-to-Speech)

[1152] System Configuration

[1153] 1. Getting user settings

[1154] A user logs into the system using a terminal.

[1155] On the settings screen, users can select their preferred voice tone (e.g., calm male voice, exciting female voice), speed, and writing style (e.g., polite expression, casual expression).

[1156] The terminal transmits the user's setting information to the server.

[1157] 2. Emotion Detection

[1158] The user inputs text or voice data and transmits it to the server via the terminal.

[1159] The server passes the received data to the emotion engine and analyzes the user's emotions.

[1160] Specifically, the emotion engine uses text analysis algorithms to identify emotions from user input, for example detecting the emotion "joy" from the text "Today was such a fun day!"

[1161] 3. Prompt generation for generative AI models

[1162] Based on the user's settings and the analysis results of the emotion engine, the server generates a prompt sentence to be input into the generative AI model.

[1163] As a specific operation, the server creates a prompt sentence saying, "If the user is expressing an emotion of joy, please generate text in a bright, cheerful voice and with casual expressions."

[1164] 4. Text Generation

[1165] The server inputs the prompt text into a generative AI model (e.g., GPT-4) and generates text that matches the user's preferences and emotions.

[1166] Specifically, the generative AI model generates the text "Today was absolutely amazing!" based on the prompt sentence.

[1167] 5. Speech synthesis

[1168] The server inputs the generated text into speech synthesis software, which generates speech in a tone and speed that matches the user's settings.

[1169] Specifically, the speech synthesis software (e.g., Amazon Polly) generates voice data in accordance with the instructions, "Generate an exciting female voice with casual expressions."

[1170] 6. Providing Output

[1171] The server transmits the generated voice data to the terminal.

[1172] The terminal reproduces the received voice data and provides it to the user.

[1173] In concrete terms, the user can play back the audio generated on the terminal and enjoy audio output tailored to his or her preferences.

[1174] Examples

[1175] Examples of prompt statements

[1176] "If the user is expressing the emotion of happiness, generate text in a cheerful, upbeat voice and with casual expressions."

[1177] "If the user indicates sadness, generate text in a calm, polite voice."

[1178] Real-world examples

[1179] A user logs into the system and selects "exciting female voice" and "casual expression."

[1180] The user types, "Today was such a fun day!"

[1181] The server's emotion engine detects "joy" and inputs the prompt sentence "Generate text in a bright, cheerful voice and casual expressions" into the generative AI model.

[1182] The server's generative AI model generates the text "Today was amazing!", and the speech synthesis software generates the audio in an exciting female voice.

[1183] The user can play back and enjoy the audio generated on the terminal.

[1184] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1185] Step 1:

[1186] A user logs into the system using a terminal. After logging in, the user selects the tone of voice, speed, and writing style he or she prefers on the settings screen. This setting information is sent from the terminal to the server.

[1187] Input: User login information, tone of voice, speed, writing style

[1188] Output: User configuration information sent to the server.

[1189] Step 2:

[1190] Users input text or voice data and send it to the server via their device. The server passes the received data to the emotion engine, which analyzes the user's emotions. The emotion engine uses text analysis algorithms to identify emotions from the user's input.

[1191] Input: User text or voice data

[1192] Output: Sentiment analysis results by the emotion engine (e.g., happy, sad)

[1193] Step 3:

[1194] The server generates a prompt sentence to be input to the generative AI model based on the user's settings and the analysis results of the emotion engine. Specifically, the server creates a prompt sentence that reads, "If the user is expressing joy, please generate text in a bright, cheerful voice and with casual expressions."

[1195] Input: User settings information, sentiment analysis results

[1196] Output: Prompt sentence to be input to the generative AI model

[1197] Step 4:

[1198] The server inputs the prompt sentence into a generative AI model (e.g. GPT-4) and generates text that matches the user's settings and emotions. The generative AI model generates text based on the prompt sentence.

[1199] Input: prompt statement

[1200] Output: The generated text (e.g. "Today was absolutely amazing!")

[1201] Step 5:

[1202] The server inputs the generated text into the speech synthesis software, which generates speech at a tone and speed that matches the user's settings. The speech synthesis software generates speech data according to the instruction, "Please generate a voice with an exciting female voice and casual expressions."

[1203] Input: Generated text, user settings information

[1204] Output: The generated audio data

[1205] Step 6:

[1206] The server transmits the generated voice data to the terminal. The terminal plays back the received voice data and provides it to the user. The user can play back the generated voice on the terminal and enjoy voice output tailored to their preferences.

[1207] Input: Generated audio data

[1208] Output: Audio played on the device

[1209] (Application example 2)

[1210] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1211] Conventional generative AI systems have limited ability to adjust voice and written expressions to suit user preferences, making it difficult to achieve natural communication based on the user's emotions. In addition, they cannot fine-tune voice properties based on user settings, making it difficult to provide an optimal experience for each individual user.

[1212] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1213] In this invention, the server includes a means for inputting various video data, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means for adjusting the voice based on the user's emotions, a means for setting voice properties based on the user's settings, and a means for recognizing the user's emotions using an emotion recognition engine. This enables natural and personalized voice output according to the user's preferences and emotions.

[1214] "Various types of video data" refers to video information in various fields such as sports, entertainment, and education.

[1215] "Generative AI" refers to artificial intelligence technology that automatically generates voice or text based on input data.

[1216] "User Preferences" refers to the individual settings, such as tone of voice, speed, writing style, etc., selected by the User.

[1217] An "emotion recognition engine" refers to technology that analyzes and recognizes emotions from a user's facial expressions, tone of voice, text, etc.

[1218] "Speech properties" refers to characteristics of speech such as tone, speed, and pitch.

[1219] "Live coverage and subtitles" refers to audio commentary and text display generated in real time for video data.

[1220] "User emotion" refers to an emotional state, such as happiness, sadness, anger, etc., expressed by a user.

[1221] "Natural communication" refers to seamless dialogue using voice and text that is tailored to the user's emotions and preferences.

[1222] The system for carrying out this invention is configured as follows: The server includes means for receiving various video data as input, means including a generation AI for generating live coverage and subtitles based on the video data, means for adjusting the output of the generation AI to suit the user's preferences, means for adjusting the voice based on the user's emotions, means for setting voice properties based on the user's settings, and means for recognizing the user's emotions using an emotion recognition engine.

[1223] Program processing explanation

[1224] The server first receives various video data as input. This video data includes video information from various fields such as sports, entertainment, and education. Next, it uses generative AI to generate live coverage and subtitles based on the input video data. This generative AI is an artificial intelligence technology that automatically generates audio and text based on the input data.

[1225] The user's device adjusts the output of the generative AI to suit the user's preferences. Specifically, it sets voice properties based on individual settings selected by the user, such as tone of voice, speed, and writing style. It also recognizes the user's emotions using an emotion recognition engine and adjusts the voice based on the user's emotions. This emotion recognition engine is a technology that analyzes and recognizes emotions from the user's facial expressions, tone of voice, text, etc.

[1226] Examples

[1227] For example, if a user selects a "calm male voice" with "polite expression" and the emotion is "joy," the following prompt sentence is input into the generative AI model.

[1228] Example prompt:

[1229] User Settings:

[1230] Tone of voice: Calm male voice

[1231] Speed: 150

[1232] Writing style: Polite

[1233] Emotion: Joy

[1234] Text:This is a sample audiobook text.

[1235] These prompts are then fed into a generative AI model that can generate voice output tailored to the user's preferences and emotions, providing a natural and personalized voice experience.

[1236] The hardware used includes servers and user devices (smartphones, tablets, PCs, etc.). The software used includes generative AI models, emotion recognition engines, and voice property setting modules. Specific software examples include GPT-3 for generative AI models, EmotionRecognizer for emotion recognition engines, and pyttsx3 for voice property setting.

[1237] The flow of the identification process in the application example 2 will be described with reference to FIG.

[1238] Step 1:

[1239] The server receives various video data as input. This video data includes video information from various fields such as sports, entertainment, and education. The input video data is converted into an appropriate format within the server and passed to the generative AI model.

[1240] Step 2:

[1241] The server uses a generative AI model to generate live commentary and subtitles based on the input video data. The generative AI model extracts important information from the video data and generates audio and text based on that information. The generated audio and text are then sent to the user's device.

[1242] Step 3:

[1243] The user sets his / her preferences through the terminal. Specifically, the user selects the tone of voice, speed, writing style, etc. This setting information is sent from the terminal to the server.

[1244] Step 4:

[1245] The server sets the voice properties based on the user's settings, specifically, by using a voice property setting module such as pyttsx3 to adjust the tone and speed of the generated voice, and the adjusted voice is sent to the user's device.

[1246] Step 5:

[1247] The user expresses his / her emotions through the terminal. The emotions are recognized from the user's facial expressions, tone of voice, text, etc. The terminal analyzes the user's emotions using an emotion recognition engine and transmits the results to the server.

[1248] Step 6:

[1249] The server adjusts the generated voice based on the emotion information received from the emotion recognition engine. Specifically, if the user is expressing a happy emotion, the tone of the voice is made brighter and the speed is made faster. Conversely, if the user is expressing a sad emotion, the tone of the voice is made calmer and the speed is made slower. This adjusted voice is sent to the user's device.

[1250] Step 7:

[1251] Users receive tailored audio through their devices, providing a natural and personalized audio experience, with audio output tailored to the user's preferences and emotions, resulting in a more satisfying experience.

[1252] Example 3

[1253] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1254] Conventional live broadcast and subtitle generation systems have difficulty in flexibly adjusting to the user's preferences and emotions, and they are insufficient in providing information in a language that is easy for the user to understand. In addition, since information is not provided based on the user's emotions, the user experience is limited. As a result, it is not possible to provide the optimal information that users desire.

[1255] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1256] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means including a generative AI model for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generative AI model to suit the user's preferences, a means for translating into the user's native language, a means for detecting the user's emotions, and a means for adjusting the text expression based on the user's emotions. This makes it possible to provide optimal information according to the user's preferences and emotions.

[1257] "Various sports and entertainment video data" refers to digital data that includes visual information of sporting events and entertainment content.

[1258] A "generative AI model" is an artificial intelligence algorithm that automatically generates text or speech based on input data.

[1259] "Means for tailoring to user preferences" refers to functionality that allows users to customize the generated content based on their individual tastes and requirements.

[1260] A "means of translation into the user's native language" is a feature that translates generated content into a language the user is most comfortable with.

[1261] "Means for detecting user emotions" refers to technology that analyzes the user's facial expressions, tone of voice, etc. to identify their emotional state at that time.

[1262] The "means for adjusting the expression of text based on the user's emotions" is a function for changing the expression method of generated content according to the detected emotions of the user.

[1263] This invention is a system that generates live coverage and subtitles according to the user's preferences and emotions. The system operates in cooperation with a server, a terminal, and a user.

[1264] Hardware and Software Used

[1265] Hardware: Servers, terminals (PCs, smartphones, tablets, etc.)

[1266] Software: generative AI models (e.g. GPT-4), emotion engines, translation software (e.g. Google Translate API)

[1267] System Operation Overview

[1268] 1. Receiving user input

[1269] A user uses the device to request live commentary or subtitle generation by entering text into the device's input field and pressing the send button.

[1270] Example: A user types, "Please give me a live broadcast of the soccer game in Japanese."

[1271] 2. Content generation using generative AI models

[1272] The server receives the user's request and sends a prompt to a generative AI model (e.g. GPT-4), which generates appropriate content based on the prompt.

[1273] Example: The server sends GPT-4 the prompt "Generate a live broadcast of a soccer match," and GPT-4 generates the sentence "The soccer match has started. The players are on the field, and the whistle has blown."

[1274] 3. Translation tailored to user preferences

[1275] The server translates the generated content into the user's native language. Translation software (e.g. Google Translate API) is used to translate the generated English text into Japanese.

[1276] Example: The server translates "The soccer match has started. The players are on the field, and the whistle has blown." to "The soccer match has started. The players are on the field, and the whistle has blown."

[1277] 4. Detect user emotions

[1278] To detect the user's emotions, the device uses the camera and microphone to analyze the user's facial expressions and tone of voice. The emotion engine analyzes this data to identify the user's emotions.

[1279] Example: The device captures a user's facial expressions with its camera, and the emotion engine detects that the user has a surprised expression.

[1280] 5. Adjusting sentence expression based on emotions

[1281] The server receives the results of the emotion engine and adjusts the generated content based on the user's emotion, for example, making the content more detailed if the user expresses surprise, and simpler if the user expresses relief.

[1282] Example: The server adjusts "The soccer game has started. The players are on the field and the whistle has blown to start the game." to "A player scores a goal! The crowd is cheering!"

[1283] 6. Delivery of Final Content

[1284] The server sends the tailored content to the user's device, where it is viewed by the user. The device displays the received content, and the user confirms it.

[1285] Example: A detailed live broadcast in Japanese is displayed on the user's device.

[1286] Examples of concrete examples and prompts

[1287] Example: A user requests a live soccer game broadcast in Japanese and expresses surprise.

[1288] User input: "Please provide live commentary of soccer games in Japanese."

[1289] Output of the generative AI model: "The soccer match has started. The players are on the field, and the whistle has blown."

[1290] Translation software translates: "The soccer game has begun. The players are on the field and the whistle has blown to start the game."

[1291] Emotion engine tuning: "The player scores a goal! The crowd is cheering!"

[1292] Example prompt:

[1293] "Generate a live broadcast of a soccer game in Japanese. The user is showing surprise emotion."

[1294] In this way, the system can provide optimal live coverage and subtitles according to the user's preferences and feelings. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1295] Step 1:

[1296] A user uses a terminal to request live broadcasting and subtitle generation. Specifically, the user enters text into the input field of the terminal and presses the send button. For example, the user enters "Please broadcast the live soccer game in Japanese." The terminal then sends the user's request to the server.

[1297] Step 2:

[1298] The server receives the user's request and sends a prompt to the generative AI model. The input contains the user's request. The server sends the prompt "Please generate a live broadcast of a soccer match" to the generative AI model. The generative AI model generates appropriate content based on the prompt, and generates the sentence "The soccer match has started. The players are on the field, and the whistle has blown." as the output.

[1299] Step 3:

[1300] The server translates the generated content into the user's native language. The input includes an English sentence that is the output from the generative AI model. The server uses translation software (e.g. Google Translate API) to translate "The soccer match has started. The players are on the field, and the whistle has blown." to "The soccer match has started. The players are on the field, and the whistle has blown.". The output is the translated Japanese sentence.

[1301] Step 4:

[1302] To detect the user's emotions, the device uses a camera and microphone to analyze the user's facial expressions and tone of voice. Inputs include real-time video and audio data of the user. The emotion engine analyzes these data to identify the user's emotions. The output is that the user is showing an emotion of surprise.

[1303] Step 5:

[1304] The server receives the results of the emotion engine and adjusts the generated content based on the user's emotions. The input includes the translated Japanese text and the emotion engine results. The server adjusts the text to include more details if the user indicates surprise, and to a simpler text if the user indicates relief. The output is the adjusted text: "The player scored a goal! The crowd is cheering!"

[1305] Step 6:

[1306] The server sends the adapted content to the user's device, where the user views it. The input includes adapted Japanese text. The device displays the received content, which the user confirms. The output is a detailed live report in Japanese on the user's device.

[1307] (Application example 3)

[1308] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1309] Conventional live broadcasting and subtitle generation systems do not support the user's native language, which can make it difficult for users to understand. In addition, they are unable to provide information according to the user's emotions, resulting in a suboptimal user experience. This can result in a decrease in the user's understanding of and satisfaction with the content they are watching.

[1310] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1311] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means for converting into the user's native language, and a means for adjusting the text expression based on the user's emotions. This allows the user to receive live coverage and subtitles in real time that are easy to understand in his or her native language, and further allows the provision of optimal information according to the user's emotions.

[1312] "Video data" is data in digital format that contains visual information such as sports and entertainment.

[1313] "Generative AI" is an artificial intelligence technology that automatically generates live commentary and subtitles based on input data.

[1314] "User Preferences" refers to the individual tastes and settings that a User has regarding the Content they view.

[1315] "Native language" refers to the language that a user understands most naturally.

[1316] "Emotion" refers to the psychological response or state a user exhibits while watching.

[1317] "Writing" refers to the content and style of the generated text.

[1318] A system for implementing the present invention is configured as follows.

[1319] System Configuration

[1320] Hardware

[1321] Servers: Servers with powerful computing power are required, including GPUs and CPUs to run generative AI models.

[1322] Devices: Includes mobile devices such as smartphones and tablets used by users.

[1323] software

[1324] Generative AI models: For example, using advanced natural language generation models such as GPT-4.

[1325] Emotion recognition engine: As an example, we use the Microsoft Azure Emotion API.

[1326] Speech Recognition API: As an example, we will use the Google Speech-to-Text API.

[1327] Translation API: As an example, we will use the Google Translate API.

[1328] Data processing and calculation

[1329] Server Processing

[1330] 1. Getting voice input:

[1331] The server receives the audio of the video data sent from the device in real time and converts the audio into text using a speech recognition API.

[1332] 2. Emotion recognition:

[1333] The server uses an emotion recognition engine to analyze the user's facial expressions and voice data sent from the device and identify the user's emotions.

[1334] 3. Text Generation:

[1335] The server uses a generative AI model to generate commentary and subtitles based on the acquired text, and adjusts the level of detail and expression of the generated text based on the results of emotion recognition.

[1336] 4. Translation:

[1337] The server uses a translation API to translate the generated text into the user's native language.

[1338] Terminal handling

[1339] 1.Display:

[1340] The terminal displays the translated text sent from the server on the screen in real time.

[1341] Examples

[1342] If the user is Japanese and is watching an English live stream, the system will work as follows:

[1343] 1. Capture the audio of the live stream and convert it to text using a speech recognition API.

[1344] 2. The emotion recognition engine detects the emotion of "surprise" from the user's facial expressions and voice.

[1345] 3. Generative AI models generate detailed commentary and subtitles.

[1346] 4. The translation API translates the generated text into Japanese.

[1347] 5. The translated Japanese text will appear on your smartphone screen.

[1348] Examples of prompt statements

[1349] If the user expresses surprise emotion, generate a detailed play-by-play. Translate the following English text into Japanese and generate a play-by-play with a detailed explanation:

[1350] In this way, users can receive the most relevant information in real time, in their native language, and according to their emotions.

[1351] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1352] Step 1:

[1353] Acquiring voice input

[1354] The server captures the audio of the video data sent from the device in real time. Specifically, it uses the device's microphone to capture the audio of the live stream and sends the audio data to the server. The input is the audio data, and the output is the audio data passed to the speech recognition API.

[1355] Step 2:

[1356] Speech to text conversion

[1357] The server uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the acquired voice data into text. The input is voice data, and the output is text data. Specifically, the server sends voice data to the speech recognition API and receives the returned text data.

[1358] Step 3:

[1359] emotion recognition

[1360] The server analyzes the user's facial and voice data sent from the device using an emotion recognition engine (e.g. Microsoft Azure Emotion API) to identify the user's emotions. The input is the user's facial and voice data, and the output is emotion data. Specifically, the server sends data to the emotion recognition engine and receives the returned emotion data.

[1361] Step 4:

[1362] Text Generation

[1363] The server uses a generative AI model (e.g., GPT-4) to generate live commentary and subtitles based on the acquired text data. Based on the results of emotion recognition, it adjusts the level of detail and expression of the generated text. The input is text data and emotion data, and the output is the generated live commentary and subtitle text data. Specifically, it sends a prompt sentence to the generative AI model and receives the returned text data.

[1364] Step 5:

[1365] translation

[1366] The server uses a translation API (e.g., Google Translate API) to translate the generated text data into the user's native language. The input is the generated text data, and the output is the translated text data. Specifically, the server sends text data to the translation API and receives the translated text in return.

[1367] Step 6:

[1368] display

[1369] The terminal displays the translated text data sent from the server on the screen in real time. The input is the translated text data, and the output is the text displayed on the terminal screen. Specifically, a UI component for displaying the text data on the terminal display is used.

[1370] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1371] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1372] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1373] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1374] [Third embodiment]

[1375] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1376] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1377] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1378] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1379] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1380] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1381] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1382] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1383] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1384] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1385] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1386] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1387] "Example 1"

[1388] One embodiment of the present invention is a system that inputs video data of various sports and entertainment. This system includes a generation AI that generates live broadcasts and subtitles based on the video data. The generation AI automatically generates live broadcasts and subtitles from the video data using AI technology such as deep learning.

[1389] "Example 2"

[1390] Furthermore, it includes a means for adjusting the output of the generating AI to suit the user's preferences. Specifically, it allows the user to set the tone of voice, speed, writing style, etc. that they prefer. For example, the user can select a "calm male voice" or an "exciting female voice," as well as a writing style such as "polite expression" or "casual expression."

[1391] "Example 3"

[1392] The AI ​​generator also has the ability to convert the content into the user's native language according to their preferences. For example, if the user is Japanese, the AI ​​generator will convert the generated live broadcast and subtitles into Japanese. This allows users to enjoy live broadcasts and subtitles in a language they can easily understand.

[1393] The process flow of each embodiment will be described below.

[1394] "Example 1"

[1395] Step 1: Users input the sports or entertainment video data they want to watch into the system.

[1396] Step 2: The system's generation AI analyzes the video data and generates live commentary and subtitles. At this time, the AI ​​understands the content of the video data and generates appropriate live commentary and subtitles.

[1397] Step 3: The generated commentary and subtitles are displayed to the user.

[1398] "Example 2"

[1399] Step 1: The user provides the system with their preferred voice tone, speed and writing style.

[1400] Step 2: The AI ​​generator adjusts the tone, speed, and writing style of the commentary and subtitles it generates based on the user's settings.

[1401] Step 3: The adjusted commentary and subtitles are then displayed to the user.

[1402] "Example 3"

[1403] Step 1: The user sets the preferred language on the system.

[1404] Step 2: The generation AI converts the live commentary and subtitles into the specified language based on the user's settings.

[1405] Step 3: The converted live commentary and subtitles will be displayed to users.

[1406] Example 1

[1407] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1408] Conventional live broadcast and subtitle generation systems based on video data require manual editing, making it difficult to adjust in real time and to meet user preferences. In addition, the analysis of video data and the quality of the generated content often lacked consistency. This resulted in a poor user experience and made it difficult to generate content efficiently.

[1409] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1410] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means for dividing the video data into frames, a means for inputting the divided frames to a generative AI model, a means for detecting important events and actions from the video data using the generative AI model and generating live coverage and subtitles, a means for overlaying the generated live coverage and subtitles on the video data, and a means for adjusting the output of the generative AI to suit the user's preferences. This enables automatic generation of high-quality live coverage and subtitles in real time, and allows easy adjustment according to the user's preferences.

[1411] "Video data" is data in digital format that contains visual information such as sports and entertainment.

[1412] A "frame" is an individual still image that makes up video data.

[1413] A "generative AI model" is an artificial intelligence algorithm that uses deep learning technology to detect important events and actions from video data and generate live commentary and subtitles.

[1414] "Significant Events and Actions" refers to occurrences or movements that are of particular interest within sports or entertainment video data.

[1415] "Live coverage" refers to commentary and comments given in real time to video data.

[1416] "Subtitles" are text information that is displayed overlaid on video data.

[1417] "User preferences" refers to individual settings such as the user's desired voice adjustments or writing style.

[1418] "Means of adjustment" refers to functions or methods for modifying the output of a generative AI model to suit the user's preferences.

[1419] The "server" is a computer system that performs processes such as analyzing video data, running generative AI models, and overlaying generated live coverage and subtitles.

[1420] This invention is a system that automatically generates live commentary and subtitles using video data of various sports and entertainment. The system includes a generative AI model and uses deep learning technology to generate live commentary and subtitles from the video data.

[1421] Hardware and Software Used

[1422] Server: A computer system that performs processes such as analyzing video data, running generative AI models, and overlaying generated live coverage and subtitles.

[1423] Terminal: A device on which a user uploads video data and views the processed video. Specifically, this applies to computers and smartphones.

[1424] Generative AI models: These are built using deep learning frameworks such as TensorFlow and PyTorch.

[1425] Data processing and calculation

[1426] 1. Input of video data

[1427] Users upload sports and entertainment video data to the system.

[1428] The terminal transmits the uploaded video data to the server.

[1429] 2. Analysis of video data

[1430] The server divides the received video data into frames. To do this, it uses a video processing library such as FFmpeg.

[1431] The server inputs the segmented frames into a generative AI model.

[1432] 3. Generative AI Processing

[1433] The generative AI model uses deep learning techniques to detect important events and actions in video data.

[1434] The generative AI model generates appropriate commentary and subtitles based on the detected events.

[1435] 4. Output of results

[1436] The server overlays the generated commentary and subtitles onto the video data, specifically by using FFmpeg to overlay the text onto the video.

[1437] The server transmits the processed video data to the terminal.

[1438] Users can watch live broadcasts and videos with added subtitles through their terminals.

[1439] Examples

[1440] For example, consider the case where a user uploads a video of a soccer match to the system. The server divides the video data into frames and inputs them into the generative AI model. The generative AI model detects goal and foul scenes, and generates commentary and subtitles such as "Goal!" and "A foul has been committed" based on the detected scenes. The server overlays the generated commentary and subtitles onto the video and sends it to the device. The user can then watch the soccer match video with the commentary and subtitles added via the device.

[1441] Examples of prompt statements

[1442] "Upload a video of a soccer game. Our generative AI will detect goals and fouls, and automatically generate commentary and subtitles."

[1443] This system allows users to enjoy high-quality live broadcasts and videos with added subtitles in real time.

[1444] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1445] Step 1:

[1446] Users upload sports and entertainment video data to the system.

[1447] Input: A video file selected by the user.

[1448] Specific actions: A user opens the system's web interface, selects a video file, and clicks the upload button.

[1449] Output: Video data is sent from the device to the server.

[1450] Step 2:

[1451] The terminal transmits the uploaded video data to the server.

[1452] Input: Video data uploaded by the user.

[1453] Specific operation: The device uploads video data to the server using an HTTP POST request.

[1454] Output: The server receives the video data.

[1455] Step 3:

[1456] The server divides the received video data into frames.

[1457] Input: Video data received by the server.

[1458] Specific operation: The server uses a video processing library such as FFmpeg to divide the video data into frames. Specifically, it executes FFmpeg commands to divide the video into frames.

[1459] ffmpeg -i input.mp4 -vf "fps=30" frame_%04d.png

[1460] Output: Segmented frame images.

[1461] Step 4:

[1462] The server inputs the segmented frames into a generative AI model.

[1463] Input: Segmented frame images.

[1464] Specific operation: The server converts the frame images into an appropriate format and inputs them into the generative AI model.

[1465] Output: The frame data fed into the generative AI model.

[1466] Step 5:

[1467] The generative AI model uses deep learning techniques to detect important events and actions in video data.

[1468] Input: The frame data fed into the generative AI model.

[1469] How it works: The generative AI model analyzes frame data to detect important events and actions, for example, goals and fouls in a soccer game.

[1470] Output: Information about the detected event and / or action.

[1471] Step 6:

[1472] The generative AI model generates appropriate commentary and subtitles based on the detected events.

[1473] Input: Information about the detected event or action.

[1474] How it works: Based on the detected events, the generative AI model generates appropriate text, such as commentary or subtitles like "Goal!" or "A foul has been committed."

[1475] Output: The generated play-by-play and subtitle text.

[1476] Step 7:

[1477] The server overlays the generated live coverage and subtitles onto the video data.

[1478] Input: The generated play-by-play or subtitle text.

[1479] What it does: The server uses FFmpeg to overlay the generated text onto the video.

[1480] ffmpeg -i input.mp4 -vf "drawtext=text='Goal!':x=10:y=10:fontsize=24:fontcolor=white" output.mp4

[1481] Output: Video data with live coverage and subtitles overlaid.

[1482] Step 8:

[1483] The server transmits the processed video data to the terminal.

[1484] Input: Video data with commentary and subtitles overlaid.

[1485] Specific operation: The server sends the processed video file to the terminal as an HTTP response.

[1486] Output: The device receives the processed video data.

[1487] Step 9:

[1488] Users can watch live broadcasts and videos with added subtitles through their terminals.

[1489] Input: The processed video data received by the device.

[1490] Specific behavior: The user uses the device's media player to play live broadcast or video with added subtitles.

[1491] Output: The video the user sees.

[1492] (Application example 1)

[1493] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1494] In conventional sports and entertainment video distribution systems, real-time live broadcasts and subtitle generation were often done manually, making it difficult to provide information quickly and accurately. In addition, it was difficult to customize to meet user preferences, and support for multiple languages ​​was limited. This led to problems such as reduced viewer satisfaction.

[1495] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1496] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means for displaying the generated live coverage and subtitles in real time, and a means for providing live coverage and subtitles in multiple languages. This enables accurate generation of live coverage and subtitles in real time, and allows customization to suit the user's preferences and support in multiple languages.

[1497] "Various sports and entertainment video data" refers to video information of sporting events and entertainment content, including live and recorded footage.

[1498] "Input means" refers to a method or device for inputting video data into the system.

[1499] "Generative AI" refers to algorithms and models that use artificial intelligence techniques such as deep learning to automatically generate live commentary and subtitles from video data.

[1500] "Means for tailoring to user preferences" refers to a method or apparatus for customizing the content or format of the generated commentary or subtitles based on a user's settings or preferences.

[1501] "Means for displaying in real time" refers to a method or apparatus for displaying the generated commentary or subtitles on a user's device immediately and without delay.

[1502] "Means for providing commentary and subtitles in multiple languages" refers to a method or apparatus for providing the generated commentary and subtitles in multiple different languages.

[1503] The system for carrying out the present invention inputs video data of various sports and entertainment, and automatically generates live broadcasts and subtitles using a generation AI. A specific embodiment of this system will be described below.

[1504] System Configuration

[1505] The system mainly uses the following hardware and software:

[1506] Hardware: Smartphone (iOS or Android), server

[1507] Software: OpenCV, TensorFlow, Transformers (Hugging Face)

[1508] Data processing and calculation

[1509] Video data input

[1510] The server receives video data of various sports and entertainment events, either live or recorded.

[1511] Video data pre-processing

[1512] The server uses OpenCV to split the video data into frames and perform pre-processing, which includes frame resizing and noise removal.

[1513] Analysis by generative AI

[1514] The server analyzes the preprocessed frames using TensorFlow and Transformers. Specifically, a generative AI model (e.g., GPT-2) is used to generate commentary and subtitles from the video data. The generative AI model takes the prompt as input and outputs appropriate commentary and subtitles.

[1515] Tailored to your preferences

[1516] The server then tailors the generated commentary and subtitles to the user's preferences, including customizing the tone of voice and writing style.

[1517] Real-time display

[1518] The device (smartphone) displays the generated live coverage and subtitles in real time, allowing users to receive information without delay.

[1519] Multiple Languages

[1520] The server provides the generated commentary and subtitles in multiple languages, allowing users who speak different languages ​​to enjoy the same content.

[1521] Examples

[1522] For example, when a goal is scored during a soccer game, the server inputs the prompt sentence "Describe the current sports event: A player is running towards the goal with the ball" into the generative AI model. Based on this, the generative AI model generates a live report and subtitles such as "Goal! Player A scores a great shot!" and displays them on the device in real time.

[1523] In this way, a system can be realized that generates live commentary and subtitles for sporting events in real time.

[1524] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1525] Step 1:

[1526] The server receives video data of various sports and entertainment events. The video data is provided in the form of live or recorded video. The input video data is prepared for processing within the server.

[1527] Step 2:

[1528] The server splits the video data into frames using OpenCV and performs preprocessing. Preprocessing includes frame resizing and noise removal. Specifically, the video data is captured at a constant frame rate and each frame is resized to a standardized size. The input is the video data and the output is the preprocessed frames.

[1529] Step 3:

[1530] The server analyzes the preprocessed frames using TensorFlow and Transformers. Specifically, a generative AI model (e.g., GPT-2) is used to generate commentary and subtitles from the video data. The generative AI model takes a prompt as input and outputs appropriate commentary and subtitles. The input is the preprocessed frames and the prompt, and the output is the generated commentary and subtitles.

[1531] Step 4:

[1532] The server adjusts the generated commentary and subtitles to suit the user's preferences. The adjustments include customizing the tone of voice and sentence expressions. Specifically, the server refers to the user's settings information and converts the generated text appropriately. The input is the generated commentary and subtitles, and the output is the adjusted commentary and subtitles.

[1533] Step 5:

[1534] The device (smartphone) displays the generated live broadcast and subtitles in real time, allowing the user to receive information without delay. Specifically, text is overlaid on the device's display. The input is the adjusted live broadcast and subtitles, and the output is the text displayed on the device's display.

[1535] Step 6:

[1536] The server provides the generated commentary and subtitles in multiple languages, allowing users who speak different languages ​​to enjoy the same content. Specifically, a generative AI model is used to translate the original text into other languages. The input is the generated commentary and subtitles, and the output is the commentary and subtitles in multiple languages.

[1537] Example 2

[1538] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1539] Conventional generative AI systems have had difficulty providing output that meets individual user preferences. In particular, they were unable to accommodate detailed settings that differ from user to user, such as tone of voice, speed, and writing style. As a result, the user experience was uniform and insufficient to customize to meet individual needs.

[1540] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1541] In this invention, the server includes a means for inputting various data, a means including a generative AI that generates information based on the data, a means for adjusting the output of the generative AI based on the user's preferences, a means for acquiring the user's settings, a means for generating a prompt sentence to be input to the generative AI model, a means for executing the generative AI model, a means for adjusting the output obtained from the generative AI model, and a means for transmitting the adjusted output to the user's terminal, thereby enabling customized output according to the individual preferences of the user.

[1542] "Various types of data" refers to data in various formats, including information entered by the user and information obtained by the system.

[1543] "Generative AI" refers to artificial intelligence that generates information based on input data.

[1544] "User preferences" refers to individual settings such as tone of voice, speed, writing style, etc. set by the User.

[1545] "Means for acquiring user settings" refers to a function that enables the system to acquire preference information that the user has set in advance.

[1546] A "prompt sentence" is an instruction sentence to be input into a generative AI model.

[1547] "Means for executing a generative AI model" refers to a function for inputting a prompt sentence into the generative AI model and generating information.

[1548] "Means for adjusting the output obtained from the generative AI model" refers to a function for adjusting the information obtained from the generative AI model based on user preferences.

[1549] A "means for transmitting the conditioned output to a user's terminal" is a function for transmitting the conditioned information to a user's terminal.

[1550] The present invention relates to a generative AI system that provides customized output according to user preferences. A specific embodiment of the system will be described below.

[1551] System Configuration

[1552] Hardware and Software

[1553] The server runs the system using the following hardware and software:

[1554] Hardware: High-performance server computer

[1555] Software: Generative AI models (e.g., GPT-3), text-to-speech software (e.g., Amazon Polly or Google Text-to-Speech)

[1556] Program Processing

[1557] Retrieving user settings

[1558] The server retrieves information about the user's predefined preferences for tone of voice, speed, writing style, etc. This information is loaded from the user's profile when they log into the system.

[1559] Example: If a user selects "calm male voice" and "polite speech," the server retrieves these preferences from a database.

[1560] Generate prompt statement

[1561] The server generates prompts based on the user's requests to feed into the generative AI model, which is then tailored to obtain an output that is tailored to the user's preferences.

[1562] Example: If a user requests, "Please explain how the generative AI model works," the server generates a prompt that reads, "Please explain the following in a calm, male voice using polite language: How the generative AI model works."

[1563] Running generative AI models

[1564] The server inputs a prompt sentence using a generative AI model (e.g., GPT-3) and generates an output according to the user's request.

[1565] Example: The server sends the generated prompt text to GPT-3, which generates an explanation of "how the generative AI model works."

[1566] Adjusting the Output

[1567] The server then adjusts the output of the generative AI model based on the user's settings, using voice synthesis software (e.g., Amazon Polly or Google Text-to-Speech) to generate speech at the user's selected tone and speed.

[1568] Example: The server sends the text obtained from GPT-3 to Amazon Polly, which generates a speech in a "calm male voice."

[1569] Send to device

[1570] The server sends the adjusted output to the user's device, where the user can view the audio and text.

[1571] Example: The server sends the generated audio file to the user's smartphone, allowing the user to play it.

[1572] Adding concrete examples

[1573] Here is a concrete example of what happens when a user selects "exciting female voice" and "casual expression."

[1574] Example prompt: "In an exciting female voice, use casual language to explain: How does a generative AI model work?"

[1575] Example output from a generative AI model: "Hey guys! Today we're going to talk about generative AI models. They're super cool!"

[1576] Speech synthesis example: Using Google Text-to-Speech, convert the above text into an "exciting female voice."

[1577] In this way, the system provides generative AI output tailored to the user's preferences.

[1578] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1579] Step 1: Get user settings

[1580] The server retrieves information such as the user's predefined preferred tone of voice, speed, writing style, etc. When a user logs into the system, the server reads the user's profile information from the database and retrieves the user's preferred settings.

[1581] Input: User login information

[1582] Data processing: Search for user settings information from the database

[1583] Output: User preferences

[1584] Step 2: Generate a prompt

[1585] The server generates a prompt sentence to be input to the generative AI model based on the user's request. The server combines the user's request content and the acquired settings to generate a prompt sentence to be input to the generative AI model.

[1586] Input: User request, user preference settings

[1587] Data processing: Generate prompts by combining the request content and configuration information

[1588] Output: The generated prompt statement

[1589] Step 3: Running the generative AI model

[1590] The server inputs a prompt sentence using a generative AI model (e.g., GPT-3) and generates an output according to the user's request. It sends the generated prompt sentence to the generative AI model and receives the generated text.

[1591] Input: Generated prompt text

[1592] Data calculation: Enter a prompt into the generative AI model to generate text

[1593] Output: The generated text

[1594] Step 4: Adjust the output

[1595] The server then adjusts the output of the generative AI model based on the user's settings, using voice synthesis software (e.g., Amazon Polly or Google Text-to-Speech) to generate speech at the user's selected tone and speed.

[1596] Input: Generated text, user preferences and settings

[1597] Data processing: Input text into speech synthesis software to generate speech

[1598] Output: Adjusted audio file

[1599] Step 5: Send to device

[1600] The server sends the adjusted output to the user's device, and the generated audio file is sent to the user's smartphone so that the user can play it.

[1601] Input: Adjusted audio file

[1602] Data processing: Send audio files to the user's device

[1603] Output: An audio file that can be played on the user's device.

[1604] In this way, the system provides generative AI output tailored to the user's preferences.

[1605] (Application example 2)

[1606] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1607] In conventional content delivery systems, it was difficult to adjust voice tone, speed, and writing style according to individual user preferences. This resulted in problems in which users could not watch or listen to content more comfortably. In particular, there is a demand for audio output that matches the user's preferences for content such as news articles, audiobooks, and podcasts, but it was difficult to achieve this with current technology.

[1608] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1609] In this invention, the server includes a means for inputting various video data, a means including a generation AI for generating voice output based on the video data, a means for adjusting the output of the generation AI according to the user's preferences, a means for allowing the user to set the tone of voice, speed, and style of writing that they prefer, a means for inputting a prompt sentence into the generation AI model, and a means for outputting the generated text by voice, which makes it possible to adjust the tone of voice, speed, and style of writing according to the individual preferences of the user.

[1610] "Video data" refers to digital data that contains visual information, such as videos and images.

[1611] "Generative AI" is a system that uses artificial intelligence technology to generate new output from input data.

[1612] "User Preferences" refers to the individual preferences of each individual User, including the particular tone of voice, speed, writing style, etc.

[1613] "Tone of voice" refers to the pitch and texture of the voice, and includes characteristics such as a calm voice or an exciting voice.

[1614] "Speed" refers to the speed at which audio is played back, usually expressed as the number of words per unit time.

[1615] "Writing style" refers to the way a sentence is written and the characteristics of the language used, and includes polite and casual expressions.

[1616] A "prompt sentence" is text data input into a generative AI and refers to the sentence that forms the basis of the generated output.

[1617] "Voice output" refers to playing back the text generated by the generation AI as audio.

[1618] A system for implementing this invention includes means for receiving various types of video data as input, means including a generation AI that generates audio output based on the video data, means for adjusting the output of the generation AI to suit the user's preferences, means for allowing the user to set the tone, speed, and style of voice expression preferred by the user, means for inputting a prompt sentence into the generation AI model, and means for audio output of the generated text.

[1619] Explanation of system program processing

[1620] The server first receives various video data as input. This video data includes content such as news articles, audiobooks, and podcasts. It then uses a generative AI to generate audio output based on this video data. For example, the generative AI uses the "pipeline" function from the "transformers" library.

[1621] Users can set their preferred voice tone, speed, and writing style through their smartphones or other devices. For this, the "pyttsx3" library is used as the voice engine. The output of the generative AI is adjusted based on the user's settings. Specifically, users can select a "calm male voice" or an "exciting female voice," as well as a writing style such as "polite expression" or "casual expression."

[1622] As a means of inputting prompt sentences into the generative AI model, the text data entered by the user is passed to the generative AI. For example, "News article: The weather is sunny today" is input as a prompt sentence for a news article. Based on this prompt sentence, the generative AI generates sentences that match the user's preferences.

[1623] The generated text is output as a voice using a voice engine. The voice engine reads the generated text aloud, adjusting the tone and speed of the voice based on the user's settings. This allows users to enjoy content in a voice that matches their preferences.

[1624] Examples

[1625] For example, if a user prefers a "calm male voice" and "polite expressions," the following prompt sentence can be input into the generative AI model.

[1626] Prompt: "News article: The weather is sunny today."

[1627] Based on this prompt, the generative AI model generates sentences that match the user's preferences, such as, "It's sunny today and the temperature is very comfortable. I hope you all have a wonderful day."

[1628] In this way, it is possible to adjust the tone, speed and writing style of speech to suit the user's individual preferences.

[1629] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1630] Step 1:

[1631] The server receives various types of video data as input, including content such as news articles, audiobooks, podcasts, etc. The input video data is stored in a database.

[1632] Step 2:

[1633] The server uses a generative AI to generate audio output based on the input video data. Specifically, it inputs a prompt sentence into a generative AI model (e.g., GPT-3) and obtains the generated text. This prompt sentence summarizes the content of the video data.

[1634] Step 3:

[1635] Users can set their preferred voice tone, speed, and writing style through the device. For example, they can use a smartphone application to select a "calm male voice" or an "exciting female voice," or "polite expressions" or "casual expressions."

[1636] Step 4:

[1637] The server adjusts the output of the generation AI based on the user's settings. Specifically, it uses a voice engine (e.g., pyttsx3) to generate voice output that matches the tone and speed of voice set by the user.

[1638] Step 5:

[1639] The server inputs a prompt sentence into the generative AI model. For example, it inputs "News article: Today's weather is sunny" as a prompt sentence for a news article. Based on this prompt sentence, the generative AI generates a sentence that matches the user's preferences.

[1640] Step 6:

[1641] The server outputs the generated text using a voice engine. The voice engine reads the generated text aloud, adjusting the tone and speed of the voice based on the user's settings. This allows the user to enjoy the content in a voice that matches their preferences.

[1642] Example 3

[1643] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1644] Conventional live broadcast and subtitle generation systems have difficulty converting the language into the user's native language in real time, which means that users cannot enjoy content in a language that is easy to understand. In addition, they do not adequately adjust to the user's preferences, making it impossible to provide an optimal experience for each individual user.

[1645] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1646] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means for receiving a prompt text input by the user, a means for acquiring information on the user's native language, a means for converting the generated text into the user's native language, and a means for transmitting the converted text to the user's terminal. This allows the user to enjoy live coverage and subtitles in real time in a language that is easy for the user to understand.

[1647] "Video data" is data in digital format that contains visual information such as sports and entertainment.

[1648] "Generative AI" is an artificial intelligence model that generates text or speech based on input data.

[1649] A "prompt" is text that contains instructions or requests that a user enters into a system.

[1650] "User's native language information" is information about the language the user is most comfortable understanding.

[1651] "Text conversion means" means a function or process for converting the generated text into the user's native language.

[1652] A "terminal" is a device used by a user, such as a computer, smartphone, or tablet.

[1653] An "HTTP request" is a communication protocol for a client to request data from a server.

[1654] An "HTTP response" is a communication protocol that enables a server to return data to a client.

[1655] "Translation API" means an application programming interface for automatically translating text into different languages.

[1656] This invention relates to a system that allows users to enjoy live broadcasts and subtitles in real time in a language that is easy for users to understand. The system takes video data of various sports and entertainment as input, generates live broadcasts and subtitles using a generative AI model, and has the ability to adjust them to suit the user's preferences.

[1657] Hardware and Software Used

[1658] Hardware

[1659] Server: A central processing unit for running the generative AI model, obtaining the user’s native language information, and converting the generated text.

[1660] Terminal: A device on which a user enters prompts and displays the converted text. Examples include computers, smartphones, and tablets.

[1661] software

[1662] Generative AI models: For example, using advanced natural language processing models such as GPT-4.

[1663] Translation API: For example, use the Google Translate API to translate the generated text into the user's native language.

[1664] Data processing and calculation

[1665] 1. Receiving a prompt: The user enters a prompt into the input field of the terminal. For example, the user enters "Please provide a commentary of this game in Japanese."

[1666] 2. Sending the prompt: The terminal sends the entered prompt to the server as an HTTP request.

[1667] 3. Get native language information: The server gets the user's profile information from the database and checks what the user's native language is.

[1668] 4. Applying the generative AI model: The server uses a generative AI model (GPT-4) to generate commentary and subtitle text based on the prompt text.

[1669] 5. Text conversion: The server converts the generated text into the user's native language using a translation API (Google Translate API).

[1670] 6. Sending the converted text: The server sends the converted text to the terminal as an HTTP response.

[1671] 7. Displaying text: The device will display the received converted text on the screen, allowing users to enjoy live commentary and subtitles in Japanese.

[1672] Examples

[1673] For example, suppose a user is watching a soccer game. The user enters a prompt sentence such as "Please provide commentary on this game in Japanese." The device sends this prompt sentence to the server, and the server uses a generative AI model to generate commentary text. The server then converts the generated text into Japanese using a translation API and sends the converted text to the device. Finally, the device displays the converted commentary on the screen, allowing the user to enjoy the commentary in Japanese.

[1674] In this way, the user can enjoy live coverage and subtitles in real time in a language that the user can easily understand. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1675] Step 1:

[1676] The user enters a prompt sentence.

[1677] The user inputs a prompt sentence into the input field of the terminal, such as "Please provide a commentary of this game in Japanese." The input prompt sentence is temporarily stored in the memory of the terminal.

[1678] Step 2:

[1679] The terminal sends a prompt to the server.

[1680] The terminal sends the prompt text entered by the user to the server as an HTTP request. Specifically, a data packet containing the prompt text is sent to the server via the Internet. The input is the prompt text, and the output is the request sent to the server.

[1681] Step 3:

[1682] The server obtains the user's native language information.

[1683] The server identifies the user's ID based on the received prompt sentence, and retrieves the user's profile information from the database. Specifically, it retrieves the user's native language information using an SQL query. The input is the user's ID, and the output is the user's native language information.

[1684] Step 4:

[1685] The server uses a generative AI model to generate text based on the prompt.

[1686] The server uses a generative AI model (e.g., GPT-4) to generate commentary and subtitle text based on the prompt sentence. The generative AI model parses the prompt sentence and generates appropriate text. The input is the prompt sentence and the output is the generated text.

[1687] Step 5:

[1688] The server translates the generated text into the user's native language.

[1689] The server translates the generated text into the user's native language using a translation API (e.g., Google translation API). Specifically, the server sends the generated text to the translation API and receives the translated text. The input is the generated text and the output is the translated text.

[1690] Step 6:

[1691] The server sends the converted text to the terminal.

[1692] The server sends the translated text to the terminal as an HTTP response. Specifically, a data packet containing the translated text is sent to the terminal over the Internet. The input is the translated text, and the output is the response sent to the terminal.

[1693] Step 7:

[1694] The terminal displays the translated text.

[1695] The terminal receives the translated text from the server and displays it on the screen, in particular by rendering the received text in a user interface for visual confirmation by the user. The input is the translated text and the output is the text displayed on the screen.

[1696] (Application example 3)

[1697] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1698] In conventional live streaming and video distribution services, users have had difficulty understanding live coverage and subtitles provided in different languages. In addition, users have had limited access to content translated in real time into their native language, limiting their viewing experience. This has created an information gap between users who speak different languages.

[1699] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1700] In this invention, the server includes a means for receiving video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means for translating the generated live coverage and subtitles into the user's native language in real time, a means for displaying the translated live coverage and subtitles, and a means for providing an interface that allows the user to select a language of their choice. This enables the user to watch live coverage and subtitles in real time in a language that they can easily understand.

[1701] "Various sports and entertainment video data" refers to video information of sporting events and entertainment content, including live streaming and recorded videos.

[1702] "Generative AI" refers to artificial intelligence technology that automatically generates live commentary and subtitles based on input video data.

[1703] "Means for tailoring to user preferences" refers to the ability to customize the content and format of the generated commentary and / or subtitles to a user's individual preferences.

[1704] "Means of translating into the user's native language in real time" refers to the function of instantly translating the generated live broadcast and subtitles into a language specified by the user.

[1705] "Means for displaying translated commentary and / or subtitles" refers to the functionality for visually displaying the translated content on a User's device.

[1706] "An interface that allows users to select their preferred language" refers to a function that provides an operation screen or menu that allows users to select their desired language.

[1707] The system for implementing this invention is configured as follows. First, the server receives video data of various sports and entertainment as input. Next, the server uses a generation AI to generate live coverage and subtitles based on the input video data. This generation AI can adjust the voice and writing style to suit the user's preferences.

[1708] The generated live commentary and subtitles are translated into the user's native language in real time. Translation services such as Google Translate API are used for this translation process. The translated live commentary and subtitles are displayed on the user's device. The user can visually check the translated content using a device such as a smartphone or tablet.

[1709] Additionally, users can specify their preferred language through a language selection interface, which is designed to be easy for users to navigate and provides language selection menus and settings screens.

[1710] As a specific example, consider the case where a user is watching a live stream in English. In this case, the server receives the English video data and uses generation AI to generate English commentary and subtitles. The generated English commentary and subtitles are then translated into Japanese in real time and displayed on the user's smartphone. The user can watch the translated Japanese commentary and subtitles by selecting Japanese through the interface.

[1711] An example of a prompt is "Please translate the English live streaming subtitles into Japanese in real time and display them." By using this prompt, users can enjoy live streaming in a language they can easily understand.

[1712] This system will eliminate the information gap between users who speak different languages, allowing users to watch live broadcasts and subtitles in their own native language in real time.

[1713] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1714] Step 1:

[1715] The server receives video data of various sports and entertainment as input. The input video data is provided in the form of live streaming, recorded video, etc. The server analyzes the format of the data and converts it into an appropriate format for processing.

[1716] Step 2:

[1717] The server uses a generation AI to generate live commentary and subtitles based on the input video data. The generation AI extracts audio and text information from the video data, and creates live commentary and subtitles in real time based on that information. Voice recognition and natural language processing technologies are used in this process. The generated live commentary and subtitles are output as text data.

[1718] Step 3:

[1719] The server adjusts the generated live broadcast and subtitles to suit the user's preferences. Specifically, it customizes the tone of voice and sentence expression based on the user's settings. This adjustment is achieved by changing the parameters of the generation AI. After adjustment, the live broadcast and subtitles are output again as text data.

[1720] Step 4:

[1721] The server translates the adjusted live broadcast and subtitles into the user's native language in real time. This translation process uses a translation service such as Google Translate API. The server sends the adjusted text data to the translation API and receives the translated text data. The translated live broadcast and subtitles are output in the language specified by the user.

[1722] Step 5:

[1723] The server sends the translated live broadcast and subtitles to the user's device. The user's device displays the received text data on the screen. The display is done on a device such as a smartphone or tablet. The user can visually check the translated content.

[1724] Step 6:

[1725] Users specify their preferred language through an interface that allows them to select their preferred language. The interface is designed to be easy for users to operate, and provides a language selection menu and a settings screen. The user's selection is sent to the server and is reflected in the next translation process.

[1726] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1727] "Example 1"

[1728] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes the user's emotions and adjusts the output of the generation AI based on those emotions. Specifically, the system estimates the user's emotions from facial expressions, tone of voice, eye movements, etc. while watching a video. Then, the system adjusts the output of the generation AI according to those emotions. For example, if the user shows an emotion of joy, the generation AI generates a more exciting live broadcast and subtitles. Conversely, if the user shows an emotion of confusion or discomfort, the generation AI generates a more understandable and polite live broadcast and subtitles.

[1729] "Example 2"

[1730] The emotion engine also adjusts the voice based on the user's emotions. Specifically, if the user is happy, the AI ​​will generate commentary and subtitles in a brighter, more cheerful voice. Conversely, if the user is sad, the AI ​​will generate commentary and subtitles in a calmer voice. This enables more natural communication that matches the user's emotions.

[1731] "Example 3"

[1732] In addition, the emotion engine adjusts the text expression based on the user's emotion. Specifically, if the user expresses surprise, the generation AI generates commentary and subtitles with more detailed explanations. Conversely, if the user expresses relief, the generation AI generates simpler commentary and subtitles. This makes it possible to provide optimal information according to the user's emotion.

[1733] The process flow of each embodiment will be described below.

[1734] "Example 1"

[1735] Step 1: The user watches the video.

[1736] Step 2: The emotion engine infers emotions from the user's facial expressions, tone of voice, eye movements, etc.

[1737] Step 3: Based on the estimated emotions, the generation AI adjusts the output and generates commentary and subtitles.

[1738] "Example 2"

[1739] Step 1: The user watches the video.

[1740] Step 2: The emotion engine estimates the user's emotion.

[1741] Step 3: Based on the estimated emotions, the generation AI adjusts the voice and generates commentary and subtitles.

[1742] "Example 3"

[1743] Step 1: The user watches the video.

[1744] Step 2: The emotion engine estimates the user's emotion.

[1745] Step 3: Based on the estimated emotions, the generation AI adjusts the sentence expression and generates live commentary and subtitles.

[1746] Example 1

[1747] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1748] Conventional live broadcast and subtitle generation systems do not adjust according to the user's emotions, and there is a problem that the user experience is not sufficiently improved. In particular, there is a demand for real-time adjustment according to the user's emotions while watching the video.

[1749] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for inputting various video data, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for collecting user emotion data, and a means for adjusting the output of the generation AI based on the emotion data. This enables the generation of real-time live coverage and subtitles according to the user's emotions.

[1750] "Video data" is data in digital format that contains visual information such as sports and entertainment.

[1751] "Generative AI" is artificial intelligence that uses deep learning technology to automatically generate live commentary and subtitles from video data.

[1752] "User emotion data" refers to information about emotions obtained from the user's facial expressions, tone of voice, eye movements, etc.

[1753] An "emotion engine" is a system that includes machine learning algorithms for analyzing a user's emotional data and inferring their emotions.

[1754] "Means for adjusting output" refers to a function that changes the output content and tone of the generation AI based on feedback from the emotion engine.

[1755] "Live coverage" refers to information conveying the progress of an event taking place in real time through audio and text.

[1756] "Subtitles" are text information added to video data, providing supplementary explanations and dialogue to the viewer.

[1757] The present invention relates to a system that takes video data of sports or entertainment as input and generates live coverage and subtitles according to the user's emotions. This system is composed of three main elements: a server, a terminal, and a user.

[1758] The server first receives various types of video data via the Internet. This video data is in digital format containing visual information such as sports matches and entertainment events. The server inputs the received video data into the generative AI. The generative AI uses deep learning frameworks such as TensorFlow and PyTorch to analyze the video data and identify important events (e.g., goal scenes, goal-scoring scenes, interviews, etc.).

[1759] While the user is watching the video, the device uses the built-in camera and microphone to collect the user's facial expressions and tone of voice. This is done using face recognition technology and voice analysis technology. For example, if the user smiles, the facial expression data is captured. The collected emotion data is sent from the device to the emotion engine. The emotion engine uses libraries such as OpenCV and Dlib to perform face recognition and estimate the user's emotion. For example, if the user is smiling, it is determined to be "joy."

[1760] The server receives feedback from the emotion engine and adjusts the output of the generation AI. Specifically, it changes the tone and content of the play-by-play and subtitles depending on the user's emotions. For example, if the user is happy, the generation AI generates an exciting play-by-play such as, "Great goal! The crowd is excited!" Conversely, if the user is confused, it provides an easy-to-understand explanation such as, "An offside was called. This is because the attacking player got ahead of the opposing defender."

[1761] The device then displays the live broadcast and subtitles to the user in real time, allowing the user to enjoy content that best suits their emotions.

[1762] As a concrete example, consider a user watching a soccer game. If the user shows a look of joy at a goal, the AI ​​will generate an exciting live broadcast such as, "What a great goal! The crowd is so excited!" On the other hand, if the user is confused by an offside call, the AI ​​will provide a clear explanation such as, "Offside was called. This is because the attacking player got ahead of the opposing defender."

[1763] An example prompt would be:

[1764] "Using video data of a soccer match as input, generate live commentary and subtitles according to the user's emotions. If the user is happy, provide an exciting commentary, and if the user is confused, provide an easy-to-understand explanation."

[1765] In this way, the present invention can provide optimal live coverage and subtitles according to the user's emotions.

[1766] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1767] Step 1: Receiving video data

[1768] The server receives various video data via the Internet. This video data is in digital format containing visual information such as sports games and entertainment events. The input is video data, and the output is video data ready to be input to the generative AI.

[1769] Step 2: Analyze the video data

[1770] The server inputs the received video data into the generation AI. The generation AI uses deep learning frameworks such as TensorFlow and PyTorch to analyze the video data and identify important events (e.g., goal scenes, scoring scenes, interviews, etc.). The input is the video data, and the output is the identified important event information.

[1771] Step 3: Collect user emotion data

[1772] The device uses a built-in camera and microphone to collect the user's facial expressions and tone of voice while the user is watching a video. This is done using facial recognition and voice analysis technologies. The input is the video and audio data the user is watching, and the output is the collected emotion data.

[1773] Step 4: Analyze the sentiment data

[1774] The terminal sends the collected emotion data to the emotion engine. The emotion engine uses libraries such as OpenCV and Dlib to perform face recognition and estimate the user's emotion. The input is the collected emotion data, and the output is the estimated user's emotion information.

[1775] Step 5: Adjusting the output of the generation AI

[1776] The server receives feedback from the emotion engine and adjusts the output of the generative AI. Specifically, it changes the tone and content of the live broadcast and subtitles according to the user's emotions. The input is the estimated user's emotion information, and the output is the adjusted content of the live broadcast and subtitles.

[1777] Step 6: Generate and display live commentary and subtitles

[1778] The server generates live coverage and subtitles based on the adjusted output of the generation AI. The terminal displays the generated live coverage and subtitles to the user in real time. The input is the adjusted live coverage and subtitle content, and the output is the live coverage and subtitles displayed to the user.

[1779] In this way, the system can provide optimal live coverage and subtitles according to the user's emotions.

[1780] (Application example 1)

[1781] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1782] Conventional sports and entertainment video distribution systems have the problem that they are unable to provide live coverage or subtitles that respond to the user's emotions, resulting in a uniform viewing experience. In addition, there are limited adjustments to suit user preferences, making it difficult to provide optimal content for each individual viewer.

[1783] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1784] In this invention, the server includes a means for inputting video data of various sports and entertainment, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI to suit the user's preferences, a means including an emotion engine for recognizing the user's emotions and adjusting the output of the generation AI based on the emotions, a means for inputting the user's facial expressions and tone of voice using a smartphone camera and microphone, and a means for the generation AI to generate live coverage and subtitles in real time according to the user's emotions. This makes it possible to provide live coverage and subtitles that are individually optimized according to the user's emotions.

[1785] "Various sports and entertainment video data" refers to video information of sporting events and entertainment content, including live and recorded footage.

[1786] "Generative AI" refers to a system that uses artificial intelligence technologies such as deep learning to automatically generate live commentary and subtitles from input video data.

[1787] "Means of adjusting to user preferences" refers to the function of customizing the output of the generative AI based on the user's viewing history and settings information.

[1788] An "emotion engine" is a system that recognizes emotions from the user's facial expressions, tone of voice, etc., and adjusts the output of the generative AI based on those emotions.

[1789] "Means of inputting a user's facial expressions and tone of voice using a smartphone's camera and microphone" refers to a function that uses a smartphone's built-in camera and microphone to capture a user's facial expressions and tone of voice in real time.

[1790] "Means of generation in real time" refers to the ability to instantly generate live commentary and subtitles based on user emotions and the context of video data, and provide them without delay.

[1791] The system for implementing this invention is configured as follows. First, the server receives video data of various sports and entertainment as input. Next, the server uses a generation AI to automatically generate live coverage and subtitles based on the input video data. This generation AI utilizes artificial intelligence technology such as deep learning.

[1792] In addition, the server is equipped with an emotion engine that recognizes the user's emotions. This emotion engine captures the user's facial expressions and tone of voice in real time using the smartphone's camera and microphone, and analyzes them. Based on the analysis results, the output of the generation AI is adjusted according to the user's emotions.

[1793] Specifically, if the user expresses joy, the AI ​​will generate more exciting commentary and subtitles. On the other hand, if the user expresses confusion or displeasure, the AI ​​will generate more understandable and detailed commentary and subtitles.

[1794] The hardware required is a smartphone camera and microphone, the software uses OpenCV to process camera input, DeepFace to recognize emotions, and the Transformers library to implement a generative AI model.

[1795] As a concrete example, consider a case where a user is watching a soccer game and expressing joy. In this case, the following prompt sentence is input to the generation AI.

[1796] Example prompt:

[1797] If the user indicates the emotion of joy, generate a play-by-play based on the following context: "The moment when team A scores a goal in a soccer game."

[1798] Based on this prompt, the generative AI will generate an exciting live broadcast and provide it to the user, allowing the user to have a more emotionally relevant viewing experience.

[1799] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1800] Step 1:

[1801] The server receives video data from various sports and entertainment sources as input. The video data can be real-time or pre-recorded. The server analyzes the data and extracts the necessary information. Specifically, it analyzes frames of the video data to detect important events (e.g., goals scored or highlights).

[1802] Step 2:

[1803] The server uses a generative AI to automatically generate commentary and subtitles based on the input video data. The generative AI uses a deep learning model to generate text from video data. The input includes frames of video data and their analysis results. The output is the generated commentary and subtitle text.

[1804] Step 3:

[1805] The server captures the user's facial expressions and tone of voice in real time using the smartphone's camera and microphone. The device sends input data from the camera and microphone to the server. The input includes video of the user's face and audio data. The output is sent to the server.

[1806] Step 4:

[1807] The server uses an emotion engine to recognize emotions from the user's facial expressions and tone of voice. The emotion engine analyzes input video and audio data using emotion recognition models such as DeepFace. Inputs include video of the user's face and audio data. Outputs include the user's emotions (e.g., happy, confused, displeased).

[1808] Step 5:

[1809] The server adjusts the output of the generative AI based on the recognized user emotions. Specifically, if the user expresses joy, the generative AI generates a more exciting commentary and subtitles. Conversely, if the user expresses confusion or displeasure, the generative AI generates a more understandable and polite commentary and subtitles. The input includes the user's emotions and the context of the video data. The output is the adjusted commentary and subtitle text.

[1810] Step 6:

[1811] The server provides the generated commentary and subtitles to the user in real time. The device displays the commentary and subtitles sent from the server. The input includes the adjusted commentary and subtitle text. The output is the commentary and subtitles displayed on the user's smartphone screen.

[1812] This allows users to have a viewing experience that is individually optimized to fit their emotions.

[1813] Example 2

[1814] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1815] Conventional generative AI systems have limited ability to adjust output to suit user preferences, making it difficult to communicate naturally according to the user's emotions. In addition, it is difficult to generate voice and written expressions that reflect the user's emotions, and there is a demand for improved user experience.

[1816] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1817] In this invention, the server includes a means for inputting various video data, a means including a generation AI for generating live coverage and subtitles based on the video data, a means for adjusting the output of the generation AI according to the user's preferences, a means for detecting the user's emotions, and a means for adjusting the output of the generation AI based on the user's emotions. This makes it possible to generate natural and appropriate voice and text expressions according to the user's preferences and emotions.

[1818] "Various types of video data" refers to various types of video information such as sports and entertainment.

[1819] "Generative AI" refers to artificial intelligence technology that generates text or speech based on input data.

[1820] "User preferences" refers to individual preferences, such as tone of voice, speed, and writing style, set by the User.

[1821] "Means of adjustment" refers to functions and methods for changing the output of the generative AI to suit the user's preferences and emotions.

[1822] "User emotion" refers to an emotional state, such as happiness, sadness, excitement, etc., exhibited by a user.

[1823] "Means for emotion detection" refers to technologies or methods for analyzing and identifying emotions from user input data.

[1824] A "prompt" is an instruction given to a generation AI to generate a specific output.

[1825] "Speech synthesis software" refers to software technology for converting text data into speech data.

[1826] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1827] The present invention relates to a system for adjusting the output of a generation AI according to the preferences and emotions of a user. A specific embodiment of this system will be described below.

[1828] Hardware and Software Used

[1829] Hardware: Servers, terminals (PCs, smartphones, tablets, etc.)

[1830] Software: Generative AI models (e.g. GPT-4), emotion engines, text-to-speech software (e.g. Amazon Polly, Google Text-to-Speech)

[1831] System Configuration

[1832] 1. Getting user settings

[1833] A user logs into the system using a terminal.

[1834] On the settings screen, users can select their preferred voice tone (e.g., calm male voice, exciting female voice), speed, and writing style (e.g., polite expression, casual expression).

[1835] The terminal transmits the user's setting information to the server.

[1836] 2. Emotion Detection

[1837] The user inputs text or voice data and transmits it to the server via the terminal.

[1838] The server passes the received data to the emotion engine and analyzes the user's emotions.

[1839] Specifically, the emotion engine uses text analysis algorithms to identify emotions from user input, for example detecting the emotion "joy" from the text "Today was such a fun day!"

[1840] 3. Prompt generation for generative AI models

[1841] Based on the user's settings and the analysis results of the emotion engine, the server generates a prompt sentence to be input into the generative AI model.

[1842] As a specific operation, the server creates a prompt sentence saying, "If the user is expressing an emotion of joy, please generate text in a bright, cheerful voice and with casual expressions."

[1843] 4. Text Generation

[1844] The server inputs the prompt text into a generative AI model (e.g., GPT-4) and generates text that matches the user's preferences and emotions.

[1845] Specifically, the generative AI model generates the text "Today was absolutely amazing!" based on the prompt sentence.

[1846] 5. Speech synthesis

[1847] The server inputs the generated text into speech synthesis software, which generates speech in a tone and speed that matches the user's settings.

[1848] Specifically, the speech synthesis software (e.g., Amazon Polly) generates voice data in accordance with the instructions, "Generate an exciting female voice with casual expressions."

[1849] 6. Providing Output

[1850] The server transmits the generated voice data to the terminal.

[1851] The terminal reproduces the received voice data and provides it to the user.

[1852] In concrete terms, the user can play back the audio generated on the terminal and enjoy audio output tailored to his or her preferences.

[1853] Examples

[1854] Examples of prompt statements

[1855] "If the user is expressing the emotion of happiness, generate text in a cheerful, upbeat voice and with casual expressions."

[1856] "If the user indicates sadness, generate text in a calm, polite voice."

[1857] Real-world examples

[1858] A user logs into the system and selects "exciting female voice" and "casual expression."

[1859] The user types, "Today was such a fun day!"

[1860] The server's emotion engine detects "joy" and inputs the prompt sentence "Generate text in a bright, cheerful voice and casual expressions" into the generative AI model.

[1861] The server's generative AI model generates the text "Today was amazing!", and the speech synthesis software generates the audio in an exciting female voice.

[1862] The user can play back and enjoy the audio generated on the terminal.

[1863] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1864] Step 1:

[1865] A user logs into the system using a terminal. After logging in, the user selects the tone of voice, speed, and writing style he or she prefers on the settings screen. This setting information is sent from the terminal to the server.

[1866] Input: User login information, tone of voice, speed, writing style

[1867] Output: User configuration information sent to the server.

[1868] Step 2:

[1869] Users input text or voice data and send it to the server via their device. The server passes the received data to the emotion engine, which analyzes the user's emotions. The emotion engine uses text analysis algorithms to identify emotions from the user's input.

[1870] Input: User text or voice data

[1871] Output: Sentiment analysis results by the emotion engine (e.g., happy, sad)

[1872] Step 3:

[1873] The server generates a prompt sentence to be input to the generative AI model based on the user's settings and the analysis results of the emotion engine. Specifically, the server creates a prompt sentence that reads, "If the user is expressing joy, please generate text in a bright, cheerful voice and with casual expressions."

[1874] Input: User settings information, sentiment analysis results

[1875] Output: Prompt sentence to be input to the generative AI model

[1876] Step 4:

[1877] The server inputs the prompt sentence into a generative AI model (e.g. GPT-4) and generates text that matches the user's settings and emotions. The generative AI model generates text based on the prompt sentence.

[1878] Input: prompt statement

[1879] Output: The generated text (e.g. "Today was absolutely amazing!")

[1880] Step 5:

[1881] The server inputs the generated text into the speech synthesis software, which generates speech at a tone and speed that matches the user's settings. The speech synthesis software generates speech data according to the instruction, "Please generate a voice with an exciting female voice and casual expressions."

[1882] Input: Generated text, user settings information

[1883] Output: The generated audio data

[1884] Step 6:

[1885] The server transmits the generated voice data to the terminal. The terminal plays back the received voice data and provides it to the user. The user can play back the generated voice on the terminal and enjoy voice output tailored to their preferences.

[1886] Input: Generated audio data

[1887] Output: Audio played on the device

[1888] (Application example 2)

[1889] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1890] Conventional generative AI systems have limited...

Claims

1. A means for acquiring video data relating to sports or entertainment; A means for receiving an instruction from a user to generate additional information including at least one of live coverage and subtitles for the acquired video data; A means for generating a prompt sentence to be input to the generative AI model based on an instruction received from a user; A means for inputting the generated prompt sentence into the generative AI model and acquiring additional information from the generative AI model; A means for outputting the acquired additional information; A system including:

2. 2. The system of claim 1, further comprising means for analyzing a user's emotion using an emotion engine and adjusting a voice of the commentary based on the analyzed emotion of the user.

3. 2. The system of claim 1, further comprising means for analyzing a user's emotion using an emotion engine and adjusting textual expressions of the subtitles based on the analyzed user's emotion.

Images