System

Abstract

A system is provided.SOLUTION: A system including means for converting audio of a video into text using a generated AI, means for analyzing the text and automatically extracting a topic and a highlight of the video, means for automatically generating a table of contents of the video based on the extracted topic and highlight, means for generating a thumbnail image associated with the table of contents, means for searching for the video using a keyword, a date, and a distributor name and providing a link for directly jumping to a corresponding part, and means for collecting viewing data and providing an analysis result to a distributor.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the number of video distributors has increased, and so has the need for viewers to efficiently view past broadcast content. However, it is not easy for viewers to find specific information from the vast amount of past broadcast videos, and it takes time and effort. Furthermore, it is difficult for broadcasters to determine which content is of most interest to viewers. An effective method to solve these problems is required. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means: First, a means for converting the audio of a video into text using a generation AI is provided, followed by a means for automatically extracting topics and highlights of the video by analyzing the text. Then, a means for automatically generating a table of contents for the video based on the extracted topics and highlights is provided, along with a means for generating thumbnail images associated with the table of contents.

[0006] The system also allows viewers to efficiently search for previously streamed videos by providing a means for searching videos by keyword, date, or streamer name and providing a link to jump directly to the relevant section. Additionally, the system provides a means for collecting viewing data and providing analysis results to streamers based on that data, enabling streamers to properly understand viewer interests and provide effective content.

[0007] "Generative AI" is an artificial intelligence model that automatically converts speech to text and then analyzes that text to extract specific topics and highlights.

[0008] A "means for converting video audio into text" is a device or system that provides a process for converting audio data in a video into text data in sentence format using generative AI.

[0009] "Means for automatic topic and highlight extraction" refers to a system that uses generative AI to automatically identify and extract important topics and noteworthy parts of videos based on text data.

[0010] The "means for automatically generating a video table of contents" is a system that automatically creates a table of contents that displays each section and important point of a video in list format based on extracted topics and highlights.

[0011] "Means for generating thumbnail images" refers to a system that automatically generates images to visually represent specific scenes in a video, and displays them in association with the table of contents or search results.

[0012] "Means to search by keyword, date, or streamer name" refers to a system that provides an interface and underlying search algorithms that allow viewers to input specific information to identify and search for the desired video or section of the video.

[0013] "Means for providing a link to jump directly to the relevant part" refers to a system that has the function of providing a link that allows viewers to directly access a specific part of the desired video based on the search results.

[0014] "Means for collecting viewing data and providing analysis results to distributors" refers to a system that collects data when viewers watch videos, analyzes that data, and provides distributors with details of viewing behavior. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system that utilizes generative AI to efficiently search and analyze the content of video streams. In this system, the server, terminal, and user elements work together to convert and analyze the audio of videos into text, and provide an interface that is visually and functionally easy to use.

[0037] Program processing explanation

[0038] Video upload and analysis

[0039] When a streamer uploads a video from their device, the server converts the audio into text using generative AI. Specifically, the server uses speech recognition technology to analyze the audio in the video in real time and convert it into text as subtitles.

[0040] The server then analyzes this text data and automatically extracts topics and highlights from the video. In this process, natural language processing technology is used to identify topics (e.g., game walkthroughs, Q&A corners, etc.) and highlight scenes (e.g., important announcements, funny moments, etc.) from the text content.

[0041] Table of contents and thumbnail generation

[0042] Based on the extracted topics and highlights, the server automatically generates a table of contents for the video, such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A session," and "20:00 - Ending."

[0043] The server also generates thumbnail images of each section, allowing viewers to easily navigate through the table of contents and jump to parts that interest them.

[0044] Providing search functionality

[0045] Users access the system through a terminal and use an interface that allows them to search for videos by specific keywords, dates, or streamer names. If a user types in a search term like "new game introductions," the server quickly identifies relevant videos and displays them as search results.

[0046] The server also provides links to relevant highlights and table of contents in search results, allowing users to jump directly to the parts they are interested in. This allows users to quickly find the information they are looking for among a vast amount of video content.

[0047] Providing analysis results

[0048] The server collects viewing data and analyzes it to determine which parts were viewed the most and which parts attracted the most viewers' attention. For example, it can collect information such as, "In the April 15, 2023 video, the Q&A session starting at 10:45 was particularly popular."

[0049] The server provides the streamer with the collected and analyzed data in the form of a dashboard. Based on this data, the streamer can plan and improve the content of their next stream. For example, it becomes easier to make decisions such as, "Viewers will be pleased if we improve the Q&A section in the next stream."

[0050] Specific examples

[0051] For example, if a streamer uploads a "walkthrough video for a new game," the server converts the audio into text and analyzes it to extract the topics "Introduction," "Game Strategy," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are presented to viewers in a visually easy-to-understand format.

[0052] When a viewer searches for "game tips," the server displays search results such as "game tips from 03:00" and also provides a link that allows the user to access that section directly. Furthermore, the server analyzes the viewing data for this video and notifies the streamer that the "game tips" section in particular is being viewed a lot. Based on this information, the streamer can reflect this in their next content creation.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The streamer uses a device to upload the video to the server.

[0056] Specifically, the streamer selects a video file on the device and clicks the upload button, and the device sends the video file to the server.

[0057] Step 2:

[0058] The server stores the received video files in a database.

[0059] Once saved, the process of converting the video's audio into text begins using generative AI.

[0060] Step 3:

[0061] The server uses generative AI to convert the video's audio data into text in real time.

[0062] Using voice recognition technology, what is said in the video is converted into text as subtitles.

[0063] Step 4:

[0064] The server analyzes the generated text data.

[0065] Using natural language processing technology, topics and highlights are automatically extracted from the text content. For example, topics such as "game strategies" and "question corners" are identified.

[0066] Step 5:

[0067] The server automatically generates a table of contents for the video based on the extracted topics and highlights.

[0068] For example, create table of contents entries such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A," and "20:00 - Ending."

[0069] Step 6:

[0070] The server generates thumbnail images corresponding to each section of the table of contents.

[0071] Thumbnail images are displayed on the viewing screen along with the table of contents, serving as a visual guide.

[0072] Step 7:

[0073] A user accesses the system through a terminal and searches for videos using a search interface.

[0074] When a user enters a specific keyword (e.g., "new game introduction"), the device sends this search query to the server.

[0075] Step 8:

[0076] The server receives the search query and searches the database to identify relevant videos and topics.

[0077] The server generates related videos and table of contents links as search results and returns them to the user.

[0078] Step 9:

[0079] Users receive search results on their devices and can jump directly to the relevant section by clicking on the displayed link.

[0080] This feature allows users to quickly access specific moments or sections.

[0081] Step 10:

[0082] The server collects viewing data.

[0083] For example, data is collected about which parts were most viewed and at what parts viewers stopped watching.

[0084] Step 11:

[0085] The server analyzes the collected viewing data and provides the analysis results to the broadcaster.

[0086] The analysis results are displayed on a dashboard dedicated to the streamer, and streamers can use this data to plan and improve their content.

[0087] In this way, through specific actions at each step, a series of processes from video uploading to viewing analysis is completed.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] Today, a huge amount of video content is being produced across a wide variety of formats. However, a lack of efficient support for searching, analyzing, and viewing this video content presents a challenge. In particular, there is a lack of functionality to convert audio within a video into text and then analyze it to quickly provide the information viewers are looking for. As a result, viewers must watch the entire video, wasting time and effort. Furthermore, streamers are unable to effectively receive feedback on the results of analyzing viewing data, making it difficult to improve the content of their next stream.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes a means for converting the audio of a video into text, a means for analyzing the text and automatically extracting topics and highlights of the video, a means for automatically generating a table of contents for the video based on the extracted topics and highlights, a means for generating images associated with the table of contents, a means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to the relevant parts, and a means for collecting viewing data and providing the analysis results to the broadcaster. This allows the server to efficiently analyze the content of the video and automatically generate a table of contents, enabling viewers to quickly access the information they desire. Furthermore, by providing broadcasters with feedback on the viewing data, the server can effectively improve the content of their next broadcast.

[0093] "Means for converting video audio into text" refers to technology that analyzes the audio data in a video and generates corresponding text data from the audio.

[0094] "Means for analyzing the text and automatically extracting topics and highlights from the video" refers to technology that analyzes the generated text data and automatically identifies important themes and noteworthy scenes.

[0095] "Means for automatically generating a video table of contents based on extracted topics and highlights" refers to a technology that organizes the start time and content of videos in list format based on the topic and highlight information obtained from the analysis results.

[0096] The "means for generating an image associated with the table of contents" refers to a technique for selecting a frame of a video corresponding to each item in the table of contents and generating it as a still image.

[0097] "Means of searching videos by keyword, date, or streamer name and providing links to jump directly to the relevant section" refers to technology that finds related videos based on the search criteria specified by the viewer and generates links that allow quick access to the relevant section.

[0098] "Means of collecting viewing data and providing analysis results to broadcasters" refers to technology that records and analyzes viewers' viewing behavior and provides feedback to broadcasters on information such as the most viewed parts and parts that viewers were most interested in.

[0099] "Means for converting to text at specified times" refers to technology that converts audio at specific points or sections of a video into text data.

[0100] "Natural language processing technology" refers to technology that allows computers to analyze and understand human language.

[0101] "Means for collecting and analyzing viewing data for specific parts" refers to technology that records viewers' viewing behavior in specific sections of a video and conducts detailed analysis based on that information.

[0102] "Means of providing distributors with information on popular viewing sections" refers to technology that notifies distributors of information on sections that are of high interest to viewers based on collected viewing data.

[0103] "An interface that allows users to search for previously distributed videos by entering specific keywords" refers to a user interface that allows viewers to search for previously distributed videos by entering keywords in a search box.

[0104] "Means of displaying related videos and their table of contents links based on a search query" refers to technology that identifies videos that match the search criteria entered by the viewer and displays them along with a table of contents for those videos.

[0105] "Means of providing links that allow viewers to jump directly from search results to specific sections of the video" refers to technology that allows viewers to access the specific part of the video they are looking for directly by simply clicking a link from the search results.

[0106] This invention is a system that utilizes generative AI to efficiently search and analyze video streaming content. In this system, the server, terminal, and user elements work together to convert and analyze the audio of videos into text, and provide an interface that is visually and functionally easy to use.

[0107] Hardware and Software Configuration

[0108] The system consists of the following main components:

[0109] Server: Use a high-performance cloud server (e.g., AWS, Google Cloud) to perform video analysis and database management.

[0110] Device: Includes any device that can connect to the internet, such as a computer, smartphone, or tablet, that a user uses.

[0111] Speech recognition technology: Converts speech to text using the Google Cloud Speech-to-Text API.

[0112] Natural language processing technology: OpenAI GPT-4 is used to analyze text data and automatically extract topics and highlights.

[0113] How to convert video audio to text

[0114] When a user uploads a video file from their device, the server stores it in Google Cloud Storage and calls the Google Cloud Speech-to-Text API to convert the audio in the video into text in real time. This text data is stored on the server.

[0115] A means for automatically extracting topics and highlights of a video by analyzing the text.

[0116] The server analyzes the stored text data using OpenAI GPT-4. Specifically, it analyzes the entire text using natural language processing technology, identifies frequently occurring words and important phrases, and automatically extracts topics (e.g., game walkthroughs, Q&A corners, etc.) and highlights (important announcements, funny moments, etc.) from the video.

[0117] A means to automatically generate a video table of contents based on extracted topics and highlights

[0118] The server automatically generates a table of contents based on the topic and highlight information obtained from the analysis, including the start time and title of each section. For example, it might generate a table of contents such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A Corner," and "20:00 - Ending."

[0119] means for generating an image associated with said table of contents;

[0120] The server extracts specific frames from the video and uses them to generate thumbnail images for each section. Specifically, it selects the frame closest to the start time of each section and processes it as the thumbnail.

[0121] A way to search for videos by keyword, date, or streamer name and provide links to jump directly to the relevant section

[0122] Users access the system through their terminals and enter keywords, dates, and broadcaster names into the search box. The server searches the database for relevant videos based on the query and displays the links, table of contents, and highlight information for the relevant videos. Users can also click on these links to directly access specific sections of the related videos.

[0123] A means of collecting viewing data and providing analysis results to broadcasters

[0124] The server records and analyzes the viewer's viewing behavior, such as the start time, end time, and length of each viewing section. The collected and analyzed data is provided to the streamer in the form of a dashboard. The streamer can use this information to plan and improve the content of their next stream.

[0125] Specific examples

[0126] For example, if a streamer wants to upload a "walkthrough video for a new game," the user selects the video from their device and begins uploading. The device sends the video file to the server, which receives the file and stores it in Google Cloud Storage. The server uses the Google Cloud Speech-to-Text API to convert the audio into text and saves it in a database. The server analyzes the saved text with OpenAI GPT-4 and extracts topics such as "Introduction," "Game Strategy," and "Conclusion." Based on these topics, a table of contents and thumbnail images for each section are automatically generated.

[0127] When a user searches for the keyword "game tips," the server displays related videos and links to those sections. The user can then access those sections directly. The server then analyzes the viewing data and notifies the streamer that the "game tips" section in particular is being viewed a lot. Based on this information, the streamer can reflect this in their next content creation.

[0128] Prompt Sentence Examples

[0129] "I have uploaded a walkthrough video for a new game. Please extract the topics of this video - introduction, game strategy, and conclusion - from their respective timestamps and automatically generate a table of contents and thumbnail images."

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Step 1:

[0132] The user selects the video file from the terminal through the system's web interface and clicks the upload button.

[0133] Input: A video file selected by the user

[0134] Output: Video file sent from device to server

[0135] Specific behavior:

[0136] When a user selects a video file on the device and clicks the upload button on the web interface, the device sends the video file to the server, which receives the video file and stores it in storage.

[0137] Step 2:

[0138] The server stores the received video file in Google Cloud Storage and calls the Google Cloud Speech-to-Text API to convert the audio in the video into text.

[0139] Input: Received video file

[0140] Output: Converted text data

[0141] Specific behavior:

[0142] The server stores the video file in Google Cloud Storage, then uses the Google Cloud Speech-to-Text API to convert the video audio into text, which is then stored in a database on the server.

[0143] Step 3:

[0144] The server analyzes the stored text data using OpenAI GPT-4 and automatically extracts topics and highlights within the video.

[0145] Input: Text data

[0146] Output: Topic and highlight information

[0147] Specific behavior:

[0148] The server receives the text data and analyzes it using OpenAI GPT-4. This analysis extracts important topics and highlights from the text. Specifically, it identifies frequently occurring words and important phrases and identifies topics and highlights based on them.

[0149] Step 4:

[0150] The server automatically generates a table of contents for the video based on the extracted topics and highlights.

[0151] Input: Topic and highlight information

[0152] Output: Table of contents data

[0153] Specific behavior:

[0154] The server then uses the obtained topic and highlight information to organize the start times and titles of each section in a table of contents format, including items such as "00:00 - Opening" and "02:15 - Introduction to the new game."

[0155] Step 5:

[0156] The server extracts frames of the video associated with the table of contents and uses them to generate thumbnail images.

[0157] Input: Table of contents data

[0158] Output: Thumbnail image

[0159] Specific behavior:

[0160] The server selects the frame closest to the start of each section and generates a thumbnail of that frame, which is then associated with the table of contents, allowing viewers to visually identify the parts of the video that interest them.

[0161] Step 6:

[0162] Users access the system through their terminal and perform a search by entering keywords, dates, and broadcaster names in the search box.

[0163] Input: Keywords, date, broadcaster name

[0164] Output: Search results (related video links, table of contents, highlight information)

[0165] Specific behavior:

[0166] When a user enters a keyword into the search box, the server searches the database for relevant video data, and displays the link to the video, its table of contents, and highlights to the user. Users can also click on the provided link to go directly to the specific section of the video they are looking for.

[0167] Step 7:

[0168] The server collects viewing data and analyzes which parts are viewed the most.

[0169] Input: Viewing data

[0170] Output: Analysis results of viewing data

[0171] Specific behavior:

[0172] The server records viewers' viewing behavior and collects and analyzes viewing data. The analysis results include the frequency of viewing each section and the parts that viewers are most interested in. This data is provided to the broadcaster in the form of a dashboard.

[0173] Step 8:

[0174] The server provides feedback to the distributor based on the results of analyzing the viewing data.

[0175] Input: Analysis results of viewing data

[0176] Output: Feedback information provided to the distributor

[0177] Specific behavior:

[0178] The server then sends the analysis results to the streamer. For example, it can provide information on which specific sections are most popular, which can help the streamer in creating the next piece of content. This feedback allows the streamer to plan and improve content that will attract viewers' attention.

[0179] (Application example 1)

[0180] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0181] With current video streaming services, the sheer volume of video content makes it difficult for users to quickly find specific information. It is also labor-intensive for streamers to use viewing data to improve their next content offering. This reduces convenience for both users and streamers, making it difficult to provide effective content. It is necessary to solve these issues and improve the efficiency of video search and streamer data utilization.

[0182] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0183] In this invention, the server includes: means for converting video audio into text using generative AI; means for analyzing the text and automatically extracting topics and highlights of the video; means for automatically generating a table of contents for the video based on the extracted topics and highlights; means for generating thumbnail images associated with the table of contents; means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to relevant parts; means for collecting viewing data and providing analysis results to broadcasters; means for generating a video timeline based on time axis information extracted from the text; and means for generating thumbnail images corresponding to each timeline entry. This allows users to quickly find parts of a video that interest them, and enables broadcasters to easily utilize viewing data to improve their next broadcast.

[0184] "Generative AI" is a technology that uses artificial intelligence technology to generate data such as text, audio, and images.

[0185] "Means for converting audio to text" refers to a technology that analyzes the audio in a video and outputs it as text information.

[0186] "Means for automatic topic and highlight extraction" refers to technology that automatically identifies important themes and points of interest from text data.

[0187] "Means for automatically generating a table of contents" refers to technology that automatically generates a table of contents to provide users with a table of contents that identifies the major sections within a video.

[0188] "Means for generating thumbnail images" refers to a technology that captures specific scenes from a video as images to be used as visual indexes.

[0189] "Means for searching and providing direct jump links" refers to technology that searches for related videos based on keywords, dates, and broadcaster names entered by the user, and generates links to directly access specific parts.

[0190] "Means of collecting viewing data and providing analytical results" refers to technology that collects and analyzes data such as video viewing history and viewing time, and provides the results to the distributor.

[0191] "Means for generating a video timeline based on time axis information" is a technology that visually represents the time axis of a video based on time information extracted from text data.

[0192] The "means for generating thumbnail images corresponding to timeline entries" is a technique for capturing an image at a specific time corresponding to each entry on the timeline and displaying it on the timeline.

[0193] The "means for proposing the next distribution plan" is a technology that automatically suggests the next distribution content to increase the viewer's interest based on the previous viewing data.

[0194] The "means for individually optimized content recommendations" is a technology that recommends individually customized content based on a user's search history and viewing history.

[0195] A system for carrying out the present invention is one in which a server, a terminal, and a user work together in cooperation with each other. A specific embodiment of the system will be described below.

[0196] Server Roles

[0197] First, when a user uploads a video from their device, the server converts the audio into text using generative AI. In this process, the server extracts audio data from the video file using a speech recognition library (e.g., speech_recognition) and a movie processing library (e.g., moviepy) and converts it into text data.

[0198] The server then analyzes this text data using a generative AI model (e.g., OpenAI's API) to automatically extract topics and highlights within the video, thereby identifying which sections are important.

[0199] Based on the extracted topics and highlights, the server automatically generates a table of contents for the video, allowing viewers to easily understand the content of the video and quickly access parts that interest them. Additionally, thumbnail images associated with the table of contents are also generated. For example, the server captures a frame at the beginning of each section and uses it as a thumbnail.

[0200] Role of user terminal

[0201] The user device provides an interface that allows viewers to search for videos by specific keywords, dates, or broadcaster names. When a user enters a search query, the server returns related videos and their table of contents links. Based on the search results, users can use links to jump directly to the parts that interest them. This allows viewers to quickly find the information they are looking for among a vast amount of video content.

[0202] Data analysis and feedback

[0203] The server also collects viewing data and provides analysis results to the broadcaster. Specifically, it makes suggestions for improving the content of the next broadcast based on data such as which parts viewers watched the most and which sections they were interested in. This feedback allows the broadcaster to create effective content to increase viewer interest.

[0204] Specific examples

[0205] For example, let's consider the case where a streamer uploads a "walkthrough video for a new game." The server converts the audio into text and analyzes it to extract topics such as "Introduction," "Game Strategy," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are presented to viewers in a visually easy-to-understand format.

[0206] When a viewer searches for "game tips," the server will display search results such as "game tips from 03:00" and provide a link that allows the user to access that section directly. An example of a prompt sentence that can be used in this case is as follows:

[0207] Prompt Sentence Examples

[0208] Extract the main topics and highlights from the following text:

[0209] "This is a walkthrough of a new game. First, I'll explain how to choose your character. Next, I'll introduce some important points for progressing through the game. Finally, I'll explain how to beat boss battles."

[0210] This allows users to quickly access important information and allows streamers to effectively plan upcoming content based on viewing data.

[0211] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0212] Step 1:

[0213] Uploading videos

[0214] The user uses the terminal to upload a video file to the server, which receives the video file, extracts the audio data, and prepares it for the next step.

[0215] Input: Video file

[0216] Output: Audio data

[0217] Step 2:

[0218] Speech-to-text

[0219] The server uses a speech recognition library (e.g., speech_recognition) to convert the audio data into text, analyzes the extracted audio data, and outputs it in text format.

[0220] Input: Audio data

[0221] Output: Text data

[0222] Step 3:

[0223] Text analytics

[0224] The server uses a generative AI model (e.g., OpenAI's API) to analyze the text data and automatically extract topics and highlights from the video. This analysis identifies the main themes and important scenes in the video.

[0225] Input: Text data

[0226] Output: Topic and highlight information

[0227] Step 4:

[0228] Automatic table of contents and thumbnail generation

[0229] The server automatically generates a table of contents for the video based on the extracted topics and highlight information, and simultaneously generates thumbnail images from the start of each section and associates them with the table of contents.

[0230] Input: Topic and highlight information

[0231] Output: Table of contents data, thumbnail images

[0232] Step 5:

[0233] Providing a search interface

[0234] The device provides the user with a search interface, displaying an input form that allows the user to search for videos by specific keywords, dates, or streamer names.

[0235] Input: None (preparation until user input)

[0236] Output: Search interface

[0237] Step 6:

[0238] Processing search queries

[0239] The user enters a search query and the device sends it to the server, which searches for related videos and table of contents links based on the query and returns the results.

[0240] Input: Search query (keyword, date, broadcaster name)

[0241] Output: Search results (related videos and table of contents links)

[0242] Step 7:

[0243] Collecting viewing data

[0244] The server collects data on users' viewing of videos (such as viewing time, viewing sections, etc.), which is later used by the distributor for analysis.

[0245] Input: Viewing activity data

[0246] Output: Recording to viewing database

[0247] Step 8:

[0248] Providing analysis results

[0249] The server analyzes the collected viewing data to determine which sections are most viewed and trends in viewer interest. The server then provides the analysis results to the broadcaster, helping them plan their next broadcast.

[0250] Input: Viewing data

[0251] Output: Analysis report

[0252] These steps allow users to quickly find what interests them and give publishers the data to improve their content next time.

[0253] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0254] This invention combines a system that utilizes generative AI to efficiently search and analyze video content with an emotion engine that recognizes user emotions. In this system, the server, terminal, and user elements work together to convert and analyze the audio of videos into text, and then provide recommended content based on the user's emotions.

[0255] Program processing explanation

[0256] Video upload and analysis

[0257] When a streamer uploads a video from their device, the server converts the audio into text using generative AI. Specifically, the server uses speech recognition technology to analyze the audio in the video in real time and convert it into text as subtitles.

[0258] The server then analyzes this text data and automatically extracts topics and highlights from the video. In this process, natural language processing technology is used to identify topics (e.g., game walkthroughs, Q&A corners, etc.) and highlight scenes (e.g., important announcements, funny moments) from the text content.

[0259] Table of contents and thumbnail generation

[0260] Based on the extracted topics and highlights, the server automatically generates a table of contents for the video, such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A session," and "20:00 - Ending."

[0261] The server also generates thumbnail images of each section, allowing viewers to easily navigate through the table of contents and jump to parts that interest them.

[0262] Providing search functionality

[0263] Users access the system through a terminal and use an interface that allows them to search for videos by specific keywords, dates, or streamer names. If a user types in a search term like "new game introductions," the server quickly identifies relevant videos and displays them as search results.

[0264] The server also provides links to relevant highlights and table of contents in search results, allowing users to jump directly to the parts they are interested in. This allows users to quickly find the information they are looking for among a vast amount of video content.

[0265] Implementing the Emotion Engine

[0266] The emotion engine uses facial recognition and voice analysis technologies to analyze the viewer's emotional state in real time while they are watching a video, determining, for example, whether they are laughing, surprised, sad, etc.

[0267] Based on the collected emotional data, the server can recommend videos and provide highlights that correspond to the user's emotional state. For example, if the server detects that the user is laughing, it can recommend more funny scenes or relaxing content.

[0268] Providing analysis results and recommended content

[0269] The server collects viewing data and emotional data, analyzes them, and generates data such as which parts were viewed the most and which parts evoked particular emotions.

[0270] The analysis results are displayed on the streamer's personal dashboard, and the streamer can use this data to plan and improve their content. Specifically, streamers can identify moments that make viewers laugh and topics that interest them and reflect this in their next stream.

[0271] Specific examples

[0272] For example, if a streamer uploads a "walkthrough video for a new game," the server converts the audio into text and analyzes it to extract the topics "Introduction," "Game Strategy," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are presented to viewers in a visually easy-to-understand format.

[0273] When a viewer searches for "game tips," the server will display search results such as "game tips from 3:00" and provide a link that allows the user to access that section directly. Furthermore, if the emotion engine detects a smile on the user's face while watching, the server will recommend funny or popular scenes from the same game. As a result, viewers can quickly access the content they want to watch, and broadcasters can maintain their viewers' interest.

[0274] The processing flow will be explained below.

[0275] Step 1:

[0276] The streamer uses a device to upload the video to the server.

[0277] Specifically, the streamer selects a video file on the device and clicks the upload button, and the device sends the video file to the server.

[0278] Step 2:

[0279] The server stores the received video files in a database.

[0280] Once saved, the process of converting the video's audio into text begins using generative AI.

[0281] Step 3:

[0282] The server uses generative AI to convert the video's audio data into text in real time.

[0283] Using voice recognition technology, what is said in the video is converted into text as subtitles.

[0284] Step 4:

[0285] The server analyzes the generated text data.

[0286] Using natural language processing technology, topics and highlights are automatically extracted from the text content. For example, topics such as "game strategies" and "question corners" are identified.

[0287] Step 5:

[0288] The server automatically generates a table of contents for the video based on the extracted topics and highlights.

[0289] For example, create table of contents entries such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A," and "20:00 - Ending."

[0290] Step 6:

[0291] The server generates thumbnail images corresponding to each section of the table of contents.

[0292] Thumbnail images are displayed on the viewing screen along with the table of contents, serving as a visual guide.

[0293] Step 7:

[0294] A user accesses the system through a terminal and searches for videos using a search interface.

[0295] When a user enters a specific keyword (e.g., "new game introduction"), the device sends this search query to the server.

[0296] Step 8:

[0297] The server receives the search query and searches the database to identify relevant videos and topics.

[0298] The server generates related videos and table of contents links as search results and returns them to the user.

[0299] Step 9:

[0300] Users receive search results on their devices and can jump directly to the relevant section by clicking on the displayed link.

[0301] This feature allows users to quickly access specific moments or sections.

[0302] Step 10:

[0303] The emotion engine analyzes the user's emotions while they are watching.

[0304] Using facial recognition and voice analysis technology, it determines in real time whether the viewer is laughing, surprised, sad, etc.

[0305] Step 11:

[0306] Based on the collected emotional data, the server provides recommended videos and highlights according to the user's emotional state.

[0307] For example, if the user is laughing, the server will suggest more funny scenes and related content.

[0308] Step 12:

[0309] The server collects and analyzes viewing data and emotion data, and generates analysis results.

[0310] Data is collected on which parts were viewed the most and how users felt about them.

[0311] Step 13:

[0312] The analysis results are displayed on a dashboard dedicated to the broadcaster.

[0313] Streamers can use this data to plan and improve their content, for example by identifying moments that made users laugh and creating strategies such as increasing the number of similar scenes in their next stream.

[0314] Example 2

[0315] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0316] With the rapid increase in video content, it is becoming increasingly difficult for viewers to efficiently search and access the information they are looking for. Furthermore, detailed viewing data is necessary for streamers to understand viewers' reactions and improve future content. However, current systems do not adequately collect and analyze not only viewing data but also emotional data during viewing, and provide content recommendations based on that data. Therefore, a comprehensive system is needed to improve the viewer experience and support streamers in content creation.

[0317] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0318] In this invention, the server includes: means for converting video audio into text using a generative AI; means for analyzing the text and automatically extracting video topics and highlights; means for automatically generating a video table of contents based on the extracted topics and highlights; means for generating thumbnail images associated with the table of contents; means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to relevant sections; means for collecting viewing data and user emotional data and providing the analysis results to the broadcaster; and means for analyzing the user's emotional state during viewing in real time using an emotion engine and recommending videos and providing highlights based on the collected emotional data. This allows viewers to efficiently access desired content, and broadcasters to understand viewer reactions in detail and improve the quality of their content in future broadcasts.

[0319] "Generative AI" refers to models or algorithms that generate data using artificial intelligence techniques.

[0320] "Text conversion" is the process of analyzing audio data and converting it into written information.

[0321] "Topic extraction" is a technique for analyzing text information and identifying key topics or themes within it.

[0322] "Highlight extraction" is a technology that automatically extracts particularly important parts or interesting scenes from video or text.

[0323] "Table of contents generation" is the process of creating easily digestible segments based on the main points of a video or piece of text.

[0324] "Thumbnail generation" is the process of extracting specific frames of a video as still images to create small visually representative images.

[0325] A "search function" is a technology that allows you to quickly find relevant information or data based on specific keywords or conditions.

[0326] "Viewing data" refers to data recorded and analyzed based on user behavior when watching videos (such as playback time and frequently viewed parts).

[0327] "Emotional data" refers to data collected by recognizing the emotional state (happiness, surprise, sadness, etc.) that viewers show while watching a video.

[0328] An "emotion engine" is a system or technology that analyzes the face and voice of a user while watching and determines their emotional state in real time.

[0329] "Recommended content" is a function that automatically suggests content that may interest a user based on the user's viewing history and emotional data.

[0330] A "viewing analysis dashboard" is a system that displays detailed viewer reactions and behavior to broadcasters based on collected viewing and emotional data.

[0331] This invention is a system for efficiently searching and analyzing video content, combining generative AI and emotion recognition technology to provide useful information to both viewers and streamers. This system is implemented primarily using the following hardware and software:

[0332] Video speech to text conversion

[0333] The server receives the video uploaded by the streamer from their device, extracts the audio using an audio processing tool (e.g., ffmpeg), and then converts the audio into text using a speech recognition service such as the Google Cloud Speech-to-Text API. The converted text is then stored in a database on the server.

[0334] Parsing text and generating a table of contents

[0335] The server analyzes the generated text using natural language processing techniques (e.g., BERT or GPT-3) to extract the topics and highlights of the video. For example, in a game walkthrough video, segments such as "Introduction," "Way of the Walkthrough," and "Conclusion" are identified. Based on this, a table of contents for the video is automatically generated, and thumbnail images corresponding to each segment are generated. OpenCV and PIL libraries are used to generate the thumbnails.

[0336] Providing search functionality

[0337] A user can access the system using a terminal and enter specific keywords into the search interface. For example, they can enter "new game introduction." The server receives this search query, matches it with the stored text data and table of contents information, identifies related videos, and displays the search results as a list. The user can click on the displayed link to jump directly to the relevant section.

[0338] Emotion recognition implementation

[0339] The emotion engine analyzes facial expressions and voice in real time via the camera and microphone while the user is watching a video. It uses FaceAPI and Emotion API to determine the user's emotions, such as smile, surprise, sadness, etc. This emotion data is sent to the server and stored in a database along with the viewing data.

[0340] Providing recommended content

[0341] The server then runs an algorithm based on the collected emotional data to recommend content that matches the user's emotional state. For example, if the user is smiling a lot, the server will recommend more entertaining videos or relaxing content. Recommended content is provided via notifications and lists.

[0342] Viewing analytics dashboard

[0343] The server generates a dashboard for streamers based on the collected viewing and emotion data. The dashboard displays the viewing time for each segment, a heat map of emotion data, and an analysis of viewer reactions. This allows streamers to understand viewer reactions in detail and use this information to improve future content creation.

[0344] Specific operation example

[0345] For example, if a streamer uploads a video explaining how to play a new game, the server converts the audio into text and analyzes it to extract topics such as "Introduction," "Game Tips," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are then provided to viewers.

[0346] When a viewer searches for "game strategies," the server displays search results such as "game strategies from 03:00," and the viewer can click on a link to directly access the relevant section. In addition, when the emotion engine detects a smile on the user's face while watching, the server recommends interesting or popular scenes from the same game, improving the user's viewing experience.

[0347] Prompt Sentence Examples

[0348] Introducing a new game

[0349] Enter: Introducing a new game

[0350] Output: A list of all videos tagged with "new game introductions"

[0351] "Tell me the highlight scenes in the video"

[0352] Input: Highlights from the video

[0353] Output: A list of highlight scenes from each video

[0354] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0355] System program processing flow

[0356] Step 1: Upload your video

[0357] The server receives a request from the distributor to upload a video file from the terminal.

[0358] Input: Video file upload request from the device

[0359] Output: The location where the uploaded video file will be saved to storage.

[0360] Specific behavior: The server receives the video file via an HTML form or API endpoint, saves it to storage, and returns a status code of 200 to confirm the upload was successful.

[0361] Step 2: Speech to text

[0362] The server extracts the audio from the stored video file and converts it into text using generative AI.

[0363] Input: Saved video file

[0364] Output: Text data

[0365] Specific operation: The server separates the audio using ffmpeg and converts the audio data to text data using the Google Cloud Speech-to-Text API. The converted text is then stored in a database.

[0366] Step 3: Parsing the text

[0367] The server analyzes the generated text data using natural language processing technology to extract topics and highlights from the video.

[0368] Input: Text data

[0369] Output: Topic and highlight information (JSON format)

[0370] Specific operation: The server uses generative AI such as BERT or GPT-3 to identify topics (e.g., "Introduction," "Game Strategy," "Conclusion") and highlight scenes, and saves the results in JSON format.

[0371] Step 4: Generate the Table of Contents and Thumbnails

[0372] The server generates a table of contents and thumbnail images for the video based on the extracted topics and highlights.

[0373] Input: Topic and highlight information (JSON format)

[0374] Output: Table of contents and thumbnail images

[0375] Specific operation: The server parses the JSON data and automatically generates a table of contents such as "00:00 - Opening" and "02:15 - Introduction to the new game." It also uses OpenCV and PIL libraries to generate and save thumbnail images for each segment.

[0376] Step 5: Keyword search

[0377] Users input specific keywords into the search interface, and the server searches for relevant videos based on the input keywords and displays the results.

[0378] Input: User's search keyword

[0379] Output: Related video list with table of contents links and thumbnails

[0380] Specific operation: When a user enters "new game introduction" into the search interface, the server references the text data and table of contents information, and displays a list of related videos and their table of contents links. The user can click on the link to directly access the relevant section.

[0381] Step 6: Emotion Recognition and Data Collection

[0382] The emotion engine uses the camera and microphone to analyze emotions in real time while the user is watching a video and transmits the results to a server.

[0383] Input: User's facial image data and voice data

[0384] Output: Emotion data (smile, surprise, sadness, etc.)

[0385] Specific operation: The emotion engine uses FaceAPI and Emotion API to determine the user's emotional state and sends the data to the server. The emotion data is stored in the database along with the viewing data.

[0386] Step 7: Providing recommended content

[0387] The server recommends optimal content to the user based on the collected emotion data.

[0388] Input: Emotion data and viewing data

[0389] Output: Recommended content

[0390] How it works: The server uses an algorithm to analyze emotion data and viewing data. For example, if the user is smiling a lot, the server will recommend more interesting videos. Recommended content is provided via notifications and the user interface.

[0391] Step 8: Displaying the analysis results on a dashboard

[0392] The server displays the collected and analyzed viewing and sentiment data on a dashboard for the broadcaster.

[0393] Input: Viewing data and emotion data

[0394] Output: Display analysis results for the broadcaster

[0395] Specific operation: The server generates a heat map of viewing time and emotional data for each segment based on viewing data and emotional data, and displays it on the dashboard. The broadcaster can use this information to improve the content of their next broadcast.

[0396] (Application example 2)

[0397] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0398] Conventional video distribution systems make it difficult for viewers to efficiently find the information they are looking for from a vast amount of video content. Furthermore, providing personalized recommended content based on the viewer's emotions is difficult, and this has not sufficiently improved the viewing experience. Therefore, there is a need for a system that not only converts audio into text and extracts video topics and highlights, but also uses viewer emotional data to recommend appropriate content.

[0399] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0400] In this invention, the server includes: means for converting video audio into text using a generative AI; means for analyzing the text and automatically extracting video topics and highlights; means for automatically generating a video table of contents based on the extracted topics and highlights; means for generating thumbnail images associated with the table of contents; means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to relevant sections; means for collecting viewing data and providing analysis results to broadcasters; an emotion engine for recognizing the user's emotional state; and means for providing recommended content to the user based on the emotion engine. This enables viewers to efficiently find desired information from a vast amount of video content and also provides personalized recommended content based on the viewer's emotions.

[0401] "Generative AI" is a generative artificial intelligence technology that primarily uses deep learning models to generate or convert text, audio, and images.

[0402] "Video audio to text conversion" refers to the process of analyzing the audio data in a video and converting it into corresponding text data.

[0403] A "topic" is a major theme or topic that summarizes the content of a video.

[0404] "Highlights" refer to particularly important moments or scenes in a video that are of great interest to viewers.

[0405] A "table of contents" is an organized list of the video's contents that provides information including the start times of each section.

[0406] A "thumbnail image" is a small image that represents a particular scene in a video and functions as a visual table of contents.

[0407] "Keywords" are specific words or phrases used to search for video content.

[0408] "Distributor name" is the name of the person or organization that published the video.

[0409] "Viewing data" refers to data obtained when a viewer watches a video, and includes information such as playback time, viewing locations, and viewing frequency.

[0410] The "emotion engine" is a technology that analyzes the user's facial expressions and voice to recognize their emotional state in real time.

[0411] "Recommended content" is highly relevant content that is selected based on the user's viewing history and emotional state and provided to the user.

[0412] To implement this invention, the following system configuration is adopted. The server implements a program that uses generative AI to convert video audio into text and analyze it. The generative AI model used in this process is, for example, Transformer-based speech recognition technology. Specific software used includes the Transformer model library and TextBlob for natural language processing. In addition, facial recognition technology and voice analysis technology are implemented in the emotion engine to analyze user emotions in real time.

[0413] The text data generated by the server is analyzed using natural language processing technology to automatically extract topics and highlights from the video. This automatically generates a table of contents and thumbnail images for the video. Users access the system using their devices and search for videos using specific keywords, dates, or broadcaster names. As a result, a link is provided that takes users directly to the relevant part of the video.

[0414] The emotion engine analyzes the user's facial expressions and voice while watching to recognize their emotional state in real time. Based on this emotion data, the server provides recommended content to the user. For example, if the user is detected as laughing, the server will recommend more funny scenes or relaxing content.

[0415] For example, when a user watches a "new game walkthrough video," the server converts the audio data into text and extracts topics such as "introduction," "game strategy," and "conclusion." If the emotion engine detects a smile while the viewer is watching, the server recommends related, interesting scenes. This allows viewers to quickly access the content they want to watch and allows streamers to sustainably capture viewers' attention.

[0416] Examples of prompt sentences include the following:

[0417] "Users are smiling while watching. Recommend content they'll enjoy."

[0418] By operating the system in this way, viewers can efficiently find the information they are looking for from a vast amount of video content, and it is also possible to provide personalized recommended content based on the viewer's emotions.

[0419] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0420] Step 1: Upload video and convert audio

[0421] A user uploads a video from their device to the server. The server then uses a generative AI model (Transformer-based speech recognition technology) to convert the audio data in the video into text data. The input is the video file, and the output is text data.

[0422] Step 2: Text analysis and topic extraction

[0423] The server analyzes the converted text data and automatically extracts topics and highlights from the video using natural language processing technology (such as TextBlob). The input is text data, and the output is the extracted topics and highlights. Specific operations include topic segmentation and extraction of important words and phrases.

[0424] Step 3: Generate the Table of Contents and Thumbnails

[0425] The server automatically generates a table of contents for the video based on the extracted topics and highlights, and also generates thumbnail images for each section. The input is the topics and highlights, and the output is the table of contents and thumbnail images. The table of contents includes the start time of each topic.

[0426] Step 4: Providing keyword search functionality

[0427] Users can search for videos by entering specific keywords, dates, or streamers on their devices. The server searches for video data corresponding to those keywords and displays related highlights and table of contents links. The input is a search query, and the output is a list of related videos and links.

[0428] Step 5: Collect and analyze emotion data

[0429] While a user is watching a video, the server uses an emotion engine to analyze the user's emotional state in real time. The input is facial expression data and voice data during viewing, and the output is emotion data (e.g., smile, surprise, etc.). Specific operations use face recognition technology and voice analysis technology.

[0430] Step 6: Generate recommended content

[0431] The server generates content recommendations appropriate for the user based on the collected emotion data. For this, it can use a generative AI model. For example, if a smile is detected, it can recommend more funny scenes or relaxing content. The input is emotion data, and the output is a list of recommended content.

[0432] Step 7: Displaying the analysis results on a dashboard

[0433] The server collects and analyzes viewing data and emotional data, and displays the results on the streamer's dashboard. This allows streamers to check data based on viewers' high interest and emotional state, and use it to plan their next content. The input is viewing data and emotional data, and the output is the analysis results on the dashboard.

[0434] The above steps will realize a system that allows viewers to efficiently find the information they are looking for from a vast amount of video content and enjoy personalized recommended content based on their emotions.

[0435] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0436] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0437] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0438] [Second embodiment]

[0439] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0440] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0441] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0442] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0443] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0444] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0445] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0446] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0447] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0448] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0449] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0450] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0451] This invention is a system that utilizes generative AI to efficiently search and analyze the content of video streams. In this system, the server, terminal, and user elements work together to convert and analyze the audio of videos into text, and provide an interface that is visually and functionally easy to use.

[0452] Program processing explanation

[0453] Video upload and analysis

[0454] When a streamer uploads a video from their device, the server converts the audio into text using generative AI. Specifically, the server uses speech recognition technology to analyze the audio in the video in real time and convert it into text as subtitles.

[0455] The server then analyzes this text data and automatically extracts topics and highlights from the video. In this process, natural language processing technology is used to identify topics (e.g., game walkthroughs, Q&A corners, etc.) and highlight scenes (e.g., important announcements, funny moments, etc.) from the text content.

[0456] Table of contents and thumbnail generation

[0457] Based on the extracted topics and highlights, the server automatically generates a table of contents for the video, such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A session," and "20:00 - Ending."

[0458] The server also generates thumbnail images of each section, allowing viewers to easily navigate through the table of contents and jump to parts that interest them.

[0459] Providing search functionality

[0460] Users access the system through a terminal and use an interface that allows them to search for videos by specific keywords, dates, or streamer names. If a user types in a search term like "new game introductions," the server quickly identifies relevant videos and displays them as search results.

[0461] The server also provides links to relevant highlights and table of contents in search results, allowing users to jump directly to the parts they are interested in. This allows users to quickly find the information they are looking for among a vast amount of video content.

[0462] Providing analysis results

[0463] The server collects viewing data and analyzes it to determine which parts were viewed the most and which parts attracted the most viewers' attention. For example, it can collect information such as, "In the April 15, 2023 video, the Q&A session starting at 10:45 was particularly popular."

[0464] The server provides the streamer with the collected and analyzed data in the form of a dashboard. Based on this data, the streamer can plan and improve the content of their next stream. For example, it becomes easier to make decisions such as, "Viewers will be pleased if we improve the Q&A section in the next stream."

[0465] Specific examples

[0466] For example, if a streamer uploads a "walkthrough video for a new game," the server converts the audio into text and analyzes it to extract the topics "Introduction," "Game Strategy," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are presented to viewers in a visually easy-to-understand format.

[0467] When a viewer searches for "game tips," the server displays search results such as "game tips from 03:00" and also provides a link that allows the user to access that section directly. Furthermore, the server analyzes the viewing data for this video and notifies the streamer that the "game tips" section in particular is being viewed a lot. Based on this information, the streamer can reflect this in their next content creation.

[0468] The processing flow will be explained below.

[0469] Step 1:

[0470] The streamer uses a device to upload the video to the server.

[0471] Specifically, the streamer selects a video file on the device and clicks the upload button, and the device sends the video file to the server.

[0472] Step 2:

[0473] The server stores the received video files in a database.

[0474] Once saved, the process of converting the video's audio into text begins using generative AI.

[0475] Step 3:

[0476] The server uses generative AI to convert the video's audio data into text in real time.

[0477] Using voice recognition technology, what is said in the video is converted into text as subtitles.

[0478] Step 4:

[0479] The server analyzes the generated text data.

[0480] Using natural language processing technology, topics and highlights are automatically extracted from the text content. For example, topics such as "game strategies" and "question corners" are identified.

[0481] Step 5:

[0482] The server automatically generates a table of contents for the video based on the extracted topics and highlights.

[0483] For example, create table of contents entries such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A," and "20:00 - Ending."

[0484] Step 6:

[0485] The server generates thumbnail images corresponding to each section of the table of contents.

[0486] Thumbnail images are displayed on the viewing screen along with the table of contents, serving as a visual guide.

[0487] Step 7:

[0488] A user accesses the system through a terminal and searches for videos using a search interface.

[0489] When a user enters a specific keyword (e.g., "new game introduction"), the device sends this search query to the server.

[0490] Step 8:

[0491] The server receives the search query and searches the database to identify relevant videos and topics.

[0492] The server generates related videos and table of contents links as search results and returns them to the user.

[0493] Step 9:

[0494] Users receive search results on their devices and can jump directly to the relevant section by clicking on the displayed link.

[0495] This feature allows users to quickly access specific moments or sections.

[0496] Step 10:

[0497] The server collects viewing data.

[0498] For example, data is collected about which parts were most viewed and at what parts viewers stopped watching.

[0499] Step 11:

[0500] The server analyzes the collected viewing data and provides the analysis results to the broadcaster.

[0501] The analysis results are displayed on a dashboard dedicated to the streamer, and streamers can use this data to plan and improve their content.

[0502] In this way, through specific actions at each step, a series of processes from video uploading to viewing analysis is completed.

[0503] Example 1

[0504] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0505] Today, a huge amount of video content is being produced across a wide variety of formats. However, a lack of efficient support for searching, analyzing, and viewing this video content presents a challenge. In particular, there is a lack of functionality to convert audio within a video into text and then analyze it to quickly provide the information viewers are looking for. As a result, viewers must watch the entire video, wasting time and effort. Furthermore, streamers are unable to effectively receive feedback on the results of analyzing viewing data, making it difficult to improve the content of their next stream.

[0506] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0507] In this invention, the server includes a means for converting the audio of a video into text, a means for analyzing the text and automatically extracting topics and highlights of the video, a means for automatically generating a table of contents for the video based on the extracted topics and highlights, a means for generating images associated with the table of contents, a means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to the relevant parts, and a means for collecting viewing data and providing the analysis results to the broadcaster. This allows the server to efficiently analyze the content of the video and automatically generate a table of contents, enabling viewers to quickly access the information they desire. Furthermore, by providing broadcasters with feedback on the viewing data, the server can effectively improve the content of their next broadcast.

[0508] "Means for converting video audio into text" refers to technology that analyzes the audio data in a video and generates corresponding text data from the audio.

[0509] "Means for analyzing the text and automatically extracting topics and highlights from the video" refers to technology that analyzes the generated text data and automatically identifies important themes and noteworthy scenes.

[0510] "Means for automatically generating a video table of contents based on extracted topics and highlights" refers to a technology that organizes the start time and content of videos in list format based on the topic and highlight information obtained from the analysis results.

[0511] The "means for generating an image associated with the table of contents" refers to a technique for selecting a frame of a video corresponding to each item in the table of contents and generating it as a still image.

[0512] "Means of searching videos by keyword, date, or streamer name and providing links to jump directly to the relevant section" refers to technology that finds related videos based on the search criteria specified by the viewer and generates links that allow quick access to the relevant section.

[0513] "Means of collecting viewing data and providing analysis results to broadcasters" refers to technology that records and analyzes viewers' viewing behavior and provides feedback to broadcasters on information such as the most viewed parts and parts that viewers were most interested in.

[0514] "Means for converting to text at specified times" refers to technology that converts audio at specific points or sections of a video into text data.

[0515] "Natural language processing technology" refers to technology that allows computers to analyze and understand human language.

[0516] "Means for collecting and analyzing viewing data for specific parts" refers to technology that records viewers' viewing behavior in specific sections of a video and conducts detailed analysis based on that information.

[0517] "Means of providing distributors with information on popular viewing sections" refers to technology that notifies distributors of information on sections that are of high interest to viewers based on collected viewing data.

[0518] "An interface that allows users to search for previously distributed videos by entering specific keywords" refers to a user interface that allows viewers to search for previously distributed videos by entering keywords in a search box.

[0519] "Means of displaying related videos and their table of contents links based on a search query" refers to technology that identifies videos that match the search criteria entered by the viewer and displays them along with a table of contents for those videos.

[0520] "Means of providing links that allow viewers to jump directly from search results to specific sections of the video" refers to technology that allows viewers to access the specific part of the video they are looking for directly by simply clicking a link from the search results.

[0521] This invention is a system that utilizes generative AI to efficiently search and analyze video streaming content. In this system, the server, terminal, and user elements work together to convert and analyze the audio of videos into text, and provide an interface that is visually and functionally easy to use.

[0522] Hardware and Software Configuration

[0523] The system consists of the following main components:

[0524] Server: Use a high-performance cloud server (e.g., AWS, Google Cloud) to perform video analysis and database management.

[0525] Device: Includes any device that can connect to the internet, such as a computer, smartphone, or tablet, that a user uses.

[0526] Speech recognition technology: Converts speech to text using the Google Cloud Speech-to-Text API.

[0527] Natural language processing technology: OpenAI GPT-4 is used to analyze text data and automatically extract topics and highlights.

[0528] How to convert video audio to text

[0529] When a user uploads a video file from their device, the server stores it in Google Cloud Storage and calls the Google Cloud Speech-to-Text API to convert the audio in the video into text in real time. This text data is stored on the server.

[0530] A means for automatically extracting topics and highlights of a video by analyzing the text.

[0531] The server analyzes the stored text data using OpenAI GPT-4. Specifically, it analyzes the entire text using natural language processing technology, identifies frequently occurring words and important phrases, and automatically extracts topics (e.g., game walkthroughs, Q&A corners, etc.) and highlights (important announcements, funny moments, etc.) from the video.

[0532] A means to automatically generate a video table of contents based on extracted topics and highlights

[0533] The server automatically generates a table of contents based on the topic and highlight information obtained from the analysis, including the start time and title of each section. For example, it might generate a table of contents such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A Corner," and "20:00 - Ending."

[0534] means for generating an image associated with said table of contents;

[0535] The server extracts specific frames from the video and uses them to generate thumbnail images for each section. Specifically, it selects the frame closest to the start time of each section and processes it as the thumbnail.

[0536] A way to search for videos by keyword, date, or streamer name and provide links to jump directly to the relevant section

[0537] Users access the system through their terminals and enter keywords, dates, and broadcaster names into the search box. The server searches the database for relevant videos based on the query and displays the links, table of contents, and highlight information for the relevant videos. Users can also click on these links to directly access specific sections of the related videos.

[0538] A means of collecting viewing data and providing analysis results to broadcasters

[0539] The server records and analyzes the viewer's viewing behavior, such as the start time, end time, and length of each viewing section. The collected and analyzed data is provided to the streamer in the form of a dashboard. The streamer can use this information to plan and improve the content of their next stream.

[0540] Specific examples

[0541] For example, if a streamer wants to upload a "walkthrough video for a new game," the user selects the video from their device and begins uploading. The device sends the video file to the server, which receives the file and stores it in Google Cloud Storage. The server uses the Google Cloud Speech-to-Text API to convert the audio into text and saves it in a database. The server analyzes the saved text with OpenAI GPT-4 and extracts topics such as "Introduction," "Game Strategy," and "Conclusion." Based on these topics, a table of contents and thumbnail images for each section are automatically generated.

[0542] When a user searches for the keyword "game tips," the server displays related videos and links to those sections. The user can then access those sections directly. The server then analyzes the viewing data and notifies the streamer that the "game tips" section in particular is being viewed a lot. Based on this information, the streamer can reflect this in their next content creation.

[0543] Prompt Sentence Examples

[0544] "I have uploaded a walkthrough video for a new game. Please extract the topics of this video - introduction, game strategy, and conclusion - from their respective timestamps and automatically generate a table of contents and thumbnail images."

[0545] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0546] Step 1:

[0547] The user selects the video file from the terminal through the system's web interface and clicks the upload button.

[0548] Input: A video file selected by the user

[0549] Output: Video file sent from device to server

[0550] Specific behavior:

[0551] When a user selects a video file on the device and clicks the upload button on the web interface, the device sends the video file to the server, which receives the video file and stores it in storage.

[0552] Step 2:

[0553] The server stores the received video file in Google Cloud Storage and calls the Google Cloud Speech-to-Text API to convert the audio in the video into text.

[0554] Input: Received video file

[0555] Output: Converted text data

[0556] Specific behavior:

[0557] The server stores the video file in Google Cloud Storage, then uses the Google Cloud Speech-to-Text API to convert the video audio into text, which is then stored in a database on the server.

[0558] Step 3:

[0559] The server analyzes the stored text data using OpenAI GPT-4 and automatically extracts topics and highlights within the video.

[0560] Input: Text data

[0561] Output: Topic and highlight information

[0562] Specific behavior:

[0563] The server receives the text data and analyzes it using OpenAI GPT-4. This analysis extracts important topics and highlights from the text. Specifically, it identifies frequently occurring words and important phrases and identifies topics and highlights based on them.

[0564] Step 4:

[0565] The server automatically generates a table of contents for the video based on the extracted topics and highlights.

[0566] Input: Topic and highlight information

[0567] Output: Table of contents data

[0568] Specific behavior:

[0569] The server then uses the obtained topic and highlight information to organize the start times and titles of each section in a table of contents format, including items such as "00:00 - Opening" and "02:15 - Introduction to the new game."

[0570] Step 5:

[0571] The server extracts frames of the video associated with the table of contents and uses them to generate thumbnail images.

[0572] Input: Table of contents data

[0573] Output: Thumbnail image

[0574] Specific behavior:

[0575] The server selects the frame closest to the start of each section and generates a thumbnail of that frame, which is then associated with the table of contents, allowing viewers to visually identify the parts of the video that interest them.

[0576] Step 6:

[0577] Users access the system through their terminal and perform a search by entering keywords, dates, and broadcaster names in the search box.

[0578] Input: Keywords, date, broadcaster name

[0579] Output: Search results (related video links, table of contents, highlight information)

[0580] Specific behavior:

[0581] When a user enters a keyword into the search box, the server searches the database for relevant video data, and displays the link to the video, its table of contents, and highlights to the user. Users can also click on the provided link to go directly to the specific section of the video they are looking for.

[0582] Step 7:

[0583] The server collects viewing data and analyzes which parts are viewed the most.

[0584] Input: Viewing data

[0585] Output: Analysis results of viewing data

[0586] Specific behavior:

[0587] The server records viewers' viewing behavior and collects and analyzes viewing data. The analysis results include the frequency of viewing each section and the parts that viewers are most interested in. This data is provided to the broadcaster in the form of a dashboard.

[0588] Step 8:

[0589] The server provides feedback to the distributor based on the results of analyzing the viewing data.

[0590] Input: Analysis results of viewing data

[0591] Output: Feedback information provided to the distributor

[0592] Specific behavior:

[0593] The server then sends the analysis results to the streamer. For example, it can provide information on which specific sections are most popular, which can help the streamer in creating the next piece of content. This feedback allows the streamer to plan and improve content that will attract viewers' attention.

[0594] (Application example 1)

[0595] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0596] With current video streaming services, the sheer volume of video content makes it difficult for users to quickly find specific information. It is also labor-intensive for streamers to use viewing data to improve their next content offering. This reduces convenience for both users and streamers, making it difficult to provide effective content. It is necessary to solve these issues and improve the efficiency of video search and streamer data utilization.

[0597] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0598] In this invention, the server includes: means for converting video audio into text using generative AI; means for analyzing the text and automatically extracting topics and highlights of the video; means for automatically generating a table of contents for the video based on the extracted topics and highlights; means for generating thumbnail images associated with the table of contents; means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to relevant parts; means for collecting viewing data and providing analysis results to broadcasters; means for generating a video timeline based on time axis information extracted from the text; and means for generating thumbnail images corresponding to each timeline entry. This allows users to quickly find parts of a video that interest them, and enables broadcasters to easily utilize viewing data to improve their next broadcast.

[0599] "Generative AI" is a technology that uses artificial intelligence technology to generate data such as text, audio, and images.

[0600] "Means for converting audio to text" refers to a technology that analyzes the audio in a video and outputs it as text information.

[0601] "Means for automatic topic and highlight extraction" refers to technology that automatically identifies important themes and points of interest from text data.

[0602] "Means for automatically generating a table of contents" refers to technology that automatically generates a table of contents to provide users with a table of contents that identifies the major sections within a video.

[0603] "Means for generating thumbnail images" refers to a technology that captures specific scenes from a video as images to be used as visual indexes.

[0604] "Means for searching and providing direct jump links" refers to technology that searches for related videos based on keywords, dates, and broadcaster names entered by the user, and generates links to directly access specific parts.

[0605] "Means of collecting viewing data and providing analytical results" refers to technology that collects and analyzes data such as video viewing history and viewing time, and provides the results to the distributor.

[0606] "Means for generating a video timeline based on time axis information" is a technology that visually represents the time axis of a video based on time information extracted from text data.

[0607] The "means for generating thumbnail images corresponding to timeline entries" is a technique for capturing an image at a specific time corresponding to each entry on the timeline and displaying it on the timeline.

[0608] The "means for proposing the next distribution plan" is a technology that automatically suggests the next distribution content to increase the viewer's interest based on the previous viewing data.

[0609] The "means for individually optimized content recommendations" is a technology that recommends individually customized content based on a user's search history and viewing history.

[0610] A system for carrying out the present invention is one in which a server, a terminal, and a user work together in cooperation with each other. A specific embodiment of the system will be described below.

[0611] Server Roles

[0612] First, when a user uploads a video from their device, the server converts the audio into text using generative AI. In this process, the server extracts audio data from the video file using a speech recognition library (e.g., speech_recognition) and a movie processing library (e.g., moviepy) and converts it into text data.

[0613] The server then analyzes this text data using a generative AI model (e.g., OpenAI's API) to automatically extract topics and highlights within the video, thereby identifying which sections are important.

[0614] Based on the extracted topics and highlights, the server automatically generates a table of contents for the video, allowing viewers to easily understand the content of the video and quickly access parts that interest them. Additionally, thumbnail images associated with the table of contents are also generated. For example, the server captures a frame at the beginning of each section and uses it as a thumbnail.

[0615] Role of user terminal

[0616] The user device provides an interface that allows viewers to search for videos by specific keywords, dates, or broadcaster names. When a user enters a search query, the server returns related videos and their table of contents links. Based on the search results, users can use links to jump directly to the parts that interest them. This allows viewers to quickly find the information they are looking for among a vast amount of video content.

[0617] Data analysis and feedback

[0618] The server also collects viewing data and provides analysis results to the broadcaster. Specifically, it makes suggestions for improving the content of the next broadcast based on data such as which parts viewers watched the most and which sections they were interested in. This feedback allows the broadcaster to create effective content to increase viewer interest.

[0619] Specific examples

[0620] For example, let's consider the case where a streamer uploads a "walkthrough video for a new game." The server converts the audio into text and analyzes it to extract topics such as "Introduction," "Game Strategy," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are presented to viewers in a visually easy-to-understand format.

[0621] When a viewer searches for "game tips," the server will display search results such as "game tips from 03:00" and provide a link that allows the user to access that section directly. An example of a prompt sentence that can be used in this case is as follows:

[0622] Prompt Sentence Examples

[0623] Extract the main topics and highlights from the following text:

[0624] "This is a walkthrough of a new game. First, I'll explain how to choose your character. Next, I'll introduce some important points for progressing through the game. Finally, I'll explain how to beat boss battles."

[0625] This allows users to quickly access important information and allows streamers to effectively plan upcoming content based on viewing data.

[0626] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0627] Step 1:

[0628] Uploading videos

[0629] The user uses the terminal to upload a video file to the server, which receives the video file, extracts the audio data, and prepares it for the next step.

[0630] Input: Video file

[0631] Output: Audio data

[0632] Step 2:

[0633] Speech-to-text

[0634] The server uses a speech recognition library (e.g., speech_recognition) to convert the audio data into text, analyzes the extracted audio data, and outputs it in text format.

[0635] Input: Audio data

[0636] Output: Text data

[0637] Step 3:

[0638] Text analytics

[0639] The server uses a generative AI model (e.g., OpenAI's API) to analyze the text data and automatically extract topics and highlights from the video. This analysis identifies the main themes and important scenes in the video.

[0640] Input: Text data

[0641] Output: Topic and highlight information

[0642] Step 4:

[0643] Automatic table of contents and thumbnail generation

[0644] The server automatically generates a table of contents for the video based on the extracted topics and highlight information, and simultaneously generates thumbnail images from the start of each section and associates them with the table of contents.

[0645] Input: Topic and highlight information

[0646] Output: Table of contents data, thumbnail images

[0647] Step 5:

[0648] Providing a search interface

[0649] The device provides the user with a search interface, displaying an input form that allows the user to search for videos by specific keywords, dates, or streamer names.

[0650] Input: None (preparation until user input)

[0651] Output: Search interface

[0652] Step 6:

[0653] Processing search queries

[0654] The user enters a search query and the device sends it to the server, which searches for related videos and table of contents links based on the query and returns the results.

[0655] Input: Search query (keyword, date, broadcaster name)

[0656] Output: Search results (related videos and table of contents links)

[0657] Step 7:

[0658] Collecting viewing data

[0659] The server collects data on users' viewing of videos (such as viewing time, viewing sections, etc.), which is later used by the distributor for analysis.

[0660] Input: Viewing activity data

[0661] Output: Recording to viewing database

[0662] Step 8:

[0663] Providing analysis results

[0664] The server analyzes the collected viewing data to determine which sections are most viewed and trends in viewer interest. The server then provides the analysis results to the broadcaster, helping them plan their next broadcast.

[0665] Input: Viewing data

[0666] Output: Analysis report

[0667] These steps allow users to quickly find what interests them and give publishers the data to improve their content next time.

[0668] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0669] This invention combines a system that utilizes generative AI to efficiently search and analyze video content with an emotion engine that recognizes user emotions. In this system, the server, terminal, and user elements work together to convert and analyze the audio of videos into text, and then provide recommended content based on the user's emotions.

[0670] Program processing explanation

[0671] Video upload and analysis

[0672] When a streamer uploads a video from their device, the server converts the audio into text using generative AI. Specifically, the server uses speech recognition technology to analyze the audio in the video in real time and convert it into text as subtitles.

[0673] The server then analyzes this text data and automatically extracts topics and highlights from the video. In this process, natural language processing technology is used to identify topics (e.g., game walkthroughs, Q&A corners, etc.) and highlight scenes (e.g., important announcements, funny moments) from the text content.

[0674] Table of contents and thumbnail generation

[0675] Based on the extracted topics and highlights, the server automatically generates a table of contents for the video, such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A session," and "20:00 - Ending."

[0676] The server also generates thumbnail images of each section, allowing viewers to easily navigate through the table of contents and jump to parts that interest them.

[0677] Providing search functionality

[0678] Users access the system through a terminal and use an interface that allows them to search for videos by specific keywords, dates, or streamer names. If a user types in a search term like "new game introductions," the server quickly identifies relevant videos and displays them as search results.

[0679] The server also provides links to relevant highlights and table of contents in search results, allowing users to jump directly to the parts they are interested in. This allows users to quickly find the information they are looking for among a vast amount of video content.

[0680] Implementing the Emotion Engine

[0681] The emotion engine uses facial recognition and voice analysis technologies to analyze the viewer's emotional state in real time while they are watching a video, determining, for example, whether they are laughing, surprised, sad, etc.

[0682] Based on the collected emotional data, the server can recommend videos and provide highlights that correspond to the user's emotional state. For example, if the server detects that the user is laughing, it can recommend more funny scenes or relaxing content.

[0683] Providing analysis results and recommended content

[0684] The server collects viewing data and emotional data, analyzes them, and generates data such as which parts were viewed the most and which parts evoked particular emotions.

[0685] The analysis results are displayed on the streamer's personal dashboard, and the streamer can use this data to plan and improve their content. Specifically, streamers can identify moments that make viewers laugh and topics that interest them and reflect this in their next stream.

[0686] Specific examples

[0687] For example, if a streamer uploads a "walkthrough video for a new game," the server converts the audio into text and analyzes it to extract the topics "Introduction," "Game Strategy," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are presented to viewers in a visually easy-to-understand format.

[0688] When a viewer searches for "game tips," the server will display search results such as "game tips from 3:00" and provide a link that allows the user to access that section directly. Furthermore, if the emotion engine detects a smile on the user's face while watching, the server will recommend funny or popular scenes from the same game. As a result, viewers can quickly access the content they want to watch, and broadcasters can maintain their viewers' interest.

[0689] The processing flow will be explained below.

[0690] Step 1:

[0691] The streamer uses a device to upload the video to the server.

[0692] Specifically, the streamer selects a video file on the device and clicks the upload button, and the device sends the video file to the server.

[0693] Step 2:

[0694] The server stores the received video files in a database.

[0695] Once saved, the process of converting the video's audio into text begins using generative AI.

[0696] Step 3:

[0697] The server uses generative AI to convert the video's audio data into text in real time.

[0698] Using voice recognition technology, what is said in the video is converted into text as subtitles.

[0699] Step 4:

[0700] The server analyzes the generated text data.

[0701] Using natural language processing technology, topics and highlights are automatically extracted from the text content. For example, topics such as "game strategies" and "question corners" are identified.

[0702] Step 5:

[0703] The server automatically generates a table of contents for the video based on the extracted topics and highlights.

[0704] For example, create table of contents entries such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A," and "20:00 - Ending."

[0705] Step 6:

[0706] The server generates thumbnail images corresponding to each section of the table of contents.

[0707] Thumbnail images are displayed on the viewing screen along with the table of contents, serving as a visual guide.

[0708] Step 7:

[0709] A user accesses the system through a terminal and searches for videos using a search interface.

[0710] When a user enters a specific keyword (e.g., "new game introduction"), the device sends this search query to the server.

[0711] Step 8:

[0712] The server receives the search query and searches the database to identify relevant videos and topics.

[0713] The server generates related videos and table of contents links as search results and returns them to the user.

[0714] Step 9:

[0715] Users receive search results on their devices and can jump directly to the relevant section by clicking on the displayed link.

[0716] This feature allows users to quickly access specific moments or sections.

[0717] Step 10:

[0718] The emotion engine analyzes the user's emotions while they are watching.

[0719] Using facial recognition and voice analysis technology, it determines in real time whether the viewer is laughing, surprised, sad, etc.

[0720] Step 11:

[0721] Based on the collected emotional data, the server provides recommended videos and highlights according to the user's emotional state.

[0722] For example, if the user is laughing, the server will suggest more funny scenes and related content.

[0723] Step 12:

[0724] The server collects and analyzes viewing data and emotion data, and generates analysis results.

[0725] Data is collected on which parts were viewed the most and how users felt about them.

[0726] Step 13:

[0727] The analysis results are displayed on a dashboard dedicated to the broadcaster.

[0728] Streamers can use this data to plan and improve their content, for example by identifying moments that made users laugh and creating strategies such as increasing the number of similar scenes in their next stream.

[0729] Example 2

[0730] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0731] With the rapid increase in video content, it is becoming increasingly difficult for viewers to efficiently search and access the information they are looking for. Furthermore, detailed viewing data is necessary for streamers to understand viewers' reactions and improve future content. However, current systems do not adequately collect and analyze not only viewing data but also emotional data during viewing, and provide content recommendations based on that data. Therefore, a comprehensive system is needed to improve the viewer experience and support streamers in content creation.

[0732] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0733] In this invention, the server includes: means for converting video audio into text using a generative AI; means for analyzing the text and automatically extracting video topics and highlights; means for automatically generating a video table of contents based on the extracted topics and highlights; means for generating thumbnail images associated with the table of contents; means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to relevant sections; means for collecting viewing data and user emotional data and providing the analysis results to the broadcaster; and means for analyzing the user's emotional state during viewing in real time using an emotion engine and recommending videos and providing highlights based on the collected emotional data. This allows viewers to efficiently access desired content, and broadcasters to understand viewer reactions in detail and improve the quality of their content in future broadcasts.

[0734] "Generative AI" refers to models or algorithms that generate data using artificial intelligence techniques.

[0735] "Text conversion" is the process of analyzing audio data and converting it into written information.

[0736] "Topic extraction" is a technique for analyzing text information and identifying key topics or themes within it.

[0737] "Highlight extraction" is a technology that automatically extracts particularly important parts or interesting scenes from video or text.

[0738] "Table of contents generation" is the process of creating easily digestible segments based on the main points of a video or piece of text.

[0739] "Thumbnail generation" is the process of extracting specific frames of a video as still images to create small visually representative images.

[0740] A "search function" is a technology that allows you to quickly find relevant information or data based on specific keywords or conditions.

[0741] "Viewing data" refers to data recorded and analyzed based on user behavior when watching videos (such as playback time and frequently viewed parts).

[0742] "Emotional data" refers to data collected by recognizing the emotional state (happiness, surprise, sadness, etc.) that viewers show while watching a video.

[0743] An "emotion engine" is a system or technology that analyzes the face and voice of a user while watching and determines their emotional state in real time.

[0744] "Recommended content" is a function that automatically suggests content that may interest a user based on the user's viewing history and emotional data.

[0745] A "viewing analysis dashboard" is a system that displays detailed viewer reactions and behavior to broadcasters based on collected viewing and emotional data.

[0746] This invention is a system for efficiently searching and analyzing video content, combining generative AI and emotion recognition technology to provide useful information to both viewers and streamers. This system is implemented primarily using the following hardware and software:

[0747] Video speech to text conversion

[0748] The server receives the video uploaded by the streamer from their device, extracts the audio using an audio processing tool (e.g., ffmpeg), and then converts the audio into text using a speech recognition service such as the Google Cloud Speech-to-Text API. The converted text is then stored in a database on the server.

[0749] Parsing text and generating a table of contents

[0750] The server analyzes the generated text using natural language processing techniques (e.g., BERT or GPT-3) to extract the topics and highlights of the video. For example, in a game walkthrough video, segments such as "Introduction," "Way of the Walkthrough," and "Conclusion" are identified. Based on this, a table of contents for the video is automatically generated, and thumbnail images corresponding to each segment are generated. OpenCV and PIL libraries are used to generate the thumbnails.

[0751] Providing search functionality

[0752] A user can access the system using a terminal and enter specific keywords into the search interface. For example, they can enter "new game introduction." The server receives this search query, matches it with the stored text data and table of contents information, identifies related videos, and displays the search results as a list. The user can click on the displayed link to jump directly to the relevant section.

[0753] Emotion recognition implementation

[0754] The emotion engine analyzes facial expressions and voice in real time via the camera and microphone while the user is watching a video. It uses FaceAPI and Emotion API to determine the user's emotions, such as smile, surprise, sadness, etc. This emotion data is sent to the server and stored in a database along with the viewing data.

[0755] Providing recommended content

[0756] The server then runs an algorithm based on the collected emotional data to recommend content that matches the user's emotional state. For example, if the user is smiling a lot, the server will recommend more entertaining videos or relaxing content. Recommended content is provided via notifications and lists.

[0757] Viewing analytics dashboard

[0758] The server generates a dashboard for streamers based on the collected viewing and emotion data. The dashboard displays the viewing time for each segment, a heat map of emotion data, and an analysis of viewer reactions. This allows streamers to understand viewer reactions in detail and use this information to improve future content creation.

[0759] Specific operation example

[0760] For example, if a streamer uploads a video explaining how to play a new game, the server converts the audio into text and analyzes it to extract topics such as "Introduction," "Game Tips," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are then provided to viewers.

[0761] When a viewer searches for "game strategies," the server displays search results such as "game strategies from 03:00," and the viewer can click on a link to directly access the relevant section. In addition, when the emotion engine detects a smile on the user's face while watching, the server recommends interesting or popular scenes from the same game, improving the user's viewing experience.

[0762] Prompt Sentence Examples

[0763] Introducing a new game

[0764] Enter: Introducing a new game

[0765] Output: A list of all videos tagged with "new game introductions"

[0766] "Tell me the highlight scenes in the video"

[0767] Input: Highlights from the video

[0768] Output: A list of highlight scenes from each video

[0769] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0770] System program processing flow

[0771] Step 1: Upload your video

[0772] The server receives a request from the distributor to upload a video file from the terminal.

[0773] Input: Video file upload request from the device

[0774] Output: The location where the uploaded video file will be saved to storage.

[0775] Specific behavior: The server receives the video file via an HTML form or API endpoint, saves it to storage, and returns a status code of 200 to confirm the upload was successful.

[0776] Step 2: Speech to text

[0777] The server extracts the audio from the stored video file and converts it into text using generative AI.

[0778] Input: Saved video file

[0779] Output: Text data

[0780] Specific operation: The server separates the audio using ffmpeg and converts the audio data to text data using the Google Cloud Speech-to-Text API. The converted text is then stored in a database.

[0781] Step 3: Parsing the text

[0782] The server analyzes the generated text data using natural language processing technology to extract topics and highlights from the video.

[0783] Input: Text data

[0784] Output: Topic and highlight information (JSON format)

[0785] Specific operation: The server uses generative AI such as BERT or GPT-3 to identify topics (e.g., "Introduction," "Game Strategy," "Conclusion") and highlight scenes, and saves the results in JSON format.

[0786] Step 4: Generate the Table of Contents and Thumbnails

[0787] The server generates a table of contents and thumbnail images for the video based on the extracted topics and highlights.

[0788] Input: Topic and highlight information (JSON format)

[0789] Output: Table of contents and thumbnail images

[0790] Specific operation: The server parses the JSON data and automatically generates a table of contents such as "00:00 - Opening" and "02:15 - Introduction to the new game." It also uses OpenCV and PIL libraries to generate and save thumbnail images for each segment.

[0791] Step 5: Keyword search

[0792] Users input specific keywords into the search interface, and the server searches for relevant videos based on the input keywords and displays the results.

[0793] Input: User's search keyword

[0794] Output: Related video list with table of contents links and thumbnails

[0795] Specific operation: When a user enters "new game introduction" into the search interface, the server references the text data and table of contents information, and displays a list of related videos and their table of contents links. The user can click on the link to directly access the relevant section.

[0796] Step 6: Emotion Recognition and Data Collection

[0797] The emotion engine uses the camera and microphone to analyze emotions in real time while the user is watching a video and transmits the results to a server.

[0798] Input: User's facial image data and voice data

[0799] Output: Emotion data (smile, surprise, sadness, etc.)

[0800] Specific operation: The emotion engine uses FaceAPI and Emotion API to determine the user's emotional state and sends the data to the server. The emotion data is stored in the database along with the viewing data.

[0801] Step 7: Providing recommended content

[0802] The server recommends optimal content to the user based on the collected emotion data.

[0803] Input: Emotion data and viewing data

[0804] Output: Recommended content

[0805] How it works: The server uses an algorithm to analyze emotion data and viewing data. For example, if the user is smiling a lot, the server will recommend more interesting videos. Recommended content is provided via notifications and the user interface.

[0806] Step 8: Displaying the analysis results on a dashboard

[0807] The server displays the collected and analyzed viewing and sentiment data on a dashboard for the broadcaster.

[0808] Input: Viewing data and emotion data

[0809] Output: Display analysis results for the broadcaster

[0810] Specific operation: The server generates a heat map of viewing time and emotional data for each segment based on viewing data and emotional data, and displays it on the dashboard. The broadcaster can use this information to improve the content of their next broadcast.

[0811] (Application example 2)

[0812] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0813] Conventional video distribution systems make it difficult for viewers to efficiently find the information they are looking for from a vast amount of video content. Furthermore, providing personalized recommended content based on the viewer's emotions is difficult, and this has not sufficiently improved the viewing experience. Therefore, there is a need for a system that not only converts audio into text and extracts video topics and highlights, but also uses viewer emotional data to recommend appropriate content.

[0814] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0815] In this invention, the server includes: means for converting video audio into text using a generative AI; means for analyzing the text and automatically extracting video topics and highlights; means for automatically generating a video table of contents based on the extracted topics and highlights; means for generating thumbnail images associated with the table of contents; means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to relevant sections; means for collecting viewing data and providing analysis results to broadcasters; an emotion engine for recognizing the user's emotional state; and means for providing recommended content to the user based on the emotion engine. This enables viewers to efficiently find desired information from a vast amount of video content and also provides personalized recommended content based on the viewer's emotions.

[0816] "Generative AI" is a generative artificial intelligence technology that primarily uses deep learning models to generate or convert text, audio, and images.

[0817] "Video audio to text conversion" refers to the process of analyzing the audio data in a video and converting it into corresponding text data.

[0818] A "topic" is a major theme or topic that summarizes the content of a video.

[0819] "Highlights" refer to particularly important moments or scenes in a video that are of great interest to viewers.

[0820] A "table of contents" is an organized list of the video's contents that provides information including the start times of each section.

[0821] A "thumbnail image" is a small image that represents a particular scene in a video and functions as a visual table of contents.

[0822] "Keywords" are specific words or phrases used to search for video content.

[0823] "Distributor name" is the name of the person or organization that published the video.

[0824] "Viewing data" refers to data obtained when a viewer watches a video, and includes information such as playback time, viewing locations, and viewing frequency.

[0825] The "emotion engine" is a technology that analyzes the user's facial expressions and voice to recognize their emotional state in real time.

[0826] "Recommended content" is highly relevant content that is selected based on the user's viewing history and emotional state and provided to the user.

[0827] To implement this invention, the following system configuration is adopted. The server implements a program that uses generative AI to convert video audio into text and analyze it. The generative AI model used in this process is, for example, Transformer-based speech recognition technology. Specific software used includes the Transformer model library and TextBlob for natural language processing. In addition, facial recognition technology and voice analysis technology are implemented in the emotion engine to analyze user emotions in real time.

[0828] The text data generated by the server is analyzed using natural language processing technology to automatically extract topics and highlights from the video. This automatically generates a table of contents and thumbnail images for the video. Users access the system using their devices and search for videos using specific keywords, dates, or broadcaster names. As a result, a link is provided that takes users directly to the relevant part of the video.

[0829] The emotion engine analyzes the user's facial expressions and voice while watching to recognize their emotional state in real time. Based on this emotion data, the server provides recommended content to the user. For example, if the user is detected as laughing, the server will recommend more funny scenes or relaxing content.

[0830] For example, when a user watches a "new game walkthrough video," the server converts the audio data into text and extracts topics such as "introduction," "game strategy," and "conclusion." If the emotion engine detects a smile while the viewer is watching, the server recommends related, interesting scenes. This allows viewers to quickly access the content they want to watch and allows streamers to sustainably capture viewers' attention.

[0831] Examples of prompt sentences include the following:

[0832] "Users are smiling while watching. Recommend content they'll enjoy."

[0833] By operating the system in this way, viewers can efficiently find the information they are looking for from a vast amount of video content, and it is also possible to provide personalized recommended content based on the viewer's emotions.

[0834] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0835] Step 1: Upload video and convert audio

[0836] A user uploads a video from their device to the server. The server then uses a generative AI model (Transformer-based speech recognition technology) to convert the audio data in the video into text data. The input is the video file, and the output is text data.

[0837] Step 2: Text analysis and topic extraction

[0838] The server analyzes the converted text data and automatically extracts topics and highlights from the video using natural language processing technology (such as TextBlob). The input is text data, and the output is the extracted topics and highlights. Specific operations include topic segmentation and extraction of important words and phrases.

[0839] Step 3: Generate the Table of Contents and Thumbnails

[0840] The server automatically generates a table of contents for the video based on the extracted topics and highlights, and also generates thumbnail images for each section. The input is the topics and highlights, and the output is the table of contents and thumbnail images. The table of contents includes the start time of each topic.

[0841] Step 4: Providing keyword search functionality

[0842] Users can search for videos by entering specific keywords, dates, or streamers on their devices. The server searches for video data corresponding to those keywords and displays related highlights and table of contents links. The input is a search query, and the output is a list of related videos and links.

[0843] Step 5: Collect and analyze emotion data

[0844] While a user is watching a video, the server uses an emotion engine to analyze the user's emotional state in real time. The input is facial expression data and voice data during viewing, and the output is emotion data (e.g., smile, surprise, etc.). Specific operations use face recognition technology and voice analysis technology.

[0845] Step 6: Generate recommended content

[0846] The server generates content recommendations appropriate for the user based on the collected emotion data. For this, it can use a generative AI model. For example, if a smile is detected, it can recommend more funny scenes or relaxing content. The input is emotion data, and the output is a list of recommended content.

[0847] Step 7: Displaying the analysis results on a dashboard

[0848] The server collects and analyzes viewing data and emotional data, and displays the results on the streamer's dashboard. This allows streamers to check data based on viewers' high interest and emotional state, and use it to plan their next content. The input is viewing data and emotional data, and the output is the analysis results on the dashboard.

[0849] The above steps will realize a system that allows viewers to efficiently find the information they are looking for from a vast amount of video content and enjoy personalized recommended content based on their emotions.

[0850] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0851] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0852] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0853] [Third embodiment]

[0854] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0855] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0856] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0857] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0858] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0859] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0860] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0861] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0862] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0863] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0864] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0865] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0866] This invention is a system that utilizes generative AI to efficiently search and analyze the content of video streams. In this system, the server, terminal, and user elements work together to convert and analyze the audio of videos into text, and provide an interface that is visually and functionally easy to use.

[0867] Program processing explanation

[0868] Video upload and analysis

[0869] When a streamer uploads a video from their device, the server converts the audio into text using generative AI. Specifically, the server uses speech recognition technology to analyze the audio in the video in real time and convert it into text as subtitles.

[0870] The server then analyzes this text data and automatically extracts topics and highlights from the video. In this process, natural language processing technology is used to identify topics (e.g., game walkthroughs, Q&A corners, etc.) and highlight scenes (e.g., important announcements, funny moments, etc.) from the text content.

[0871] Table of contents and thumbnail generation

[0872] Based on the extracted topics and highlights, the server automatically generates a table of contents for the video, such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A session," and "20:00 - Ending."

[0873] The server also generates thumbnail images of each section, allowing viewers to easily navigate through the table of contents and jump to parts that interest them.

[0874] Providing search functionality

[0875] Users access the system through a terminal and use an interface that allows them to search for videos by specific keywords, dates, or streamer names. If a user types in a search term like "new game introductions," the server quickly identifies relevant videos and displays them as search results.

[0876] The server also provides links to relevant highlights and table of contents in search results, allowing users to jump directly to the parts they are interested in. This allows users to quickly find the information they are looking for among a vast amount of video content.

[0877] Providing analysis results

[0878] The server collects viewing data and analyzes it to determine which parts were viewed the most and which parts attracted the most viewers' attention. For example, it can collect information such as, "In the April 15, 2023 video, the Q&A session starting at 10:45 was particularly popular."

[0879] The server provides the streamer with the collected and analyzed data in the form of a dashboard. Based on this data, the streamer can plan and improve the content of their next stream. For example, it becomes easier to make decisions such as, "Viewers will be pleased if we improve the Q&A section in the next stream."

[0880] Specific examples

[0881] For example, if a streamer uploads a "walkthrough video for a new game," the server converts the audio into text and analyzes it to extract the topics "Introduction," "Game Strategy," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are presented to viewers in a visually easy-to-understand format.

[0882] When a viewer searches for "game tips," the server displays search results such as "game tips from 03:00" and also provides a link that allows the user to access that section directly. Furthermore, the server analyzes the viewing data for this video and notifies the streamer that the "game tips" section in particular is being viewed a lot. Based on this information, the streamer can reflect this in their next content creation.

[0883] The processing flow will be explained below.

[0884] Step 1:

[0885] The streamer uses a device to upload the video to the server.

[0886] Specifically, the streamer selects a video file on the device and clicks the upload button, and the device sends the video file to the server.

[0887] Step 2:

[0888] The server stores the received video files in a database.

[0889] Once saved, the process of converting the video's audio into text begins using generative AI.

[0890] Step 3:

[0891] The server uses generative AI to convert the video's audio data into text in real time.

[0892] Using voice recognition technology, what is said in the video is converted into text as subtitles.

[0893] Step 4:

[0894] The server analyzes the generated text data.

[0895] Using natural language processing technology, topics and highlights are automatically extracted from the text content. For example, topics such as "game strategies" and "question corners" are identified.

[0896] Step 5:

[0897] The server automatically generates a table of contents for the video based on the extracted topics and highlights.

[0898] For example, create table of contents entries such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A," and "20:00 - Ending."

[0899] Step 6:

[0900] The server generates thumbnail images corresponding to each section of the table of contents.

[0901] Thumbnail images are displayed on the viewing screen along with the table of contents, serving as a visual guide.

[0902] Step 7:

[0903] A user accesses the system through a terminal and searches for videos using a search interface.

[0904] When a user enters a specific keyword (e.g., "new game introduction"), the device sends this search query to the server.

[0905] Step 8:

[0906] The server receives the search query and searches the database to identify relevant videos and topics.

[0907] The server generates related videos and table of contents links as search results and returns them to the user.

[0908] Step 9:

[0909] Users receive search results on their devices and can jump directly to the relevant section by clicking on the displayed link.

[0910] This feature allows users to quickly access specific moments or sections.

[0911] Step 10:

[0912] The server collects viewing data.

[0913] For example, data is collected about which parts were most viewed and at what parts viewers stopped watching.

[0914] Step 11:

[0915] The server analyzes the collected viewing data and provides the analysis results to the broadcaster.

[0916] The analysis results are displayed on a dashboard dedicated to the streamer, and streamers can use this data to plan and improve their content.

[0917] In this way, through specific actions at each step, a series of processes from video uploading to viewing analysis is completed.

[0918] Example 1

[0919] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0920] Today, a huge amount of video content is being produced across a wide variety of formats. However, a lack of efficient support for searching, analyzing, and viewing this video content presents a challenge. In particular, there is a lack of functionality to convert audio within a video into text and then analyze it to quickly provide the information viewers are looking for. As a result, viewers must watch the entire video, wasting time and effort. Furthermore, streamers are unable to effectively receive feedback on the results of analyzing viewing data, making it difficult to improve the content of their next stream.

[0921] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0922] In this invention, the server includes a means for converting the audio of a video into text, a means for analyzing the text and automatically extracting topics and highlights of the video, a means for automatically generating a table of contents for the video based on the extracted topics and highlights, a means for generating images associated with the table of contents, a means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to the relevant parts, and a means for collecting viewing data and providing the analysis results to the broadcaster. This allows the server to efficiently analyze the content of the video and automatically generate a table of contents, enabling viewers to quickly access the information they desire. Furthermore, by providing broadcasters with feedback on the viewing data, the server can effectively improve the content of their next broadcast.

[0923] "Means for converting video audio into text" refers to technology that analyzes the audio data in a video and generates corresponding text data from the audio.

[0924] "Means for analyzing the text and automatically extracting topics and highlights from the video" refers to technology that analyzes the generated text data and automatically identifies important themes and noteworthy scenes.

[0925] "Means for automatically generating a video table of contents based on extracted topics and highlights" refers to a technology that organizes the start time and content of videos in list format based on the topic and highlight information obtained from the analysis results.

[0926] The "means for generating an image associated with the table of contents" refers to a technique for selecting a frame of a video corresponding to each item in the table of contents and generating it as a still image.

[0927] "Means of searching videos by keyword, date, or streamer name and providing links to jump directly to the relevant section" refers to technology that finds related videos based on the search criteria specified by the viewer and generates links that allow quick access to the relevant section.

[0928] "Means of collecting viewing data and providing analysis results to broadcasters" refers to technology that records and analyzes viewers' viewing behavior and provides feedback to broadcasters on information such as the most viewed parts and parts that viewers were most interested in.

[0929] "Means for converting to text at specified times" refers to technology that converts audio at specific points or sections of a video into text data.

[0930] "Natural language processing technology" refers to technology that allows computers to analyze and understand human language.

[0931] "Means for collecting and analyzing viewing data for specific parts" refers to technology that records viewers' viewing behavior in specific sections of a video and conducts detailed analysis based on that information.

[0932] "Means of providing distributors with information on popular viewing sections" refers to technology that notifies distributors of information on sections that are of high interest to viewers based on collected viewing data.

[0933] "An interface that allows users to search for previously distributed videos by entering specific keywords" refers to a user interface that allows viewers to search for previously distributed videos by entering keywords in a search box.

[0934] "Means of displaying related videos and their table of contents links based on a search query" refers to technology that identifies videos that match the search criteria entered by the viewer and displays them along with a table of contents for those videos.

[0935] "Means of providing links that allow viewers to jump directly from search results to specific sections of the video" refers to technology that allows viewers to access the specific part of the video they are looking for directly by simply clicking a link from the search results.

[0936] This invention is a system that utilizes generative AI to efficiently search and analyze video streaming content. In this system, the server, terminal, and user elements work together to convert and analyze the audio of videos into text, and provide an interface that is visually and functionally easy to use.

[0937] Hardware and Software Configuration

[0938] The system consists of the following main components:

[0939] Server: Use a high-performance cloud server (e.g., AWS, Google Cloud) to perform video analysis and database management.

[0940] Device: Includes any device that can connect to the internet, such as a computer, smartphone, or tablet, that a user uses.

[0941] Speech recognition technology: Converts speech to text using the Google Cloud Speech-to-Text API.

[0942] Natural language processing technology: OpenAI GPT-4 is used to analyze text data and automatically extract topics and highlights.

[0943] How to convert video audio to text

[0944] When a user uploads a video file from their device, the server stores it in Google Cloud Storage and calls the Google Cloud Speech-to-Text API to convert the audio in the video into text in real time. This text data is stored on the server.

[0945] A means for automatically extracting topics and highlights of a video by analyzing the text.

[0946] The server analyzes the stored text data using OpenAI GPT-4. Specifically, it analyzes the entire text using natural language processing technology, identifies frequently occurring words and important phrases, and automatically extracts topics (e.g., game walkthroughs, Q&A corners, etc.) and highlights (important announcements, funny moments, etc.) from the video.

[0947] A means to automatically generate a video table of contents based on extracted topics and highlights

[0948] The server automatically generates a table of contents based on the topic and highlight information obtained from the analysis, including the start time and title of each section. For example, it might generate a table of contents such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A Corner," and "20:00 - Ending."

[0949] means for generating an image associated with said table of contents;

[0950] The server extracts specific frames from the video and uses them to generate thumbnail images for each section. Specifically, it selects the frame closest to the start time of each section and processes it as the thumbnail.

[0951] A way to search for videos by keyword, date, or streamer name and provide links to jump directly to the relevant section

[0952] Users access the system through their terminals and enter keywords, dates, and broadcaster names into the search box. The server searches the database for relevant videos based on the query and displays the links, table of contents, and highlight information for the relevant videos. Users can also click on these links to directly access specific sections of the related videos.

[0953] A means of collecting viewing data and providing analysis results to broadcasters

[0954] The server records and analyzes the viewer's viewing behavior, such as the start time, end time, and length of each viewing section. The collected and analyzed data is provided to the streamer in the form of a dashboard. The streamer can use this information to plan and improve the content of their next stream.

[0955] Specific examples

[0956] For example, if a streamer wants to upload a "walkthrough video for a new game," the user selects the video from their device and begins uploading. The device sends the video file to the server, which receives the file and stores it in Google Cloud Storage. The server uses the Google Cloud Speech-to-Text API to convert the audio into text and saves it in a database. The server analyzes the saved text with OpenAI GPT-4 and extracts topics such as "Introduction," "Game Strategy," and "Conclusion." Based on these topics, a table of contents and thumbnail images for each section are automatically generated.

[0957] When a user searches for the keyword "game tips," the server displays related videos and links to those sections. The user can then access those sections directly. The server then analyzes the viewing data and notifies the streamer that the "game tips" section in particular is being viewed a lot. Based on this information, the streamer can reflect this in their next content creation.

[0958] Prompt Sentence Examples

[0959] "I have uploaded a walkthrough video for a new game. Please extract the topics of this video - introduction, game strategy, and conclusion - from their respective timestamps and automatically generate a table of contents and thumbnail images."

[0960] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0961] Step 1:

[0962] The user selects the video file from the terminal through the system's web interface and clicks the upload button.

[0963] Input: A video file selected by the user

[0964] Output: Video file sent from device to server

[0965] Specific behavior:

[0966] When a user selects a video file on the device and clicks the upload button on the web interface, the device sends the video file to the server, which receives the video file and stores it in storage.

[0967] Step 2:

[0968] The server stores the received video file in Google Cloud Storage and calls the Google Cloud Speech-to-Text API to convert the audio in the video into text.

[0969] Input: Received video file

[0970] Output: Converted text data

[0971] Specific behavior:

[0972] The server stores the video file in Google Cloud Storage, then uses the Google Cloud Speech-to-Text API to convert the video audio into text, which is then stored in a database on the server.

[0973] Step 3:

[0974] The server analyzes the stored text data using OpenAI GPT-4 and automatically extracts topics and highlights within the video.

[0975] Input: Text data

[0976] Output: Topic and highlight information

[0977] Specific behavior:

[0978] The server receives the text data and analyzes it using OpenAI GPT-4. This analysis extracts important topics and highlights from the text. Specifically, it identifies frequently occurring words and important phrases and identifies topics and highlights based on them.

[0979] Step 4:

[0980] The server automatically generates a table of contents for the video based on the extracted topics and highlights.

[0981] Input: Topic and highlight information

[0982] Output: Table of contents data

[0983] Specific behavior:

[0984] The server then uses the obtained topic and highlight information to organize the start times and titles of each section in a table of contents format, including items such as "00:00 - Opening" and "02:15 - Introduction to the new game."

[0985] Step 5:

[0986] The server extracts frames of the video associated with the table of contents and uses them to generate thumbnail images.

[0987] Input: Table of contents data

[0988] Output: Thumbnail image

[0989] Specific behavior:

[0990] The server selects the frame closest to the start of each section and generates a thumbnail of that frame, which is then associated with the table of contents, allowing viewers to visually identify the parts of the video that interest them.

[0991] Step 6:

[0992] Users access the system through their terminal and perform a search by entering keywords, dates, and broadcaster names in the search box.

[0993] Input: Keywords, date, broadcaster name

[0994] Output: Search results (related video links, table of contents, highlight information)

[0995] Specific behavior:

[0996] When a user enters a keyword into the search box, the server searches the database for relevant video data, and displays the link to the video, its table of contents, and highlights to the user. Users can also click on the provided link to go directly to the specific section of the video they are looking for.

[0997] Step 7:

[0998] The server collects viewing data and analyzes which parts are viewed the most.

[0999] Input: Viewing data

[1000] Output: Analysis results of viewing data

[1001] Specific behavior:

[1002] The server records viewers' viewing behavior and collects and analyzes viewing data. The analysis results include the frequency of viewing each section and the parts that viewers are most interested in. This data is provided to the broadcaster in the form of a dashboard.

[1003] Step 8:

[1004] The server provides feedback to the distributor based on the results of analyzing the viewing data.

[1005] Input: Analysis results of viewing data

[1006] Output: Feedback information provided to the distributor

[1007] Specific behavior:

[1008] The server then sends the analysis results to the streamer. For example, it can provide information on which specific sections are most popular, which can help the streamer in creating the next piece of content. This feedback allows the streamer to plan and improve content that will attract viewers' attention.

[1009] (Application example 1)

[1010] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1011] With current video streaming services, the sheer volume of video content makes it difficult for users to quickly find specific information. It is also labor-intensive for streamers to use viewing data to improve their next content offering. This reduces convenience for both users and streamers, making it difficult to provide effective content. It is necessary to solve these issues and improve the efficiency of video search and streamer data utilization.

[1012] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1013] In this invention, the server includes: means for converting video audio into text using generative AI; means for analyzing the text and automatically extracting topics and highlights of the video; means for automatically generating a table of contents for the video based on the extracted topics and highlights; means for generating thumbnail images associated with the table of contents; means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to relevant parts; means for collecting viewing data and providing analysis results to broadcasters; means for generating a video timeline based on time axis information extracted from the text; and means for generating thumbnail images corresponding to each timeline entry. This allows users to quickly find parts of a video that interest them, and enables broadcasters to easily utilize viewing data to improve their next broadcast.

[1014] "Generative AI" is a technology that uses artificial intelligence technology to generate data such as text, audio, and images.

[1015] "Means for converting audio to text" refers to a technology that analyzes the audio in a video and outputs it as text information.

[1016] "Means for automatic topic and highlight extraction" refers to technology that automatically identifies important themes and points of interest from text data.

[1017] "Means for automatically generating a table of contents" refers to technology that automatically generates a table of contents to provide users with a table of contents that identifies the major sections within a video.

[1018] "Means for generating thumbnail images" refers to a technology that captures specific scenes from a video as images to be used as visual indexes.

[1019] "Means for searching and providing direct jump links" refers to technology that searches for related videos based on keywords, dates, and broadcaster names entered by the user, and generates links to directly access specific parts.

[1020] "Means of collecting viewing data and providing analytical results" refers to technology that collects and analyzes data such as video viewing history and viewing time, and provides the results to the distributor.

[1021] "Means for generating a video timeline based on time axis information" is a technology that visually represents the time axis of a video based on time information extracted from text data.

[1022] The "means for generating thumbnail images corresponding to timeline entries" is a technique for capturing an image at a specific time corresponding to each entry on the timeline and displaying it on the timeline.

[1023] The "means for proposing the next distribution plan" is a technology that automatically suggests the next distribution content to increase the viewer's interest based on the previous viewing data.

[1024] The "means for individually optimized content recommendations" is a technology that recommends individually customized content based on a user's search history and viewing history.

[1025] A system for carrying out the present invention is one in which a server, a terminal, and a user work together in cooperation with each other. A specific embodiment of the system will be described below.

[1026] Server Roles

[1027] First, when a user uploads a video from their device, the server converts the audio into text using generative AI. In this process, the server extracts audio data from the video file using a speech recognition library (e.g., speech_recognition) and a movie processing library (e.g., moviepy) and converts it into text data.

[1028] The server then analyzes this text data using a generative AI model (e.g., OpenAI's API) to automatically extract topics and highlights within the video, thereby identifying which sections are important.

[1029] Based on the extracted topics and highlights, the server automatically generates a table of contents for the video, allowing viewers to easily understand the content of the video and quickly access parts that interest them. Additionally, thumbnail images associated with the table of contents are also generated. For example, the server captures a frame at the beginning of each section and uses it as a thumbnail.

[1030] Role of user terminal

[1031] The user device provides an interface that allows viewers to search for videos by specific keywords, dates, or broadcaster names. When a user enters a search query, the server returns related videos and their table of contents links. Based on the search results, users can use links to jump directly to the parts that interest them. This allows viewers to quickly find the information they are looking for among a vast amount of video content.

[1032] Data analysis and feedback

[1033] The server also collects viewing data and provides analysis results to the broadcaster. Specifically, it makes suggestions for improving the content of the next broadcast based on data such as which parts viewers watched the most and which sections they were interested in. This feedback allows the broadcaster to create effective content to increase viewer interest.

[1034] Specific examples

[1035] For example, let's consider the case where a streamer uploads a "walkthrough video for a new game." The server converts the audio into text and analyzes it to extract topics such as "Introduction," "Game Strategy," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are presented to viewers in a visually easy-to-understand format.

[1036] When a viewer searches for "game tips," the server will display search results such as "game tips from 03:00" and provide a link that allows the user to access that section directly. An example of a prompt sentence that can be used in this case is as follows:

[1037] Prompt Sentence Examples

[1038] Extract the main topics and highlights from the following text:

[1039] "This is a walkthrough of a new game. First, I'll explain how to choose your character. Next, I'll introduce some important points for progressing through the game. Finally, I'll explain how to beat boss battles."

[1040] This allows users to quickly access important information and allows streamers to effectively plan upcoming content based on viewing data.

[1041] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1042] Step 1:

[1043] Uploading videos

[1044] The user uses the terminal to upload a video file to the server, which receives the video file, extracts the audio data, and prepares it for the next step.

[1045] Input: Video file

[1046] Output: Audio data

[1047] Step 2:

[1048] Speech-to-text

[1049] The server uses a speech recognition library (e.g., speech_recognition) to convert the audio data into text, analyzes the extracted audio data, and outputs it in text format.

[1050] Input: Audio data

[1051] Output: Text data

[1052] Step 3:

[1053] Text analytics

[1054] The server uses a generative AI model (e.g., OpenAI's API) to analyze the text data and automatically extract topics and highlights from the video. This analysis identifies the main themes and important scenes in the video.

[1055] Input: Text data

[1056] Output: Topic and highlight information

[1057] Step 4:

[1058] Automatic table of contents and thumbnail generation

[1059] The server automatically generates a table of contents for the video based on the extracted topics and highlight information, and simultaneously generates thumbnail images from the start of each section and associates them with the table of contents.

[1060] Input: Topic and highlight information

[1061] Output: Table of contents data, thumbnail images

[1062] Step 5:

[1063] Providing a search interface

[1064] The device provides the user with a search interface, displaying an input form that allows the user to search for videos by specific keywords, dates, or streamer names.

[1065] Input: None (preparation until user input)

[1066] Output: Search interface

[1067] Step 6:

[1068] Processing search queries

[1069] The user enters a search query and the device sends it to the server, which searches for related videos and table of contents links based on the query and returns the results.

[1070] Input: Search query (keyword, date, broadcaster name)

[1071] Output: Search results (related videos and table of contents links)

[1072] Step 7:

[1073] Collecting viewing data

[1074] The server collects data on users' viewing of videos (such as viewing time, viewing sections, etc.), which is later used by the distributor for analysis.

[1075] Input: Viewing activity data

[1076] Output: Recording to viewing database

[1077] Step 8:

[1078] Providing analysis results

[1079] The server analyzes the collected viewing data to determine which sections are most viewed and trends in viewer interest. The server then provides the analysis results to the broadcaster, helping them plan their next broadcast.

[1080] Input: Viewing data

[1081] Output: Analysis report

[1082] These steps allow users to quickly find what interests them and give publishers the data to improve their content next time.

[1083] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1084] This invention combines a system that utilizes generative AI to efficiently search and analyze video content with an emotion engine that recognizes user emotions. In this system, the server, terminal, and user elements work together to convert and analyze the audio of videos into text, and then provide recommended content based on the user's emotions.

[1085] Program processing explanation

[1086] Video upload and analysis

[1087] When a streamer uploads a video from their device, the server converts the audio into text using generative AI. Specifically, the server uses speech recognition technology to analyze the audio in the video in real time and convert it into text as subtitles.

[1088] The server then analyzes this text data and automatically extracts topics and highlights from the video. In this process, natural language processing technology is used to identify topics (e.g., game walkthroughs, Q&A corners, etc.) and highlight scenes (e.g., important announcements, funny moments) from the text content.

[1089] Table of contents and thumbnail generation

[1090] Based on the extracted topics and highlights, the server automatically generates a table of contents for the video, such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A session," and "20:00 - Ending."

[1091] The server also generates thumbnail images of each section, allowing viewers to easily navigate through the table of contents and jump to parts that interest them.

[1092] Providing search functionality

[1093] Users access the system through a terminal and use an interface that allows them to search for videos by specific keywords, dates, or streamer names. If a user types in a search term like "new game introductions," the server quickly identifies relevant videos and displays them as search results.

[1094] The server also provides links to relevant highlights and table of contents in search results, allowing users to jump directly to the parts they are interested in. This allows users to quickly find the information they are looking for among a vast amount of video content.

[1095] Implementing the Emotion Engine

[1096] The emotion engine uses facial recognition and voice analysis technologies to analyze the viewer's emotional state in real time while they are watching a video, determining, for example, whether they are laughing, surprised, sad, etc.

[1097] Based on the collected emotional data, the server can recommend videos and provide highlights that correspond to the user's emotional state. For example, if the server detects that the user is laughing, it can recommend more funny scenes or relaxing content.

[1098] Providing analysis results and recommended content

[1099] The server collects viewing data and emotional data, analyzes them, and generates data such as which parts were viewed the most and which parts evoked particular emotions.

[1100] The analysis results are displayed on the streamer's personal dashboard, and the streamer can use this data to plan and improve their content. Specifically, streamers can identify moments that make viewers laugh and topics that interest them and reflect this in their next stream.

[1101] Specific examples

[1102] For example, if a streamer uploads a "walkthrough video for a new game," the server converts the audio into text and analyzes it to extract the topics "Introduction," "Game Strategy," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are presented to viewers in a visually easy-to-understand format.

[1103] When a viewer searches for "game tips," the server will display search results such as "game tips from 3:00" and provide a link that allows the user to access that section directly. Furthermore, if the emotion engine detects a smile on the user's face while watching, the server will recommend funny or popular scenes from the same game. As a result, viewers can quickly access the content they want to watch, and broadcasters can maintain their viewers' interest.

[1104] The processing flow will be explained below.

[1105] Step 1:

[1106] The streamer uses a device to upload the video to the server.

[1107] Specifically, the streamer selects a video file on the device and clicks the upload button, and the device sends the video file to the server.

[1108] Step 2:

[1109] The server stores the received video files in a database.

[1110] Once saved, the process of converting the video's audio into text begins using generative AI.

[1111] Step 3:

[1112] The server uses generative AI to convert the video's audio data into text in real time.

[1113] Using voice recognition technology, what is said in the video is converted into text as subtitles.

[1114] Step 4:

[1115] The server analyzes the generated text data.

[1116] Using natural language processing technology, topics and highlights are automatically extracted from the text content. For example, topics such as "game strategies" and "question corners" are identified.

[1117] Step 5:

[1118] The server automatically generates a table of contents for the video based on the extracted topics and highlights.

[1119] For example, create table of contents entries such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A," and "20:00 - Ending."

[1120] Step 6:

[1121] The server generates thumbnail images corresponding to each section of the table of contents.

[1122] Thumbnail images are displayed on the viewing screen along with the table of contents, serving as a visual guide.

[1123] Step 7:

[1124] A user accesses the system through a terminal and searches for videos using a search interface.

[1125] When a user enters a specific keyword (e.g., "new game introduction"), the device sends this search query to the server.

[1126] Step 8:

[1127] The server receives the search query and searches the database to identify relevant videos and topics.

[1128] The server generates related videos and table of contents links as search results and returns them to the user.

[1129] Step 9:

[1130] Users receive search results on their devices and can jump directly to the relevant section by clicking on the displayed link.

[1131] This feature allows users to quickly access specific moments or sections.

[1132] Step 10:

[1133] The emotion engine analyzes the user's emotions while they are watching.

[1134] Using facial recognition and voice analysis technology, it determines in real time whether the viewer is laughing, surprised, sad, etc.

[1135] Step 11:

[1136] Based on the collected emotional data, the server provides recommended videos and highlights according to the user's emotional state.

[1137] For example, if the user is laughing, the server will suggest more funny scenes and related content.

[1138] Step 12:

[1139] The server collects and analyzes viewing data and emotion data, and generates analysis results.

[1140] Data is collected on which parts were viewed the most and how users felt about them.

[1141] Step 13:

[1142] The analysis results are displayed on a dashboard dedicated to the broadcaster.

[1143] Streamers can use this data to plan and improve their content, for example by identifying moments that made users laugh and creating strategies such as increasing the number of similar scenes in their next stream.

[1144] Example 2

[1145] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1146] With the rapid increase in video content, it is becoming increasingly difficult for viewers to efficiently search and access the information they are looking for. Furthermore, detailed viewing data is necessary for streamers to understand viewers' reactions and improve future content. However, current systems do not adequately collect and analyze not only viewing data but also emotional data during viewing, and provide content recommendations based on that data. Therefore, a comprehensive system is needed to improve the viewer experience and support streamers in content creation.

[1147] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1148] In this invention, the server includes: means for converting video audio into text using a generative AI; means for analyzing the text and automatically extracting video topics and highlights; means for automatically generating a video table of contents based on the extracted topics and highlights; means for generating thumbnail images associated with the table of contents; means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to relevant sections; means for collecting viewing data and user emotional data and providing the analysis results to the broadcaster; and means for analyzing the user's emotional state during viewing in real time using an emotion engine and recommending videos and providing highlights based on the collected emotional data. This allows viewers to efficiently access desired content, and broadcasters to understand viewer reactions in detail and improve the quality of their content in future broadcasts.

[1149] "Generative AI" refers to models or algorithms that generate data using artificial intelligence techniques.

[1150] "Text conversion" is the process of analyzing audio data and converting it into written information.

[1151] "Topic extraction" is a technique for analyzing text information and identifying key topics or themes within it.

[1152] "Highlight extraction" is a technology that automatically extracts particularly important parts or interesting scenes from video or text.

[1153] "Table of contents generation" is the process of creating easily digestible segments based on the main points of a video or piece of text.

[1154] "Thumbnail generation" is the process of extracting specific frames of a video as still images to create small visually representative images.

[1155] A "search function" is a technology that allows you to quickly find relevant information or data based on specific keywords or conditions.

[1156] "Viewing data" refers to data recorded and analyzed based on user behavior when watching videos (such as playback time and frequently viewed parts).

[1157] "Emotional data" refers to data collected by recognizing the emotional state (happiness, surprise, sadness, etc.) that viewers show while watching a video.

[1158] An "emotion engine" is a system or technology that analyzes the face and voice of a user while watching and determines their emotional state in real time.

[1159] "Recommended content" is a function that automatically suggests content that may interest a user based on the user's viewing history and emotional data.

[1160] A "viewing analysis dashboard" is a system that displays detailed viewer reactions and behavior to broadcasters based on collected viewing and emotional data.

[1161] This invention is a system for efficiently searching and analyzing video content, combining generative AI and emotion recognition technology to provide useful information to both viewers and streamers. This system is implemented primarily using the following hardware and software:

[1162] Video speech to text conversion

[1163] The server receives the video uploaded by the streamer from their device, extracts the audio using an audio processing tool (e.g., ffmpeg), and then converts the audio into text using a speech recognition service such as the Google Cloud Speech-to-Text API. The converted text is then stored in a database on the server.

[1164] Parsing text and generating a table of contents

[1165] The server analyzes the generated text using natural language processing techniques (e.g., BERT or GPT-3) to extract the topics and highlights of the video. For example, in a game walkthrough video, segments such as "Introduction," "Way of the Walkthrough," and "Conclusion" are identified. Based on this, a table of contents for the video is automatically generated, and thumbnail images corresponding to each segment are generated. OpenCV and PIL libraries are used to generate the thumbnails.

[1166] Providing search functionality

[1167] A user can access the system using a terminal and enter specific keywords into the search interface. For example, they can enter "new game introduction." The server receives this search query, matches it with the stored text data and table of contents information, identifies related videos, and displays the search results as a list. The user can click on the displayed link to jump directly to the relevant section.

[1168] Emotion recognition implementation

[1169] The emotion engine analyzes facial expressions and voice in real time via the camera and microphone while the user is watching a video. It uses FaceAPI and Emotion API to determine the user's emotions, such as smile, surprise, sadness, etc. This emotion data is sent to the server and stored in a database along with the viewing data.

[1170] Providing recommended content

[1171] The server then runs an algorithm based on the collected emotional data to recommend content that matches the user's emotional state. For example, if the user is smiling a lot, the server will recommend more entertaining videos or relaxing content. Recommended content is provided via notifications and lists.

[1172] Viewing analytics dashboard

[1173] The server generates a dashboard for streamers based on the collected viewing and emotion data. The dashboard displays the viewing time for each segment, a heat map of emotion data, and an analysis of viewer reactions. This allows streamers to understand viewer reactions in detail and use this information to improve future content creation.

[1174] Specific operation example

[1175] For example, if a streamer uploads a video explaining how to play a new game, the server converts the audio into text and analyzes it to extract topics such as "Introduction," "Game Tips," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are then provided to viewers.

[1176] When a viewer searches for "game strategies," the server displays search results such as "game strategies from 03:00," and the viewer can click on a link to directly access the relevant section. In addition, when the emotion engine detects a smile on the user's face while watching, the server recommends interesting or popular scenes from the same game, improving the user's viewing experience.

[1177] Prompt Sentence Examples

[1178] Introducing a new game

[1179] Enter: Introducing a new game

[1180] Output: A list of all videos tagged with "new game introductions"

[1181] "Tell me the highlight scenes in the video"

[1182] Input: Highlights from the video

[1183] Output: A list of highlight scenes from each video

[1184] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1185] System program processing flow

[1186] Step 1: Upload your video

[1187] The server receives a request from the distributor to upload a video file from the terminal.

[1188] Input: Video file upload request from the device

[1189] Output: The location where the uploaded video file will be saved to storage.

[1190] Specific behavior: The server receives the video file via an HTML form or API endpoint, saves it to storage, and returns a status code of 200 to confirm the upload was successful.

[1191] Step 2: Speech to text

[1192] The server extracts the audio from the stored video file and converts it into text using generative AI.

[1193] Input: Saved video file

[1194] Output: Text data

[1195] Specific operation: The server separates the audio using ffmpeg and converts the audio data to text data using the Google Cloud Speech-to-Text API. The converted text is then stored in a database.

[1196] Step 3: Parsing the text

[1197] The server analyzes the generated text data using natural language processing technology to extract topics and highlights from the video.

[1198] Input: Text data

[1199] Output: Topic and highlight information (JSON format)

[1200] Specific operation: The server uses generative AI such as BERT or GPT-3 to identify topics (e.g., "Introduction," "Game Strategy," "Conclusion") and highlight scenes, and saves the results in JSON format.

[1201] Step 4: Generate the Table of Contents and Thumbnails

[1202] The server generates a table of contents and thumbnail images for the video based on the extracted topics and highlights.

[1203] Input: Topic and highlight information (JSON format)

[1204] Output: Table of contents and thumbnail images

[1205] Specific operation: The server parses the JSON data and automatically generates a table of contents such as "00:00 - Opening" and "02:15 - Introduction to the new game." It also uses OpenCV and PIL libraries to generate and save thumbnail images for each segment.

[1206] Step 5: Keyword search

[1207] Users input specific keywords into the search interface, and the server searches for relevant videos based on the input keywords and displays the results.

[1208] Input: User's search keyword

[1209] Output: Related video list with table of contents links and thumbnails

[1210] Specific operation: When a user enters "new game introduction" into the search interface, the server references the text data and table of contents information, and displays a list of related videos and their table of contents links. The user can click on the link to directly access the relevant section.

[1211] Step 6: Emotion Recognition and Data Collection

[1212] The emotion engine uses the camera and microphone to analyze emotions in real time while the user is watching a video and transmits the results to a server.

[1213] Input: User's facial image data and voice data

[1214] Output: Emotion data (smile, surprise, sadness, etc.)

[1215] Specific operation: The emotion engine uses FaceAPI and Emotion API to determine the user's emotional state and sends the data to the server. The emotion data is stored in the database along with the viewing data.

[1216] Step 7: Providing recommended content

[1217] The server recommends optimal content to the user based on the collected emotion data.

[1218] Input: Emotion data and viewing data

[1219] Output: Recommended content

[1220] How it works: The server uses an algorithm to analyze emotion data and viewing data. For example, if the user is smiling a lot, the server will recommend more interesting videos. Recommended content is provided via notifications and the user interface.

[1221] Step 8: Displaying the analysis results on a dashboard

[1222] The server displays the collected and analyzed viewing and sentiment data on a dashboard for the broadcaster.

[1223] Input: Viewing data and emotion data

[1224] Output: Display analysis results for the broadcaster

[1225] Specific operation: The server generates a heat map of viewing time and emotional data for each segment based on viewing data and emotional data, and displays it on the dashboard. The broadcaster can use this information to improve the content of their next broadcast.

[1226] (Application example 2)

[1227] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1228] Conventional video distribution systems make it difficult for viewers to efficiently find the information they are looking for from a vast amount of video content. Furthermore, providing personalized recommended content based on the viewer's emotions is difficult, and this has not sufficiently improved the viewing experience. Therefore, there is a need for a system that not only converts audio into text and extracts video topics and highlights, but also uses viewer emotional data to recommend appropriate content.

[1229] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1230] In this invention, the server includes: means for converting video audio into text using a generative AI; means for analyzing the text and automatically extracting video topics and highlights; means for automatically generating a video table of contents based on the extracted topics and highlights; means for generating thumbnail images associated with the table of contents; means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to relevant sections; means for collecting viewing data and providing analysis results to broadcasters; an emotion engine for recognizing the user's emotional state; and means for providing recommended content to the user based on the emotion engine. This enables viewers to efficiently find desired information from a vast amount of video content and also provides personalized recommended content based on the viewer's emotions.

[1231] "Generative AI" is a generative artificial intelligence technology that primarily uses deep learning models to generate or convert text, audio, and images.

[1232] "Video audio to text conversion" refers to the process of analyzing the audio data in a video and converting it into corresponding text data.

[1233] A "topic" is a major theme or topic that summarizes the content of a video.

[1234] "Highlights" refer to particularly important moments or scenes in a video that are of great interest to viewers.

[1235] A "table of contents" is an organized list of the video's contents that provides information including the start times of each section.

[1236] A "thumbnail image" is a small image that represents a particular scene in a video and functions as a visual table of contents.

[1237] "Keywords" are specific words or phrases used to search for video content.

[1238] "Distributor name" is the name of the person or organization that published the video.

[1239] "Viewing data" refers to data obtained when a viewer watches a video, and includes information such as playback time, viewing locations, and viewing frequency.

[1240] The "emotion engine" is a technology that analyzes the user's facial expressions and voice to recognize their emotional state in real time.

[1241] "Recommended content" is highly relevant content that is selected based on the user's viewing history and emotional state and provided to the user.

[1242] To implement this invention, the following system configuration is adopted. The server implements a program that uses generative AI to convert video audio into text and analyze it. The generative AI model used in this process is, for example, Transformer-based speech recognition technology. Specific software used includes the Transformer model library and TextBlob for natural language processing. In addition, facial recognition technology and voice analysis technology are implemented in the emotion engine to analyze user emotions in real time.

[1243] The text data generated by the server is analyzed using natural language processing technology to automatically extract topics and highlights from the video. This automatically generates a table of contents and thumbnail images for the video. Users access the system using their devices and search for videos using specific keywords, dates, or broadcaster names. As a result, a link is provided that takes users directly to the relevant part of the video.

[1244] The emotion engine analyzes the user's facial expressions and voice while watching to recognize their emotional state in real time. Based on this emotion data, the server provides recommended content to the user. For example, if the user is detected as laughing, the server will recommend more funny scenes or relaxing content.

[1245] For example, when a user watches a "new game walkthrough video," the server converts the audio data into text and extracts topics such as "introduction," "game strategy," and "conclusion." If the emotion engine detects a smile while the viewer is watching, the server recommends related, interesting scenes. This allows viewers to quickly access the content they want to watch and allows streamers to sustainably capture viewers' attention.

[1246] Examples of prompt sentences include the following:

[1247] "Users are smiling while watching. Recommend content they'll enjoy."

[1248] By operating the system in this way, viewers can efficiently find the information they are looking for from a vast amount of video content, and it is also possible to provide personalized recommended content based on the viewer's emotions.

[1249] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1250] Step 1: Upload video and convert audio

[1251] A user uploads a video from their device to the server. The server then uses a generative AI model (Transformer-based speech recognition technology) to convert the audio data in the video into text data. The input is the video file, and the output is text data.

[1252] Step 2: Text analysis and topic extraction

[1253] The server analyzes the converted text data and automatically extracts topics and highlights from the video using natural language processing technology (such as TextBlob). The input is text data, and the output is the extracted topics and highlights. Specific operations include topic segmentation and extraction of important words and phrases.

[1254] Step 3: Generate the Table of Contents and Thumbnails

[1255] The server automatically generates a table of contents for the video based on the extracted topics and highlights, and also generates thumbnail images for each section. The input is the topics and highlights, and the output is the table of contents and thumbnail images. The table of contents includes the start time of each topic.

[1256] Step 4: Providing keyword search functionality

[1257] Users can search for videos by entering specific keywords, dates, or streamers on their devices. The server searches for video data corresponding to those keywords and displays related highlights and table of contents links. The input is a search query, and the output is a list of related videos and links.

[1258] Step 5: Collect and analyze emotion data

[1259] While a user is watching a video, the server uses an emotion engine to analyze the user's emotional state in real time. The input is facial expression data and voice data during viewing, and the output is emotion data (e.g., smile, surprise, etc.). Specific operations use face recognition technology and voice analysis technology.

[1260] Step 6: Generate recommended content

[1261] The server generates content recommendations appropriate for the user based on the collected emotion data. For this, it can use a generative AI model. For example, if a smile is detected, it can recommend more funny scenes or relaxing content. The input is emotion data, and the output is a list of recommended content.

[1262] Step 7: Displaying the analysis results on a dashboard

[1263] The server collects and analyzes viewing data and emotional data, and displays the results on the streamer's dashboard. This allows streamers to check data based on viewers' high interest and emotional state, and use it to plan their next content. The input is viewing data and emotional data, and the output is the analysis results on the dashboard.

[1264] The above steps will realize a system that allows viewers to efficiently find the information they are looking for from a vast amount of video content and enjoy personalized recommended content based on their emotions.

[1265] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1266] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1267] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1268] [Fourth embodiment]

[1269] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1270] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1271] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1272] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1273] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1274] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1275] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1276] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1277] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1278] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1279] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1280] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1281] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1282] This invention is a system that utilizes generative AI to efficiently search and analyze the content of video streams. In this system, the server, terminal, and user elements work together to convert and analyze the audio of videos into text, and provide an interface that is visually and functionally easy to use.

[1283] Program processing explanation

[1284] Video upload and analysis

[1285] When a streamer uploads a video from their device, the server converts the audio into text using generative AI. Specifically, the server uses speech recognition technology to analyze the audio in the video in real time and convert it into text as subtitles.

[1286] The server then analyzes this text data and automatically extracts topics and highlights from the video. In this process, natural language processing technology is used to identify topics (e.g., game walkthroughs, Q&A corners, etc.) and highlight scenes (e.g., important announcements, funny moments, etc.) from the text content.

[1287] Table of contents and thumbnail generation

[1288] Based on the extracted topics and highlights, the server automatically generates a table of contents for the video, such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A session," and "20:00 - Ending."

[1289] The server also generates thumbnail images of each section, allowing viewers to easily navigate through the table of contents and jump to parts that interest them.

[1290] Providing search functionality

[1291] Users access the system through a terminal and use an interface that allows them to search for videos by specific keywords, dates, or streamer names. If a user types in a search term like "new game introductions," the server quickly identifies relevant videos and displays them as search results.

[1292] The server also provides links to relevant highlights and table of contents in search results, allowing users to jump directly to the parts they are interested in. This allows users to quickly find the information they are looking for among a vast amount of video content.

[1293] Providing analysis results

[1294] The server collects viewing data and analyzes it to determine which parts were viewed the most and which parts attracted the most viewers' attention. For example, it can collect information such as, "In the April 15, 2023 video, the Q&A session starting at 10:45 was particularly popular."

[1295] The server provides the streamer with the collected and analyzed data in the form of a dashboard. Based on this data, the streamer can plan and improve the content of their next stream. For example, it becomes easier to make decisions such as, "Viewers will be pleased if we improve the Q&A section in the next stream."

[1296] Specific examples

[1297] For example, if a streamer uploads a "walkthrough video for a new game," the server converts the audio into text and analyzes it to extract the topics "Introduction," "Game Strategy," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are presented to viewers in a visually easy-to-understand format.

[1298] When a viewer searches for "game tips," the server displays search results such as "game tips from 03:00" and also provides a link that allows the user to access that section directly. Furthermore, the server analyzes the viewing data for this video and notifies the streamer that the "game tips" section in particular is being viewed a lot. Based on this information, the streamer can reflect this in their next content creation.

[1299] The processing flow will be explained below.

[1300] Step 1:

[1301] The streamer uses a device to upload the video to the server.

[1302] Specifically, the streamer selects a video file on the device and clicks the upload button, and the device sends the video file to the server.

[1303] Step 2:

[1304] The server stores the received video files in a database.

[1305] Once saved, the process of converting the video's audio into text begins using generative AI.

[1306] Step 3:

[1307] The server uses generative AI to convert the video's audio data into text in real time.

[1308] Using voice recognition technology, what is said in the video is converted into text as subtitles.

[1309] Step 4:

[1310] The server analyzes the generated text data.

[1311] Using natural language processing technology, topics and highlights are automatically extracted from the text content. For example, topics such as "game strategies" and "question corners" are identified.

[1312] Step 5:

[1313] The server automatically generates a table of contents for the video based on the extracted topics and highlights.

[1314] For example, create table of contents entries such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A," and "20:00 - Ending."

[1315] Step 6:

[1316] The server generates thumbnail images corresponding to each section of the table of contents.

[1317] Thumbnail images are displayed on the viewing screen along with the table of contents, serving as a visual guide.

[1318] Step 7:

[1319] A user accesses the system through a terminal and searches for videos using a search interface.

[1320] When a user enters a specific keyword (e.g., "new game introduction"), the device sends this search query to the server.

[1321] Step 8:

[1322] The server receives the search query and searches the database to identify relevant videos and topics.

[1323] The server generates related videos and table of contents links as search results and returns them to the user.

[1324] Step 9:

[1325] Users receive search results on their devices and can jump directly to the relevant section by clicking on the displayed link.

[1326] This feature allows users to quickly access specific moments or sections.

[1327] Step 10:

[1328] The server collects viewing data.

[1329] For example, data is collected about which parts were most viewed and at what parts viewers stopped watching.

[1330] Step 11:

[1331] The server analyzes the collected viewing data and provides the analysis results to the broadcaster.

[1332] The analysis results are displayed on a dashboard dedicated to the streamer, and streamers can use this data to plan and improve their content.

[1333] In this way, through specific actions at each step, a series of processes from video uploading to viewing analysis is completed.

[1334] Example 1

[1335] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1336] Today, a huge amount of video content is being produced across a wide variety of formats. However, a lack of efficient support for searching, analyzing, and viewing this video content presents a challenge. In particular, there is a lack of functionality to convert audio within a video into text and then analyze it to quickly provide the information viewers are looking for. As a result, viewers must watch the entire video, wasting time and effort. Furthermore, streamers are unable to effectively receive feedback on the results of analyzing viewing data, making it difficult to improve the content of their next stream.

[1337] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1338] In this invention, the server includes a means for converting the audio of a video into text, a means for analyzing the text and automatically extracting topics and highlights of the video, a means for automatically generating a table of contents for the video based on the extracted topics and highlights, a means for generating images associated with the table of contents, a means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to the relevant parts, and a means for collecting viewing data and providing the analysis results to the broadcaster. This allows the server to efficiently analyze the content of the video and automatically generate a table of contents, enabling viewers to quickly access the information they desire. Furthermore, by providing broadcasters with feedback on the viewing data, the server can effectively improve the content of their next broadcast.

[1339] "Means for converting video audio into text" refers to technology that analyzes the audio data in a video and generates corresponding text data from the audio.

[1340] "Means for analyzing the text and automatically extracting topics and highlights from the video" refers to technology that analyzes the generated text data and automatically identifies important themes and noteworthy scenes.

[1341] "Means for automatically generating a video table of contents based on extracted topics and highlights" refers to a technology that organizes the start time and content of videos in list format based on the topic and highlight information obtained from the analysis results.

[1342] The "means for generating an image associated with the table of contents" refers to a technique for selecting a frame of a video corresponding to each item in the table of contents and generating it as a still image.

[1343] "Means of searching videos by keyword, date, or streamer name and providing links to jump directly to the relevant section" refers to technology that finds related videos based on the search criteria specified by the viewer and generates links that allow quick access to the relevant section.

[1344] "Means of collecting viewing data and providing analysis results to broadcasters" refers to technology that records and analyzes viewers' viewing behavior and provides feedback to broadcasters on information such as the most viewed parts and parts that viewers were most interested in.

[1345] "Means for converting to text at specified times" refers to technology that converts audio at specific points or sections of a video into text data.

[1346] "Natural language processing technology" refers to technology that allows computers to analyze and understand human language.

[1347] "Means for collecting and analyzing viewing data for specific parts" refers to technology that records viewers' viewing behavior in specific sections of a video and conducts detailed analysis based on that information.

[1348] "Means of providing distributors with information on popular viewing sections" refers to technology that notifies distributors of information on sections that are of high interest to viewers based on collected viewing data.

[1349] "An interface that allows users to search for previously distributed videos by entering specific keywords" refers to a user interface that allows viewers to search for previously distributed videos by entering keywords in a search box.

[1350] "Means of displaying related videos and their table of contents links based on a search query" refers to technology that identifies videos that match the search criteria entered by the viewer and displays them along with a table of contents for those videos.

[1351] "Means of providing links that allow viewers to jump directly from search results to specific sections of the video" refers to technology that allows viewers to access the specific part of the video they are looking for directly by simply clicking a link from the search results.

[1352] This invention is a system that utilizes generative AI to efficiently search and analyze video streaming content. In this system, the server, terminal, and user elements work together to convert and analyze the audio of videos into text, and provide an interface that is visually and functionally easy to use.

[1353] Hardware and Software Configuration

[1354] The system consists of the following main components:

[1355] Server: Use a high-performance cloud server (e.g., AWS, Google Cloud) to perform video analysis and database management.

[1356] Device: Includes any device that can connect to the internet, such as a computer, smartphone, or tablet, that a user uses.

[1357] Speech recognition technology: Converts speech to text using the Google Cloud Speech-to-Text API.

[1358] Natural language processing technology: OpenAI GPT-4 is used to analyze text data and automatically extract topics and highlights.

[1359] How to convert video audio to text

[1360] When a user uploads a video file from their device, the server stores it in Google Cloud Storage and calls the Google Cloud Speech-to-Text API to convert the audio in the video into text in real time. This text data is stored on the server.

[1361] A means for automatically extracting topics and highlights of a video by analyzing the text.

[1362] The server analyzes the stored text data using OpenAI GPT-4. Specifically, it analyzes the entire text using natural language processing technology, identifies frequently occurring words and important phrases, and automatically extracts topics (e.g., game walkthroughs, Q&A corners, etc.) and highlights (important announcements, funny moments, etc.) from the video.

[1363] A means to automatically generate a video table of contents based on extracted topics and highlights

[1364] The server automatically generates a table of contents based on the topic and highlight information obtained from the analysis, including the start time and title of each section. For example, it might generate a table of contents such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A Corner," and "20:00 - Ending."

[1365] means for generating an image associated with said table of contents;

[1366] The server extracts specific frames from the video and uses them to generate thumbnail images for each section. Specifically, it selects the frame closest to the start time of each section and processes it as the thumbnail.

[1367] A way to search for videos by keyword, date, or streamer name and provide links to jump directly to the relevant section

[1368] Users access the system through their terminals and enter keywords, dates, and broadcaster names into the search box. The server searches the database for relevant videos based on the query and displays the links, table of contents, and highlight information for the relevant videos. Users can also click on these links to directly access specific sections of the related videos.

[1369] A means of collecting viewing data and providing analysis results to broadcasters

[1370] The server records and analyzes the viewer's viewing behavior, such as the start time, end time, and length of each viewing section. The collected and analyzed data is provided to the streamer in the form of a dashboard. The streamer can use this information to plan and improve the content of their next stream.

[1371] Specific examples

[1372] For example, if a streamer wants to upload a "walkthrough video for a new game," the user selects the video from their device and begins uploading. The device sends the video file to the server, which receives the file and stores it in Google Cloud Storage. The server uses the Google Cloud Speech-to-Text API to convert the audio into text and saves it in a database. The server analyzes the saved text with OpenAI GPT-4 and extracts topics such as "Introduction," "Game Strategy," and "Conclusion." Based on these topics, a table of contents and thumbnail images for each section are automatically generated.

[1373] When a user searches for the keyword "game tips," the server displays related videos and links to those sections. The user can then access those sections directly. The server then analyzes the viewing data and notifies the streamer that the "game tips" section in particular is being viewed a lot. Based on this information, the streamer can reflect this in their next content creation.

[1374] Prompt Sentence Examples

[1375] "I have uploaded a walkthrough video for a new game. Please extract the topics of this video - introduction, game strategy, and conclusion - from their respective timestamps and automatically generate a table of contents and thumbnail images."

[1376] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1377] Step 1:

[1378] The user selects the video file from the terminal through the system's web interface and clicks the upload button.

[1379] Input: A video file selected by the user

[1380] Output: Video file sent from device to server

[1381] Specific behavior:

[1382] When a user selects a video file on the device and clicks the upload button on the web interface, the device sends the video file to the server, which receives the video file and stores it in storage.

[1383] Step 2:

[1384] The server stores the received video file in Google Cloud Storage and calls the Google Cloud Speech-to-Text API to convert the audio in the video into text.

[1385] Input: Received video file

[1386] Output: Converted text data

[1387] Specific behavior:

[1388] The server stores the video file in Google Cloud Storage, then uses the Google Cloud Speech-to-Text API to convert the video audio into text, which is then stored in a database on the server.

[1389] Step 3:

[1390] The server analyzes the stored text data using OpenAI GPT-4 and automatically extracts topics and highlights within the video.

[1391] Input: Text data

[1392] Output: Topic and highlight information

[1393] Specific behavior:

[1394] The server receives the text data and analyzes it using OpenAI GPT-4. This analysis extracts important topics and highlights from the text. Specifically, it identifies frequently occurring words and important phrases and identifies topics and highlights based on them.

[1395] Step 4:

[1396] The server automatically generates a table of contents for the video based on the extracted topics and highlights.

[1397] Input: Topic and highlight information

[1398] Output: Table of contents data

[1399] Specific behavior:

[1400] The server then uses the obtained topic and highlight information to organize the start times and titles of each section in a table of contents format, including items such as "00:00 - Opening" and "02:15 - Introduction to the new game."

[1401] Step 5:

[1402] The server extracts frames of the video associated with the table of contents and uses them to generate thumbnail images.

[1403] Input: Table of contents data

[1404] Output: Thumbnail image

[1405] Specific behavior:

[1406] The server selects the frame closest to the start of each section and generates a thumbnail of that frame, which is then associated with the table of contents, allowing viewers to visually identify the parts of the video that interest them.

[1407] Step 6:

[1408] Users access the system through their terminal and perform a search by entering keywords, dates, and broadcaster names in the search box.

[1409] Input: Keywords, date, broadcaster name

[1410] Output: Search results (related video links, table of contents, highlight information)

[1411] Specific behavior:

[1412] When a user enters a keyword into the search box, the server searches the database for relevant video data, and displays the link to the video, its table of contents, and highlights to the user. Users can also click on the provided link to go directly to the specific section of the video they are looking for.

[1413] Step 7:

[1414] The server collects viewing data and analyzes which parts are viewed the most.

[1415] Input: Viewing data

[1416] Output: Analysis results of viewing data

[1417] Specific behavior:

[1418] The server records viewers' viewing behavior and collects and analyzes viewing data. The analysis results include the frequency of viewing each section and the parts that viewers are most interested in. This data is provided to the broadcaster in the form of a dashboard.

[1419] Step 8:

[1420] The server provides feedback to the distributor based on the results of analyzing the viewing data.

[1421] Input: Analysis results of viewing data

[1422] Output: Feedback information provided to the distributor

[1423] Specific behavior:

[1424] The server then sends the analysis results to the streamer. For example, it can provide information on which specific sections are most popular, which can help the streamer in creating the next piece of content. This feedback allows the streamer to plan and improve content that will attract viewers' attention.

[1425] (Application example 1)

[1426] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1427] With current video streaming services, the sheer volume of video content makes it difficult for users to quickly find specific information. It is also labor-intensive for streamers to use viewing data to improve their next content offering. This reduces convenience for both users and streamers, making it difficult to provide effective content. It is necessary to solve these issues and improve the efficiency of video search and streamer data utilization.

[1428] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1429] In this invention, the server includes: means for converting video audio into text using generative AI; means for analyzing the text and automatically extracting topics and highlights of the video; means for automatically generating a table of contents for the video based on the extracted topics and highlights; means for generating thumbnail images associated with the table of contents; means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to relevant parts; means for collecting viewing data and providing analysis results to broadcasters; means for generating a video timeline based on time axis information extracted from the text; and means for generating thumbnail images corresponding to each timeline entry. This allows users to quickly find parts of a video that interest them, and enables broadcasters to easily utilize viewing data to improve their next broadcast.

[1430] "Generative AI" is a technology that uses artificial intelligence technology to generate data such as text, audio, and images.

[1431] "Means for converting audio to text" refers to a technology that analyzes the audio in a video and outputs it as text information.

[1432] "Means for automatic topic and highlight extraction" refers to technology that automatically identifies important themes and points of interest from text data.

[1433] "Means for automatically generating a table of contents" refers to technology that automatically generates a table of contents to provide users with a table of contents that identifies the major sections within a video.

[1434] "Means for generating thumbnail images" refers to a technology that captures specific scenes from a video as images to be used as visual indexes.

[1435] "Means for searching and providing direct jump links" refers to technology that searches for related videos based on keywords, dates, and broadcaster names entered by the user, and generates links to directly access specific parts.

[1436] "Means of collecting viewing data and providing analytical results" refers to technology that collects and analyzes data such as video viewing history and viewing time, and provides the results to the distributor.

[1437] "Means for generating a video timeline based on time axis information" is a technology that visually represents the time axis of a video based on time information extracted from text data.

[1438] The "means for generating thumbnail images corresponding to timeline entries" is a technique for capturing an image at a specific time corresponding to each entry on the timeline and displaying it on the timeline.

[1439] The "means for proposing the next distribution plan" is a technology that automatically suggests the next distribution content to increase the viewer's interest based on the previous viewing data.

[1440] The "means for individually optimized content recommendations" is a technology that recommends individually customized content based on a user's search history and viewing history.

[1441] A system for carrying out the present invention is one in which a server, a terminal, and a user work together in cooperation with each other. A specific embodiment of the system will be described below.

[1442] Server Roles

[1443] First, when a user uploads a video from their device, the server converts the audio into text using generative AI. In this process, the server extracts audio data from the video file using a speech recognition library (e.g., speech_recognition) and a movie processing library (e.g., moviepy) and converts it into text data.

[1444] The server then analyzes this text data using a generative AI model (e.g., OpenAI's API) to automatically extract topics and highlights within the video, thereby identifying which sections are important.

[1445] Based on the extracted topics and highlights, the server automatically generates a table of contents for the video, allowing viewers to easily understand the content of the video and quickly access parts that interest them. Additionally, thumbnail images associated with the table of contents are also generated. For example, the server captures a frame at the beginning of each section and uses it as a thumbnail.

[1446] Role of user terminal

[1447] The user device provides an interface that allows viewers to search for videos by specific keywords, dates, or broadcaster names. When a user enters a search query, the server returns related videos and their table of contents links. Based on the search results, users can use links to jump directly to the parts that interest them. This allows viewers to quickly find the information they are looking for among a vast amount of video content.

[1448] Data analysis and feedback

[1449] The server also collects viewing data and provides analysis results to the broadcaster. Specifically, it makes suggestions for improving the content of the next broadcast based on data such as which parts viewers watched the most and which sections they were interested in. This feedback allows the broadcaster to create effective content to increase viewer interest.

[1450] Specific examples

[1451] For example, let's consider the case where a streamer uploads a "walkthrough video for a new game." The server converts the audio into text and analyzes it to extract topics such as "Introduction," "Game Strategy," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are presented to viewers in a visually easy-to-understand format.

[1452] When a viewer searches for "game tips," the server will display search results such as "game tips from 03:00" and provide a link that allows the user to access that section directly. An example of a prompt sentence that can be used in this case is as follows:

[1453] Prompt Sentence Examples

[1454] Extract the main topics and highlights from the following text:

[1455] "This is a walkthrough of a new game. First, I'll explain how to choose your character. Next, I'll introduce some important points for progressing through the game. Finally, I'll explain how to beat boss battles."

[1456] This allows users to quickly access important information and allows streamers to effectively plan upcoming content based on viewing data.

[1457] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1458] Step 1:

[1459] Uploading videos

[1460] The user uses the terminal to upload a video file to the server, which receives the video file, extracts the audio data, and prepares it for the next step.

[1461] Input: Video file

[1462] Output: Audio data

[1463] Step 2:

[1464] Speech-to-text

[1465] The server uses a speech recognition library (e.g., speech_recognition) to convert the audio data into text, analyzes the extracted audio data, and outputs it in text format.

[1466] Input: Audio data

[1467] Output: Text data

[1468] Step 3:

[1469] Text analytics

[1470] The server uses a generative AI model (e.g., OpenAI's API) to analyze the text data and automatically extract topics and highlights from the video. This analysis identifies the main themes and important scenes in the video.

[1471] Input: Text data

[1472] Output: Topic and highlight information

[1473] Step 4:

[1474] Automatic table of contents and thumbnail generation

[1475] The server automatically generates a table of contents for the video based on the extracted topics and highlight information, and simultaneously generates thumbnail images from the start of each section and associates them with the table of contents.

[1476] Input: Topic and highlight information

[1477] Output: Table of contents data, thumbnail images

[1478] Step 5:

[1479] Providing a search interface

[1480] The device provides the user with a search interface, displaying an input form that allows the user to search for videos by specific keywords, dates, or streamer names.

[1481] Input: None (preparation until user input)

[1482] Output: Search interface

[1483] Step 6:

[1484] Processing search queries

[1485] The user enters a search query and the device sends it to the server, which searches for related videos and table of contents links based on the query and returns the results.

[1486] Input: Search query (keyword, date, broadcaster name)

[1487] Output: Search results (related videos and table of contents links)

[1488] Step 7:

[1489] Collecting viewing data

[1490] The server collects data on users' viewing of videos (such as viewing time, viewing sections, etc.), which is later used by the distributor for analysis.

[1491] Input: Viewing activity data

[1492] Output: Recording to viewing database

[1493] Step 8:

[1494] Providing analysis results

[1495] The server analyzes the collected viewing data to determine which sections are most viewed and trends in viewer interest. The server then provides the analysis results to the broadcaster, helping them plan their next broadcast.

[1496] Input: Viewing data

[1497] Output: Analysis report

[1498] These steps allow users to quickly find what interests them and give publishers the data to improve their content next time.

[1499] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1500] This invention combines a system that utilizes generative AI to efficiently search and analyze video content with an emotion engine that recognizes user emotions. In this system, the server, terminal, and user elements work together to convert and analyze the audio of videos into text, and then provide recommended content based on the user's emotions.

[1501] Program processing explanation

[1502] Video upload and analysis

[1503] When a streamer uploads a video from their device, the server converts the audio into text using generative AI. Specifically, the server uses speech recognition technology to analyze the audio in the video in real time and convert it into text as subtitles.

[1504] The server then analyzes this text data and automatically extracts topics and highlights from the video. In this process, natural language processing technology is used to identify topics (e.g., game walkthroughs, Q&A corners, etc.) and highlight scenes (e.g., important announcements, funny moments) from the text content.

[1505] Table of contents and thumbnail generation

[1506] Based on the extracted topics and highlights, the server automatically generates a table of contents for the video, such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A session," and "20:00 - Ending."

[1507] The server also generates thumbnail images of each section, allowing viewers to easily navigate through the table of contents and jump to parts that interest them.

[1508] Providing search functionality

[1509] Users access the system through a terminal and use an interface that allows them to search for videos by specific keywords, dates, or streamer names. If a user types in a search term like "new game introductions," the server quickly identifies relevant videos and displays them as search results.

[1510] The server also provides links to relevant highlights and table of contents in search results, allowing users to jump directly to the parts they are interested in. This allows users to quickly find the information they are looking for among a vast amount of video content.

[1511] Implementing the Emotion Engine

[1512] The emotion engine uses facial recognition and voice analysis technologies to analyze the viewer's emotional state in real time while they are watching a video, determining, for example, whether they are laughing, surprised, sad, etc.

[1513] Based on the collected emotional data, the server can recommend videos and provide highlights that correspond to the user's emotional state. For example, if the server detects that the user is laughing, it can recommend more funny scenes or relaxing content.

[1514] Providing analysis results and recommended content

[1515] The server collects viewing data and emotional data, analyzes them, and generates data such as which parts were viewed the most and which parts evoked particular emotions.

[1516] The analysis results are displayed on the streamer's personal dashboard, and the streamer can use this data to plan and improve their content. Specifically, streamers can identify moments that make viewers laugh and topics that interest them and reflect this in their next stream.

[1517] Specific examples

[1518] For example, if a streamer uploads a "walkthrough video for a new game," the server converts the audio into text and analyzes it to extract the topics "Introduction," "Game Strategy," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are presented to viewers in a visually easy-to-understand format.

[1519] When a viewer searches for "game tips," the server will display search results such as "game tips from 3:00" and provide a link that allows the user to access that section directly. Furthermore, if the emotion engine detects a smile on the user's face while watching, the server will recommend funny or popular scenes from the same game. As a result, viewers can quickly access the content they want to watch, and broadcasters can maintain their viewers' interest.

[1520] The processing flow will be explained below.

[1521] Step 1:

[1522] The streamer uses a device to upload the video to the server.

[1523] Specifically, the streamer selects a video file on the device and clicks the upload button, and the device sends the video file to the server.

[1524] Step 2:

[1525] The server stores the received video files in a database.

[1526] Once saved, the process of converting the video's audio into text begins using generative AI.

[1527] Step 3:

[1528] The server uses generative AI to convert the video's audio data into text in real time.

[1529] Using voice recognition technology, what is said in the video is converted into text as subtitles.

[1530] Step 4:

[1531] The server analyzes the generated text data.

[1532] Using natural language processing technology, topics and highlights are automatically extracted from the text content. For example, topics such as "game strategies" and "question corners" are identified.

[1533] Step 5:

[1534] The server automatically generates a table of contents for the video based on the extracted topics and highlights.

[1535] For example, create table of contents entries such as "00:00 - Opening," "02:15 - Introduction to the new game," "10:45 - Q&A," and "20:00 - Ending."

[1536] Step 6:

[1537] The server generates thumbnail images corresponding to each section of the table of contents.

[1538] Thumbnail images are displayed on the viewing screen along with the table of contents, serving as a visual guide.

[1539] Step 7:

[1540] A user accesses the system through a terminal and searches for videos using a search interface.

[1541] When a user enters a specific keyword (e.g., "new game introduction"), the device sends this search query to the server.

[1542] Step 8:

[1543] The server receives the search query and searches the database to identify relevant videos and topics.

[1544] The server generates related videos and table of contents links as search results and returns them to the user.

[1545] Step 9:

[1546] Users receive search results on their devices and can jump directly to the relevant section by clicking on the displayed link.

[1547] This feature allows users to quickly access specific moments or sections.

[1548] Step 10:

[1549] The emotion engine analyzes the user's emotions while they are watching.

[1550] Using facial recognition and voice analysis technology, it determines in real time whether the viewer is laughing, surprised, sad, etc.

[1551] Step 11:

[1552] Based on the collected emotional data, the server provides recommended videos and highlights according to the user's emotional state.

[1553] For example, if the user is laughing, the server will suggest more funny scenes and related content.

[1554] Step 12:

[1555] The server collects and analyzes viewing data and emotion data, and generates analysis results.

[1556] Data is collected on which parts were viewed the most and how users felt about them.

[1557] Step 13:

[1558] The analysis results are displayed on a dashboard dedicated to the broadcaster.

[1559] Streamers can use this data to plan and improve their content, for example by identifying moments that made users laugh and creating strategies such as increasing the number of similar scenes in their next stream.

[1560] Example 2

[1561] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1562] With the rapid increase in video content, it is becoming increasingly difficult for viewers to efficiently search and access the information they are looking for. Furthermore, detailed viewing data is necessary for streamers to understand viewers' reactions and improve future content. However, current systems do not adequately collect and analyze not only viewing data but also emotional data during viewing, and provide content recommendations based on that data. Therefore, a comprehensive system is needed to improve the viewer experience and support streamers in content creation.

[1563] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1564] In this invention, the server includes: means for converting video audio into text using a generative AI; means for analyzing the text and automatically extracting video topics and highlights; means for automatically generating a video table of contents based on the extracted topics and highlights; means for generating thumbnail images associated with the table of contents; means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to relevant sections; means for collecting viewing data and user emotional data and providing the analysis results to the broadcaster; and means for analyzing the user's emotional state during viewing in real time using an emotion engine and recommending videos and providing highlights based on the collected emotional data. This allows viewers to efficiently access desired content, and broadcasters to understand viewer reactions in detail and improve the quality of their content in future broadcasts.

[1565] "Generative AI" refers to models or algorithms that generate data using artificial intelligence techniques.

[1566] "Text conversion" is the process of analyzing audio data and converting it into written information.

[1567] "Topic extraction" is a technique for analyzing text information and identifying key topics or themes within it.

[1568] "Highlight extraction" is a technology that automatically extracts particularly important parts or interesting scenes from video or text.

[1569] "Table of contents generation" is the process of creating easily digestible segments based on the main points of a video or piece of text.

[1570] "Thumbnail generation" is the process of extracting specific frames of a video as still images to create small visually representative images.

[1571] A "search function" is a technology that allows you to quickly find relevant information or data based on specific keywords or conditions.

[1572] "Viewing data" refers to data recorded and analyzed based on user behavior when watching videos (such as playback time and frequently viewed parts).

[1573] "Emotional data" refers to data collected by recognizing the emotional state (happiness, surprise, sadness, etc.) that viewers show while watching a video.

[1574] An "emotion engine" is a system or technology that analyzes the face and voice of a user while watching and determines their emotional state in real time.

[1575] "Recommended content" is a function that automatically suggests content that may interest a user based on the user's viewing history and emotional data.

[1576] A "viewing analysis dashboard" is a system that displays detailed viewer reactions and behavior to broadcasters based on collected viewing and emotional data.

[1577] This invention is a system for efficiently searching and analyzing video content, combining generative AI and emotion recognition technology to provide useful information to both viewers and streamers. This system is implemented primarily using the following hardware and software:

[1578] Video speech to text conversion

[1579] The server receives the video uploaded by the streamer from their device, extracts the audio using an audio processing tool (e.g., ffmpeg), and then converts the audio into text using a speech recognition service such as the Google Cloud Speech-to-Text API. The converted text is then stored in a database on the server.

[1580] Parsing text and generating a table of contents

[1581] The server analyzes the generated text using natural language processing techniques (e.g., BERT or GPT-3) to extract the topics and highlights of the video. For example, in a game walkthrough video, segments such as "Introduction," "Way of the Walkthrough," and "Conclusion" are identified. Based on this, a table of contents for the video is automatically generated, and thumbnail images corresponding to each segment are generated. OpenCV and PIL libraries are used to generate the thumbnails.

[1582] Providing search functionality

[1583] A user can access the system using a terminal and enter specific keywords into the search interface. For example, they can enter "new game introduction." The server receives this search query, matches it with the stored text data and table of contents information, identifies related videos, and displays the search results as a list. The user can click on the displayed link to jump directly to the relevant section.

[1584] Emotion recognition implementation

[1585] The emotion engine analyzes facial expressions and voice in real time via the camera and microphone while the user is watching a video. It uses FaceAPI and Emotion API to determine the user's emotions, such as smile, surprise, sadness, etc. This emotion data is sent to the server and stored in a database along with the viewing data.

[1586] Providing recommended content

[1587] The server then runs an algorithm based on the collected emotional data to recommend content that matches the user's emotional state. For example, if the user is smiling a lot, the server will recommend more entertaining videos or relaxing content. Recommended content is provided via notifications and lists.

[1588] Viewing analytics dashboard

[1589] The server generates a dashboard for streamers based on the collected viewing and emotion data. The dashboard displays the viewing time for each segment, a heat map of emotion data, and an analysis of viewer reactions. This allows streamers to understand viewer reactions in detail and use this information to improve future content creation.

[1590] Specific operation example

[1591] For example, if a streamer uploads a video explaining how to play a new game, the server converts the audio into text and analyzes it to extract topics such as "Introduction," "Game Tips," and "Conclusion." Based on this, the server automatically generates a table of contents and thumbnail images, which are then provided to viewers.

[1592] When a viewer searches for "game strategies," the server displays search results such as "game strategies from 03:00," and the viewer can click on a link to directly access the relevant section. In addition, when the emotion engine detects a smile on the user's face while watching, the server recommends interesting or popular scenes from the same game, improving the user's viewing experience.

[1593] Prompt Sentence Examples

[1594] Introducing a new game

[1595] Enter: Introducing a new game

[1596] Output: A list of all videos tagged with "new game introductions"

[1597] "Tell me the highlight scenes in the video"

[1598] Input: Highlights from the video

[1599] Output: A list of highlight scenes from each video

[1600] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1601] System program processing flow

[1602] Step 1: Upload your video

[1603] The server receives a request from the distributor to upload a video file from the terminal.

[1604] Input: Video file upload request from the device

[1605] Output: The location where the uploaded video file will be saved to storage.

[1606] Specific behavior: The server receives the video file via an HTML form or API endpoint, saves it to storage, and returns a status code of 200 to confirm the upload was successful.

[1607] Step 2: Speech to text

[1608] The server extracts the audio from the stored video file and converts it into text using generative AI.

[1609] Input: Saved video file

[1610] Output: Text data

[1611] Specific operation: The server separates the audio using ffmpeg and converts the audio data to text data using the Google Cloud Speech-to-Text API. The converted text is then stored in a database.

[1612] Step 3: Parsing the text

[1613] The server analyzes the generated text data using natural language processing technology to extract topics and highlights from the video.

[1614] Input: Text data

[1615] Output: Topic and highlight information (JSON format)

[1616] Specific operation: The server uses generative AI such as BERT or GPT-3 to identify topics (e.g., "Introduction," "Game Strategy," "Conclusion") and highlight scenes, and saves the results in JSON format.

[1617] Step 4: Generate the Table of Contents and Thumbnails

[1618] The server generates a table of contents and thumbnail images for the video based on the extracted topics and highlights.

[1619] Input: Topic and highlight information (JSON format)

[1620] Output: Table of contents and thumbnail images

[1621] Specific operation: The server parses the JSON data and automatically generates a table of contents such as "00:00 - Opening" and "02:15 - Introduction to the new game." It also uses OpenCV and PIL libraries to generate and save thumbnail images for each segment.

[1622] Step 5: Keyword search

[1623] Users input specific keywords into the search interface, and the server searches for relevant videos based on the input keywords and displays the results.

[1624] Input: User's search keyword

[1625] Output: Related video list with table of contents links and thumbnails

[1626] Specific operation: When a user enters "new game introduction" into the search interface, the server references the text data and table of contents information, and displays a list of related videos and their table of contents links. The user can click on the link to directly access the relevant section.

[1627] Step 6: Emotion Recognition and Data Collection

[1628] The emotion engine uses the camera and microphone to analyze emotions in real time while the user is watching a video and transmits the results to a server.

[1629] Input: User's facial image data and voice data

[1630] Output: Emotion data (smile, surprise, sadness, etc.)

[1631] Specific operation: The emotion engine uses FaceAPI and Emotion API to determine the user's emotional state and sends the data to the server. The emotion data is stored in the database along with the viewing data.

[1632] Step 7: Providing recommended content

[1633] The server recommends optimal content to the user based on the collected emotion data.

[1634] Input: Emotion data and viewing data

[1635] Output: Recommended content

[1636] How it works: The server uses an algorithm to analyze emotion data and viewing data. For example, if the user is smiling a lot, the server will recommend more interesting videos. Recommended content is provided via notifications and the user interface.

[1637] Step 8: Displaying the analysis results on a dashboard

[1638] The server displays the collected and analyzed viewing and sentiment data on a dashboard for the broadcaster.

[1639] Input: Viewing data and emotion data

[1640] Output: Display analysis results for the broadcaster

[1641] Specific operation: The server generates a heat map of viewing time and emotional data for each segment based on viewing data and emotional data, and displays it on the dashboard. The broadcaster can use this information to improve the content of their next broadcast.

[1642] (Application example 2)

[1643] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1644] Conventional video distribution systems make it difficult for viewers to efficiently find the information they are looking for from a vast amount of video content. Furthermore, providing personalized recommended content based on the viewer's emotions is difficult, and this has not sufficiently improved the viewing experience. Therefore, there is a need for a system that not only converts audio into text and extracts video topics and highlights, but also uses viewer emotional data to recommend appropriate content.

[1645] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1646] In this invention, the server includes: means for converting video audio into text using a generative AI; means for analyzing the text and automatically extracting video topics and highlights; means for automatically generating a video table of contents based on the extracted topics and highlights; means for generating thumbnail images associated with the table of contents; means for searching videos by keyword, date, and broadcaster name and providing links to jump directly to relevant sections; means for collecting viewing data and providing analysis results to broadcasters; an emotion engine for recognizing the user's emotional state; and means for providing recommended content to the user based on the emotion engine. This enables viewers to efficiently find desired information from a vast amount of video content and also provides personalized recommended content based on the viewer's emotions.

[1647] "Generative AI" is a generative artificial intelligence technology that primarily uses deep learning models to generate or convert text, audio, and images.

[1648] "Video audio to text conversion" refers to the process of analyzing the audio data in a video and converting it into corresponding text data.

[1649] A "topic" is a major theme or topic that summarizes the content of a video.

[1650] "Highlights" refer to particularly important moments or scenes in a video that are of great interest to viewers.

[1651] A "table of contents" is an organized list of the video's contents that provides information including the start times of each section.

[1652] A "thumbnail image" is a small image that represents a particular scene in a video and functions as a visual table of contents.

[1653] "Keywords" are specific words or phrases used to search for video content.

[1654] "Distributor name" is the name of the person or organization that published the video.

[1655] "Viewing data" refers to data obtained when a viewer watches a video, and includes information such as playback time, viewing locations, and viewing frequency.

[1656] The "emotion engine" is a technology that analyzes the user's facial expressions and voice to recognize their emotional state in real time.

[1657] "Recommended content" is highly relevant content that is selected based on the user's viewing history and emotional state and provided to the user.

[1658] To implement this invention, the following system configuration is adopted. The server implements a program that uses generative AI to convert video audio into text and analyze it. The generative AI model used in this process is, for example, Transformer-based speech recognition technology. Specific software used includes the Transformer model library and TextBlob for natural language processing. In addition, facial recognition technology and voice analysis technology are implemented in the emotion engine to analyze user emotions in real time.

[1659] The text data generated by the server is analyzed using natural language processing technology to automatically extract topics and highlights from the video. This automatically generates a table of contents and thumbnail images for the video. Users access the system using their devices and search for videos using specific keywords, dates, or broadcaster names. As a result, a link is provided that takes users directly to the relevant part of the video.

[1660] The emotion engine analyzes the user's facial expressions and voice while watching to recognize their emotional state in real time. Based on this emotion data, the server provides recommended content to the user. For example, if the user is detected as laughing, the server will recommend more funny scenes or relaxing content.

[1661] For example, when a user watches a "new game walkthrough video," the server converts the audio data into text and extracts topics such as "introduction," "game strategy," and "conclusion." If the emotion engine detects a smile while the viewer is watching, the server recommends related, interesting scenes. This allows viewers to quickly access the content they want to watch and allows streamers to sustainably capture viewers' attention.

[1662] Examples of prompt sentences include the following:

[1663] "Users are smiling while watching. Recommend content they'll enjoy."

[1664] By operating the system in this way, viewers can efficiently find the information they are looking for from a vast amount of video content, and it is also possible to provide personalized recommended content based on the viewer's emotions.

[1665] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1666] Step 1: Upload video and convert audio

[1667] A user uploads a video from their device to the server. The server then uses a generative AI model (Transformer-based speech recognition technology) to convert the audio data in the video into text data. The input is the video file, and the output is text data.

[1668] Step 2: Text analysis and topic extraction

[1669] The server analyzes the converted text data and automatically extracts topics and highlights from the video using natural language processing technology (such as TextBlob). The input is text data, and the output is the extracted topics and highlights. Specific operations include topic segmentation and extraction of important words and phrases.

[1670] Step 3: Generate the Table of Contents and Thumbnails

[1671] The server automatically generates a table of contents for the video based on the extracted topics and highlights, and also generates thumbnail images for each section. The input is the topics and highlights, and the output is the table of contents and thumbnail images. The table of contents includes the start time of each topic.

[1672] Step 4: Providing keyword search functionality

[1673] Users can search for videos by entering specific keywords, dates, or streamers on their devices. The server searches for video data corresponding to those keywords and displays related highlights and table of contents links. The input is a search query, and the output is a list of related videos and links.

[1674] Step 5: Collect and analyze emotion data

[1675] While a user is watching a video, the server uses an emotion engine to analyze the user's emotional state in real time. The input is facial expression data and voice data during viewing, and the output is emotion data (e.g., smile, surprise, etc.). Specific operations use face recognition technology and voice analysis technology.

[1676] Step 6: Generate recommended content

[1677] The server generates content recommendations appropriate for the user based on the collected emotion data. For this, it can use a generative AI model. For example, if a smile is detected, it can recommend more funny scenes or relaxing content. The input is emotion data, and the output is a list of recommended content.

[1678] Step 7: Displaying the analysis results on a dashboard

[1679] The server collects and analyzes viewing data and emotional data, and displays the results on the streamer's dashboard. This allows streamers to check data based on viewers' high interest and emotional state, and use it to plan their next content. The input is viewing data and emotional data, and the output is the analysis results on the dashboard.

[1680] The above steps will realize a system that allows viewers to efficiently find the information they are looking for from a vast amount of video content and enjoy personalized recommended content based on their emotions.

[1681] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1682] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1683] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1684] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1685] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1686] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1687] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1688] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1689] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1690] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1691] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1692] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1693] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1694] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1695] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1696] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1697] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1698] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1699] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1700] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1701] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1702] The following is further disclosed regarding the above embodiment.

[1703] (Claim 1)

[1704] A means of converting video audio into text using generative AI,

[1705] means for analyzing the text and automatically extracting topics and highlights of the video;

[1706] A means for automatically generating a video table of contents based on the extracted topics and highlights;

[1707] means for generating thumbnail images associated with the table of contents;

[1708] A way to search for videos by keyword, date, or broadcaster name and provide links to jump directly to the relevant part,

[1709] A means of collecting viewing data and providing analysis results to distributors;

[1710] A system including:

[1711] (Claim 2)

[1712] A method to convert the audio of the distributed video into text at a specified timing and then automatically detect important topics.

[1713] means for collecting and analyzing viewing data for specific portions of the video;

[1714] A means for providing information on popular viewing parts to distributors based on the collected viewing data;

[1715] 10. The system of claim 1, comprising:

[1716] (Claim 3)

[1717] A means to provide an interface that allows viewers to search for past broadcast videos by entering specific keywords, and

[1718] A means to display related videos and their table of contents links based on the search query;

[1719] A way to provide links that allow viewers to jump directly from search results to specific sections of your video;

[1720] 10. The system of claim 1, comprising:

[1721] "Example 1"

[1722] (Claim 1)

[1723] A way to convert video audio into text,

[1724] means for analyzing the text and automatically extracting topics and highlights of the video;

[1725] A means for automatically generating a video table of contents based on the extracted topics and highlights;

[1726] means for generating an image associated with the table of contents;

[1727] A way to search for videos by keyword, date, or broadcaster name and provide links to jump directly to the relevant part,

[1728] A means of collecting viewing data and providing analysis results to distributors;

[1729] A system including:

[1730] (Claim 2)

[1731] A means to convert the audio of the distributed video into text at a specified timing,

[1732] A means for automatically detecting important topics using natural language processing technology;

[1733] means for collecting and analyzing viewing data for specific portions of the video;

[1734] A means for providing information on popular viewing parts to distributors based on the collected viewing data;

[1735] 10. The system of claim 1, comprising:

[1736] (Claim 3)

[1737] A means to provide an interface that allows viewers to search for past broadcast videos by entering specific keywords, and

[1738] A means to display related videos and their table of contents links based on the search query;

[1739] A way to provide links that allow viewers to jump directly from search results to specific sections of your video;

[1740] 10. The system of claim 1, comprising:

[1741] "Application Example 1"

[1742] (Claim 1)

[1743] A means of converting video audio into text using generative AI,

[1744] means for analyzing the text and automatically extracting topics and highlights of the video;

[1745] A means for automatically generating a video table of contents based on the extracted topics and highlights;

[1746] means for generating thumbnail images associated with the table of contents;

[1747] A way to search for videos by keyword, date, or broadcaster name and provide links to jump directly to the relevant part,

[1748] A means of collecting viewing data and providing analysis results to distributors;

[1749] means for generating a timeline of the video based on the time axis information extracted from the text;

[1750] means for generating a thumbnail image corresponding to each timeline entry;

[1751] A system including:

[1752] (Claim 2)

[1753] A method to convert the audio of the distributed video into text at a specified timing and then automatically detect important topics.

[1754] means for collecting and analyzing viewing data for specific portions of the video;

[1755] A means for providing information on popular viewing parts to distributors based on the collected viewing data;

[1756] A method to propose the next distribution plan based on the parts of the video that interest the viewer, and

[1757] 10. The system of claim 1, comprising:

[1758] (Claim 3)

[1759] A means to provide an interface that allows viewers to search for past broadcast videos by entering specific keywords, and

[1760] A means to display related videos and their table of contents links based on the search query;

[1761] A way to provide links that allow viewers to jump directly from search results to specific sections of your video;

[1762] a means for making individually optimized content recommendations based on search history obtained from a viewer;

[1763] 10. The system of claim 1, comprising:

[1764] "Example 2: Combining Emotion Engines"

[1765] (Claim 1)

[1766] A means of converting video audio into text using generative AI,

[1767] means for analyzing the text and automatically extracting topics and highlights of the video;

[1768] A means for automatically generating a video table of contents based on the extracted topics and highlights;

[1769] means for generating thumbnail images associated with the table of contents;

[1770] A way to search for videos by keyword, date, or broadcaster name and provide links to jump directly to the relevant part,

[1771] A means for collecting viewing data and user emotion data and providing analysis results to the distributor;

[1772] A means for analyzing the emotional state of a user while watching in real time using an emotion engine and recommending videos and providing highlights based on the collected emotion data;

[1773] A system including:

[1774] (Claim 2)

[1775] A method to convert the audio of the distributed video into text at a specified timing and then automatically detect important topics.

[1776] means for collecting and analyzing viewing and sentiment data for specific portions of the video;

[1777] A means for providing information to a distributor about parts of the content that are frequently viewed and parts of the content that are frequently detected with specific emotions based on the collected viewing data and emotion data;

[1778] 10. The system of claim 1, comprising:

[1779] (Claim 3)

[1780] A means to provide an interface that allows viewers to search for past broadcast videos by entering specific keywords, and

[1781] A means for displaying related videos and their table of contents links and thumbnails based on a search query;

[1782] A way to provide links that allow viewers to jump directly from search results to specific sections of your video;

[1783] 10. The system of claim 1, comprising:

[1784] "Application example 2 when combining emotion engines"

[1785] (Claim 1)

[1786] A means of converting video audio into text using generative AI,

[1787] means for analyzing the text and automatically extracting topics and highlights of the video;

[1788] A means for automatically generating a video table of contents based on the extracted topics and highlights;

[1789] means for generating thumbnail images associated with the table of contents;

[1790] A way to search for videos by keyword, date, or broadcaster name and provide links to jump directly to the relevant part,

[1791] A means of collecting viewing data and providing analysis results to distributors;

[1792] an emotion engine that recognizes the user's emotional state;

[1793] means for providing recommended content to a user based on the emotion engine;

[1794] A system including:

[1795] (Claim 2)

[1796] A method to convert the audio of the distributed video into text at a specified timing and then automatically detect important topics.

[1797] means for collecting and analyzing viewing data for specific portions of the video;

[1798] A means for providing information on popular viewing parts to a broadcaster based on the collected viewing data and emotion data;

[1799] 10. The system of claim 1, comprising:

[1800] (Claim 3)

[1801] A means to provide an interface that allows viewers to search for past broadcast videos by entering specific keywords, and

[1802] A means to display related videos and their table of contents links based on the search query;

[1803] A way to provide links that allow viewers to jump directly from search results to specific sections of your video;

[1804] A means for analyzing the emotional state of a user while viewing using an emotion engine;

[1805] 10. The system of claim 1, comprising: [Explanation of symbols]

[1806] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of converting video audio into text using generative AI, means for analyzing the text and automatically extracting topics and highlights of the video; A means for automatically generating a video table of contents based on the extracted topics and highlights; means for generating thumbnail images associated with the table of contents; A way to search for videos by keyword, date, or broadcaster name and provide links to jump directly to the relevant part, A means of collecting viewing data and providing analysis results to distributors; A system including:

2. A method to convert the audio of the distributed video into text at a specified timing and then automatically detect important topics. means for collecting and analyzing viewing data for specific portions of the video; A means for providing information on popular viewing parts to distributors based on the collected viewing data; The system of claim 1 , comprising:

3. A means to provide an interface that allows viewers to search for past broadcast videos by entering specific keywords, and A means to display related videos and their table of contents links based on the search query; A way to provide links that allow viewers to jump directly from search results to specific sections of your video; The system of claim 1 , comprising:

Images