system

Abstract

Provide a system. [Solution] A system is provided that includes: a means for a user to upload a video; a means for a server to receive the video and store it in temporary storage; a means for the server to convert the video's audio data into text using a generative artificial intelligence engine; a means for the server to analyze the video's video data and divide scenes to create a summary; a means for the server to create chapter candidates based on the generated summary and scenes; a means for the server to send the chapter candidates to a terminal; a means for the terminal to display the chapter candidates received to the user; a means for the user to input a specific keyword into a search bar on the terminal; a means for the terminal to send the keyword to the server; a means for the server to analyze the text data of the video and identify scenes containing the keyword; a means for the server to send the scene timestamp to the terminal; and a means for the terminal to display the timestamp to the user, allowing the user to access the specified scene.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, many users have to watch the entire video to efficiently obtain the information they need, which takes a lot of time and effort. Furthermore, it is difficult to identify the appropriate section or scene to find specific information, making information search within videos extremely inefficient. Therefore, there is a need for a system that automatically analyzes video content, presents summaries and chapter suggestions, and also provides a system that can perform reverse search using keywords. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including: means for a user to upload a video; means for a server to receive the video and store it in temporary storage; means for the server to convert the video's audio data into text using a generative artificial intelligence engine; means for the server to analyze the video data of the video, divide the scenes and create a summary; means for the server to create chapter candidates based on the generated summary and scenes; means for the server to send the chapter candidates to a terminal; means for the terminal to display the chapter candidates received to the user; means for the user to input a specific keyword into a search bar on the terminal; means for the terminal to send the keyword to the server; means for the server to analyze the text data of the video and identify scenes containing the keyword; means for the server to send the scene timestamp to the terminal; and means for the terminal to display the timestamp to the user, allowing the user to access the specified scene.

[0006] 1. "User" refers to the end user who uploads videos and uses the search functionality of the System.

[0007] 2. "Video" means a digital file containing video and audio.

[0008] 3. "Server" refers to a computer system that receives, analyzes, stores, and processes video data.

[0009] 4. "Terminal" refers to a device such as a computer or mobile device used by a user, which communicates with the server and displays data.

[0010] 5. "Generative AI engine" refers to an AI system that analyzes video content and performs tasks such as text conversion, summary generation, and chapter division.

[0011] 6. "Audio data" refers to digitized data of audio information played within a video.

[0012] 7. "Convert to text" refers to the process of analyzing and converting audio data into human-readable text.

[0013] 8. "Video data" refers to the digitized visual information reproduced within a video.

[0014] 9. "Splitting scenes" refers to the process of analyzing the visual data of a video and dividing it into different sections or scenes.

[0015] 10. "Summary" refers to a brief summary of the video content that includes the main points and themes.

[0016] 11. "Chapter candidate" refers to a segment that is a candidate for automatically generating a chapter that indicates a specific segment within a video.

[0017] 12. "Search bar" refers to an input field where users can enter specific keywords to search for information.

[0018] 13. "Timestamp" refers to a digital marker that indicates the start time or location of a particular scene in a video.

[0019] 14. "Reverse word search" refers to the function that identifies and displays relevant scenes within a video based on keywords entered by the user.

[0020] 15. "JSON Format" means a lightweight text format for data exchange, based on JavaScript® Object Notation. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The present invention is a system that automatically analyzes the content of a video and can pinpoint the information that a user needs. Specific embodiments for carrying out the present invention will be described below.

[0043] System Overview

[0044] This system consists of users, a server, and terminals. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the videos by entering specific keywords.

[0045] Uploading and saving videos

[0046] First, the user selects a video file and uploads it to the system. Once the video file is uploaded, the device sends it to the server. The server then stores the received video file in temporary storage and prepares it for analysis.

[0047] Video analysis and summary generation

[0048] The server then launches a generative AI engine to convert the video's audio data into text. Through this speech recognition process, the video's content is extracted as text data. The server then analyzes the video data and segments each scene. The generative AI engine then summarizes the content of each scene and extracts the main points.

[0049] Creating and viewing chapter suggestions

[0050] The server creates chapter candidates based on the generated summaries and scenes. This automatically divides the video into multiple sections, each with a title and summary. The server sends the chapter candidates in JSON format to the device, which then displays them visually to the user.

[0051] Reverse search function from words

[0052] The user enters a specific keyword in the search bar and clicks the search button. The device sends the keyword to the server. The server analyzes the text data of the video and identifies the scene in which the keyword appears. Once the timestamp of the relevant scene is identified, the server sends that information to the device. The device displays the search results, including the timestamp, to the user, allowing the user to directly access the specified scene.

[0053] Specific examples

[0054] For example, consider a user who uploads a video about "Programming Basics," which includes examples of multiple programming languages ​​and their usage.

[0055] 1. Upload and analyze

[0056] A user uploads a video on "Programming Basics."

[0057] The terminal transmits the selected video file to the server.

[0058] The server receives the video, stores it in temporary storage, and begins the analysis process.

[0059] 2. Summarizing and Chapter Creation

[0060] The server uses a generative artificial intelligence engine to summarize the content of the video.

[0061] The server creates chapters based on each language segment (e.g. "Python Basics", "JavaScript Basics", etc.).

[0062] 3. Display results

[0063] The server sends the summary and chapter information to the device in JSON format.

[0064] The terminal displays this to the user in a list format.

[0065] 4. Reverse Search Feature

[0066] A user types "Python" into the search bar and clicks the search button.

[0067] The device sends the keyword to the server.

[0068] The server analyzes the video transcription data, identifies scenes in which "Python" appears, and sends relevant information to the device.

[0069] The device displays the search results in a list format, allowing the user to directly access related scenes.

[0070] This system allows users to quickly access the information they need without having to play the entire video, making it possible to use videos efficiently.

[0071] The processing flow will be explained below.

[0072] Step 1:

[0073] The user selects a video file and uploads it to the system. The user selects a video file from their computer using the file selection dialog and clicks the "Upload" button.

[0074] Step 2:

[0075] The device sends the selected video file to the server. The device obtains the binary data of the video file and sends it to the server using an HTTP POST request.

[0076] Step 3:

[0077] The server receives the video file and stores it in temporary storage. The server receives the video file, checks the integrity of the data, and then stores it in a temporary storage folder.

[0078] Step 4:

[0079] The server starts the generative artificial intelligence engine. The server calls the generative artificial intelligence engine to start the process of converting audio data into text for the prepared video file.

[0080] Step 5:

[0081] The server converts the audio data into text, and a generative AI engine analyzes the audio in the video and generates recognized text data.

[0082] Step 6:

[0083] The server analyzes the video data, extracts each frame of the video, and segments it into scenes using a video recognition algorithm.

[0084] Step 7:

[0085] The server generates a summary for each scene using the generated text and video data. The content is reviewed to ensure that key points and themes are included.

[0086] Step 8:

[0087] The server creates chapter candidates. Based on the generated summary and scene information, the server assigns a title and summary to each segment and creates chapter candidates.

[0088] Step 9:

[0089] The server sends the chapter candidates to the device. The server encodes the completed chapter candidates in JSON format and sends them to the device as an HTTP response.

[0090] Step 10:

[0091] The device parses the received chapter suggestions and displays them on the user interface. The device parses the JSON data and displays them to the user in list or timeline format.

[0092] Step 11:

[0093] A user enters a specific keyword into the search bar and clicks the search button.A user enters a keyword into the input field of the search bar and presses the button with the intention of a search command.

[0094] Step 12:

[0095] The device sends the keyword to the server. The device creates an HTTP GET request including the user's keyword and sends it to the server.

[0096] Step 13:

[0097] The server analyzes the text data of the video and identifies the scenes in which the keywords appear. Using a full-text search algorithm, the server extracts text fragments containing the keywords and obtains the timestamps of the corresponding scenes.

[0098] Step 14:

[0099] The server sends the timestamp to the device, and the server returns the identified timestamp and scene information in JSON format to the device as search results.

[0100] Step 15:

[0101] The device displays the search results to the user. The device displays the received timestamps and scene information in a list format, and provides a "Play" button for each entry.

[0102] Step 16:

[0103] The user clicks the play button to access the specified scene. When the user clicks the play button in the displayed list, the video player starts playing from the corresponding scene.

[0104] Example 1

[0105] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0106] In today's world, the proliferation of video content has made it difficult for users to quickly and efficiently obtain the information they need. Long video playback times also create inefficiencies in information retrieval. Furthermore, the lack of a way for users to quickly search for specific information of interest within a video limits the viewing experience.

[0107] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0108] In this invention, the server includes: means for a user to upload a video; means for a terminal to transmit a video file to the server; means for the server to store the video file in temporary storage; means for the server to convert audio data of the video into text using speech recognition technology; means for the server to segment video data of the video by scene using video analysis technology; means for the server to generate a summary from the text data and scene information using a generative artificial intelligence engine; means for the server to create chapter candidates based on the generated summary and scene information; means for the server to transmit the chapter candidates to the terminal in JSON format; means for the terminal to visually display the received chapter candidates to the user; means for the user to input a specific keyword into a search bar of the terminal; means for the terminal to transmit the keyword to the server; means for the server to analyze the text data and identify scenes containing the keyword; means for the server to transmit scene timestamp information to the terminal; and means for the terminal to display the timestamp information to the user and allow the user to access the specified scene. This allows the user to quickly access necessary information without playing the entire video.

[0109] "User" means an individual or entity that uses the system to upload videos and search for specific information.

[0110] A "terminal" is an electronic device, such as a computer or smartphone, that a user uses to upload videos or view search results.

[0111] A "server" is a computer system that stores videos sent by users and performs analysis processing.

[0112] "Temporary storage" refers to data storage used by the server to temporarily store video files.

[0113] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[0114] "Video analysis technology" is a technology that analyzes video data and identifies specific scenes and frames.

[0115] A "generative AI engine" is a type of artificial intelligence used to generate new information or summaries based on input data.

[0116] "Text data" is character-based data that represents the content of a video extracted using voice recognition technology.

[0117] A "scene" is a segment that separates parts of a video where the content changes.

[0118] "Chapter candidates" are multiple sections generated based on the video content, and each section is given a title and a summary.

[0119] The "JSON format" is one of the standard formats for structuring and describing data in text format.

[0120] The "search bar" is an input area where users can input specific keywords to search for related information within a video.

[0121] A "timestamp" is information that indicates a specific time point within a video.

[0122] The present invention is a system that automatically analyzes the content of a video and can pinpoint the information that a user needs. Specific embodiments for carrying out the present invention will be described below.

[0123] System Overview

[0124] This system consists of users, a server, and terminals. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the videos by entering specific keywords.

[0125] Uploading and saving videos

[0126] First, the user selects a video file and uploads it to the system. To do this, the user uses a device such as a PC or smartphone. The device then sends the selected video file to the server. The server then stores the received video file in temporary storage (for example, Amazon S3 or Google® Cloud Storage) and prepares it for analysis.

[0127] Video analysis and summary generation

[0128] The server then analyzes the video using voice recognition and video analysis technology. The specific process is as follows:

[0129] 1. Audio to text conversion:

[0130] The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to convert the audio data in the video into text data. Through this process, the content of the video is extracted as a string of characters.

[0131] 2. Scene segmentation of video data:

[0132] The server uses video analysis technology (e.g., OpenCV library) to analyze the video data and segment each scene. Specifically, it detects change points between frames in the video and recognizes different scenes.

[0133] 3. Generate a summary:

[0134] The server uses a generative artificial intelligence engine (e.g., OpenAI® GPT) to generate a summary of each scene based on the extracted text data and scene information, concisely summarizing the key points and content.

[0135] Creating and viewing chapter suggestions

[0136] Based on the generated summary and scene information, the server creates chapter candidates that divide the video into multiple sections. Each section is then given a title and summary. The server converts the chapter candidates into JSON format and sends them to the device. The device parses the received JSON data and displays the chapter candidates in a list format on the user interface. The user can click on one of them to directly access a specific scene.

[0137] Keyword search function

[0138] The user enters a specific keyword into the system's search bar and clicks the search button. The device sends this keyword to the server. The server analyzes the text data of the video and identifies the scenes in which the keyword appears. It obtains the timestamp information of the relevant scenes and sends it to the device in JSON format. The device analyzes the received data and displays the search results in a list format on the user interface. The user can click on this to directly access the relevant scenes.

[0139] Specific examples

[0140] Here, we will use a video on "Programming Basics" as an example.

[0141] 1. Upload and analyze:

[0142] A user uploads a video on "Programming Basics."

[0143] The device sends the selected video to the server, which stores the video in temporary storage.

[0144] The server converts the audio data into text using the Google Cloud Speech-to-Text API, and splits the video data into scenes using OpenCV.

[0145] The server creates chapter candidates based on the summary generated using OpenAI GPT and sends them to the device.

[0146] 2. Results display:

[0147] The device receives the chapter information in JSON format and displays it in a list format on the user interface.

[0148] 3. Keyword Search:

[0149] A user types "Python" into the search bar and clicks the search button.

[0150] The device sends the keyword to the server, and the server analyzes the text data of the video to identify the relevant scene, obtains timestamp information, and sends it to the device.

[0151] The device displays the search results in a list format, allowing the user to directly access related scenes.

[0152] Prompt Sentence Examples

[0153] Here are some example prompts to enter into a generative AI engine:

[0154] "This video explains the basics of programming. Please summarize the basic usage of each programming language scene by scene."

[0155] "Please summarize the sections on Python, JavaScript, and Java® basics for each scene."

[0156] This system allows users to quickly access the information they need without having to play the entire video.

[0157] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0158] Step 1:

[0159] The user selects a video file and uploads it to the system via their device. The user selects a video file through a web browser on a device such as a PC or smartphone. The selected video file is sent from the device to the server. The input is the video file, and the output is the transmission of the video file to the server.

[0160] Step 2:

[0161] The device sends the selected video file to the server. Specifically, it uploads the video file to the server using an HTTP POST request. The input is the video file selected by the user, and the output is the video file sent to the server.

[0162] Step 3:

[0163] The server stores the received video file in temporary storage. The server stores the received video file in storage such as Amazon S3 or Google Cloud Storage, and records the file path and file name in a database. The input is the video file received from the device, and the output is saving to storage and recording to the database.

[0164] Step 4:

[0165] The server uses speech recognition technology to convert the audio data in the video into text. Specifically, it uses the Google Cloud Speech-to-Text API to convert the audio data in the video file into text. The input is the audio data in the video file, and the output is the converted text data.

[0166] Step 5:

[0167] The server uses video analysis technology to divide the video data into scenes. Specifically, it uses the OpenCV library to analyze the video data frame by frame and detect different scenes. The input is the video data of the video file, and the output is the divided scene information.

[0168] Step 6:

[0169] The server uses a generative AI engine to generate summaries based on the text data and scene information. Specifically, it uses OpenAI GPT to summarize the key points of each scene. The input is the text data for each scene, and the output is a summary of each scene.

[0170] Step 7:

[0171] The server creates chapter candidates based on the generated summaries and scene information. The server combines the summaries and scene information, divides the video into multiple sections, and assigns titles and summaries to each section. The input is the summaries and scene information, and the output is chapter candidates.

[0172] Step 8:

[0173] The server sends chapter candidates to the device in JSON format. The server converts the generated chapter information into JSON format and sends it to the device as an HTTP response. The input is the chapter candidates, and the output is the JSON data sent to the device.

[0174] Step 9:

[0175] The device displays the received chapter candidates on the user interface. The device parses the received JSON data and displays the chapter candidates in list format on the user interface. The input is JSON data, and the output is a visual representation of the chapter candidates.

[0176] Step 10:

[0177] The user enters a specific keyword into the search bar of the device. The user enters the keyword of interest and clicks the search button. The input is the keyword, and the output is the transmission of the keyword.

[0178] Step 11:

[0179] The terminal sends a keyword to the server. The terminal sends the entered keyword to the server via an HTTP POST request. The input is the keyword, and the output is the keyword sent to the server.

[0180] Step 12:

[0181] The server analyzes the text data to identify scenes containing keywords. The server analyzes the text data of the video and obtains the scenes in which the keywords appear and their timestamp information. The input is the keywords and text data, and the output is the timestamp information of the identified scenes.

[0182] Step 13:

[0183] The server sends the scene timestamp information to the terminal. The server converts the acquired timestamp information into JSON format and sends it to the terminal as an HTTP response. The input is the timestamp information, and the output is the JSON data sent to the terminal.

[0184] Step 14:

[0185] The terminal displays the timestamp information on the user interface, and the user accesses the specified scene. The terminal parses the received JSON data and displays the search results in a list format. The user can click on it to directly access the related scene. The input is JSON data, and the output is a visually displayed search result.

[0186] This allows users to quickly access the information they need without having to play the entire video.

[0187] (Application example 1)

[0188] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0189] In conventional video content distribution systems, users must watch the entire video, making it difficult to quickly access the information they need. Furthermore, the functionality for searching for specific topics or scenes was insufficient, preventing efficient information retrieval. This made organizing and searching for information cumbersome for long videos, such as educational content or documentaries, and reduced usability.

[0190] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0191] In this invention, the server includes a means for converting audio data from a video into text using a generative artificial intelligence engine, a means for analyzing the video data of the video, dividing the scenes, and creating summaries, and a means for creating chapter candidates based on the generated summaries and scenes. This allows users to quickly search for related scenes within a video by entering specific keywords. Furthermore, by adding a means for generating prompt sentences and sending them to the server, the generated summaries and chapters can be easily understood with high accuracy. This system allows users to efficiently access the information they need without having to watch the entire video.

[0192] "Means for uploading videos" is a function that allows users to send any video file to the system and have it saved on the server.

[0193] "Means for the server to receive the video and store it in temporary storage" refers to a function in which the server receives the uploaded video file and temporarily stores it for subsequent analysis processing.

[0194] "Means for converting video audio data into text using a generative artificial intelligence engine" is a function that uses generative artificial intelligence technology to automatically transcribe audio within a video.

[0195] "Means for analyzing video data of a video, dividing the video into scenes, and creating summaries" refers to a function that analyzes video data, divides the content into multiple scenes, and extracts the important content of each scene as a summary.

[0196] "Means for creating chapter candidates based on the generated summary and scenes" is a function that divides a video into multiple sections (chapters) based on the generated summary and scene information, and assigns titles and summaries to each section.

[0197] The "means for transmitting chapter candidates to a terminal" is a function for transmitting chapter information generated by the server to a user's terminal.

[0198] The "means for displaying chapter candidates received by the terminal to the user" is a function for visually displaying chapter information received on the user's terminal.

[0199] "Means for a user to input a specific keyword into the search bar of a terminal" refers to a function that allows a user to input a keyword they want to search for into the search bar on the terminal.

[0200] The "means for the terminal to transmit a keyword to the server" is a function for transmitting a keyword input from the terminal to the server.

[0201] "Means for analyzing the text data of a video and identifying scenes containing keywords" is a function for analyzing the generated text data and identifying scenes in which the input keywords appear.

[0202] The "means for transmitting the time stamp of the scene to the terminal" is a function for transmitting the time information (time stamp) of the identified scene to the terminal.

[0203] "Means for the terminal to display the timestamp to the user and for the user to access a specified scene" is a function that displays timestamp information on the terminal and allows the user to directly access a specific scene based on that information.

[0204] "Means for generating video summaries using a generative AI model" refers to a function that uses a generative AI model to automatically summarize the content of a video.

[0205] "Means for generating prompt sentences and sending them to the server" is a function that automatically generates input sentences (prompt sentences) for the generative AI model and sends them to the server.

[0206] The present invention provides a function for efficiently viewing and searching video content. Specific embodiments of the present invention will be described below.

[0207] System Overview

[0208] This system combines a server, a user device, and a generative AI model engine. Users upload videos through their devices, and the server analyzes the videos and generates summaries. Users can also search for related scenes within videos using specific keywords.

[0209] Hardware and Software

[0210] Hardware: User devices such as smartphones, tablets, and PCs, as well as servers.

[0211] Software: ffmpeg, moviepy, speech_recognition, transformers library, and generative AI model engine.

[0212] Data processing and calculation

[0213] 1. Upload and save your video

[0214] The user selects a video file and uploads it from the device to the server, which then stores the received video file in temporary storage.

[0215] 2. Speech-to-text

[0216] The server extracts the audio data from the received video and converts it into text using speech recognition technology, using the speech_recognition library.

[0217] 3. Video analysis and summary generation

[0218] The server analyzes the video data and segments it into scenes, using the ffmpeg and moviepy libraries.

[0219] The server uses a generative AI model (e.g., the summarization model from the transformers library) to summarize the key points for each scene.

[0220] 4. Creating chapter suggestions

[0221] The server automatically creates chapter candidates based on the generated summary and scene information, and sends the chapter information to the user device in JSON format.

[0222] 5. Chapter display and search

[0223] The user terminal displays the received chapter information in list form.

[0224] When a user enters a specific keyword into the search bar of the device and clicks the search button, the device sends the keyword to the server.

[0225] The server analyzes the text data of the video, identifies scenes containing the keywords, and sends the timestamp information of those scenes to the terminal.

[0226] The terminal displays the timestamp information to the user, and the user can directly access the specified scene based on the information.

[0227] Specific examples

[0228] For example, suppose a user uploads a video on an educational video streaming service titled "Python Basics." Using this system, the user can:

[0229] Upload a video and its content will be automatically summarized using a generative AI model.

[0230] Each chapter (e.g., "Basic Python Syntax," "Data Structures," and "Functions") is automatically generated and displayed as a list.

[0231] When a user searches for "loop," relevant scenes are displayed with timestamps.

[0232] Users can click on the displayed timestamp to view the corresponding scene directly.

[0233] Prompt Sentence Examples

[0234] The prompt sentence can be summarized as follows: "Python is an easy-to-use and powerful programming language. Many beginners are amazed by its simplicity."

[0235] "Write a summary of this text: 'The programming language Python is easy to use and is the language of choice for many beginners.'"

[0236] This system will make viewing and searching video content more efficient and improve usability.

[0237] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0238] Step 1:

[0239] The user selects and uploads a video.

[0240] Input: The user selects a video file from local storage.

[0241] Output: The video file is sent to the user's device and uploaded to the server.

[0242] Specific operation: The user clicks the upload button on the device and selects a video file from the file selection dialog. The selected video file is sent from the device to the server.

[0243] Step 2:

[0244] The server receives the video and stores it in temporary storage.

[0245] Input: The video file sent from the user device.

[0246] Output: The video file is saved to the server's temporary storage.

[0247] Specific operation: The server stores the received video file in temporary storage and prepares it for subsequent analysis processing.

[0248] Step 3:

[0249] The server converts the video's audio data into text.

[0250] Input: Video file saved in temporary storage.

[0251] Output: Text data generated from the audio data.

[0252] What it does: The server uses the ffmpeg library to extract audio data from the video, then uses the speech_recognition library to convert the audio data to text.

[0253] Step 4:

[0254] The server analyzes the video data, divides the scenes, and creates a summary.

[0255] Input: Video files stored in temporary storage and generated text data.

[0256] Output: Summary text and chapter information for each scene.

[0257] How it works: The server uses the moviepy library to analyze the video data and segment it into scenes, then uses a generative AI model (e.g., the summarization model from the transformers library) to automatically summarize the content of each scene.

[0258] Step 5:

[0259] The server creates chapter candidates based on the generated summary and scenes.

[0260] Input: Summary text and scene information.

[0261] Output: Chapter candidate information (title, summary, timestamp, etc.).

[0262] Specific operation: The server creates chapter candidates in JSON format based on the generated summary and scene information.

[0263] Step 6:

[0264] The server transmits the chapter candidates to the terminal.

[0265] Input: Chapter candidate information.

[0266] Output: Chapter suggestion information sent to the device.

[0267] Specific operation: The server transmits the generated chapter candidate information to the user terminal.

[0268] Step 7:

[0269] The terminal displays the received chapter candidates to the user.

[0270] Input: Chapter candidate information received from the server.

[0271] Output: The chapter list displayed in the user interface.

[0272] Specific operation: The terminal displays the chapter candidate information in a list format on the user interface, allowing access to each chapter.

[0273] Step 8:

[0274] A user types a specific keyword into the search bar.

[0275] Input: The keyword entered by the user.

[0276] Output: Search keywords displayed on the device.

[0277] Specific operation: The user enters a specific keyword into the search bar of the device and clicks the search button.

[0278] Step 9:

[0279] The device sends the keyword to the server.

[0280] Input: The keyword entered in the search bar.

[0281] Output: The search terms sent to the server.

[0282] Specific operation: The terminal sends the keyword entered by the user to the server.

[0283] Step 10:

[0284] The server analyzes the text data of the video and identifies scenes that contain the keywords.

[0285] Input: Search keywords sent to the server and generated text data.

[0286] Output: Timestamp information of identified scenes.

[0287] Specific operation: The server analyzes the generated text data, identifies scenes that contain the search keywords, and generates timestamp information for the identified scenes.

[0288] Step 11:

[0289] The server transmits the timestamp of the identified scene to the terminal.

[0290] Input: The timestamp information of the identified scene.

[0291] Output: Timestamp information sent to the terminal.

[0292] Specific operation: The server transmits the timestamp information of the identified scene to the user terminal.

[0293] Step 12:

[0294] The terminal displays the timestamp to the user, and the user accesses the specified scene.

[0295] Input: The timestamp information received from the server.

[0296] Output: Timestamp information and associated scenes displayed in the user interface.

[0297] Specific operation: The device displays the timestamp information in the user interface, and the user can click on the displayed timestamp to directly access a specific scene.

[0298] The above are the specific processing steps for carrying out the invention.

[0299] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0300] The present invention is a system that automatically analyzes the content of videos and can pinpoint the information a user needs, and combines it with an emotion engine that recognizes the user's emotions to provide more advanced and personalized information. Specific embodiments for carrying out the present invention are described below.

[0301] System Overview

[0302] This system consists of a user, a server, a terminal, and an emotion engine. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the videos by entering specific keywords. Furthermore, the emotion engine recognizes the user's emotions and dynamically adjusts the video playback content and chapter display based on that information.

[0303] Uploading and saving videos

[0304] First, the user selects a video file and uploads it to the system. Once the video file is uploaded, the device sends it to the server. The server then stores the received video file in temporary storage and prepares it for analysis.

[0305] Video analysis and summary generation

[0306] The server then launches a generative AI engine to convert the video's audio data into text. Through this speech recognition process, the video's content is extracted as text data. The server then analyzes the video data and segments each scene. The generative AI engine then summarizes the content of each scene and extracts the main points.

[0307] Creating and viewing chapter suggestions

[0308] The server creates chapter candidates based on the generated summaries and scenes. This automatically divides the video into multiple sections, each with a title and summary. The server sends the chapter candidates in JSON format to the device, which then displays them visually to the user.

[0309] Reverse search function from words

[0310] The user enters a specific keyword in the search bar and clicks the search button. The device sends the keyword to the server. The server analyzes the text data of the video and identifies the scene in which the keyword appears. Once the timestamp of the relevant scene is identified, the server sends that information to the device. The device displays the search results, including the timestamp, to the user, allowing the user to directly access the specified scene.

[0311] Emotion recognition function

[0312] The system further enhances the user experience by incorporating an emotion engine. While the user is watching a video, a camera built into the device captures real-time video of the user. This video data is sent to a server, which then uses the emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial expressions and movements to identify their emotional state (e.g., joy, sadness, excitement, etc.).

[0313] The server can dynamically adjust the video playback and chapters displayed based on the emotional data it acquires. For example, if it determines that the user is interested, it can provide more information or supplemental videos on related topics. On the other hand, if it determines that the user is bored, it can present the option to skip to important scenes.

[0314] Specific examples

[0315] For example, if a user uploads a video about "latest tech trends," the process goes like this:

[0316] 1. Upload and analyze

[0317] A user uploads a video about "latest tech trends."

[0318] The device sends the video file to the server.

[0319] The server receives the video, stores it in temporary storage, and begins analysis.

[0320] 2. Summarizing and Chapter Creation

[0321] The server uses a generative artificial intelligence engine to generate a summary of the video.

[0322] The server creates chapter candidates based on each technical topic.

[0323] 3. Display results

[0324] The server sends the summary and chapter information to the device in JSON format.

[0325] The terminal displays this to the user in a list format.

[0326] 4. Reverse Search Feature

[0327] A user types "blockchain" into the search bar and clicks the search button.

[0328] The device sends the keyword to the server.

[0329] The server analyzes the video transcript, identifies scenes in which "blockchain" appears, and sends that information to the device.

[0330] The terminal displays the search results to the user in a list format, allowing the user to directly access related scenes.

[0331] 5. Emotion recognition

[0332] While the user is watching the video, the device's camera captures real-time video.

[0333] The server analyzes the video data and uses an emotion engine to recognize the user's emotions.

[0334] Based on the emotional data, the video playback content and chapter display are dynamically adjusted.

[0335] The system not only allows users to instantly access the information they need without having to play the entire video, but also provides a personalized viewing experience through emotion recognition.

[0336] The processing flow will be explained below.

[0337] Step 1:

[0338] The user selects a video file and uploads it to the system. The user uses a file selection dialog to select a video file from their computer and clicks the "Upload" button.

[0339] Step 2:

[0340] The device sends the selected video file to the server. The device obtains the binary data of the video file and sends it to the server using an HTTP POST request.

[0341] Step 3:

[0342] The server receives the video file and stores it in temporary storage. The server receives the video file, checks the integrity of the data, and then stores it in a temporary storage folder.

[0343] Step 4:

[0344] The server starts the generative artificial intelligence engine. The server calls the generative artificial intelligence engine to start the process of converting audio data into text for the prepared video file.

[0345] Step 5:

[0346] The server converts the audio data into text, and a generative AI engine analyzes the audio in the video and generates recognized text data.

[0347] Step 6:

[0348] The server analyzes the video data, extracts each frame of the video, and segments it into scenes using a video recognition algorithm.

[0349] Step 7:

[0350] The server generates a summary for each scene using the generated text and video data. The content is reviewed to ensure that key points and themes are included.

[0351] Step 8:

[0352] The server creates chapter candidates. Based on the generated summaries and scene information, the server assigns titles and summaries to each segment and creates chapter candidates.

[0353] Step 9:

[0354] The server sends the chapter candidates to the device. The server encodes the completed chapter candidates in JSON format and sends them to the device as an HTTP response.

[0355] Step 10:

[0356] The device parses the received chapter suggestions and displays them on the user interface. The device parses the JSON data and displays them to the user in list or timeline format.

[0357] Step 11:

[0358] A user enters a specific keyword into the search bar and clicks the search button.A user enters a keyword into the input field of the search bar and presses the button to execute the search command.

[0359] Step 12:

[0360] The device sends the keyword to the server. The device creates an HTTP GET request including the user's keyword and sends it to the server.

[0361] Step 13:

[0362] The server analyzes the text data of the video and identifies the scenes in which the keywords appear. Using a full-text search algorithm, the server extracts text fragments containing the keywords and obtains the timestamps of the corresponding scenes.

[0363] Step 14:

[0364] The server sends the timestamp to the device. The server encodes the timestamp and scene information identified as search results in JSON format and sends it to the device as an HTTP response.

[0365] Step 15:

[0366] The device displays the search results to the user. The device displays the received timestamps and scene information in a list format, and provides a "Play" button for each entry.

[0367] Step 16:

[0368] The user clicks the play button to access the specified scene. When the user clicks the play button in the displayed list, the video player starts playing from the corresponding scene.

[0369] Step 17:

[0370] The device acquires real-time video data of the user. The device's camera captures the user's face and movements and sends the data to the server.

[0371] Step 18:

[0372] The server starts the emotion engine. The server invokes the emotion engine to analyze the received video data.

[0373] Step 19:

[0374] The server analyzes the user's emotions, and the emotion engine analyzes the user's facial expressions and movements to identify their emotional state, such as joy, sadness, or excitement.

[0375] Step 20:

[0376] The server uses the emotion analysis results to dynamically adjust video playback and chapter display based on the emotion data.

[0377] Step 21:

[0378] The server sends adjustments based on emotion data to the device, and the server sends dynamically adjusted information to the device, which then visually displays it.

[0379] Step 22:

[0380] The device displays a display that reflects the emotional data to the user. The device displays a list that reflects the emotional data and adjusts the video player to the user, allowing the user to have an optimal viewing experience.

[0381] Example 2

[0382] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0383] While conventional video analysis systems have the ability to summarize videos and search for specific scenes, they lack the ability to dynamically change content based on the user's emotional state. Furthermore, while they enable users to quickly obtain specific information, they have the problem of being unable to respond to the user's interests and emotions while watching, because the video playback content is static. Therefore, there is a need for a system that can individually optimize the user's viewing experience and efficiently provide the information they need.

[0384] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0385] In this invention, the server includes means for converting audio data of a video into text using a generative artificial intelligence engine, means for analyzing video data of the video and dividing it into scenes to create a summary, means for creating chapter candidates based on the created summary and scenes, and means for recognizing the user's emotional state using an emotion engine and dynamically adjusting the video playback content and chapter display. This enables a personalized viewing experience according to the user's emotions and makes it possible to efficiently provide necessary information.

[0386] "Video" means a digital file containing audio and video information that a user uploads to the system.

[0387] The "server" is a central processing unit that receives video, temporarily stores it, and analyzes it using a generative artificial intelligence engine and an emotion engine.

[0388] A "generative artificial intelligence engine" is an AI technology that converts audio data from videos into text and analyzes video data to create scene summaries.

[0389] "Voice recognition technology" is a technology for converting audio data from video into text data.

[0390] "Video data" refers to digital data that constitutes the visual information of a moving image.

[0391] "Scene segmentation" is a method of dividing a continuous video sequence into multiple stages and analyzing each section independently.

[0392] A "summary" is text information that briefly summarizes the main content of each scene in a video.

[0393] "Chapter candidates" refer to the individual parts of a video divided into multiple sections based on summarized scene information.

[0394] "Emotion recognition" is the process of analyzing a user's facial expressions, movements, and other facial expressions to identify their emotional state.

[0395] "Terminal" means a device through which a user accesses the system to upload, search, and play videos.

[0396] A "keyword" is a character string that a user enters to search for specific information.

[0397] A "timestamp" is information that indicates the start time of a particular scene in a video.

[0398] "Dynamic adjustment" refers to changing the video playback content and chapter display in real time based on the user's emotion recognition results.

[0399] The present invention is an advanced information provision system that combines video analysis and emotion recognition, and specific embodiments are described in detail below. This system allows users to upload videos and automatically analyze the content of those videos to efficiently obtain the information users need. The present invention includes a server, a terminal, and an emotion engine. Ultimately, it realizes dynamic information provision according to the user's emotions.

[0400] Uploading and saving videos

[0401] A user uses a device to select and upload a video file. The device then sends the video file to a server. The server receives the video file and stores it in temporary storage, preparing the video for analysis. For example, if a user uploads a video of a seminar on technology trends, the device sends the file to the server, which stores it.

[0402] Video analysis and summary generation

[0403] The server launches a generative AI engine to begin analyzing the video. First, the audio data is converted into text using speech recognition technology (e.g., Google Speech-to-Text API). Next, the video data is analyzed and divided into scenes. For each divided scene, a generative AI model (e.g., GPT-4 (registered trademark)) is used to generate a summary. This allows the main points of the video to be concisely extracted.

[0404] Creating chapter suggestions

[0405] The server creates chapter candidates based on the generated summary and split scenes. Each chapter is divided into sections and given a title and summary. The server sends this information to the device in JSON format, and the device displays it to the user. For example, if a video on "latest technology trends" is uploaded, the server generates chapters based on technology topics and displays them to the user.

[0406] Keyword search function

[0407] The user enters a specific keyword into the search bar on the device and clicks the search button. The device then sends the keyword to the server. The server analyzes the text data of the video and identifies the scene in which the keyword appears. The timestamp information of the relevant scene is sent to the device and displayed to the user. The user can then directly access the relevant scene from the search results.

[0408] Emotion recognition function

[0409] The system incorporates an emotion engine that analyzes the user's emotions in real time. While the user is watching a video, a camera built into the device captures real-time footage of the user and sends it to the server. The server uses the emotion engine to analyze the user's emotional state and dynamically adjusts the video playback content and chapter display. For example, if the system determines that the user is excited, it provides related information or supplementary videos. If the system determines that the user is bored, it presents an option to skip to important scenes.

[0410] As a concrete example, when a user uploads a video about "latest technology trends," the process proceeds as follows:

[0411] 1. The user uploads a video and the device sends it to the server.

[0412] 2. The server stores the video and begins analysis, converting the audio data into text and analyzing the video data to generate a summary.

[0413] 3. The server creates chapter candidates and sends them to the device, which displays them to the user.

[0414] 4. The user searches by keyword, and the server sends the timestamp of the corresponding scene to the device. The device displays the search results, and the user can access the corresponding scene.

[0415] 5. While the user is watching the video, the device camera captures their emotions. The server analyzes this using an emotion engine and dynamically adjusts the playback content.

[0416] In this way, the system allows users to quickly access the information they need without having to play the entire video, and also provides a personalized viewing experience through emotion recognition.

[0417] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0418] Step 1:

[0419] A user selects a video file on a terminal and uploads it to the system. Specifically, the user selects a video file from the user interface and presses the upload button. The input at this time is the selected video file, and the output is the terminal sending the video file to the server.

[0420] Step 2:

[0421] The device sends the video file selected by the user to the server. It receives the video file uploaded by the user as input and sends it to the server as output. Specifically, the device generates an HTTP request, attaches the video file, and sends it to the server.

[0422] Step 3:

[0423] The server receives the video file and stores it in temporary storage. It receives the video file sent from the terminal as input and stores it in temporary storage as output. Specifically, the server uses a file management system to store the video file in a specified directory.

[0424] Step 4:

[0425] The server starts a generative artificial intelligence engine and converts the video's audio data into text using speech recognition technology. It receives the saved video file as input and obtains the generated text data as output. Specifically, the server calls the speech recognition API, sends the audio data as input, and receives the text data.

[0426] Step 5:

[0427] The server analyzes the video data, segments the scenes, and creates summaries. It receives the video file and generated text data as input, and generates segmented scenes and their summaries as output. Specifically, the server applies a video analysis algorithm to identify the video frames for each scene and creates a summary using a generative AI model.

[0428] Step 6:

[0429] The server creates chapter candidates based on the generated summaries and scenes. It receives scenes and their summaries as input and generates chapter information as output. Specifically, the server aggregates the summaries for each scene and generates chapter titles and summaries in JSON format.

[0430] Step 7:

[0431] The server sends the generated chapter information to the terminal. It receives the generated chapter information as input and sends it to the terminal as output. Specifically, the server generates an HTTP response, attaches the chapter information, and sends it to the terminal.

[0432] Step 8:

[0433] The terminal displays the received chapter candidates to the user. It receives the chapter information received from the server as input and displays it on the user interface as output. Specifically, the terminal analyzes the chapter information and displays it in list format.

[0434] Step 9:

[0435] The user enters a specific keyword into the search bar of the device and clicks the search button. The keyword is received as input and sent to the server as output. The specific operation is that the user enters a keyword into the search bar and clicks the search button.

[0436] Step 10:

[0437] The terminal sends the keyword to the server. It receives the keyword entered by the user as input and sends it to the server as output. Specifically, the terminal generates an HTTP request, attaches the keyword, and sends it to the server.

[0438] Step 11:

[0439] The server analyzes the text data of the video to identify scenes containing the keywords. Using the keywords received as input and the text data of the video, it generates a timestamp for the relevant scene as output. Specifically, the server searches the text data for keywords and identifies the start time of the relevant scene.

[0440] Step 12:

[0441] The server sends the scene timestamp to the terminal. It receives the timestamp of the specified scene as input and sends it to the terminal as output. As a specific operation, the server generates an HTTP response, attaches the timestamp information, and sends it to the terminal.

[0442] Step 13:

[0443] The terminal displays the timestamp to the user, and the user accesses the specified scene. The terminal receives the timestamp information from the server as input, displays it on the user interface as output, and accesses the specified scene. As a specific operation, the terminal uses a playback player to play the video from the specified scene.

[0444] Step 14:

[0445] The camera built into the device captures real-time video of the user and sends it to the server. The device acquires real-time video of the user as input and sends it to the server as output. Specifically, the device activates the camera module, captures video data, and sends it to the server.

[0446] Step 15:

[0447] The server uses an emotion engine to recognize the user's emotional state and dynamically adjust the video playback content and chapter display. It receives the transmitted video data as input and generates emotion recognition results and dynamic adjustment results as output. Specifically, the emotion engine analyzes the video data, identifies the user's emotional state, and changes the video playback content accordingly.

[0448] Step 16:

[0449] Based on the emotion recognition, the server dynamically adjusts the video playback content and chapter display and sends them to the terminal. The emotion recognition results are received as input, and the adjustments are sent to the terminal as output. Specifically, the server generates adjustment instructions and sends them to the terminal, which then changes the video playback content as appropriate.

[0450] (Application example 2)

[0451] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0452] The challenge is to provide users with a more effective and personalized advertising experience by appropriately recognizing users' interests and emotions while watching advertising videos and dynamically adjusting the advertising content based on that information. Conventional advertising systems provide uniform advertising content without considering the user's emotional state, making it difficult to maintain user interest. Furthermore, there is no mechanism for users to quickly search for related information using specific keywords, resulting in a poor user experience.

[0453] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for automatically analyzing the content of the advertising video being viewed by the user, means for capturing real-time video of the user using the camera of the terminal, means for the server to analyze the video data and recognize the user's emotions using an emotion engine, and means for the server to dynamically adjust the advertising content based on the recognized emotion information. This enables a personalized advertising experience that matches the user's emotional state.

[0454] "User" means a person or organization that uses the system to upload videos and receive the analyzed results.

[0455] "Server" is a central computer system that processes and analyzes video data received from users, and stores and transmits the results.

[0456] "Temporary storage" is a storage device that temporarily stores video when the server receives it.

[0457] A "generative artificial intelligence engine" is an analytical engine equipped with AI technology that analyzes the content of videos, converts audio data into text, and generates summaries.

[0458] "Audio data" is digital data that includes audio information emitted within a video.

[0459] "Text data" refers to data obtained by converting voice data into text format.

[0460] "Video data" is digital data that contains visual information within a moving image.

[0461] A "scene" refers to a specific time range within a video, and is a video portion with independent content.

[0462] "Chapter candidates" are sections divided into scenes that are generated as a result of analyzing a video, and each section is given a title and a summary.

[0463] "Terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[0464] A "search bar" is a UI component that allows a user to enter specific keywords.

[0465] "Keywords" refer to specific words or phrases that users use to search for specific scenes within a video.

[0466] A "timestamp" is digital information that indicates the start time of a particular scene.

[0467] The "Emotion Engine" is an analytical engine equipped with AI technology that analyzes real-time video footage of users and identifies their emotional state.

[0468] An "advertising video" is a video created to promote a particular product or service.

[0469] "Real-time video" refers to video data of a user captured in real time.

[0470] "Emotion information" is data that indicates the user's emotional state analyzed by the emotion engine.

[0471] "Dynamic adjustment" means changing the content and presentation in real time depending on the situation.

[0472] Specific embodiments for carrying out the present invention are described below.

[0473] System configuration

[0474] The system consists of a user, a server, a terminal, and an emotion engine. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the video by entering specific keywords. Furthermore, the emotion engine recognizes the user's emotions and dynamically adjusts the playback content and chapter display of the advertising video based on that information.

[0475] Uploading and saving videos

[0476] The user uploads the advertising video to the device. The device then sends the video file to the server, which stores it in temporary storage. In this process, the server and device hardware are typically general-purpose computers or smartphones.

[0477] Video analysis and summary generation

[0478] The server then launches a generative AI engine to convert the audio data from the ad video into text. Through this speech recognition process, the video content is extracted as text data. The server then analyzes the video data and segments each scene. The generative AI engine generates a summary of each scene and extracts the main points. Transformers from Hugging Face are used for speech recognition.

[0479] Creating and viewing chapter suggestions

[0480] The server creates chapter candidates based on the generated summaries and scenes. This automatically divides the video into multiple sections, each with a title and summary. The chapter information is sent to the device in JSON format, and the device displays it visually to the user.

[0481] Keyword search function

[0482] The user enters a specific keyword into the device's search bar and clicks the search button. The device then sends the keyword to the server. The server analyzes the text data in the video and identifies scenes that contain the keyword. Once the timestamp of the relevant scene is identified, that information is sent to the device. The device then displays the search results, including the timestamp, to the user, allowing the user to directly access the specified scene.

[0483] Emotion recognition and dynamic ad adjustment

[0484] The system further enhances the user experience by incorporating an emotion engine. While the user is watching the advertising video, a camera built into the device captures real-time video of the user. This video data is sent to a server, which then uses an emotion engine to recognize the user's emotions. The emotion engine uses services such as Amazon Rekognition to analyze the user's facial expressions and movements to identify their emotional state.

[0485] The server dynamically adjusts the content of the ad video and the chapters displayed based on the acquired emotional data. For example, if the server determines that the user is interested, it may provide more information or supplemental videos on related topics. On the other hand, if the server determines that the user is bored, it may present the option to skip to important scenes.

[0486] Specific examples

[0487] As an example, if a user uploads an advertising video about "latest smartphone technology," the process would proceed as follows:

[0488] 1. Upload and analyze

[0489] A user uploads an advertising video for the "latest smartphone technology."

[0490] The device sends the video file to the server.

[0491] The server receives the video, stores it in temporary storage, and begins analysis.

[0492] 2. Summarizing and Chapter Creation

[0493] The server uses a generative artificial intelligence engine to generate a summary of the video.

[0494] The server creates chapter candidates based on each technical topic.

[0495] 3. Display results

[0496] The server sends the summary and chapter information to the device in JSON format.

[0497] The terminal displays this to the user in a list format.

[0498] 4. Reverse Search Feature

[0499] A user types "camera features" into the search bar and clicks the search button.

[0500] The device sends the keyword to the server.

[0501] The server analyzes the video transcription data, identifies scenes in which the "camera function" appears, and sends that information to the device.

[0502] The terminal displays the search results to the user in a list format, allowing the user to directly access related scenes.

[0503] 5. Emotion recognition

[0504] While the user is watching the advertising video, the camera on the terminal captures real-time video.

[0505] The server analyzes the video data and uses an emotion engine to recognize the user's emotions.

[0506] Based on the emotional data, the playback content and chapter display of the advertising video are dynamically adjusted, improving the user experience and maximizing the effectiveness of the advertisement.

[0507] Example prompts

[0508] The prompt sentences below are examples used in the emotion-sensitive ad viewer.

[0509] "Identify exciting moments in advertising videos and highlight relevant technology demos."

[0510] "If the user expresses interest, provide additional information related to the topic."

[0511] This system not only allows users to instantly access the information they need without having to watch the entire advertisement, but also provides a personalized viewing experience through emotion recognition.

[0512] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0513] Step 1:

[0514] A user uploads an advertising video.

[0515] The user selects an advertising video file through the terminal interface and uploads it to the system, at which point the terminal sends the selected video file to the server, where the input is the video file provided by the user and the output is the video file received by the server.

[0516] Step 2:

[0517] The server stores the video in temporary storage.

[0518] The server temporarily stores the received video file in temporary storage. The input is the video file sent from the device, and the output is the video file saved in the temporary storage.

[0519] Step 3:

[0520] The server launches a generative artificial intelligence engine to convert the video's audio data into text.

[0521] The server uses a generative artificial intelligence engine to convert the audio data in the video into text data. The input is the video file loaded from temporary storage, and the output is text data. This process uses voice recognition technology.

[0522] Step 4:

[0523] The server analyzes the video data, divides the scenes, and creates summaries.

[0524] The server analyzes the video data and divides it into multiple scenes. It then uses a generative AI engine to create a summary of each scene and extract key points. The input is text data and video data, and the output is a summary of each scene and chapter suggestions.

[0525] Step 5:

[0526] The server creates chapter candidates based on the generated summary and scenes.

[0527] The server generates chapter candidates, each containing a title and summary, based on the generated summaries and the divided scenes. The input is scene summary data, and the output is chapter candidate data.

[0528] Step 6:

[0529] The server transmits the chapter candidates to the terminal.

[0530] The server sends the created chapter candidates to the terminal. The input is the chapter candidate data, and the output is the chapter candidate data received by the terminal.

[0531] Step 7:

[0532] The terminal displays the received chapter candidates to the user.

[0533] The terminal visually displays the chapter candidates received from the server to the user in the form of a text list. The input is the received chapter candidate data, and the output is the chapter list displayed to the user.

[0534] Step 8:

[0535] The user enters a specific keyword into the device's search bar.

[0536] The user inputs a specific keyword through the search bar and clicks the search button. The input is the keyword input by the user, and the output is the keyword data sent from the terminal to the server.

[0537] Step 9:

[0538] The device sends the keyword to the server.

[0539] The terminal sends the keyword received from the user to the server. The input is the keyword entered by the user, and the output is the keyword data sent to the server.

[0540] Step 10:

[0541] The server analyzes the text data of the video and identifies scenes that contain the keywords.

[0542] The server analyzes the text data of the video and identifies scenes that contain the keywords entered by the user. The input is keyword data and text data, and the output is scene information that contains the keywords.

[0543] Step 11:

[0544] The server sends the scene timestamp to the device.

[0545] The server sends the timestamp of the identified scene to the terminal. The input is the scene information, and the output is the timestamp data sent to the terminal.

[0546] Step 12:

[0547] The terminal displays the timestamp to the user, and the user accesses the specified scene.

[0548] The terminal displays the received timestamp to the user, allowing the user to directly access the specified scene. The input is the timestamp data received from the server, and the output is the timestamp information displayed to the user.

[0549] Step 13:

[0550] A real-time image of the user is acquired using the device's camera.

[0551] While the user is watching the advertisement video, the camera of the terminal captures the user's real-time video. The input is the video data from the camera, and the output is the video data sent to the server.

[0552] Step 14:

[0553] The server analyzes the video data and uses an emotion engine to recognize the user's emotions.

[0554] The server analyzes the acquired video data using an emotion engine to identify the user's emotional state. The input is the video data received from the device, and the output is the emotion recognition result.

[0555] Step 15:

[0556] The server dynamically adjusts the advertising content based on the recognized emotion information.

[0557] The server dynamically adjusts the playback content of the advertising video and the chapters to be displayed based on the emotional information analyzed by the emotion engine. The input is the emotion recognition result, and the output is the adjusted advertising content.

[0558] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0559] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search<url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0560] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0561] [Second embodiment]

[0562] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0563] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0564] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0565] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0566] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0567] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0568] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0569] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0570] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0571] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0572] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0573] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0574] The present invention is a system that automatically analyzes the content of a video and can pinpoint the information that a user needs. Specific embodiments for carrying out the present invention will be described below.

[0575] System Overview

[0576] This system consists of users, a server, and terminals. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the videos by entering specific keywords.

[0577] Uploading and saving videos

[0578] First, the user selects a video file and uploads it to the system. Once the video file is uploaded, the device sends it to the server. The server then stores the received video file in temporary storage and prepares it for analysis.

[0579] Video analysis and summary generation

[0580] The server then launches a generative AI engine to convert the video's audio data into text. Through this speech recognition process, the video's content is extracted as text data. The server then analyzes the video data and segments each scene. The generative AI engine then summarizes the content of each scene and extracts the main points.

[0581] Creating and viewing chapter suggestions

[0582] The server creates chapter candidates based on the generated summaries and scenes. This automatically divides the video into multiple sections, each with a title and summary. The server sends the chapter candidates in JSON format to the device, which then displays them visually to the user.

[0583] Reverse search function from words

[0584] The user enters a specific keyword in the search bar and clicks the search button. The device sends the keyword to the server. The server analyzes the text data of the video and identifies the scene in which the keyword appears. Once the timestamp of the relevant scene is identified, the server sends that information to the device. The device displays the search results, including the timestamp, to the user, allowing the user to directly access the specified scene.

[0585] Specific examples

[0586] For example, consider a user who uploads a video about "Programming Basics," which includes examples of multiple programming languages ​​and their usage.

[0587] 1. Upload and analyze

[0588] A user uploads a video on "Programming Basics."

[0589] The terminal transmits the selected video file to the server.

[0590] The server receives the video, stores it in temporary storage, and begins the analysis process.

[0591] 2. Summarizing and Chapter Creation

[0592] The server uses a generative artificial intelligence engine to summarize the content of the video.

[0593] The server creates chapters based on each language segment (e.g. "Python Basics", "JavaScript Basics", etc.).

[0594] 3. Display results

[0595] The server sends the summary and chapter information to the device in JSON format.

[0596] The terminal displays this to the user in a list format.

[0597] 4. Reverse Search Feature

[0598] A user types "Python" into the search bar and clicks the search button.

[0599] The device sends the keyword to the server.

[0600] The server analyzes the video transcription data, identifies scenes in which "Python" appears, and sends relevant information to the device.

[0601] The device displays the search results in a list format, allowing the user to directly access related scenes.

[0602] This system allows users to quickly access the information they need without having to play the entire video, making it possible to use videos efficiently.

[0603] The processing flow will be explained below.

[0604] Step 1:

[0605] The user selects a video file and uploads it to the system. The user selects a video file from their computer using the file selection dialog and clicks the "Upload" button.

[0606] Step 2:

[0607] The device sends the selected video file to the server. The device obtains the binary data of the video file and sends it to the server using an HTTP POST request.

[0608] Step 3:

[0609] The server receives the video file and stores it in temporary storage. The server receives the video file, checks the integrity of the data, and then stores it in a temporary storage folder.

[0610] Step 4:

[0611] The server starts the generative artificial intelligence engine. The server calls the generative artificial intelligence engine to start the process of converting audio data into text for the prepared video file.

[0612] Step 5:

[0613] The server converts the audio data into text, and a generative AI engine analyzes the audio in the video and generates recognized text data.

[0614] Step 6:

[0615] The server analyzes the video data, extracts each frame of the video, and segments it into scenes using a video recognition algorithm.

[0616] Step 7:

[0617] The server generates a summary for each scene using the generated text and video data. The content is reviewed to ensure that key points and themes are included.

[0618] Step 8:

[0619] The server creates chapter candidates. Based on the generated summary and scene information, the server assigns a title and summary to each segment and creates chapter candidates.

[0620] Step 9:

[0621] The server sends the chapter candidates to the device. The server encodes the completed chapter candidates in JSON format and sends them to the device as an HTTP response.

[0622] Step 10:

[0623] The device parses the received chapter suggestions and displays them on the user interface. The device parses the JSON data and displays them to the user in list or timeline format.

[0624] Step 11:

[0625] A user enters a specific keyword into the search bar and clicks the search button.A user enters a keyword into the input field of the search bar and presses the button with the intention of a search command.

[0626] Step 12:

[0627] The device sends the keyword to the server. The device creates an HTTP GET request including the user's keyword and sends it to the server.

[0628] Step 13:

[0629] The server analyzes the text data of the video and identifies the scenes in which the keywords appear. Using a full-text search algorithm, the server extracts text fragments containing the keywords and obtains the timestamps of the corresponding scenes.

[0630] Step 14:

[0631] The server sends the timestamp to the device, and the server returns the identified timestamp and scene information in JSON format to the device as search results.

[0632] Step 15:

[0633] The device displays the search results to the user. The device displays the received timestamps and scene information in a list format, and provides a "Play" button for each entry.

[0634] Step 16:

[0635] The user clicks the play button to access the specified scene. When the user clicks the play button in the displayed list, the video player starts playing from the corresponding scene.

[0636] Example 1

[0637] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0638] In today's world, the proliferation of video content has made it difficult for users to quickly and efficiently obtain the information they need. Long video playback times also create inefficiencies in information retrieval. Furthermore, the lack of a way for users to quickly search for specific information of interest within a video limits the viewing experience.

[0639] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0640] In this invention, the server includes: means for a user to upload a video; means for a terminal to transmit a video file to the server; means for the server to store the video file in temporary storage; means for the server to convert audio data of the video into text using speech recognition technology; means for the server to segment video data of the video by scene using video analysis technology; means for the server to generate a summary from the text data and scene information using a generative artificial intelligence engine; means for the server to create chapter candidates based on the generated summary and scene information; means for the server to transmit the chapter candidates to the terminal in JSON format; means for the terminal to visually display the received chapter candidates to the user; means for the user to input a specific keyword into a search bar of the terminal; means for the terminal to transmit the keyword to the server; means for the server to analyze the text data and identify scenes containing the keyword; means for the server to transmit scene timestamp information to the terminal; and means for the terminal to display the timestamp information to the user and allow the user to access the specified scene. This allows the user to quickly access necessary information without playing the entire video.

[0641] "User" means an individual or entity that uses the system to upload videos and search for specific information.

[0642] A "terminal" is an electronic device, such as a computer or smartphone, that a user uses to upload videos or view search results.

[0643] A "server" is a computer system that stores videos sent by users and performs analysis processing.

[0644] "Temporary storage" refers to data storage used by the server to temporarily store video files.

[0645] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[0646] "Video analysis technology" is a technology that analyzes video data and identifies specific scenes and frames.

[0647] A "generative AI engine" is a type of artificial intelligence used to generate new information or summaries based on input data.

[0648] "Text data" is character-based data that represents the content of a video extracted using voice recognition technology.

[0649] A "scene" is a segment that separates parts of a video where the content changes.

[0650] "Chapter candidates" are multiple sections generated based on the video content, and each section is given a title and a summary.

[0651] The "JSON format" is one of the standard formats for structuring and describing data in text format.

[0652] The "search bar" is an input area where users can input specific keywords to search for related information within a video.

[0653] A "timestamp" is information that indicates a specific time point within a video.

[0654] The present invention is a system that automatically analyzes the content of a video and can pinpoint the information that a user needs. Specific embodiments for carrying out the present invention will be described below.

[0655] System Overview

[0656] This system consists of users, a server, and terminals. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the videos by entering specific keywords.

[0657] Uploading and saving videos

[0658] First, the user selects a video file and uploads it to the system. To do this, the user uses a device such as a PC or smartphone. The device then sends the selected video file to the server. The server then stores the received video file in temporary storage (for example, Amazon S3 or Google Cloud Storage) and prepares it for analysis.

[0659] Video analysis and summary generation

[0660] The server then analyzes the video using voice recognition and video analysis technology. The specific process is as follows:

[0661] 1. Audio to text conversion:

[0662] The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to convert the audio data in the video into text data. Through this process, the content of the video is extracted as a string of characters.

[0663] 2. Scene segmentation of video data:

[0664] The server uses video analysis technology (e.g., OpenCV library) to analyze the video data and segment each scene. Specifically, it detects change points between frames in the video and recognizes different scenes.

[0665] 3. Generate a summary:

[0666] The server uses a generative artificial intelligence engine (e.g., OpenAI GPT) to generate a summary of each scene based on the extracted text data and scene information, concisely summarizing the key points and content.

[0667] Creating and viewing chapter suggestions

[0668] Based on the generated summary and scene information, the server creates chapter candidates that divide the video into multiple sections. Each section is then given a title and summary. The server converts the chapter candidates into JSON format and sends them to the device. The device parses the received JSON data and displays the chapter candidates in a list format on the user interface. The user can click on one of them to directly access a specific scene.

[0669] Keyword search function

[0670] The user enters a specific keyword into the system's search bar and clicks the search button. The device sends this keyword to the server. The server analyzes the text data of the video and identifies the scenes in which the keyword appears. It obtains the timestamp information of the relevant scenes and sends it to the device in JSON format. The device analyzes the received data and displays the search results in a list format on the user interface. The user can click on this to directly access the relevant scenes.

[0671] Specific examples

[0672] Here, we will use a video on "Programming Basics" as an example.

[0673] 1. Upload and analyze:

[0674] A user uploads a video on "Programming Basics."

[0675] The device sends the selected video to the server, which stores the video in temporary storage.

[0676] The server converts the audio data into text using the Google Cloud Speech-to-Text API, and splits the video data into scenes using OpenCV.

[0677] The server creates chapter candidates based on the summary generated using OpenAI GPT and sends them to the device.

[0678] 2. Results display:

[0679] The device receives the chapter information in JSON format and displays it in a list format on the user interface.

[0680] 3. Keyword Search:

[0681] A user types "Python" into the search bar and clicks the search button.

[0682] The device sends the keyword to the server, and the server analyzes the text data of the video to identify the relevant scene, obtains timestamp information, and sends it to the device.

[0683] The device displays the search results in a list format, allowing the user to directly access related scenes.

[0684] Prompt Sentence Examples

[0685] Here are some example prompts to enter into a generative AI engine:

[0686] "This video explains the basics of programming. Please summarize the basic usage of each programming language scene by scene."

[0687] "Please summarize the sections on Python, JavaScript, and Java basics for each scene."

[0688] This system allows users to quickly access the information they need without having to play the entire video.

[0689] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0690] Step 1:

[0691] The user selects a video file and uploads it to the system via their device. The user selects a video file through a web browser on a device such as a PC or smartphone. The selected video file is sent from the device to the server. The input is the video file, and the output is the transmission of the video file to the server.

[0692] Step 2:

[0693] The device sends the selected video file to the server. Specifically, it uploads the video file to the server using an HTTP POST request. The input is the video file selected by the user, and the output is the video file sent to the server.

[0694] Step 3:

[0695] The server stores the received video file in temporary storage. The server stores the received video file in storage such as Amazon S3 or Google Cloud Storage, and records the file path and file name in a database. The input is the video file received from the device, and the output is saving to storage and recording to the database.

[0696] Step 4:

[0697] The server uses speech recognition technology to convert the audio data in the video into text. Specifically, it uses the Google Cloud Speech-to-Text API to convert the audio data in the video file into text. The input is the audio data in the video file, and the output is the converted text data.

[0698] Step 5:

[0699] The server uses video analysis technology to divide the video data into scenes. Specifically, it uses the OpenCV library to analyze the video data frame by frame and detect different scenes. The input is the video data of the video file, and the output is the divided scene information.

[0700] Step 6:

[0701] The server uses a generative AI engine to generate summaries based on the text data and scene information. Specifically, it uses OpenAI GPT to summarize the key points of each scene. The input is the text data for each scene, and the output is a summary of each scene.

[0702] Step 7:

[0703] The server creates chapter candidates based on the generated summaries and scene information. The server combines the summaries and scene information, divides the video into multiple sections, and assigns titles and summaries to each section. The input is the summaries and scene information, and the output is chapter candidates.

[0704] Step 8:

[0705] The server sends chapter candidates to the device in JSON format. The server converts the generated chapter information into JSON format and sends it to the device as an HTTP response. The input is the chapter candidates, and the output is the JSON data sent to the device.

[0706] Step 9:

[0707] The device displays the received chapter candidates on the user interface. The device parses the received JSON data and displays the chapter candidates in list format on the user interface. The input is JSON data, and the output is a visual representation of the chapter candidates.

[0708] Step 10:

[0709] The user enters a specific keyword into the search bar of the device. The user enters the keyword of interest and clicks the search button. The input is the keyword, and the output is the transmission of the keyword.

[0710] Step 11:

[0711] The terminal sends a keyword to the server. The terminal sends the entered keyword to the server via an HTTP POST request. The input is the keyword, and the output is the keyword sent to the server.

[0712] Step 12:

[0713] The server analyzes the text data to identify scenes containing keywords. The server analyzes the text data of the video and obtains the scenes in which the keywords appear and their timestamp information. The input is the keywords and text data, and the output is the timestamp information of the identified scenes.

[0714] Step 13:

[0715] The server sends the scene timestamp information to the terminal. The server converts the acquired timestamp information into JSON format and sends it to the terminal as an HTTP response. The input is the timestamp information, and the output is the JSON data sent to the terminal.

[0716] Step 14:

[0717] The terminal displays the timestamp information on the user interface, and the user accesses the specified scene. The terminal parses the received JSON data and displays the search results in a list format. The user can click on it to directly access the related scene. The input is JSON data, and the output is a visually displayed search result.

[0718] This allows users to quickly access the information they need without having to play the entire video.

[0719] (Application example 1)

[0720] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0721] In conventional video content distribution systems, users must watch the entire video, making it difficult to quickly access the information they need. Furthermore, the functionality for searching for specific topics or scenes was insufficient, preventing efficient information retrieval. This made organizing and searching for information cumbersome for long videos, such as educational content or documentaries, and reduced usability.

[0722] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0723] In this invention, the server includes a means for converting audio data from a video into text using a generative artificial intelligence engine, a means for analyzing the video data of the video, dividing the scenes, and creating summaries, and a means for creating chapter candidates based on the generated summaries and scenes. This allows users to quickly search for related scenes within a video by entering specific keywords. Furthermore, by adding a means for generating prompt sentences and sending them to the server, the generated summaries and chapters can be easily understood with high accuracy. This system allows users to efficiently access the information they need without having to watch the entire video.

[0724] "Means for uploading videos" is a function that allows users to send any video file to the system and have it saved on the server.

[0725] "Means for the server to receive the video and store it in temporary storage" refers to a function in which the server receives the uploaded video file and temporarily stores it for subsequent analysis processing.

[0726] "Means for converting video audio data into text using a generative artificial intelligence engine" is a function that uses generative artificial intelligence technology to automatically transcribe audio within a video.

[0727] "Means for analyzing video data of a video, dividing the video into scenes, and creating summaries" refers to a function that analyzes video data, divides the content into multiple scenes, and extracts the important content of each scene as a summary.

[0728] "Means for creating chapter candidates based on the generated summary and scenes" is a function that divides a video into multiple sections (chapters) based on the generated summary and scene information, and assigns titles and summaries to each section.

[0729] The "means for transmitting chapter candidates to a terminal" is a function for transmitting chapter information generated by the server to a user's terminal.

[0730] The "means for displaying chapter candidates received by the terminal to the user" is a function for visually displaying chapter information received on the user's terminal.

[0731] "Means for a user to input a specific keyword into the search bar of a terminal" refers to a function that allows a user to input a keyword they want to search for into the search bar on the terminal.

[0732] The "means for the terminal to transmit a keyword to the server" is a function for transmitting a keyword input from the terminal to the server.

[0733] "Means for analyzing the text data of a video and identifying scenes containing keywords" is a function for analyzing the generated text data and identifying scenes in which the input keywords appear.

[0734] The "means for transmitting the time stamp of the scene to the terminal" is a function for transmitting the time information (time stamp) of the identified scene to the terminal.

[0735] "Means for the terminal to display the timestamp to the user and for the user to access a specified scene" is a function that displays timestamp information on the terminal and allows the user to directly access a specific scene based on that information.

[0736] "Means for generating video summaries using a generative AI model" refers to a function that uses a generative AI model to automatically summarize the content of a video.

[0737] "Means for generating prompt sentences and sending them to the server" is a function that automatically generates input sentences (prompt sentences) for the generative AI model and sends them to the server.

[0738] The present invention provides a function for efficiently viewing and searching video content. Specific embodiments of the present invention will be described below.

[0739] System Overview

[0740] This system combines a server, a user device, and a generative AI model engine. Users upload videos through their devices, and the server analyzes the videos and generates summaries. Users can also search for related scenes within videos using specific keywords.

[0741] Hardware and Software

[0742] Hardware: User devices such as smartphones, tablets, and PCs, as well as servers.

[0743] Software: ffmpeg, moviepy, speech_recognition, transformers library, and generative AI model engine.

[0744] Data processing and calculation

[0745] 1. Upload and save your video

[0746] The user selects a video file and uploads it from the device to the server, which then stores the received video file in temporary storage.

[0747] 2. Speech-to-text

[0748] The server extracts the audio data from the received video and converts it into text using speech recognition technology, using the speech_recognition library.

[0749] 3. Video analysis and summary generation

[0750] The server analyzes the video data and segments it into scenes, using the ffmpeg and moviepy libraries.

[0751] The server uses a generative AI model (e.g., the summarization model from the transformers library) to summarize the key points for each scene.

[0752] 4. Creating chapter suggestions

[0753] The server automatically creates chapter candidates based on the generated summary and scene information, and sends the chapter information to the user device in JSON format.

[0754] 5. Chapter display and search

[0755] The user terminal displays the received chapter information in list form.

[0756] When a user enters a specific keyword into the search bar of the device and clicks the search button, the device sends the keyword to the server.

[0757] The server analyzes the text data of the video, identifies scenes containing the keywords, and sends the timestamp information of those scenes to the terminal.

[0758] The terminal displays the timestamp information to the user, and the user can directly access the specified scene based on the information.

[0759] Specific examples

[0760] For example, suppose a user uploads a video on an educational video streaming service titled "Python Basics." Using this system, the user can:

[0761] Upload a video and its content will be automatically summarized using a generative AI model.

[0762] Each chapter (e.g., "Basic Python Syntax," "Data Structures," and "Functions") is automatically generated and displayed as a list.

[0763] When a user searches for "loop," relevant scenes are displayed with timestamps.

[0764] Users can click on the displayed timestamp to view the corresponding scene directly.

[0765] Prompt Sentence Examples

[0766] The prompt sentence can be summarized as follows: "Python is an easy-to-use and powerful programming language. Many beginners are amazed by its simplicity."

[0767] "Write a summary of this text: 'The programming language Python is easy to use and is the language of choice for many beginners.'"

[0768] This system will make viewing and searching video content more efficient and improve usability.

[0769] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0770] Step 1:

[0771] The user selects and uploads a video.

[0772] Input: The user selects a video file from local storage.

[0773] Output: The video file is sent to the user's device and uploaded to the server.

[0774] Specific operation: The user clicks the upload button on the device and selects a video file from the file selection dialog. The selected video file is sent from the device to the server.

[0775] Step 2:

[0776] The server receives the video and stores it in temporary storage.

[0777] Input: The video file sent from the user device.

[0778] Output: The video file is saved to the server's temporary storage.

[0779] Specific operation: The server stores the received video file in temporary storage and prepares it for subsequent analysis processing.

[0780] Step 3:

[0781] The server converts the video's audio data into text.

[0782] Input: Video file saved in temporary storage.

[0783] Output: Text data generated from the audio data.

[0784] What it does: The server uses the ffmpeg library to extract audio data from the video, then uses the speech_recognition library to convert the audio data to text.

[0785] Step 4:

[0786] The server analyzes the video data, divides the scenes, and creates a summary.

[0787] Input: Video files stored in temporary storage and generated text data.

[0788] Output: Summary text and chapter information for each scene.

[0789] How it works: The server uses the moviepy library to analyze the video data and segment it into scenes, then uses a generative AI model (e.g., the summarization model from the transformers library) to automatically summarize the content of each scene.

[0790] Step 5:

[0791] The server creates chapter candidates based on the generated summary and scenes.

[0792] Input: Summary text and scene information.

[0793] Output: Chapter candidate information (title, summary, timestamp, etc.).

[0794] Specific operation: The server creates chapter candidates in JSON format based on the generated summary and scene information.

[0795] Step 6:

[0796] The server transmits the chapter candidates to the terminal.

[0797] Input: Chapter candidate information.

[0798] Output: Chapter suggestion information sent to the device.

[0799] Specific operation: The server transmits the generated chapter candidate information to the user terminal.

[0800] Step 7:

[0801] The terminal displays the received chapter candidates to the user.

[0802] Input: Chapter candidate information received from the server.

[0803] Output: The chapter list displayed in the user interface.

[0804] Specific operation: The terminal displays the chapter candidate information in a list format on the user interface, allowing access to each chapter.

[0805] Step 8:

[0806] A user types a specific keyword into the search bar.

[0807] Input: The keyword entered by the user.

[0808] Output: Search keywords displayed on the device.

[0809] Specific operation: The user enters a specific keyword into the search bar of the device and clicks the search button.

[0810] Step 9:

[0811] The device sends the keyword to the server.

[0812] Input: The keyword entered in the search bar.

[0813] Output: The search terms sent to the server.

[0814] Specific operation: The terminal sends the keyword entered by the user to the server.

[0815] Step 10:

[0816] The server analyzes the text data of the video and identifies scenes that contain the keywords.

[0817] Input: Search keywords sent to the server and generated text data.

[0818] Output: Timestamp information of identified scenes.

[0819] Specific operation: The server analyzes the generated text data, identifies scenes that contain the search keywords, and generates timestamp information for the identified scenes.

[0820] Step 11:

[0821] The server transmits the timestamp of the identified scene to the terminal.

[0822] Input: The timestamp information of the identified scene.

[0823] Output: Timestamp information sent to the terminal.

[0824] Specific operation: The server transmits the timestamp information of the identified scene to the user terminal.

[0825] Step 12:

[0826] The terminal displays the timestamp to the user, and the user accesses the specified scene.

[0827] Input: The timestamp information received from the server.

[0828] Output: Timestamp information and associated scenes displayed in the user interface.

[0829] Specific operation: The device displays the timestamp information in the user interface, and the user can click on the displayed timestamp to directly access a specific scene.

[0830] The above are the specific processing steps for carrying out the invention.

[0831] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0832] The present invention is a system that automatically analyzes the content of videos and can pinpoint the information a user needs, and combines it with an emotion engine that recognizes the user's emotions to provide more advanced and personalized information. Specific embodiments for carrying out the present invention are described below.

[0833] System Overview

[0834] This system consists of a user, a server, a terminal, and an emotion engine. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the videos by entering specific keywords. Furthermore, the emotion engine recognizes the user's emotions and dynamically adjusts the video playback content and chapter display based on that information.

[0835] Uploading and saving videos

[0836] First, the user selects a video file and uploads it to the system. Once the video file is uploaded, the device sends it to the server. The server then stores the received video file in temporary storage and prepares it for analysis.

[0837] Video analysis and summary generation

[0838] The server then launches a generative AI engine to convert the video's audio data into text. Through this speech recognition process, the video's content is extracted as text data. The server then analyzes the video data and segments each scene. The generative AI engine then summarizes the content of each scene and extracts the main points.

[0839] Creating and viewing chapter suggestions

[0840] The server creates chapter candidates based on the generated summaries and scenes. This automatically divides the video into multiple sections, each with a title and summary. The server sends the chapter candidates in JSON format to the device, which then displays them visually to the user.

[0841] Reverse search function from words

[0842] The user enters a specific keyword in the search bar and clicks the search button. The device sends the keyword to the server. The server analyzes the text data of the video and identifies the scene in which the keyword appears. Once the timestamp of the relevant scene is identified, the server sends that information to the device. The device displays the search results, including the timestamp, to the user, allowing the user to directly access the specified scene.

[0843] Emotion recognition function

[0844] The system further enhances the user experience by incorporating an emotion engine. While the user is watching a video, a camera built into the device captures real-time video of the user. This video data is sent to a server, which then uses the emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial expressions and movements to identify their emotional state (e.g., joy, sadness, excitement, etc.).

[0845] The server can dynamically adjust the video playback and chapters displayed based on the emotional data it acquires. For example, if it determines that the user is interested, it can provide more information or supplemental videos on related topics. On the other hand, if it determines that the user is bored, it can present the option to skip to important scenes.

[0846] Specific examples

[0847] For example, if a user uploads a video about "latest tech trends," the process goes like this:

[0848] 1. Upload and analyze

[0849] A user uploads a video about "latest tech trends."

[0850] The device sends the video file to the server.

[0851] The server receives the video, stores it in temporary storage, and begins analysis.

[0852] 2. Summarizing and Chapter Creation

[0853] The server uses a generative artificial intelligence engine to generate a summary of the video.

[0854] The server creates chapter candidates based on each technical topic.

[0855] 3. Display results

[0856] The server sends the summary and chapter information to the device in JSON format.

[0857] The terminal displays this to the user in a list format.

[0858] 4. Reverse Search Feature

[0859] A user types "blockchain" into the search bar and clicks the search button.

[0860] The device sends the keyword to the server.

[0861] The server analyzes the video transcript, identifies scenes in which "blockchain" appears, and sends that information to the device.

[0862] The terminal displays the search results to the user in a list format, allowing the user to directly access related scenes.

[0863] 5. Emotion recognition

[0864] While the user is watching the video, the device's camera captures real-time video.

[0865] The server analyzes the video data and uses an emotion engine to recognize the user's emotions.

[0866] Based on the emotional data, the video playback content and chapter display are dynamically adjusted.

[0867] The system not only allows users to instantly access the information they need without having to play the entire video, but also provides a personalized viewing experience through emotion recognition.

[0868] The processing flow will be explained below.

[0869] Step 1:

[0870] The user selects a video file and uploads it to the system. The user uses a file selection dialog to select a video file from their computer and clicks the "Upload" button.

[0871] Step 2:

[0872] The device sends the selected video file to the server. The device obtains the binary data of the video file and sends it to the server using an HTTP POST request.

[0873] Step 3:

[0874] The server receives the video file and stores it in temporary storage. The server receives the video file, checks the integrity of the data, and then stores it in a temporary storage folder.

[0875] Step 4:

[0876] The server starts the generative artificial intelligence engine. The server calls the generative artificial intelligence engine to start the process of converting audio data into text for the prepared video file.

[0877] Step 5:

[0878] The server converts the audio data into text, and a generative AI engine analyzes the audio in the video and generates recognized text data.

[0879] Step 6:

[0880] The server analyzes the video data, extracts each frame of the video, and segments it into scenes using a video recognition algorithm.

[0881] Step 7:

[0882] The server generates a summary for each scene using the generated text and video data. The content is reviewed to ensure that key points and themes are included.

[0883] Step 8:

[0884] The server creates chapter candidates. Based on the generated summaries and scene information, the server assigns titles and summaries to each segment and creates chapter candidates.

[0885] Step 9:

[0886] The server sends the chapter candidates to the device. The server encodes the completed chapter candidates in JSON format and sends them to the device as an HTTP response.

[0887] Step 10:

[0888] The device parses the received chapter suggestions and displays them on the user interface. The device parses the JSON data and displays them to the user in list or timeline format.

[0889] Step 11:

[0890] A user enters a specific keyword into the search bar and clicks the search button.A user enters a keyword into the input field of the search bar and presses the button to execute the search command.

[0891] Step 12:

[0892] The device sends the keyword to the server. The device creates an HTTP GET request including the user's keyword and sends it to the server.

[0893] Step 13:

[0894] The server analyzes the text data of the video and identifies the scenes in which the keywords appear. Using a full-text search algorithm, the server extracts text fragments containing the keywords and obtains the timestamps of the corresponding scenes.

[0895] Step 14:

[0896] The server sends the timestamp to the device. The server encodes the timestamp and scene information identified as search results in JSON format and sends it to the device as an HTTP response.

[0897] Step 15:

[0898] The device displays the search results to the user. The device displays the received timestamps and scene information in a list format, and provides a "Play" button for each entry.

[0899] Step 16:

[0900] The user clicks the play button to access the specified scene. When the user clicks the play button in the displayed list, the video player starts playing from the corresponding scene.

[0901] Step 17:

[0902] The device acquires real-time video data of the user. The device's camera captures the user's face and movements and sends the data to the server.

[0903] Step 18:

[0904] The server starts the emotion engine. The server invokes the emotion engine to analyze the received video data.

[0905] Step 19:

[0906] The server analyzes the user's emotions, and the emotion engine analyzes the user's facial expressions and movements to identify their emotional state, such as joy, sadness, or excitement.

[0907] Step 20:

[0908] The server uses the emotion analysis results to dynamically adjust video playback and chapter display based on the emotion data.

[0909] Step 21:

[0910] The server sends adjustments based on emotion data to the device, and the server sends dynamically adjusted information to the device, which then visually displays it.

[0911] Step 22:

[0912] The device displays a display that reflects the emotional data to the user. The device displays a list that reflects the emotional data and adjusts the video player to the user, allowing the user to have an optimal viewing experience.

[0913] Example 2

[0914] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0915] While conventional video analysis systems have the ability to summarize videos and search for specific scenes, they lack the ability to dynamically change content based on the user's emotional state. Furthermore, while they enable users to quickly obtain specific information, they have the problem of being unable to respond to the user's interests and emotions while watching, because the video playback content is static. Therefore, there is a need for a system that can individually optimize the user's viewing experience and efficiently provide the information they need.

[0916] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0917] In this invention, the server includes means for converting audio data of a video into text using a generative artificial intelligence engine, means for analyzing video data of the video and dividing it into scenes to create a summary, means for creating chapter candidates based on the created summary and scenes, and means for recognizing the user's emotional state using an emotion engine and dynamically adjusting the video playback content and chapter display. This enables a personalized viewing experience according to the user's emotions and makes it possible to efficiently provide necessary information.

[0918] "Video" means a digital file containing audio and video information that a user uploads to the system.

[0919] The "server" is a central processing unit that receives video, temporarily stores it, and analyzes it using a generative artificial intelligence engine and an emotion engine.

[0920] A "generative artificial intelligence engine" is an AI technology that converts audio data from videos into text and analyzes video data to create scene summaries.

[0921] "Voice recognition technology" is a technology for converting audio data from video into text data.

[0922] "Video data" refers to digital data that constitutes the visual information of a moving image.

[0923] "Scene segmentation" is a method of dividing a continuous video sequence into multiple stages and analyzing each section independently.

[0924] A "summary" is text information that briefly summarizes the main content of each scene in a video.

[0925] "Chapter candidates" refer to the individual parts of a video divided into multiple sections based on summarized scene information.

[0926] "Emotion recognition" is the process of analyzing a user's facial expressions, movements, and other facial expressions to identify their emotional state.

[0927] "Terminal" means a device through which a user accesses the system to upload, search, and play videos.

[0928] A "keyword" is a character string that a user enters to search for specific information.

[0929] A "timestamp" is information that indicates the start time of a particular scene in a video.

[0930] "Dynamic adjustment" refers to changing the video playback content and chapter display in real time based on the user's emotion recognition results.

[0931] The present invention is an advanced information provision system that combines video analysis and emotion recognition, and specific embodiments are described in detail below. This system allows users to upload videos and automatically analyze the content of those videos to efficiently obtain the information users need. The present invention includes a server, a terminal, and an emotion engine. Ultimately, it realizes dynamic information provision according to the user's emotions.

[0932] Uploading and saving videos

[0933] A user uses a device to select and upload a video file. The device then sends the video file to a server. The server receives the video file and stores it in temporary storage, preparing the video for analysis. For example, if a user uploads a video of a seminar on technology trends, the device sends the file to the server, which stores it.

[0934] Video analysis and summary generation

[0935] The server launches a generative AI engine to begin analyzing the video. First, the audio data is converted into text using speech recognition technology (e.g., Google Speech-to-Text API). Next, the video data is analyzed and divided into scenes. For each divided scene, a generative AI model (e.g., GPT-4) is used to generate a summary. This allows the main points of the video to be concisely extracted.

[0936] Creating chapter suggestions

[0937] The server creates chapter candidates based on the generated summary and split scenes. Each chapter is divided into sections and given a title and summary. The server sends this information to the device in JSON format, and the device displays it to the user. For example, if a video on "latest technology trends" is uploaded, the server generates chapters based on technology topics and displays them to the user.

[0938] Keyword search function

[0939] The user enters a specific keyword into the search bar on the device and clicks the search button. The device then sends the keyword to the server. The server analyzes the text data of the video and identifies the scene in which the keyword appears. The timestamp information of the relevant scene is sent to the device and displayed to the user. The user can then directly access the relevant scene from the search results.

[0940] Emotion recognition function

[0941] The system incorporates an emotion engine that analyzes the user's emotions in real time. While the user is watching a video, a camera built into the device captures real-time footage of the user and sends it to the server. The server uses the emotion engine to analyze the user's emotional state and dynamically adjusts the video playback content and chapter display. For example, if the system determines that the user is excited, it provides related information or supplementary videos. If the system determines that the user is bored, it presents an option to skip to important scenes.

[0942] As a concrete example, when a user uploads a video about "latest technology trends," the process proceeds as follows:

[0943] 1. The user uploads a video and the device sends it to the server.

[0944] 2. The server stores the video and begins analysis, converting the audio data into text and analyzing the video data to generate a summary.

[0945] 3. The server creates chapter candidates and sends them to the device, which displays them to the user.

[0946] 4. The user searches by keyword, and the server sends the timestamp of the corresponding scene to the device. The device displays the search results, and the user can access the corresponding scene.

[0947] 5. While the user is watching the video, the device camera captures their emotions. The server analyzes this using an emotion engine and dynamically adjusts the playback content.

[0948] In this way, the system allows users to quickly access the information they need without having to play the entire video, and also provides a personalized viewing experience through emotion recognition.

[0949] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0950] Step 1:

[0951] A user selects a video file on a terminal and uploads it to the system. Specifically, the user selects a video file from the user interface and presses the upload button. The input at this time is the selected video file, and the output is the terminal sending the video file to the server.

[0952] Step 2:

[0953] The device sends the video file selected by the user to the server. It receives the video file uploaded by the user as input and sends it to the server as output. Specifically, the device generates an HTTP request, attaches the video file, and sends it to the server.

[0954] Step 3:

[0955] The server receives the video file and stores it in temporary storage. It receives the video file sent from the terminal as input and stores it in temporary storage as output. Specifically, the server uses a file management system to store the video file in a specified directory.

[0956] Step 4:

[0957] The server starts a generative artificial intelligence engine and converts the video's audio data into text using speech recognition technology. It receives the saved video file as input and obtains the generated text data as output. Specifically, the server calls the speech recognition API, sends the audio data as input, and receives the text data.

[0958] Step 5:

[0959] The server analyzes the video data, segments the scenes, and creates summaries. It receives the video file and generated text data as input, and generates segmented scenes and their summaries as output. Specifically, the server applies a video analysis algorithm to identify the video frames for each scene and creates a summary using a generative AI model.

[0960] Step 6:

[0961] The server creates chapter candidates based on the generated summaries and scenes. It receives scenes and their summaries as input and generates chapter information as output. Specifically, the server aggregates the summaries for each scene and generates chapter titles and summaries in JSON format.

[0962] Step 7:

[0963] The server sends the generated chapter information to the terminal. It receives the generated chapter information as input and sends it to the terminal as output. Specifically, the server generates an HTTP response, attaches the chapter information, and sends it to the terminal.

[0964] Step 8:

[0965] The terminal displays the received chapter candidates to the user. It receives the chapter information received from the server as input and displays it on the user interface as output. Specifically, the terminal analyzes the chapter information and displays it in list format.

[0966] Step 9:

[0967] The user enters a specific keyword into the search bar of the device and clicks the search button. The keyword is received as input and sent to the server as output. The specific operation is that the user enters a keyword into the search bar and clicks the search button.

[0968] Step 10:

[0969] The terminal sends the keyword to the server. It receives the keyword entered by the user as input and sends it to the server as output. Specifically, the terminal generates an HTTP request, attaches the keyword, and sends it to the server.

[0970] Step 11:

[0971] The server analyzes the text data of the video to identify scenes containing the keywords. Using the keywords received as input and the text data of the video, it generates a timestamp for the relevant scene as output. Specifically, the server searches the text data for keywords and identifies the start time of the relevant scene.

[0972] Step 12:

[0973] The server sends the scene timestamp to the terminal. It receives the timestamp of the specified scene as input and sends it to the terminal as output. As a specific operation, the server generates an HTTP response, attaches the timestamp information, and sends it to the terminal.

[0974] Step 13:

[0975] The terminal displays the timestamp to the user, and the user accesses the specified scene. The terminal receives the timestamp information from the server as input, displays it on the user interface as output, and accesses the specified scene. As a specific operation, the terminal uses a playback player to play the video from the specified scene.

[0976] Step 14:

[0977] The camera built into the device captures real-time video of the user and sends it to the server. The device acquires real-time video of the user as input and sends it to the server as output. Specifically, the device activates the camera module, captures video data, and sends it to the server.

[0978] Step 15:

[0979] The server uses an emotion engine to recognize the user's emotional state and dynamically adjust the video playback content and chapter display. It receives the transmitted video data as input and generates emotion recognition results and dynamic adjustment results as output. Specifically, the emotion engine analyzes the video data, identifies the user's emotional state, and changes the video playback content accordingly.

[0980] Step 16:

[0981] Based on the emotion recognition, the server dynamically adjusts the video playback content and chapter display and sends them to the terminal. The emotion recognition results are received as input, and the adjustments are sent to the terminal as output. Specifically, the server generates adjustment instructions and sends them to the terminal, which then changes the video playback content as appropriate.

[0982] (Application example 2)

[0983] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0984] The challenge is to provide users with a more effective and personalized advertising experience by appropriately recognizing users' interests and emotions while watching advertising videos and dynamically adjusting the advertising content based on that information. Conventional advertising systems provide uniform advertising content without considering the user's emotional state, making it difficult to maintain user interest. Furthermore, there is no mechanism for users to quickly search for related information using specific keywords, resulting in a poor user experience.

[0985] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for automatically analyzing the content of the advertising video being viewed by the user, means for capturing real-time video of the user using the camera of the terminal, means for the server to analyze the video data and recognize the user's emotions using an emotion engine, and means for the server to dynamically adjust the advertising content based on the recognized emotion information. This enables a personalized advertising experience that matches the user's emotional state.

[0986] "User" means a person or organization that uses the system to upload videos and receive the analyzed results.

[0987] "Server" is a central computer system that processes and analyzes video data received from users, and stores and transmits the results.

[0988] "Temporary storage" is a storage device that temporarily stores video when the server receives it.

[0989] A "generative artificial intelligence engine" is an analytical engine equipped with AI technology that analyzes the content of videos, converts audio data into text, and generates summaries.

[0990] "Audio data" is digital data that includes audio information emitted within a video.

[0991] "Text data" refers to data obtained by converting voice data into text format.

[0992] "Video data" is digital data that contains visual information within a moving image.

[0993] A "scene" refers to a specific time range within a video, and is a video portion with independent content.

[0994] "Chapter candidates" are sections divided into scenes that are generated as a result of analyzing a video, and each section is given a title and a summary.

[0995] "Terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[0996] A "search bar" is a UI component that allows a user to enter specific keywords.

[0997] "Keywords" refer to specific words or phrases that users use to search for specific scenes within a video.

[0998] A "timestamp" is digital information that indicates the start time of a particular scene.

[0999] The "Emotion Engine" is an analytical engine equipped with AI technology that analyzes real-time video footage of users and identifies their emotional state.

[1000] An "advertising video" is a video created to promote a particular product or service.

[1001] "Real-time video" refers to video data of a user captured in real time.

[1002] "Emotion information" is data that indicates the user's emotional state analyzed by the emotion engine.

[1003] "Dynamic adjustment" means changing the content and presentation in real time depending on the situation.

[1004] Specific embodiments for carrying out the present invention are described below.

[1005] System configuration

[1006] The system consists of a user, a server, a terminal, and an emotion engine. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the video by entering specific keywords. Furthermore, the emotion engine recognizes the user's emotions and dynamically adjusts the playback content and chapter display of the advertising video based on that information.

[1007] Uploading and saving videos

[1008] The user uploads the advertising video to the device. The device then sends the video file to the server, which stores it in temporary storage. In this process, the server and device hardware are typically general-purpose computers or smartphones.

[1009] Video analysis and summary generation

[1010] The server then launches a generative AI engine to convert the audio data from the ad video into text. Through this speech recognition process, the video content is extracted as text data. The server then analyzes the video data and segments each scene. The generative AI engine generates a summary of each scene and extracts the main points. Transformers from Hugging Face are used for speech recognition.

[1011] Creating and viewing chapter suggestions

[1012] The server creates chapter candidates based on the generated summaries and scenes. This automatically divides the video into multiple sections, each with a title and summary. The chapter information is sent to the device in JSON format, and the device displays it visually to the user.

[1013] Keyword search function

[1014] The user enters a specific keyword into the device's search bar and clicks the search button. The device then sends the keyword to the server. The server analyzes the text data in the video and identifies scenes that contain the keyword. Once the timestamp of the relevant scene is identified, that information is sent to the device. The device then displays the search results, including the timestamp, to the user, allowing the user to directly access the specified scene.

[1015] Emotion recognition and dynamic ad adjustment

[1016] The system further enhances the user experience by incorporating an emotion engine. While the user is watching the advertising video, a camera built into the device captures real-time video of the user. This video data is sent to a server, which then uses an emotion engine to recognize the user's emotions. The emotion engine uses services such as Amazon Rekognition to analyze the user's facial expressions and movements to identify their emotional state.

[1017] The server dynamically adjusts the content of the ad video and the chapters displayed based on the acquired emotional data. For example, if the server determines that the user is interested, it may provide more information or supplemental videos on related topics. On the other hand, if the server determines that the user is bored, it may present the option to skip to important scenes.

[1018] Specific examples

[1019] As an example, if a user uploads an advertising video about "latest smartphone technology," the process would proceed as follows:

[1020] 1. Upload and analyze

[1021] A user uploads an advertising video for the "latest smartphone technology."

[1022] The device sends the video file to the server.

[1023] The server receives the video, stores it in temporary storage, and begins analysis.

[1024] 2. Summarizing and Chapter Creation

[1025] The server uses a generative artificial intelligence engine to generate a summary of the video.

[1026] The server creates chapter candidates based on each technical topic.

[1027] 3. Display results

[1028] The server sends the summary and chapter information to the device in JSON format.

[1029] The terminal displays this to the user in a list format.

[1030] 4. Reverse Search Feature

[1031] A user types "camera features" into the search bar and clicks the search button.

[1032] The device sends the keyword to the server.

[1033] The server analyzes the video transcription data, identifies scenes in which the "camera function" appears, and sends that information to the device.

[1034] The terminal displays the search results to the user in a list format, allowing the user to directly access related scenes.

[1035] 5. Emotion recognition

[1036] While the user is watching the advertising video, the camera on the terminal captures real-time video.

[1037] The server analyzes the video data and uses an emotion engine to recognize the user's emotions.

[1038] Based on the emotional data, the playback content and chapter display of the advertising video are dynamically adjusted, improving the user experience and maximizing the effectiveness of the advertisement.

[1039] Example prompts

[1040] The prompt sentences below are examples used in the emotion-sensitive ad viewer.

[1041] "Identify exciting moments in advertising videos and highlight relevant technology demos."

[1042] "If the user expresses interest, provide additional information related to the topic."

[1043] This system not only allows users to instantly access the information they need without having to watch the entire advertisement, but also provides a personalized viewing experience through emotion recognition.

[1044] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1045] Step 1:

[1046] A user uploads an advertising video.

[1047] The user selects an advertising video file through the terminal interface and uploads it to the system, at which point the terminal sends the selected video file to the server, where the input is the video file provided by the user and the output is the video file received by the server.

[1048] Step 2:

[1049] The server stores the video in temporary storage.

[1050] The server temporarily stores the received video file in temporary storage. The input is the video file sent from the device, and the output is the video file saved in the temporary storage.

[1051] Step 3:

[1052] The server launches a generative artificial intelligence engine to convert the video's audio data into text.

[1053] The server uses a generative artificial intelligence engine to convert the audio data in the video into text data. The input is the video file loaded from temporary storage, and the output is text data. This process uses voice recognition technology.

[1054] Step 4:

[1055] The server analyzes the video data, divides the scenes, and creates summaries.

[1056] The server analyzes the video data and divides it into multiple scenes. It then uses a generative AI engine to create a summary of each scene and extract key points. The input is text data and video data, and the output is a summary of each scene and chapter suggestions.

[1057] Step 5:

[1058] The server creates chapter candidates based on the generated summary and scenes.

[1059] The server generates chapter candidates, each containing a title and summary, based on the generated summaries and the divided scenes. The input is scene summary data, and the output is chapter candidate data.

[1060] Step 6:

[1061] The server transmits the chapter candidates to the terminal.

[1062] The server sends the created chapter candidates to the terminal. The input is the chapter candidate data, and the output is the chapter candidate data received by the terminal.

[1063] Step 7:

[1064] The terminal displays the received chapter candidates to the user.

[1065] The terminal visually displays the chapter candidates received from the server to the user in the form of a text list. The input is the received chapter candidate data, and the output is the chapter list displayed to the user.

[1066] Step 8:

[1067] The user enters a specific keyword into the device's search bar.

[1068] The user inputs a specific keyword through the search bar and clicks the search button. The input is the keyword input by the user, and the output is the keyword data sent from the terminal to the server.

[1069] Step 9:

[1070] The device sends the keyword to the server.

[1071] The terminal sends the keyword received from the user to the server. The input is the keyword entered by the user, and the output is the keyword data sent to the server.

[1072] Step 10:

[1073] The server analyzes the text data of the video and identifies scenes that contain the keywords.

[1074] The server analyzes the text data of the video and identifies scenes that contain the keywords entered by the user. The input is keyword data and text data, and the output is scene information that contains the keywords.

[1075] Step 11:

[1076] The server sends the scene timestamp to the device.

[1077] The server sends the timestamp of the identified scene to the terminal. The input is the scene information, and the output is the timestamp data sent to the terminal.

[1078] Step 12:

[1079] The terminal displays the timestamp to the user, and the user accesses the specified scene.

[1080] The terminal displays the received timestamp to the user, allowing the user to directly access the specified scene. The input is the timestamp data received from the server, and the output is the timestamp information displayed to the user.

[1081] Step 13:

[1082] A real-time image of the user is acquired using the device's camera.

[1083] While the user is watching the advertisement video, the camera of the terminal captures the user's real-time video. The input is the video data from the camera, and the output is the video data sent to the server.

[1084] Step 14:

[1085] The server analyzes the video data and uses an emotion engine to recognize the user's emotions.

[1086] The server analyzes the acquired video data using an emotion engine to identify the user's emotional state. The input is the video data received from the device, and the output is the emotion recognition result.

[1087] Step 15:

[1088] The server dynamically adjusts the advertising content based on the recognized emotion information.

[1089] The server dynamically adjusts the playback content of the advertising video and the chapters to be displayed based on the emotional information analyzed by the emotion engine. The input is the emotion recognition result, and the output is the adjusted advertising content.

[1090] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1091] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1092] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1093] [Third embodiment]

[1094] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1095] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1096] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1097] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1098] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1099] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1100] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1101] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1102] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1103] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1104] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1105] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1106] The present invention is a system that automatically analyzes the content of a video and can pinpoint the information that a user needs. Specific embodiments for carrying out the present invention will be described below.

[1107] System Overview

[1108] This system consists of users, a server, and terminals. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the videos by entering specific keywords.

[1109] Uploading and saving videos

[1110] First, the user selects a video file and uploads it to the system. Once the video file is uploaded, the device sends it to the server. The server then stores the received video file in temporary storage and prepares it for analysis.

[1111] Video analysis and summary generation

[1112] The server then launches a generative AI engine to convert the video's audio data into text. Through this speech recognition process, the video's content is extracted as text data. The server then analyzes the video data and segments each scene. The generative AI engine then summarizes the content of each scene and extracts the main points.

[1113] Creating and viewing chapter suggestions

[1114] The server creates chapter candidates based on the generated summaries and scenes. This automatically divides the video into multiple sections, each with a title and summary. The server sends the chapter candidates in JSON format to the device, which then displays them visually to the user.

[1115] Reverse search function from words

[1116] The user enters a specific keyword in the search bar and clicks the search button. The device sends the keyword to the server. The server analyzes the text data of the video and identifies the scene in which the keyword appears. Once the timestamp of the relevant scene is identified, the server sends that information to the device. The device displays the search results, including the timestamp, to the user, allowing the user to directly access the specified scene.

[1117] Specific examples

[1118] For example, consider a user who uploads a video about "Programming Basics," which includes examples of multiple programming languages ​​and their usage.

[1119] 1. Upload and analyze

[1120] A user uploads a video on "Programming Basics."

[1121] The terminal transmits the selected video file to the server.

[1122] The server receives the video, stores it in temporary storage, and begins the analysis process.

[1123] 2. Summarizing and Chapter Creation

[1124] The server uses a generative artificial intelligence engine to summarize the content of the video.

[1125] The server creates chapters based on each language segment (e.g. "Python Basics", "JavaScript Basics", etc.).

[1126] 3. Display results

[1127] The server sends the summary and chapter information to the device in JSON format.

[1128] The terminal displays this to the user in a list format.

[1129] 4. Reverse Search Feature

[1130] A user types "Python" into the search bar and clicks the search button.

[1131] The device sends the keyword to the server.

[1132] The server analyzes the video transcription data, identifies scenes in which "Python" appears, and sends relevant information to the device.

[1133] The device displays the search results in a list format, allowing the user to directly access related scenes.

[1134] This system allows users to quickly access the information they need without having to play the entire video, making it possible to use videos efficiently.

[1135] The processing flow will be explained below.

[1136] Step 1:

[1137] The user selects a video file and uploads it to the system. The user selects a video file from their computer using the file selection dialog and clicks the "Upload" button.

[1138] Step 2:

[1139] The device sends the selected video file to the server. The device obtains the binary data of the video file and sends it to the server using an HTTP POST request.

[1140] Step 3:

[1141] The server receives the video file and stores it in temporary storage. The server receives the video file, checks the integrity of the data, and then stores it in a temporary storage folder.

[1142] Step 4:

[1143] The server starts the generative artificial intelligence engine. The server calls the generative artificial intelligence engine to start the process of converting audio data into text for the prepared video file.

[1144] Step 5:

[1145] The server converts the audio data into text, and a generative AI engine analyzes the audio in the video and generates recognized text data.

[1146] Step 6:

[1147] The server analyzes the video data, extracts each frame of the video, and segments it into scenes using a video recognition algorithm.

[1148] Step 7:

[1149] The server generates a summary for each scene using the generated text and video data. The content is reviewed to ensure that key points and themes are included.

[1150] Step 8:

[1151] The server creates chapter candidates. Based on the generated summary and scene information, the server assigns a title and summary to each segment and creates chapter candidates.

[1152] Step 9:

[1153] The server sends the chapter candidates to the device. The server encodes the completed chapter candidates in JSON format and sends them to the device as an HTTP response.

[1154] Step 10:

[1155] The device parses the received chapter suggestions and displays them on the user interface. The device parses the JSON data and displays them to the user in list or timeline format.

[1156] Step 11:

[1157] A user enters a specific keyword into the search bar and clicks the search button.A user enters a keyword into the input field of the search bar and presses the button with the intention of a search command.

[1158] Step 12:

[1159] The device sends the keyword to the server. The device creates an HTTP GET request including the user's keyword and sends it to the server.

[1160] Step 13:

[1161] The server analyzes the text data of the video and identifies the scenes in which the keywords appear. Using a full-text search algorithm, the server extracts text fragments containing the keywords and obtains the timestamps of the corresponding scenes.

[1162] Step 14:

[1163] The server sends the timestamp to the device, and the server returns the identified timestamp and scene information in JSON format to the device as search results.

[1164] Step 15:

[1165] The device displays the search results to the user. The device displays the received timestamps and scene information in a list format, and provides a "Play" button for each entry.

[1166] Step 16:

[1167] The user clicks the play button to access the specified scene. When the user clicks the play button in the displayed list, the video player starts playing from the corresponding scene.

[1168] Example 1

[1169] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1170] In today's world, the proliferation of video content has made it difficult for users to quickly and efficiently obtain the information they need. Long video playback times also create inefficiencies in information retrieval. Furthermore, the lack of a way for users to quickly search for specific information of interest within a video limits the viewing experience.

[1171] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1172] In this invention, the server includes: means for a user to upload a video; means for a terminal to transmit a video file to the server; means for the server to store the video file in temporary storage; means for the server to convert audio data of the video into text using speech recognition technology; means for the server to segment video data of the video by scene using video analysis technology; means for the server to generate a summary from the text data and scene information using a generative artificial intelligence engine; means for the server to create chapter candidates based on the generated summary and scene information; means for the server to transmit the chapter candidates to the terminal in JSON format; means for the terminal to visually display the received chapter candidates to the user; means for the user to input a specific keyword into a search bar of the terminal; means for the terminal to transmit the keyword to the server; means for the server to analyze the text data and identify scenes containing the keyword; means for the server to transmit scene timestamp information to the terminal; and means for the terminal to display the timestamp information to the user and allow the user to access the specified scene. This allows the user to quickly access necessary information without playing the entire video.

[1173] "User" means an individual or entity that uses the system to upload videos and search for specific information.

[1174] A "terminal" is an electronic device, such as a computer or smartphone, that a user uses to upload videos or view search results.

[1175] A "server" is a computer system that stores videos sent by users and performs analysis processing.

[1176] "Temporary storage" refers to data storage used by the server to temporarily store video files.

[1177] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[1178] "Video analysis technology" is a technology that analyzes video data and identifies specific scenes and frames.

[1179] A "generative AI engine" is a type of artificial intelligence used to generate new information or summaries based on input data.

[1180] "Text data" is character-based data that represents the content of a video extracted using voice recognition technology.

[1181] A "scene" is a segment that separates parts of a video where the content changes.

[1182] "Chapter candidates" are multiple sections generated based on the video content, and each section is given a title and a summary.

[1183] The "JSON format" is one of the standard formats for structuring and describing data in text format.

[1184] The "search bar" is an input area where users can input specific keywords to search for related information within a video.

[1185] A "timestamp" is information that indicates a specific time point within a video.

[1186] The present invention is a system that automatically analyzes the content of a video and can pinpoint the information that a user needs. Specific embodiments for carrying out the present invention will be described below.

[1187] System Overview

[1188] This system consists of users, a server, and terminals. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the videos by entering specific keywords.

[1189] Uploading and saving videos

[1190] First, the user selects a video file and uploads it to the system. To do this, the user uses a device such as a PC or smartphone. The device then sends the selected video file to the server. The server then stores the received video file in temporary storage (for example, Amazon S3 or Google Cloud Storage) and prepares it for analysis.

[1191] Video analysis and summary generation

[1192] The server then analyzes the video using voice recognition and video analysis technology. The specific process is as follows:

[1193] 1. Audio to text conversion:

[1194] The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to convert the audio data in the video into text data. Through this process, the content of the video is extracted as a string of characters.

[1195] 2. Scene segmentation of video data:

[1196] The server uses video analysis technology (e.g., OpenCV library) to analyze the video data and segment each scene. Specifically, it detects change points between frames in the video and recognizes different scenes.

[1197] 3. Generate a summary:

[1198] The server uses a generative artificial intelligence engine (e.g., OpenAI GPT) to generate a summary of each scene based on the extracted text data and scene information, concisely summarizing the key points and content.

[1199] Creating and viewing chapter suggestions

[1200] Based on the generated summary and scene information, the server creates chapter candidates that divide the video into multiple sections. Each section is then given a title and summary. The server converts the chapter candidates into JSON format and sends them to the device. The device parses the received JSON data and displays the chapter candidates in a list format on the user interface. The user can click on one of them to directly access a specific scene.

[1201] Keyword search function

[1202] The user enters a specific keyword into the system's search bar and clicks the search button. The device sends this keyword to the server. The server analyzes the text data of the video and identifies the scenes in which the keyword appears. It obtains the timestamp information of the relevant scenes and sends it to the device in JSON format. The device analyzes the received data and displays the search results in a list format on the user interface. The user can click on this to directly access the relevant scenes.

[1203] Specific examples

[1204] Here, we will use a video on "Programming Basics" as an example.

[1205] 1. Upload and analyze:

[1206] A user uploads a video on "Programming Basics."

[1207] The device sends the selected video to the server, which stores the video in temporary storage.

[1208] The server converts the audio data into text using the Google Cloud Speech-to-Text API, and splits the video data into scenes using OpenCV.

[1209] The server creates chapter candidates based on the summary generated using OpenAI GPT and sends them to the device.

[1210] 2. Results display:

[1211] The device receives the chapter information in JSON format and displays it in a list format on the user interface.

[1212] 3. Keyword Search:

[1213] A user types "Python" into the search bar and clicks the search button.

[1214] The device sends the keyword to the server, and the server analyzes the text data of the video to identify the relevant scene, obtains timestamp information, and sends it to the device.

[1215] The device displays the search results in a list format, allowing the user to directly access related scenes.

[1216] Prompt Sentence Examples

[1217] Here are some example prompts to enter into a generative AI engine:

[1218] "This video explains the basics of programming. Please summarize the basic usage of each programming language scene by scene."

[1219] "Please summarize the sections on Python, JavaScript, and Java basics for each scene."

[1220] This system allows users to quickly access the information they need without having to play the entire video.

[1221] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1222] Step 1:

[1223] The user selects a video file and uploads it to the system via their device. The user selects a video file through a web browser on a device such as a PC or smartphone. The selected video file is sent from the device to the server. The input is the video file, and the output is the transmission of the video file to the server.

[1224] Step 2:

[1225] The device sends the selected video file to the server. Specifically, it uploads the video file to the server using an HTTP POST request. The input is the video file selected by the user, and the output is the video file sent to the server.

[1226] Step 3:

[1227] The server stores the received video file in temporary storage. The server stores the received video file in storage such as Amazon S3 or Google Cloud Storage, and records the file path and file name in a database. The input is the video file received from the device, and the output is saving to storage and recording to the database.

[1228] Step 4:

[1229] The server uses speech recognition technology to convert the audio data in the video into text. Specifically, it uses the Google Cloud Speech-to-Text API to convert the audio data in the video file into text. The input is the audio data in the video file, and the output is the converted text data.

[1230] Step 5:

[1231] The server uses video analysis technology to divide the video data into scenes. Specifically, it uses the OpenCV library to analyze the video data frame by frame and detect different scenes. The input is the video data of the video file, and the output is the divided scene information.

[1232] Step 6:

[1233] The server uses a generative AI engine to generate summaries based on the text data and scene information. Specifically, it uses OpenAI GPT to summarize the key points of each scene. The input is the text data for each scene, and the output is a summary of each scene.

[1234] Step 7:

[1235] The server creates chapter candidates based on the generated summaries and scene information. The server combines the summaries and scene information, divides the video into multiple sections, and assigns titles and summaries to each section. The input is the summaries and scene information, and the output is chapter candidates.

[1236] Step 8:

[1237] The server sends chapter candidates to the device in JSON format. The server converts the generated chapter information into JSON format and sends it to the device as an HTTP response. The input is the chapter candidates, and the output is the JSON data sent to the device.

[1238] Step 9:

[1239] The device displays the received chapter candidates on the user interface. The device parses the received JSON data and displays the chapter candidates in list format on the user interface. The input is JSON data, and the output is a visual representation of the chapter candidates.

[1240] Step 10:

[1241] The user enters a specific keyword into the search bar of the device. The user enters the keyword of interest and clicks the search button. The input is the keyword, and the output is the transmission of the keyword.

[1242] Step 11:

[1243] The terminal sends a keyword to the server. The terminal sends the entered keyword to the server via an HTTP POST request. The input is the keyword, and the output is the keyword sent to the server.

[1244] Step 12:

[1245] The server analyzes the text data to identify scenes containing keywords. The server analyzes the text data of the video and obtains the scenes in which the keywords appear and their timestamp information. The input is the keywords and text data, and the output is the timestamp information of the identified scenes.

[1246] Step 13:

[1247] The server sends the scene timestamp information to the terminal. The server converts the acquired timestamp information into JSON format and sends it to the terminal as an HTTP response. The input is the timestamp information, and the output is the JSON data sent to the terminal.

[1248] Step 14:

[1249] The terminal displays the timestamp information on the user interface, and the user accesses the specified scene. The terminal parses the received JSON data and displays the search results in a list format. The user can click on it to directly access the related scene. The input is JSON data, and the output is a visually displayed search result.

[1250] This allows users to quickly access the information they need without having to play the entire video.

[1251] (Application example 1)

[1252] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1253] In conventional video content distribution systems, users must watch the entire video, making it difficult to quickly access the information they need. Furthermore, the functionality for searching for specific topics or scenes was insufficient, preventing efficient information retrieval. This made organizing and searching for information cumbersome for long videos, such as educational content or documentaries, and reduced usability.

[1254] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1255] In this invention, the server includes a means for converting audio data from a video into text using a generative artificial intelligence engine, a means for analyzing the video data of the video, dividing the scenes, and creating summaries, and a means for creating chapter candidates based on the generated summaries and scenes. This allows users to quickly search for related scenes within a video by entering specific keywords. Furthermore, by adding a means for generating prompt sentences and sending them to the server, the generated summaries and chapters can be easily understood with high accuracy. This system allows users to efficiently access the information they need without having to watch the entire video.

[1256] "Means for uploading videos" is a function that allows users to send any video file to the system and have it saved on the server.

[1257] "Means for the server to receive the video and store it in temporary storage" refers to a function in which the server receives the uploaded video file and temporarily stores it for subsequent analysis processing.

[1258] "Means for converting video audio data into text using a generative artificial intelligence engine" is a function that uses generative artificial intelligence technology to automatically transcribe audio within a video.

[1259] "Means for analyzing video data of a video, dividing the video into scenes, and creating summaries" refers to a function that analyzes video data, divides the content into multiple scenes, and extracts the important content of each scene as a summary.

[1260] "Means for creating chapter candidates based on the generated summary and scenes" is a function that divides a video into multiple sections (chapters) based on the generated summary and scene information, and assigns titles and summaries to each section.

[1261] The "means for transmitting chapter candidates to a terminal" is a function for transmitting chapter information generated by the server to a user's terminal.

[1262] The "means for displaying chapter candidates received by the terminal to the user" is a function for visually displaying chapter information received on the user's terminal.

[1263] "Means for a user to input a specific keyword into the search bar of a terminal" refers to a function that allows a user to input a keyword they want to search for into the search bar on the terminal.

[1264] The "means for the terminal to transmit a keyword to the server" is a function for transmitting a keyword input from the terminal to the server.

[1265] "Means for analyzing the text data of a video and identifying scenes containing keywords" is a function for analyzing the generated text data and identifying scenes in which the input keywords appear.

[1266] The "means for transmitting the time stamp of the scene to the terminal" is a function for transmitting the time information (time stamp) of the identified scene to the terminal.

[1267] "Means for the terminal to display the timestamp to the user and for the user to access a specified scene" is a function that displays timestamp information on the terminal and allows the user to directly access a specific scene based on that information.

[1268] "Means for generating video summaries using a generative AI model" refers to a function that uses a generative AI model to automatically summarize the content of a video.

[1269] "Means for generating prompt sentences and sending them to the server" is a function that automatically generates input sentences (prompt sentences) for the generative AI model and sends them to the server.

[1270] The present invention provides a function for efficiently viewing and searching video content. Specific embodiments of the present invention will be described below.

[1271] System Overview

[1272] This system combines a server, a user device, and a generative AI model engine. Users upload videos through their devices, and the server analyzes the videos and generates summaries. Users can also search for related scenes within videos using specific keywords.

[1273] Hardware and Software

[1274] Hardware: User devices such as smartphones, tablets, and PCs, as well as servers.

[1275] Software: ffmpeg, moviepy, speech_recognition, transformers library, and generative AI model engine.

[1276] Data processing and calculation

[1277] 1. Upload and save your video

[1278] The user selects a video file and uploads it from the device to the server, which then stores the received video file in temporary storage.

[1279] 2. Speech-to-text

[1280] The server extracts the audio data from the received video and converts it into text using speech recognition technology, using the speech_recognition library.

[1281] 3. Video analysis and summary generation

[1282] The server analyzes the video data and segments it into scenes, using the ffmpeg and moviepy libraries.

[1283] The server uses a generative AI model (e.g., the summarization model from the transformers library) to summarize the key points for each scene.

[1284] 4. Creating chapter suggestions

[1285] The server automatically creates chapter candidates based on the generated summary and scene information, and sends the chapter information to the user device in JSON format.

[1286] 5. Chapter display and search

[1287] The user terminal displays the received chapter information in list form.

[1288] When a user enters a specific keyword into the search bar of the device and clicks the search button, the device sends the keyword to the server.

[1289] The server analyzes the text data of the video, identifies scenes containing the keywords, and sends the timestamp information of those scenes to the terminal.

[1290] The terminal displays the timestamp information to the user, and the user can directly access the specified scene based on the information.

[1291] Specific examples

[1292] For example, suppose a user uploads a video on an educational video streaming service titled "Python Basics." Using this system, the user can:

[1293] Upload a video and its content will be automatically summarized using a generative AI model.

[1294] Each chapter (e.g., "Basic Python Syntax," "Data Structures," and "Functions") is automatically generated and displayed as a list.

[1295] When a user searches for "loop," relevant scenes are displayed with timestamps.

[1296] Users can click on the displayed timestamp to view the corresponding scene directly.

[1297] Prompt Sentence Examples

[1298] The prompt sentence can be summarized as follows: "Python is an easy-to-use and powerful programming language. Many beginners are amazed by its simplicity."

[1299] "Write a summary of this text: 'The programming language Python is easy to use and is the language of choice for many beginners.'"

[1300] This system will make viewing and searching video content more efficient and improve usability.

[1301] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1302] Step 1:

[1303] The user selects and uploads a video.

[1304] Input: The user selects a video file from local storage.

[1305] Output: The video file is sent to the user's device and uploaded to the server.

[1306] Specific operation: The user clicks the upload button on the device and selects a video file from the file selection dialog. The selected video file is sent from the device to the server.

[1307] Step 2:

[1308] The server receives the video and stores it in temporary storage.

[1309] Input: The video file sent from the user device.

[1310] Output: The video file is saved to the server's temporary storage.

[1311] Specific operation: The server stores the received video file in temporary storage and prepares it for subsequent analysis processing.

[1312] Step 3:

[1313] The server converts the video's audio data into text.

[1314] Input: Video file saved in temporary storage.

[1315] Output: Text data generated from the audio data.

[1316] What it does: The server uses the ffmpeg library to extract audio data from the video, then uses the speech_recognition library to convert the audio data to text.

[1317] Step 4:

[1318] The server analyzes the video data, divides the scenes, and creates a summary.

[1319] Input: Video files stored in temporary storage and generated text data.

[1320] Output: Summary text and chapter information for each scene.

[1321] How it works: The server uses the moviepy library to analyze the video data and segment it into scenes, then uses a generative AI model (e.g., the summarization model from the transformers library) to automatically summarize the content of each scene.

[1322] Step 5:

[1323] The server creates chapter candidates based on the generated summary and scenes.

[1324] Input: Summary text and scene information.

[1325] Output: Chapter candidate information (title, summary, timestamp, etc.).

[1326] Specific operation: The server creates chapter candidates in JSON format based on the generated summary and scene information.

[1327] Step 6:

[1328] The server transmits the chapter candidates to the terminal.

[1329] Input: Chapter candidate information.

[1330] Output: Chapter suggestion information sent to the device.

[1331] Specific operation: The server transmits the generated chapter candidate information to the user terminal.

[1332] Step 7:

[1333] The terminal displays the received chapter candidates to the user.

[1334] Input: Chapter candidate information received from the server.

[1335] Output: The chapter list displayed in the user interface.

[1336] Specific operation: The terminal displays the chapter candidate information in a list format on the user interface, allowing access to each chapter.

[1337] Step 8:

[1338] A user types a specific keyword into the search bar.

[1339] Input: The keyword entered by the user.

[1340] Output: Search keywords displayed on the device.

[1341] Specific operation: The user enters a specific keyword into the search bar of the device and clicks the search button.

[1342] Step 9:

[1343] The device sends the keyword to the server.

[1344] Input: The keyword entered in the search bar.

[1345] Output: The search terms sent to the server.

[1346] Specific operation: The terminal sends the keyword entered by the user to the server.

[1347] Step 10:

[1348] The server analyzes the text data of the video and identifies scenes that contain the keywords.

[1349] Input: Search keywords sent to the server and generated text data.

[1350] Output: Timestamp information of identified scenes.

[1351] Specific operation: The server analyzes the generated text data, identifies scenes that contain the search keywords, and generates timestamp information for the identified scenes.

[1352] Step 11:

[1353] The server transmits the timestamp of the identified scene to the terminal.

[1354] Input: The timestamp information of the identified scene.

[1355] Output: Timestamp information sent to the terminal.

[1356] Specific operation: The server transmits the timestamp information of the identified scene to the user terminal.

[1357] Step 12:

[1358] The terminal displays the timestamp to the user, and the user accesses the specified scene.

[1359] Input: The timestamp information received from the server.

[1360] Output: Timestamp information and associated scenes displayed in the user interface.

[1361] Specific operation: The device displays the timestamp information in the user interface, and the user can click on the displayed timestamp to directly access a specific scene.

[1362] The above are the specific processing steps for carrying out the invention.

[1363] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1364] The present invention is a system that automatically analyzes the content of videos and can pinpoint the information a user needs, and combines it with an emotion engine that recognizes the user's emotions to provide more advanced and personalized information. Specific embodiments for carrying out the present invention are described below.

[1365] System Overview

[1366] This system consists of a user, a server, a terminal, and an emotion engine. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the videos by entering specific keywords. Furthermore, the emotion engine recognizes the user's emotions and dynamically adjusts the video playback content and chapter display based on that information.

[1367] Uploading and saving videos

[1368] First, the user selects a video file and uploads it to the system. Once the video file is uploaded, the device sends it to the server. The server then stores the received video file in temporary storage and prepares it for analysis.

[1369] Video analysis and summary generation

[1370] The server then launches a generative AI engine to convert the video's audio data into text. Through this speech recognition process, the video's content is extracted as text data. The server then analyzes the video data and segments each scene. The generative AI engine then summarizes the content of each scene and extracts the main points.

[1371] Creating and viewing chapter suggestions

[1372] The server creates chapter candidates based on the generated summaries and scenes. This automatically divides the video into multiple sections, each with a title and summary. The server sends the chapter candidates in JSON format to the device, which then displays them visually to the user.

[1373] Reverse search function from words

[1374] The user enters a specific keyword in the search bar and clicks the search button. The device sends the keyword to the server. The server analyzes the text data of the video and identifies the scene in which the keyword appears. Once the timestamp of the relevant scene is identified, the server sends that information to the device. The device displays the search results, including the timestamp, to the user, allowing the user to directly access the specified scene.

[1375] Emotion recognition function

[1376] The system further enhances the user experience by incorporating an emotion engine. While the user is watching a video, a camera built into the device captures real-time video of the user. This video data is sent to a server, which then uses the emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial expressions and movements to identify their emotional state (e.g., joy, sadness, excitement, etc.).

[1377] The server can dynamically adjust the video playback and chapters displayed based on the emotional data it acquires. For example, if it determines that the user is interested, it can provide more information or supplemental videos on related topics. On the other hand, if it determines that the user is bored, it can present the option to skip to important scenes.

[1378] Specific examples

[1379] For example, if a user uploads a video about "latest tech trends," the process goes like this:

[1380] 1. Upload and analyze

[1381] A user uploads a video about "latest tech trends."

[1382] The device sends the video file to the server.

[1383] The server receives the video, stores it in temporary storage, and begins analysis.

[1384] 2. Summarizing and Chapter Creation

[1385] The server uses a generative artificial intelligence engine to generate a summary of the video.

[1386] The server creates chapter candidates based on each technical topic.

[1387] 3. Display results

[1388] The server sends the summary and chapter information to the device in JSON format.

[1389] The terminal displays this to the user in a list format.

[1390] 4. Reverse Search Feature

[1391] A user types "blockchain" into the search bar and clicks the search button.

[1392] The device sends the keyword to the server.

[1393] The server analyzes the video transcript, identifies scenes in which "blockchain" appears, and sends that information to the device.

[1394] The terminal displays the search results to the user in a list format, allowing the user to directly access related scenes.

[1395] 5. Emotion recognition

[1396] While the user is watching the video, the device's camera captures real-time video.

[1397] The server analyzes the video data and uses an emotion engine to recognize the user's emotions.

[1398] Based on the emotional data, the video playback content and chapter display are dynamically adjusted.

[1399] The system not only allows users to instantly access the information they need without having to play the entire video, but also provides a personalized viewing experience through emotion recognition.

[1400] The processing flow will be explained below.

[1401] Step 1:

[1402] The user selects a video file and uploads it to the system. The user uses a file selection dialog to select a video file from their computer and clicks the "Upload" button.

[1403] Step 2:

[1404] The device sends the selected video file to the server. The device obtains the binary data of the video file and sends it to the server using an HTTP POST request.

[1405] Step 3:

[1406] The server receives the video file and stores it in temporary storage. The server receives the video file, checks the integrity of the data, and then stores it in a temporary storage folder.

[1407] Step 4:

[1408] The server starts the generative artificial intelligence engine. The server calls the generative artificial intelligence engine to start the process of converting audio data into text for the prepared video file.

[1409] Step 5:

[1410] The server converts the audio data into text, and a generative AI engine analyzes the audio in the video and generates recognized text data.

[1411] Step 6:

[1412] The server analyzes the video data, extracts each frame of the video, and segments it into scenes using a video recognition algorithm.

[1413] Step 7:

[1414] The server generates a summary for each scene using the generated text and video data. The content is reviewed to ensure that key points and themes are included.

[1415] Step 8:

[1416] The server creates chapter candidates. Based on the generated summaries and scene information, the server assigns titles and summaries to each segment and creates chapter candidates.

[1417] Step 9:

[1418] The server sends the chapter candidates to the device. The server encodes the completed chapter candidates in JSON format and sends them to the device as an HTTP response.

[1419] Step 10:

[1420] The device parses the received chapter suggestions and displays them on the user interface. The device parses the JSON data and displays them to the user in list or timeline format.

[1421] Step 11:

[1422] A user enters a specific keyword into the search bar and clicks the search button.A user enters a keyword into the input field of the search bar and presses the button to execute the search command.

[1423] Step 12:

[1424] The device sends the keyword to the server. The device creates an HTTP GET request including the user's keyword and sends it to the server.

[1425] Step 13:

[1426] The server analyzes the text data of the video and identifies the scenes in which the keywords appear. Using a full-text search algorithm, the server extracts text fragments containing the keywords and obtains the timestamps of the corresponding scenes.

[1427] Step 14:

[1428] The server sends the timestamp to the device. The server encodes the timestamp and scene information identified as search results in JSON format and sends it to the device as an HTTP response.

[1429] Step 15:

[1430] The device displays the search results to the user. The device displays the received timestamps and scene information in a list format, and provides a "Play" button for each entry.

[1431] Step 16:

[1432] The user clicks the play button to access the specified scene. When the user clicks the play button in the displayed list, the video player starts playing from the corresponding scene.

[1433] Step 17:

[1434] The device acquires real-time video data of the user. The device's camera captures the user's face and movements and sends the data to the server.

[1435] Step 18:

[1436] The server starts the emotion engine. The server invokes the emotion engine to analyze the received video data.

[1437] Step 19:

[1438] The server analyzes the user's emotions, and the emotion engine analyzes the user's facial expressions and movements to identify their emotional state, such as joy, sadness, or excitement.

[1439] Step 20:

[1440] The server uses the emotion analysis results to dynamically adjust video playback and chapter display based on the emotion data.

[1441] Step 21:

[1442] The server sends adjustments based on emotion data to the device, and the server sends dynamically adjusted information to the device, which then visually displays it.

[1443] Step 22:

[1444] The device displays a display that reflects the emotional data to the user. The device displays a list that reflects the emotional data and adjusts the video player to the user, allowing the user to have an optimal viewing experience.

[1445] Example 2

[1446] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1447] While conventional video analysis systems have the ability to summarize videos and search for specific scenes, they lack the ability to dynamically change content based on the user's emotional state. Furthermore, while they enable users to quickly obtain specific information, they have the problem of being unable to respond to the user's interests and emotions while watching, because the video playback content is static. Therefore, there is a need for a system that can individually optimize the user's viewing experience and efficiently provide the information they need.

[1448] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1449] In this invention, the server includes means for converting audio data of a video into text using a generative artificial intelligence engine, means for analyzing video data of the video and dividing it into scenes to create a summary, means for creating chapter candidates based on the created summary and scenes, and means for recognizing the user's emotional state using an emotion engine and dynamically adjusting the video playback content and chapter display. This enables a personalized viewing experience according to the user's emotions and makes it possible to efficiently provide necessary information.

[1450] "Video" means a digital file containing audio and video information that a user uploads to the system.

[1451] The "server" is a central processing unit that receives video, temporarily stores it, and analyzes it using a generative artificial intelligence engine and an emotion engine.

[1452] A "generative artificial intelligence engine" is an AI technology that converts audio data from videos into text and analyzes video data to create scene summaries.

[1453] "Voice recognition technology" is a technology for converting audio data from video into text data.

[1454] "Video data" refers to digital data that constitutes the visual information of a moving image.

[1455] "Scene segmentation" is a method of dividing a continuous video sequence into multiple stages and analyzing each section independently.

[1456] A "summary" is text information that briefly summarizes the main content of each scene in a video.

[1457] "Chapter candidates" refer to the individual parts of a video divided into multiple sections based on summarized scene information.

[1458] "Emotion recognition" is the process of analyzing a user's facial expressions, movements, and other facial expressions to identify their emotional state.

[1459] "Terminal" means a device through which a user accesses the system to upload, search, and play videos.

[1460] A "keyword" is a character string that a user enters to search for specific information.

[1461] A "timestamp" is information that indicates the start time of a particular scene in a video.

[1462] "Dynamic adjustment" refers to changing the video playback content and chapter display in real time based on the user's emotion recognition results.

[1463] The present invention is an advanced information provision system that combines video analysis and emotion recognition, and specific embodiments are described in detail below. This system allows users to upload videos and automatically analyze the content of those videos to efficiently obtain the information users need. The present invention includes a server, a terminal, and an emotion engine. Ultimately, it realizes dynamic information provision according to the user's emotions.

[1464] Uploading and saving videos

[1465] A user uses a device to select and upload a video file. The device then sends the video file to a server. The server receives the video file and stores it in temporary storage, preparing the video for analysis. For example, if a user uploads a video of a seminar on technology trends, the device sends the file to the server, which stores it.

[1466] Video analysis and summary generation

[1467] The server launches a generative AI engine to begin analyzing the video. First, the audio data is converted into text using speech recognition technology (e.g., Google Speech-to-Text API). Next, the video data is analyzed and divided into scenes. For each divided scene, a generative AI model (e.g., GPT-4) is used to generate a summary. This allows the main points of the video to be concisely extracted.

[1468] Creating chapter suggestions

[1469] The server creates chapter candidates based on the generated summary and split scenes. Each chapter is divided into sections and given a title and summary. The server sends this information to the device in JSON format, and the device displays it to the user. For example, if a video on "latest technology trends" is uploaded, the server generates chapters based on technology topics and displays them to the user.

[1470] Keyword search function

[1471] The user enters a specific keyword into the search bar on the device and clicks the search button. The device then sends the keyword to the server. The server analyzes the text data of the video and identifies the scene in which the keyword appears. The timestamp information of the relevant scene is sent to the device and displayed to the user. The user can then directly access the relevant scene from the search results.

[1472] Emotion recognition function

[1473] The system incorporates an emotion engine that analyzes the user's emotions in real time. While the user is watching a video, a camera built into the device captures real-time footage of the user and sends it to the server. The server uses the emotion engine to analyze the user's emotional state and dynamically adjusts the video playback content and chapter display. For example, if the system determines that the user is excited, it provides related information or supplementary videos. If the system determines that the user is bored, it presents an option to skip to important scenes.

[1474] As a concrete example, when a user uploads a video about "latest technology trends," the process proceeds as follows:

[1475] 1. The user uploads a video and the device sends it to the server.

[1476] 2. The server stores the video and begins analysis, converting the audio data into text and analyzing the video data to generate a summary.

[1477] 3. The server creates chapter candidates and sends them to the device, which displays them to the user.

[1478] 4. The user searches by keyword, and the server sends the timestamp of the corresponding scene to the device. The device displays the search results, and the user can access the corresponding scene.

[1479] 5. While the user is watching the video, the device camera captures their emotions. The server analyzes this using an emotion engine and dynamically adjusts the playback content.

[1480] In this way, the system allows users to quickly access the information they need without having to play the entire video, and also provides a personalized viewing experience through emotion recognition.

[1481] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1482] Step 1:

[1483] A user selects a video file on a terminal and uploads it to the system. Specifically, the user selects a video file from the user interface and presses the upload button. The input at this time is the selected video file, and the output is the terminal sending the video file to the server.

[1484] Step 2:

[1485] The device sends the video file selected by the user to the server. It receives the video file uploaded by the user as input and sends it to the server as output. Specifically, the device generates an HTTP request, attaches the video file, and sends it to the server.

[1486] Step 3:

[1487] The server receives the video file and stores it in temporary storage. It receives the video file sent from the terminal as input and stores it in temporary storage as output. Specifically, the server uses a file management system to store the video file in a specified directory.

[1488] Step 4:

[1489] The server starts a generative artificial intelligence engine and converts the video's audio data into text using speech recognition technology. It receives the saved video file as input and obtains the generated text data as output. Specifically, the server calls the speech recognition API, sends the audio data as input, and receives the text data.

[1490] Step 5:

[1491] The server analyzes the video data, segments the scenes, and creates summaries. It receives the video file and generated text data as input, and generates segmented scenes and their summaries as output. Specifically, the server applies a video analysis algorithm to identify the video frames for each scene and creates a summary using a generative AI model.

[1492] Step 6:

[1493] The server creates chapter candidates based on the generated summaries and scenes. It receives scenes and their summaries as input and generates chapter information as output. Specifically, the server aggregates the summaries for each scene and generates chapter titles and summaries in JSON format.

[1494] Step 7:

[1495] The server sends the generated chapter information to the terminal. It receives the generated chapter information as input and sends it to the terminal as output. Specifically, the server generates an HTTP response, attaches the chapter information, and sends it to the terminal.

[1496] Step 8:

[1497] The terminal displays the received chapter candidates to the user. It receives the chapter information received from the server as input and displays it on the user interface as output. Specifically, the terminal analyzes the chapter information and displays it in list format.

[1498] Step 9:

[1499] The user enters a specific keyword into the search bar of the device and clicks the search button. The keyword is received as input and sent to the server as output. The specific operation is that the user enters a keyword into the search bar and clicks the search button.

[1500] Step 10:

[1501] The terminal sends the keyword to the server. It receives the keyword entered by the user as input and sends it to the server as output. Specifically, the terminal generates an HTTP request, attaches the keyword, and sends it to the server.

[1502] Step 11:

[1503] The server analyzes the text data of the video to identify scenes containing the keywords. Using the keywords received as input and the text data of the video, it generates a timestamp for the relevant scene as output. Specifically, the server searches the text data for keywords and identifies the start time of the relevant scene.

[1504] Step 12:

[1505] The server sends the scene timestamp to the terminal. It receives the timestamp of the specified scene as input and sends it to the terminal as output. As a specific operation, the server generates an HTTP response, attaches the timestamp information, and sends it to the terminal.

[1506] Step 13:

[1507] The terminal displays the timestamp to the user, and the user accesses the specified scene. The terminal receives the timestamp information from the server as input, displays it on the user interface as output, and accesses the specified scene. As a specific operation, the terminal uses a playback player to play the video from the specified scene.

[1508] Step 14:

[1509] The camera built into the device captures real-time video of the user and sends it to the server. The device acquires real-time video of the user as input and sends it to the server as output. Specifically, the device activates the camera module, captures video data, and sends it to the server.

[1510] Step 15:

[1511] The server uses an emotion engine to recognize the user's emotional state and dynamically adjust the video playback content and chapter display. It receives the transmitted video data as input and generates emotion recognition results and dynamic adjustment results as output. Specifically, the emotion engine analyzes the video data, identifies the user's emotional state, and changes the video playback content accordingly.

[1512] Step 16:

[1513] Based on the emotion recognition, the server dynamically adjusts the video playback content and chapter display and sends them to the terminal. The emotion recognition results are received as input, and the adjustments are sent to the terminal as output. Specifically, the server generates adjustment instructions and sends them to the terminal, which then changes the video playback content as appropriate.

[1514] (Application example 2)

[1515] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1516] The challenge is to provide users with a more effective and personalized advertising experience by appropriately recognizing users' interests and emotions while watching advertising videos and dynamically adjusting the advertising content based on that information. Conventional advertising systems provide uniform advertising content without considering the user's emotional state, making it difficult to maintain user interest. Furthermore, there is no mechanism for users to quickly search for related information using specific keywords, resulting in a poor user experience.

[1517] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for automatically analyzing the content of the advertising video being viewed by the user, means for capturing real-time video of the user using the camera of the terminal, means for the server to analyze the video data and recognize the user's emotions using an emotion engine, and means for the server to dynamically adjust the advertising content based on the recognized emotion information. This enables a personalized advertising experience that matches the user's emotional state.

[1518] "User" means a person or organization that uses the system to upload videos and receive the analyzed results.

[1519] "Server" is a central computer system that processes and analyzes video data received from users, and stores and transmits the results.

[1520] "Temporary storage" is a storage device that temporarily stores video when the server receives it.

[1521] A "generative artificial intelligence engine" is an analytical engine equipped with AI technology that analyzes the content of videos, converts audio data into text, and generates summaries.

[1522] "Audio data" is digital data that includes audio information emitted within a video.

[1523] "Text data" refers to data obtained by converting voice data into text format.

[1524] "Video data" is digital data that contains visual information within a moving image.

[1525] A "scene" refers to a specific time range within a video, and is a video portion with independent content.

[1526] "Chapter candidates" are sections divided into scenes that are generated as a result of analyzing a video, and each section is given a title and a summary.

[1527] "Terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[1528] A "search bar" is a UI component that allows a user to enter specific keywords.

[1529] "Keywords" refer to specific words or phrases that users use to search for specific scenes within a video.

[1530] A "timestamp" is digital information that indicates the start time of a particular scene.

[1531] The "Emotion Engine" is an analytical engine equipped with AI technology that analyzes real-time video footage of users and identifies their emotional state.

[1532] An "advertising video" is a video created to promote a particular product or service.

[1533] "Real-time video" refers to video data of a user captured in real time.

[1534] "Emotion information" is data that indicates the user's emotional state analyzed by the emotion engine.

[1535] "Dynamic adjustment" means changing the content and presentation in real time depending on the situation.

[1536] Specific embodiments for carrying out the present invention are described below.

[1537] System configuration

[1538] The system consists of a user, a server, a terminal, and an emotion engine. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the video by entering specific keywords. Furthermore, the emotion engine recognizes the user's emotions and dynamically adjusts the playback content and chapter display of the advertising video based on that information.

[1539] Uploading and saving videos

[1540] The user uploads the advertising video to the device. The device then sends the video file to the server, which stores it in temporary storage. In this process, the server and device hardware are typically general-purpose computers or smartphones.

[1541] Video analysis and summary generation

[1542] The server then launches a generative AI engine to convert the audio data from the ad video into text. Through this speech recognition process, the video content is extracted as text data. The server then analyzes the video data and segments each scene. The generative AI engine generates a summary of each scene and extracts the main points. Transformers from Hugging Face are used for speech recognition.

[1543] Creating and viewing chapter suggestions

[1544] The server creates chapter candidates based on the generated summaries and scenes. This automatically divides the video into multiple sections, each with a title and summary. The chapter information is sent to the device in JSON format, and the device displays it visually to the user.

[1545] Keyword search function

[1546] The user enters a specific keyword into the device's search bar and clicks the search button. The device then sends the keyword to the server. The server analyzes the text data in the video and identifies scenes that contain the keyword. Once the timestamp of the relevant scene is identified, that information is sent to the device. The device then displays the search results, including the timestamp, to the user, allowing the user to directly access the specified scene.

[1547] Emotion recognition and dynamic ad adjustment

[1548] The system further enhances the user experience by incorporating an emotion engine. While the user is watching the advertising video, a camera built into the device captures real-time video of the user. This video data is sent to a server, which then uses an emotion engine to recognize the user's emotions. The emotion engine uses services such as Amazon Rekognition to analyze the user's facial expressions and movements to identify their emotional state.

[1549] The server dynamically adjusts the content of the ad video and the chapters displayed based on the acquired emotional data. For example, if the server determines that the user is interested, it may provide more information or supplemental videos on related topics. On the other hand, if the server determines that the user is bored, it may present the option to skip to important scenes.

[1550] Specific examples

[1551] As an example, if a user uploads an advertising video about "latest smartphone technology," the process would proceed as follows:

[1552] 1. Upload and analyze

[1553] A user uploads an advertising video for the "latest smartphone technology."

[1554] The device sends the video file to the server.

[1555] The server receives the video, stores it in temporary storage, and begins analysis.

[1556] 2. Summarizing and Chapter Creation

[1557] The server uses a generative artificial intelligence engine to generate a summary of the video.

[1558] The server creates chapter candidates based on each technical topic.

[1559] 3. Display results

[1560] The server sends the summary and chapter information to the device in JSON format.

[1561] The terminal displays this to the user in a list format.

[1562] 4. Reverse Search Feature

[1563] A user types "camera features" into the search bar and clicks the search button.

[1564] The device sends the keyword to the server.

[1565] The server analyzes the video transcription data, identifies scenes in which the "camera function" appears, and sends that information to the device.

[1566] The terminal displays the search results to the user in a list format, allowing the user to directly access related scenes.

[1567] 5. Emotion recognition

[1568] While the user is watching the advertising video, the camera on the terminal captures real-time video.

[1569] The server analyzes the video data and uses an emotion engine to recognize the user's emotions.

[1570] Based on the emotional data, the playback content and chapter display of the advertising video are dynamically adjusted, improving the user experience and maximizing the effectiveness of the advertisement.

[1571] Example prompts

[1572] The prompt sentences below are examples used in the emotion-sensitive ad viewer.

[1573] "Identify exciting moments in advertising videos and highlight relevant technology demos."

[1574] "If the user expresses interest, provide additional information related to the topic."

[1575] This system not only allows users to instantly access the information they need without having to watch the entire advertisement, but also provides a personalized viewing experience through emotion recognition.

[1576] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1577] Step 1:

[1578] A user uploads an advertising video.

[1579] The user selects an advertising video file through the terminal interface and uploads it to the system, at which point the terminal sends the selected video file to the server, where the input is the video file provided by the user and the output is the video file received by the server.

[1580] Step 2:

[1581] The server stores the video in temporary storage.

[1582] The server temporarily stores the received video file in temporary storage. The input is the video file sent from the device, and the output is the video file saved in the temporary storage.

[1583] Step 3:

[1584] The server launches a generative artificial intelligence engine to convert the video's audio data into text.

[1585] The server uses a generative artificial intelligence engine to convert the audio data in the video into text data. The input is the video file loaded from temporary storage, and the output is text data. This process uses voice recognition technology.

[1586] Step 4:

[1587] The server analyzes the video data, divides the scenes, and creates summaries.

[1588] The server analyzes the video data and divides it into multiple scenes. It then uses a generative AI engine to create a summary of each scene and extract key points. The input is text data and video data, and the output is a summary of each scene and chapter suggestions.

[1589] Step 5:

[1590] The server creates chapter candidates based on the generated summary and scenes.

[1591] The server generates chapter candidates, each containing a title and summary, based on the generated summaries and the divided scenes. The input is scene summary data, and the output is chapter candidate data.

[1592] Step 6:

[1593] The server transmits the chapter candidates to the terminal.

[1594] The server sends the created chapter candidates to the terminal. The input is the chapter candidate data, and the output is the chapter candidate data received by the terminal.

[1595] Step 7:

[1596] The terminal displays the received chapter candidates to the user.

[1597] The terminal visually displays the chapter candidates received from the server to the user in the form of a text list. The input is the received chapter candidate data, and the output is the chapter list displayed to the user.

[1598] Step 8:

[1599] The user enters a specific keyword into the device's search bar.

[1600] The user inputs a specific keyword through the search bar and clicks the search button. The input is the keyword input by the user, and the output is the keyword data sent from the terminal to the server.

[1601] Step 9:

[1602] The device sends the keyword to the server.

[1603] The terminal sends the keyword received from the user to the server. The input is the keyword entered by the user, and the output is the keyword data sent to the server.

[1604] Step 10:

[1605] The server analyzes the text data of the video and identifies scenes that contain the keywords.

[1606] The server analyzes the text data of the video and identifies scenes that contain the keywords entered by the user. The input is keyword data and text data, and the output is scene information that contains the keywords.

[1607] Step 11:

[1608] The server sends the scene timestamp to the device.

[1609] The server sends the timestamp of the identified scene to the terminal. The input is the scene information, and the output is the timestamp data sent to the terminal.

[1610] Step 12:

[1611] The terminal displays the timestamp to the user, and the user accesses the specified scene.

[1612] The terminal displays the received timestamp to the user, allowing the user to directly access the specified scene. The input is the timestamp data received from the server, and the output is the timestamp information displayed to the user.

[1613] Step 13:

[1614] A real-time image of the user is acquired using the device's camera.

[1615] While the user is watching the advertisement video, the camera of the terminal captures the user's real-time video. The input is the video data from the camera, and the output is the video data sent to the server.

[1616] Step 14:

[1617] The server analyzes the video data and uses an emotion engine to recognize the user's emotions.

[1618] The server analyzes the acquired video data using an emotion engine to identify the user's emotional state. The input is the video data received from the device, and the output is the emotion recognition result.

[1619] Step 15:

[1620] The server dynamically adjusts the advertising content based on the recognized emotion information.

[1621] The server dynamically adjusts the playback content of the advertising video and the chapters to be displayed based on the emotional information analyzed by the emotion engine. The input is the emotion recognition result, and the output is the adjusted advertising content.

[1622] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1623] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1624] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1625] [Fourth embodiment]

[1626] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1627] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1628] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1629] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1630] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1631] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1632] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1633] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1634] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1635] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1636] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1637] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1638] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1639] The present invention is a system that automatically analyzes the content of a video and can pinpoint the information that a user needs. Specific embodiments for carrying out the present invention will be described below.

[1640] System Overview

[1641] This system consists of users, a server, and terminals. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the videos by entering specific keywords.

[1642] Uploading and saving videos

[1643] First, the user selects a video file and uploads it to the system. Once the video file is uploaded, the device sends it to the server. The server then stores the received video file in temporary storage and prepares it for analysis.

[1644] Video analysis and summary generation

[1645] The server then launches a generative AI engine to convert the video's audio data into text. Through this speech recognition process, the video's content is extracted as text data. The server then analyzes the video data and segments each scene. The generative AI engine then summarizes the content of each scene and extracts the main points.

[1646] Creating and viewing chapter suggestions

[1647] The server creates chapter candidates based on the generated summaries and scenes. This automatically divides the video into multiple sections, each with a title and summary. The server sends the chapter candidates in JSON format to the device, which then displays them visually to the user.

[1648] Reverse search function from words

[1649] The user enters a specific keyword in the search bar and clicks the search button. The device sends the keyword to the server. The server analyzes the text data of the video and identifies the scene in which the keyword appears. Once the timestamp of the relevant scene is identified, the server sends that information to the device. The device displays the search results, including the timestamp, to the user, allowing the user to directly access the specified scene.

[1650] Specific examples

[1651] For example, consider a user who uploads a video about "Programming Basics," which includes examples of multiple programming languages ​​and their usage.

[1652] 1. Upload and analyze

[1653] A user uploads a video on "Programming Basics."

[1654] The terminal transmits the selected video file to the server.

[1655] The server receives the video, stores it in temporary storage, and begins the analysis process.

[1656] 2. Summarizing and Chapter Creation

[1657] The server uses a generative artificial intelligence engine to summarize the content of the video.

[1658] The server creates chapters based on each language segment (e.g. "Python Basics", "JavaScript Basics", etc.).

[1659] 3. Display results

[1660] The server sends the summary and chapter information to the device in JSON format.

[1661] The terminal displays this to the user in a list format.

[1662] 4. Reverse Search Feature

[1663] A user types "Python" into the search bar and clicks the search button.

[1664] The device sends the keyword to the server.

[1665] The server analyzes the video transcription data, identifies scenes in which "Python" appears, and sends relevant information to the device.

[1666] The device displays the search results in a list format, allowing the user to directly access related scenes.

[1667] This system allows users to quickly access the information they need without having to play the entire video, making it possible to use videos efficiently.

[1668] The processing flow will be explained below.

[1669] Step 1:

[1670] The user selects a video file and uploads it to the system. The user selects a video file from their computer using the file selection dialog and clicks the "Upload" button.

[1671] Step 2:

[1672] The device sends the selected video file to the server. The device obtains the binary data of the video file and sends it to the server using an HTTP POST request.

[1673] Step 3:

[1674] The server receives the video file and stores it in temporary storage. The server receives the video file, checks the integrity of the data, and then stores it in a temporary storage folder.

[1675] Step 4:

[1676] The server starts the generative artificial intelligence engine. The server calls the generative artificial intelligence engine to start the process of converting audio data into text for the prepared video file.

[1677] Step 5:

[1678] The server converts the audio data into text, and a generative AI engine analyzes the audio in the video and generates recognized text data.

[1679] Step 6:

[1680] The server analyzes the video data, extracts each frame of the video, and segments it into scenes using a video recognition algorithm.

[1681] Step 7:

[1682] The server generates a summary for each scene using the generated text and video data. The content is reviewed to ensure that key points and themes are included.

[1683] Step 8:

[1684] The server creates chapter candidates. Based on the generated summary and scene information, the server assigns a title and summary to each segment and creates chapter candidates.

[1685] Step 9:

[1686] The server sends the chapter candidates to the device. The server encodes the completed chapter candidates in JSON format and sends them to the device as an HTTP response.

[1687] Step 10:

[1688] The device parses the received chapter suggestions and displays them on the user interface. The device parses the JSON data and displays them to the user in list or timeline format.

[1689] Step 11:

[1690] A user enters a specific keyword into the search bar and clicks the search button.A user enters a keyword into the input field of the search bar and presses the button with the intention of a search command.

[1691] Step 12:

[1692] The device sends the keyword to the server. The device creates an HTTP GET request including the user's keyword and sends it to the server.

[1693] Step 13:

[1694] The server analyzes the text data of the video and identifies the scenes in which the keywords appear. Using a full-text search algorithm, the server extracts text fragments containing the keywords and obtains the timestamps of the corresponding scenes.

[1695] Step 14:

[1696] The server sends the timestamp to the device, and the server returns the identified timestamp and scene information in JSON format to the device as search results.

[1697] Step 15:

[1698] The device displays the search results to the user. The device displays the received timestamps and scene information in a list format, and provides a "Play" button for each entry.

[1699] Step 16:

[1700] The user clicks the play button to access the specified scene. When the user clicks the play button in the displayed list, the video player starts playing from the corresponding scene.

[1701] Example 1

[1702] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1703] In today's world, the proliferation of video content has made it difficult for users to quickly and efficiently obtain the information they need. Long video playback times also create inefficiencies in information retrieval. Furthermore, the lack of a way for users to quickly search for specific information of interest within a video limits the viewing experience.

[1704] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1705] In this invention, the server includes: means for a user to upload a video; means for a terminal to transmit a video file to the server; means for the server to store the video file in temporary storage; means for the server to convert audio data of the video into text using speech recognition technology; means for the server to segment video data of the video by scene using video analysis technology; means for the server to generate a summary from the text data and scene information using a generative artificial intelligence engine; means for the server to create chapter candidates based on the generated summary and scene information; means for the server to transmit the chapter candidates to the terminal in JSON format; means for the terminal to visually display the received chapter candidates to the user; means for the user to input a specific keyword into a search bar of the terminal; means for the terminal to transmit the keyword to the server; means for the server to analyze the text data and identify scenes containing the keyword; means for the server to transmit scene timestamp information to the terminal; and means for the terminal to display the timestamp information to the user and allow the user to access the specified scene. This allows the user to quickly access necessary information without playing the entire video.

[1706] "User" means an individual or entity that uses the system to upload videos and search for specific information.

[1707] A "terminal" is an electronic device, such as a computer or smartphone, that a user uses to upload videos or view search results.

[1708] A "server" is a computer system that stores videos sent by users and performs analysis processing.

[1709] "Temporary storage" refers to data storage used by the server to temporarily store video files.

[1710] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[1711] "Video analysis technology" is a technology that analyzes video data and identifies specific scenes and frames.

[1712] A "generative AI engine" is a type of artificial intelligence used to generate new information or summaries based on input data.

[1713] "Text data" is character-based data that represents the content of a video extracted using voice recognition technology.

[1714] A "scene" is a segment that separates parts of a video where the content changes.

[1715] "Chapter candidates" are multiple sections generated based on the video content, and each section is given a title and a summary.

[1716] The "JSON format" is one of the standard formats for structuring and describing data in text format.

[1717] The "search bar" is an input area where users can input specific keywords to search for related information within a video.

[1718] A "timestamp" is information that indicates a specific time point within a video.

[1719] The present invention is a system that automatically analyzes the content of a video and can pinpoint the information that a user needs. Specific embodiments for carrying out the present invention will be described below.

[1720] System Overview

[1721] This system consists of users, a server, and terminals. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the videos by entering specific keywords.

[1722] Uploading and saving videos

[1723] First, the user selects a video file and uploads it to the system. To do this, the user uses a device such as a PC or smartphone. The device then sends the selected video file to the server. The server then stores the received video file in temporary storage (for example, Amazon S3 or Google Cloud Storage) and prepares it for analysis.

[1724] Video analysis and summary generation

[1725] The server then analyzes the video using voice recognition and video analysis technology. The specific process is as follows:

[1726] 1. Audio to text conversion:

[1727] The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to convert the audio data in the video into text data. Through this process, the content of the video is extracted as a string of characters.

[1728] 2. Scene segmentation of video data:

[1729] The server uses video analysis technology (e.g., OpenCV library) to analyze the video data and segment each scene. Specifically, it detects change points between frames in the video and recognizes different scenes.

[1730] 3. Generate a summary:

[1731] The server uses a generative artificial intelligence engine (e.g., OpenAI GPT) to generate a summary of each scene based on the extracted text data and scene information, concisely summarizing the key points and content.

[1732] Creating and viewing chapter suggestions

[1733] Based on the generated summary and scene information, the server creates chapter candidates that divide the video into multiple sections. Each section is then given a title and summary. The server converts the chapter candidates into JSON format and sends them to the device. The device parses the received JSON data and displays the chapter candidates in a list format on the user interface. The user can click on one of them to directly access a specific scene.

[1734] Keyword search function

[1735] The user enters a specific keyword into the system's search bar and clicks the search button. The device sends this keyword to the server. The server analyzes the text data of the video and identifies the scenes in which the keyword appears. It obtains the timestamp information of the relevant scenes and sends it to the device in JSON format. The device analyzes the received data and displays the search results in a list format on the user interface. The user can click on this to directly access the relevant scenes.

[1736] Specific examples

[1737] Here, we will use a video on "Programming Basics" as an example.

[1738] 1. Upload and analyze:

[1739] A user uploads a video on "Programming Basics."

[1740] The device sends the selected video to the server, which stores the video in temporary storage.

[1741] The server converts the audio data into text using the Google Cloud Speech-to-Text API, and splits the video data into scenes using OpenCV.

[1742] The server creates chapter candidates based on the summary generated using OpenAI GPT and sends them to the device.

[1743] 2. Results display:

[1744] The device receives the chapter information in JSON format and displays it in a list format on the user interface.

[1745] 3. Keyword Search:

[1746] A user types "Python" into the search bar and clicks the search button.

[1747] The device sends the keyword to the server, and the server analyzes the text data of the video to identify the relevant scene, obtains timestamp information, and sends it to the device.

[1748] The device displays the search results in a list format, allowing the user to directly access related scenes.

[1749] Prompt Sentence Examples

[1750] Here are some example prompts to enter into a generative AI engine:

[1751] "This video explains the basics of programming. Please summarize the basic usage of each programming language scene by scene."

[1752] "Please summarize the sections on Python, JavaScript, and Java basics for each scene."

[1753] This system allows users to quickly access the information they need without having to play the entire video.

[1754] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1755] Step 1:

[1756] The user selects a video file and uploads it to the system via their device. The user selects a video file through a web browser on a device such as a PC or smartphone. The selected video file is sent from the device to the server. The input is the video file, and the output is the transmission of the video file to the server.

[1757] Step 2:

[1758] The device sends the selected video file to the server. Specifically, it uploads the video file to the server using an HTTP POST request. The input is the video file selected by the user, and the output is the video file sent to the server.

[1759] Step 3:

[1760] The server stores the received video file in temporary storage. The server stores the received video file in storage such as Amazon S3 or Google Cloud Storage, and records the file path and file name in a database. The input is the video file received from the device, and the output is saving to storage and recording to the database.

[1761] Step 4:

[1762] The server uses speech recognition technology to convert the audio data in the video into text. Specifically, it uses the Google Cloud Speech-to-Text API to convert the audio data in the video file into text. The input is the audio data in the video file, and the output is the converted text data.

[1763] Step 5:

[1764] The server uses video analysis technology to divide the video data into scenes. Specifically, it uses the OpenCV library to analyze the video data frame by frame and detect different scenes. The input is the video data of the video file, and the output is the divided scene information.

[1765] Step 6:

[1766] The server uses a generative AI engine to generate summaries based on the text data and scene information. Specifically, it uses OpenAI GPT to summarize the key points of each scene. The input is the text data for each scene, and the output is a summary of each scene.

[1767] Step 7:

[1768] The server creates chapter candidates based on the generated summaries and scene information. The server combines the summaries and scene information, divides the video into multiple sections, and assigns titles and summaries to each section. The input is the summaries and scene information, and the output is chapter candidates.

[1769] Step 8:

[1770] The server sends chapter candidates to the device in JSON format. The server converts the generated chapter information into JSON format and sends it to the device as an HTTP response. The input is the chapter candidates, and the output is the JSON data sent to the device.

[1771] Step 9:

[1772] The device displays the received chapter candidates on the user interface. The device parses the received JSON data and displays the chapter candidates in list format on the user interface. The input is JSON data, and the output is a visual representation of the chapter candidates.

[1773] Step 10:

[1774] The user enters a specific keyword into the search bar of the device. The user enters the keyword of interest and clicks the search button. The input is the keyword, and the output is the transmission of the keyword.

[1775] Step 11:

[1776] The terminal sends a keyword to the server. The terminal sends the entered keyword to the server via an HTTP POST request. The input is the keyword, and the output is the keyword sent to the server.

[1777] Step 12:

[1778] The server analyzes the text data to identify scenes containing keywords. The server analyzes the text data of the video and obtains the scenes in which the keywords appear and their timestamp information. The input is the keywords and text data, and the output is the timestamp information of the identified scenes.

[1779] Step 13:

[1780] The server sends the scene timestamp information to the terminal. The server converts the acquired timestamp information into JSON format and sends it to the terminal as an HTTP response. The input is the timestamp information, and the output is the JSON data sent to the terminal.

[1781] Step 14:

[1782] The terminal displays the timestamp information on the user interface, and the user accesses the specified scene. The terminal parses the received JSON data and displays the search results in a list format. The user can click on it to directly access the related scene. The input is JSON data, and the output is a visually displayed search result.

[1783] This allows users to quickly access the information they need without having to play the entire video.

[1784] (Application example 1)

[1785] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1786] In conventional video content distribution systems, users must watch the entire video, making it difficult to quickly access the information they need. Furthermore, the functionality for searching for specific topics or scenes was insufficient, preventing efficient information retrieval. This made organizing and searching for information cumbersome for long videos, such as educational content or documentaries, and reduced usability.

[1787] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1788] In this invention, the server includes a means for converting audio data from a video into text using a generative artificial intelligence engine, a means for analyzing the video data of the video, dividing the scenes, and creating summaries, and a means for creating chapter candidates based on the generated summaries and scenes. This allows users to quickly search for related scenes within a video by entering specific keywords. Furthermore, by adding a means for generating prompt sentences and sending them to the server, the generated summaries and chapters can be easily understood with high accuracy. This system allows users to efficiently access the information they need without having to watch the entire video.

[1789] "Means for uploading videos" is a function that allows users to send any video file to the system and have it saved on the server.

[1790] "Means for the server to receive the video and store it in temporary storage" refers to a function in which the server receives the uploaded video file and temporarily stores it for subsequent analysis processing.

[1791] "Means for converting video audio data into text using a generative artificial intelligence engine" is a function that uses generative artificial intelligence technology to automatically transcribe audio within a video.

[1792] "Means for analyzing video data of a video, dividing the video into scenes, and creating summaries" refers to a function that analyzes video data, divides the content into multiple scenes, and extracts the important content of each scene as a summary.

[1793] "Means for creating chapter candidates based on the generated summary and scenes" is a function that divides a video into multiple sections (chapters) based on the generated summary and scene information, and assigns titles and summaries to each section.

[1794] The "means for transmitting chapter candidates to a terminal" is a function for transmitting chapter information generated by the server to a user's terminal.

[1795] The "means for displaying chapter candidates received by the terminal to the user" is a function for visually displaying chapter information received on the user's terminal.

[1796] "Means for a user to input a specific keyword into the search bar of a terminal" refers to a function that allows a user to input a keyword they want to search for into the search bar on the terminal.

[1797] The "means for the terminal to transmit a keyword to the server" is a function for transmitting a keyword input from the terminal to the server.

[1798] "Means for analyzing the text data of a video and identifying scenes containing keywords" is a function for analyzing the generated text data and identifying scenes in which the input keywords appear.

[1799] The "means for transmitting the time stamp of the scene to the terminal" is a function for transmitting the time information (time stamp) of the identified scene to the terminal.

[1800] "Means for the terminal to display the timestamp to the user and for the user to access a specified scene" is a function that displays timestamp information on the terminal and allows the user to directly access a specific scene based on that information.

[1801] "Means for generating video summaries using a generative AI model" refers to a function that uses a generative AI model to automatically summarize the content of a video.

[1802] "Means for generating prompt sentences and sending them to the server" is a function that automatically generates input sentences (prompt sentences) for the generative AI model and sends them to the server.

[1803] The present invention provides a function for efficiently viewing and searching video content. Specific embodiments of the present invention will be described below.

[1804] System Overview

[1805] This system combines a server, a user device, and a generative AI model engine. Users upload videos through their devices, and the server analyzes the videos and generates summaries. Users can also search for related scenes within videos using specific keywords.

[1806] Hardware and Software

[1807] Hardware: User devices such as smartphones, tablets, and PCs, as well as servers.

[1808] Software: ffmpeg, moviepy, speech_recognition, transformers library, and generative AI model engine.

[1809] Data processing and calculation

[1810] 1. Upload and save your video

[1811] The user selects a video file and uploads it from the device to the server, which then stores the received video file in temporary storage.

[1812] 2. Speech-to-text

[1813] The server extracts the audio data from the received video and converts it into text using speech recognition technology, using the speech_recognition library.

[1814] 3. Video analysis and summary generation

[1815] The server analyzes the video data and segments it into scenes, using the ffmpeg and moviepy libraries.

[1816] The server uses a generative AI model (e.g., the summarization model from the transformers library) to summarize the key points for each scene.

[1817] 4. Creating chapter suggestions

[1818] The server automatically creates chapter candidates based on the generated summary and scene information, and sends the chapter information to the user device in JSON format.

[1819] 5. Chapter display and search

[1820] The user terminal displays the received chapter information in list form.

[1821] When a user enters a specific keyword into the search bar of the device and clicks the search button, the device sends the keyword to the server.

[1822] The server analyzes the text data of the video, identifies scenes containing the keywords, and sends the timestamp information of those scenes to the terminal.

[1823] The terminal displays the timestamp information to the user, and the user can directly access the specified scene based on the information.

[1824] Specific examples

[1825] For example, suppose a user uploads a video on an educational video streaming service titled "Python Basics." Using this system, the user can:

[1826] Upload a video and its content will be automatically summarized using a generative AI model.

[1827] Each chapter (e.g., "Basic Python Syntax," "Data Structures," and "Functions") is automatically generated and displayed as a list.

[1828] When a user searches for "loop," relevant scenes are displayed with timestamps.

[1829] Users can click on the displayed timestamp to view the corresponding scene directly.

[1830] Prompt Sentence Examples

[1831] The prompt sentence can be summarized as follows: "Python is an easy-to-use and powerful programming language. Many beginners are amazed by its simplicity."

[1832] "Write a summary of this text: 'The programming language Python is easy to use and is the language of choice for many beginners.'"

[1833] This system will make viewing and searching video content more efficient and improve usability.

[1834] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1835] Step 1:

[1836] The user selects and uploads a video.

[1837] Input: The user selects a video file from local storage.

[1838] Output: The video file is sent to the user's device and uploaded to the server.

[1839] Specific operation: The user clicks the upload button on the device and selects a video file from the file selection dialog. The selected video file is sent from the device to the server.

[1840] Step 2:

[1841] The server receives the video and stores it in temporary storage.

[1842] Input: The video file sent from the user device.

[1843] Output: The video file is saved to the server's temporary storage.

[1844] Specific operation: The server stores the received video file in temporary storage and prepares it for subsequent analysis processing.

[1845] Step 3:

[1846] The server converts the video's audio data into text.

[1847] Input: Video file saved in temporary storage.

[1848] Output: Text data generated from the audio data.

[1849] What it does: The server uses the ffmpeg library to extract audio data from the video, then uses the speech_recognition library to convert the audio data to text.

[1850] Step 4:

[1851] The server analyzes the video data, divides the scenes, and creates a summary.

[1852] Input: Video files stored in temporary storage and generated text data.

[1853] Output: Summary text and chapter information for each scene.

[1854] How it works: The server uses the moviepy library to analyze the video data and segment it into scenes, then uses a generative AI model (e.g., the summarization model from the transformers library) to automatically summarize the content of each scene.

[1855] Step 5:

[1856] The server creates chapter candidates based on the generated summary and scenes.

[1857] Input: Summary text and scene information.

[1858] Output: Chapter candidate information (title, summary, timestamp, etc.).

[1859] Specific operation: The server creates chapter candidates in JSON format based on the generated summary and scene information.

[1860] Step 6:

[1861] The server transmits the chapter candidates to the terminal.

[1862] Input: Chapter candidate information.

[1863] Output: Chapter suggestion information sent to the device.

[1864] Specific operation: The server transmits the generated chapter candidate information to the user terminal.

[1865] Step 7:

[1866] The terminal displays the received chapter candidates to the user.

[1867] Input: Chapter candidate information received from the server.

[1868] Output: The chapter list displayed in the user interface.

[1869] Specific operation: The terminal displays the chapter candidate information in a list format on the user interface, allowing access to each chapter.

[1870] Step 8:

[1871] A user types a specific keyword into the search bar.

[1872] Input: The keyword entered by the user.

[1873] Output: Search keywords displayed on the device.

[1874] Specific operation: The user enters a specific keyword into the search bar of the device and clicks the search button.

[1875] Step 9:

[1876] The device sends the keyword to the server.

[1877] Input: The keyword entered in the search bar.

[1878] Output: The search terms sent to the server.

[1879] Specific operation: The terminal sends the keyword entered by the user to the server.

[1880] Step 10:

[1881] The server analyzes the text data of the video and identifies scenes that contain the keywords.

[1882] Input: Search keywords sent to the server and generated text data.

[1883] Output: Timestamp information of identified scenes.

[1884] Specific operation: The server analyzes the generated text data, identifies scenes that contain the search keywords, and generates timestamp information for the identified scenes.

[1885] Step 11:

[1886] The server transmits the timestamp of the identified scene to the terminal.

[1887] Input: The timestamp information of the identified scene.

[1888] Output: Timestamp information sent to the terminal.

[1889] Specific operation: The server transmits the timestamp information of the identified scene to the user terminal.

[1890] Step 12:

[1891] The terminal displays the timestamp to the user, and the user accesses the specified scene.

[1892] Input: The timestamp information received from the server.

[1893] Output: Timestamp information and associated scenes displayed in the user interface.

[1894] Specific operation: The device displays the timestamp information in the user interface, and the user can click on the displayed timestamp to directly access a specific scene.

[1895] The above are the specific processing steps for carrying out the invention.

[1896] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1897] The present invention is a system that automatically analyzes the content of videos and can pinpoint the information a user needs, and combines it with an emotion engine that recognizes the user's emotions to provide more advanced and personalized information. Specific embodiments for carrying out the present invention are described below.

[1898] System Overview

[1899] This system consists of a user, a server, a terminal, and an emotion engine. Users upload videos to the system and use the function to automatically generate summaries and chapters for the videos. Users can also quickly search for related scenes within the videos by entering specific keywords. Furthermore, the emotion engine recognizes the user's emotions and dynamically adjusts the video playback content and chapter display based on that information.

[1900] Uploading and saving videos

[1901] First, the user selects a video file and uploads it to the system. Once the video file is uploaded, the device sends it to the server. The server then stores the received video file in temporary storage and prepares it for analysis.

[1902] Video analysis and summary generation

[1903] The server then launches a generative AI engine to convert the video's audio data into text. Through this speech recognition process, the video's content is extracted as text data. The server then analyzes the video data and segments each scene. The generative AI engine then summarizes the content of each scene and extracts the main points.

[1904] Creating and viewing chapter suggestions

[1905] The server creates chapter candidates based on the generated summaries and scenes. This automatically divides the video into multiple sections, each with a title and summary. The server sends the chapter candidates in JSON format to the device, which then displays them visually to the user.

[1906] Reverse search function from words

[1907] The user enters a specific keyword in the search bar and clicks the search button. The device sends the keyword to the server. The server analyzes the text data of the video and identifies the scene in which the keyword appears. Once the timestamp of the relevant scene is identified, the server sends that information to the device. The device displays the search results, including the timestamp, to the user, allowing the user to directly access the specified scene.

[1908] Emotion recognition function

[1909] The system further enhances the user experience by incorporating an emotion engine. While the user is watching a video, a camera built into the device captures real-time video of the user. This video data is sent to a server, which then uses the emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial expressions and movements to identify their emotional state (e.g., joy, sadness, excitement, etc.).

[1910] The server can dynamically adjust the video playback and chapters displayed based on the emotional data it acquires. For example, if it determines that the user is interested, it can provide more information or supplemental videos on related topics. On the other hand, if it determines that the user is bored, it can present the option to skip to important scenes.

[1911] Specific examples

[1912] For example, if a user uploads a video about "latest tech trends," the process goes like this:

[1913] 1. Upload and analyze

[1914] A user uploads a video about "latest tech trends."

[1915] The device sends the video file to the server.

[1916] The server receives the video, stores it in temporary storage, and begins analysis.

[1917] 2. Summarizing and Chapter Creation

[1918] The server uses a generative artificial intelligence engine to generate a summary of the video.

[1919] The server creates chapter candidates based on each technical topic.

[1920] 3. Display results

[1921] The server sends the summary and chapter information to the device in JSON format.

[1922] The terminal displays this to the user in a list format.

[1923] 4. Reverse Search Feature

[1924] A user types "blockchain" into the search bar and clicks the search button.

[1925] The device sends the keyword to the server.

[1926] The server analyzes the video transcript, identifies scenes in which "blockchain" appears, and sends that information to the device.

[1927] The terminal displays the search results to the user in a list format, allowing the user to directly access related scenes.

[1928] 5. Emotion recognition

[1929] While the user is watching the video, the device's camera captures real-time video.

[1930] The server analyzes the video data and uses an emotion engine to recognize the user's emotions.

[1931] Based on the emotional data, the video playback content and chapter display are dynamically adjusted.

[1932] The system not only allows users to instantly access the information they need without having to play the entire video, but also provides a personalized viewing experience through emotion recognition.

[1933] The processing flow will be explained below.

[1934] Step 1:

[1935] The user selects a video file and uploads it to the system. The user uses a file selection dialog to select a video file from their computer and clicks the "Upload" button.

[1936] Step 2:

[1937] The device sends the selected video file to the server. The device obtains the binary data of the video file and sends it to the server using an HTTP POST request.

[1938] Step 3:

[1939] The server receives the video file and stores it in temporary storage. The server receives the video file, checks the integrity of the data, and then stores it in a temporary storage folder.

[1940] Step 4:

[1941] The server starts the generative artificial intelligence engine. The server calls the generative artificial intelligence engine to start the process of converting audio data into text for the prepared video file.

[1942] Step 5:

[1943] The server converts the audio data into text, and a generative AI engine analyzes the audio in the video and generates recognized text data.

[1944] Step 6:

[1945] The server analyzes the video data, extracts each frame of the video, and segments it into scenes using a video recognition algorithm.

[1946] Step 7:

[1947] The server generates a summary for each scene using the generated text and video data. The content is reviewed to ensure that key points and themes are included.

[1948] Step 8:

[1949] The server creates chapter candidates. Based on the generated summaries and scene information, the server assigns titles and summaries to each segment and creates chapter candidates.

[1950] Step 9:

[1951] The server sends the chapter candidates to the device. The server encodes the completed chapter candidates in JSON format and sends them to the device as an HTTP response.

[1952] Step 10:

[1953] The device parses the received chapter suggestions and displays them on the user interface. The device parses the JSON data and displays them to the user in list or timeline format.

[1954] Step 11:

[1955] A user enters a specific keyword into the search bar and clicks the search button.A user enters a keyword into the input field of the search bar and presses the button to execute the search command.

[1956] Step 12:

[1957] The device sends the keyword to the server. The device creates an HTTP GET request including the user's keyword and sends it to the server.

[1958] Step 13:

[1959] The server analyzes the text data of the video and identifies the scenes in which the keywords appear. Using a full-text search algorithm, the server extracts text fragments containing the keywords and obtains the timestamps of the corresponding scenes.

[1960] Step 14:

[1961] The server sends the timestamp to the device. The server encodes the timestamp and scene information identified as search results in JSON format and sends it to the device as an HTTP response.

[1962] Step 15:

[1963] The device displays the search results to the user. The device displays the received timestamps and scene information in a list format, and provides a "Play" button for each entry.

[1964] Step 16:

[1965] The user clicks the play button to access the specified scene. When the user clicks the play button in the displayed list, the video player starts playing from the corresponding scene.

[1966] Step 17:

[1967] The device acquires real-time video data of the user. The device's camera captures the user's face and movements and sends the data to the server.

[1968] Step 18:

[1969] The server starts the emotion engine. The server invokes the emotion engine to analyze the received video data.

[1970] Step 19:

[1971] The server analyzes the user's emotions, and the emotion engine analyzes the user's facial expressions and movements to identify their emotional state, such as joy, sadness, or excitement.

[1972] Step 20:

[1973] The server uses the emotion analysis results to dynamically adjust video playback and chapter display based on the emotion data.

[1974] Step 21:

[1975] The server sends adjustments based on emotion data to the device, and the server sends dynamically adjusted information to the device, which then visually displays it.

[1976] Step 22:

[1977] The device displays a display that reflects the emotional data to the user. The device displays a list that reflects the emotional data and adjusts the video player to the user, allowing the user to have an optimal viewing experience.

[1978] Example 2

[1979] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1980] While conventional video analysis systems have the ability to summarize videos and search for specific scenes, they lack the ability to dynamically change content based on the user's emotional state. Furthermore, while they enable users to quickly obtain specific information, they have the problem of being unable to respond to the user's interests and emotions while watching, because the video playback content is static. Therefore, there is a need for a system that can individually optimize the user's viewing experience and efficiently provide the information they need.

[1981] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1982] In this invention, the server includes means for converting audio data of a video into text using a generative artificial intelligence engine, means for analyzing video data of the video and dividing it into scenes to create a summary, means for creating chapter candidates based on the created summary and scenes, and means for recognizing the user's emotional state using an emotion engine and dynamically adjusting the video playback content and chapter display. This enables a personalized viewing experience according to the user's emotions and makes it possible to efficiently provide necessary information.

[1983] "Video" means a digital file containing audio and video information that a user uploads to the system.

[1984] The "server" is a central processing unit that receives video, temporarily stores it, and analyzes it using a generative artificial intelligence engine and an emotion engine.

[1985] A "generative artificial intelligence engine" is an AI technology that converts audio data from videos into text and analyzes video data to create scene summaries.

[1986] "Voice recognition technology" is a technology for converting audio data from video into text data.

[1987] "Video data" refers to digital data that constitutes the visual information of a moving image.

[1988] "Scene segmentation" is a method of dividing a continuous video sequence into multiple stages and analyzing each section independently.

[1989] A "summary" is text information that briefly summarizes the main content of each scene in a video.

[1990] "Chapter candidates" refer to the individual parts of a video divided into multiple sections based on summarized scene information.

[1991] "Emotion recognition" is the process of analyzing a user's facial expressions, movements, and other facial expressions to identify their emotional state.

[1992] "Terminal" means a device through which a user accesses the system to upload, search, and play videos.

[1993] A "keyword" is a character string that a user enters to search for specific information.

[1994] A "timestamp" is information that indicates the start time of a particular scene in a video.

[1995] "Dynamic adjustment" refers to changing the video playback content and chapter display in real time based on the user's emotion recognition results.

[1996] The present invention is an advanced information provision system that combines video analysis and emotion recognition, and specific embodiments are described in detail below. This system allows users to upload videos and automatically analyze the content of those videos to efficiently obtain the information users need. The present invention includes a server, a terminal, and an emotion engine. Ultimately, it realizes dynamic information provision according to the user's emotions.

[1997] Uploading and saving videos

[1998] A user uses a device to select and upload a video file. The device then sends the video file to a server. The server receives the video file and stores it in temporary storage, preparing the video for analysis. For example, if a user uploads a video of a seminar on technology trends, the device sends the file to the server, which stores it.

[1999] Video analysis and summary generation

[2000] The server launches a generative AI engine to begin analyzing the video. First, the audio data is converted into text using speech recognition technology (e.g., Google Speech-to-Text API). Next, the video data is analyzed and divided into scenes. For each divided scene, a generative AI model (e.g., GPT-4) is used to generate a summary. This allows the main points of the video to be concisely extracted.

[2001] Creating chapter suggestions

[2002] The server creates chapter candidates based on the generated summary and split scenes. Each chapter is divided into sections and given a title and summary. The server sends this information to the device in JSON format, and the device displays it to the user. For example, if a video on "latest technology trends" is uploaded, the server generates chapters based on technology topics and displays them to the user.

[2003] Keyword search function

[2004] The user enters a specific keyword into the search bar on the device and clicks the search button. The device then sends the keyword to the server. The server analyzes the text data of the video and identifies the scene in which the keyword appears. The timestamp information of the relevant scene is sent to the device and displayed to the user. The user can then directly access the relevant scene from the search results.

[2005] Emotion recognition function

[2006] The system incorporates an emotion engine that analyzes the user's emotions in real time. While the user is watching a video, a camera built into the device captures real-time footage of the user and sends it to the server. The server uses the emotion engine to analyze the user's emotional state and dynamically adjusts the video playback content and chapter display. For example, if the system determines that the user is excited, it provides related information or supplementary videos. If the system determines that the user is bored, it presents an option to skip to important scenes.

[2007] As a concrete example, when a user uploads a video about "latest technology trends," the process proceeds as follows:

[2008] 1. The user uploads a video and the device sends it to the server.

[2009] 2. The server stores the video and begins analysis, converting the audio data into text and analyzing the video data to generate a summary.

[2010] 3. The server creates chapter candidates and sends them to the device, which displays them to the user.

[2011] 4. The user searches by keyword, and the server sends the timestamp of the corresponding scene to the device. The device displays the search results, and the user can access the corresponding scene.

[2012] 5. While the user is watching the video, the device camera captures their emotions. The server analyzes this using an emotion engine and dynamically adjusts the playback content.

[2013] In this way, the system allows users to quickly access the information they need without having to play the entire video, and also provides a personalized viewing experience through emotion recognition.

[2014] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2015] Step 1:

[2016] A user selects a video file on a terminal and uploads it to the system. Specifically, the user selects a video file from the user interface and presses the upload button. The input at this time is the selected video file, and the output is the terminal sending the video file to the server.

[2017] Step 2:

[2018] The device sends the video file selected by the user to the server. It receives the video file uploaded by the user as input and sends it to the server as output. Specifically, the device generates an HTTP request, attaches the video file, and sends it to the server.

[2019] Step 3:

[2020] The server receives the video file and stores it in temporary storage. It receives the video file sent from the terminal as input and stores it in temporary storage as output. Specifically, the server uses a file management system to store the video file in a speci...

Claims

1. a means for users to upload videos; A means for the server to receive and store the video in temporary storage; A means for the server to convert the audio data of the video into text using a generative artificial intelligence engine; A server analyzes the video data, divides the video into scenes, and creates a summary. a means for the server to generate chapter candidates based on the generated summaries and scenes; A means for the server to transmit chapter candidates to the terminal; a means for displaying the chapter candidates received by the terminal to a user; A means for a user to input a specific keyword into a search bar on the device; A means for the terminal to transmit the keyword to the server; A means for the server to analyze text data of the video and identify scenes containing keywords; A means for the server to transmit a timestamp of the scene to the terminal; a means for the terminal to display the timestamp to the user and for the user to access the specified scene; A system including:

2. 2. The system according to claim 1, further comprising means for the server to analyze video data of the video using a generative artificial intelligence engine and to generate chapter candidates based on the analysis.

3. 2. The system of claim 1, wherein the server includes means for converting audio data of the video into text using voice recognition technology.

Images