System

Abstract

A system is provided.SOLUTION: The system includes means for receiving a video file, means for extracting audio data from the received video file, means for converting the audio data into text data, means for extracting main topics and key points from the text data, means for dividing the video file into chapters based on the extracted topics or key points, means for generating a summary for each chapter, means for storing the generated chapters and summaries, means for displaying the stored chapters and summaries, and means for performing a search by inputting a specific keyword and displaying a search result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today, demand for video content is increasing, and young people, in particular, need information quickly. However, searching for the necessary information in long videos can be time-consuming and labor-intensive. This creates a need for tools that allow users to efficiently obtain the necessary information without having to replay the entire video. Businesses also need effective methods for quickly extracting important information from videos and generating meeting minutes and procedure manuals. [Means for solving the problem]

[0005] To solve this problem, the present invention provides the following means: First, a video file is received and audio data is extracted from the received video file. Then, the audio data is converted into text data, and major topics and key points are extracted from the text data. Next, the video file is divided into chapters based on the extracted topics or key points, and a summary is generated for each chapter. These generated chapters and summaries are saved and displayed. Finally, a user can enter specific keywords to perform a search and display the search results, allowing them to quickly obtain the information they need.

[0006] A "video file" is a media file in digital format that contains audio and video.

[0007] "Audio data" is data that represents audio information extracted from a video file in digital format.

[0008] "Text data" is digital data that has been analyzed and converted into text information.

[0009] "Major topics" are important themes or topics that make up the content of a video, extracted from text data.

[0010] "Key points" are the most important points or important statements in the content of a video that are extracted from text data.

[0011] "Chapters" are content-based segments of a video file, each related to a specific topic or key point.

[0012] A "summary" is a sentence or piece of information that concisely summarizes the contents of each chapter.

[0013] "Natural language processing technology" refers to computer science techniques for analyzing, understanding, and manipulating text data.

[0014] A "keyword" is a word or phrase that a user enters to search for specific information.

[0015] "Search results" refers to the information the system provides to display related chapters and their contents in response to the entered keywords. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a system for acquiring "necessary information" from video content in a "short time." The specific processing and operation of each step will be described below.

[0038] Importing videos

[0039] A user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the system, and the server receives the video file and saves it in the specified directory. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[0040] Video Analysis

[0041] The server extracts the audio track from the stored video file, for example using a tool such as FFmpeg. The server then sends the extracted audio data to a speech recognition engine (for example, Google® Cloud Speech-to-Text or DeepSpeech) to convert it into text. The obtained text data is saved as an intermediate file.

[0042] Natural Language Processing

[0043] The server reads the text data and uses natural language processing libraries (e.g., NLTK or spaCy) to extract major topics and key points, clarifying the content of the video and establishing a foundation for the next steps.

[0044] Chapter generation and summarization

[0045] The server divides the video into multiple chapters based on topics and key points. It identifies the start and end times of each chapter and lists the chapter information. The server then generates a summary of each chapter's text data using a summarization tool (e.g., GPT-3 (registered trademark) or BERT). The generated chapters and summaries are stored in a database.

[0046] Providing a user interface

[0047] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the device in JSON format or similar. The device then displays information for each chapter on the user interface based on the analysis results received. This information includes the start time, summary, and thumbnail of each chapter. This allows the user to select the part they want to watch and play it.

[0048] Reverse search function

[0049] The user enters a specific keyword in the search bar and clicks the search button. The server performs a full-text search of the text data within the video based on the received keyword (using ElasticSearch (registered trademark) or similar). The server extracts the relevant chapter information as search results and sends it to the device. The device displays the received search results in list format on the user interface. The user selects the part they want to view from the displayed search results and clicks the play button. The device then plays the video from the specific part of the chapter they clicked.

[0050] Specific examples

[0051] For example, consider the case where a university student uploads a recorded lecture. When the user uploads the lecture video file to the system, the server extracts the audio data and converts it into text data using a speech recognition engine. Then, natural language processing technology is used to extract key topics from the text data and divide the video into chapters based on this. The content of each chapter is summarized and displayed on the device, allowing the user to select the chapter they want to view and review it efficiently. Users can also quickly find the information they need by searching for specific keywords.

[0052] In this way, the present invention provides an effective means for efficiently obtaining necessary information from video content.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] Uploading videos

[0056] The user logs into the system, selects a video file, and clicks the upload button.

[0057] The terminal transmits the selected video file to the system.

[0058] Step 2:

[0059] Saving video files

[0060] The server saves the received video file in the specified directory in the internal storage.

[0061] The server registers the destination path and related metadata (file name, upload date and time, user ID, etc.) in a database.

[0062] Step 3:

[0063] Extracting audio data

[0064] The server extracts the audio track from the stored video file using a tool such as FFmpeg.

[0065] Step 4:

[0066] Voice Recognition

[0067] The server sends the extracted voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech) and converts it into text data.

[0068] The server stores the converted text data as an intermediate file.

[0069] Step 5:

[0070] Natural Language Processing

[0071] The server reads the text data stored as intermediate files and extracts major topics and key points using natural language processing libraries (e.g., NLTK or spaCy).

[0072] Step 6:

[0073] Chapter Generation

[0074] The server divides the video into chapters based on the extracted topics and key points, identifies the start and end times of each chapter, and lists the chapter information.

[0075] Step 7:

[0076] Generate a summary

[0077] Based on each chapter, the server generates a summary for each chapter using a summarization tool (e.g., GPT-3 or BERT).

[0078] The server stores the generated chapters and summaries in a database.

[0079] Step 8:

[0080] Sending analysis results

[0081] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the terminal in JSON format or similar.

[0082] Step 9:

[0083] View Chapters

[0084] Based on the analysis results, the device displays information for each chapter on the user interface, including the start time, summary, and thumbnail of each chapter.

[0085] Step 10:

[0086] Play video

[0087] The user selects the chapter they want to watch and clicks the play button.

[0088] The device will play the video from the start time of the selected chapter using HTML5 Video Player or VLC.

[0089] Step 11:

[0090] Keyword search

[0091] The user enters a specific keyword into the search bar and clicks the search button.

[0092] The server performs a full-text search of the text data in the video based on the received keywords (e.g., using Elasticsearch).

[0093] The server extracts the relevant chapter information as a search result and transmits it to the terminal.

[0094] Step 12:

[0095] Displaying search results

[0096] The device displays the received search results in a list format, with a timestamp and summary of the chapters in each search result.

[0097] The user selects the part they want to watch from the search results and clicks the play button.

[0098] The device will play the video from the specific part of the chapter you clicked.

[0099] Example 1

[0100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] In recent years, the amount of video content has rapidly increased, creating a demand for users to quickly obtain the information they need from videos. However, conventional technologies are inefficient because they require users to watch the entire video to find the information they need. Furthermore, they lack the ability to search for specific content within a video, which means it takes a long time to find the desired information. Furthermore, video content summarization and automatic chapter division are performed manually, which is cumbersome and time-consuming.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0103] In this invention, the server includes means for receiving a video file, means for extracting audio data from the video file, means for converting the audio data into text data, means for extracting major topics and key points from the text data, means for dividing the video file into chapters based on the extracted topics or key points, means for generating summaries for each chapter, means for saving the generated chapters and summaries, means for displaying the saved chapters and summaries, means for inputting a specific keyword to perform a search and displaying the search results, and means for extracting relevant chapter information as the search results and playing the content from the first section. This allows users to efficiently obtain the information they need from videos and quickly search for and play specific content.

[0104] The "means for receiving video files" is a mechanism by which a user uploads video files to the system and the server receives the video files.

[0105] A "means for extracting audio data from video files" is a tool or program used to separate the audio track from a saved video file and obtain the audio data.

[0106] The "means for converting voice data into text data" refers to a mechanism that analyzes the extracted voice data and converts it into text data, such as using a voice recognition engine.

[0107] "Means for extracting major topics and key points from text data" refers to natural language processing techniques for analyzing text data and identifying important topics and key points from its content.

[0108] The "means for dividing a video file into chapters based on extracted topics or key points" refers to a mechanism for dividing the content of a video based on the analysis results and identifying each section.

[0109] The "means for generating a summary for each chapter" is a technology for automatically creating a summary that concisely summarizes the contents of each chapter.

[0110] The "means for saving the generated chapters and summaries" is a mechanism for storing the divided chapter information and summaries thereof in a database or the like.

[0111] The "means for displaying saved chapters and summaries" is a mechanism for displaying the analysis results on a terminal, allowing the user to view the information.

[0112] "Means for searching by entering specific keywords and displaying search results" is a mechanism for searching the text data of videos based on keywords entered by the user and displaying matching results.

[0113] "Means for extracting relevant chapter information as a search result and playing from the first section" refers to a mechanism that starts playing the video from the relevant point based on the chapter information obtained by the search.

[0114] This invention is a system that enables a user to quickly and efficiently obtain necessary information from video content. The specific processing and operation of each step will be described in detail below.

[0115] First, the user logs in to the system using a terminal and clicks the upload button to specify a video file. When the user selects a video file and executes upload, the terminal sends the selected video file to the server. The server receives the video file and saves it in the specified directory. At this time, metadata such as the save path and file name, upload date and time, and user ID are registered in the database.

[0116] Next, the server extracts the audio track from the saved video file. This process is performed using a tool such as FFmpeg, specifically by executing FFmpeg commands to separate the audio data. The server then sends the extracted audio data to a speech recognition engine (such as Google Cloud Speech-to-Text or DeepSpeech) for analysis. The speech recognition engine converts the audio data into text data and returns the results to the server. The server then saves the obtained text data as an intermediate file.

[0117] The server reads the stored text data and extracts major topics and key points using natural language processing libraries (e.g., NLTK or spaCy). This process involves tokenizing the text data, tagging it with POS tags, and recognizing entities. The results of this analysis are used in the next step.

[0118] The server divides the video into chapters based on the extracted topics and key points. It calculates the start and end times of each chapter and lists the chapter information. The server then generates a summary of each chapter's text data using a summarization tool (such as GPT-3 or BERT). This allows for a concise summary of the content of each chapter. The generated chapters and summaries are stored in a database.

[0119] The server then retrieves the generated chapter and summary information from the database. The retrieved data is converted to JSON format and sent to the device. The device then displays information for each chapter on the user interface based on the analysis results. The displayed information includes the chapter start time, summary, thumbnail, etc. Through the interface, the user can select the part they want to watch and play it.

[0120] Users can also enter specific keywords into the search bar and click the search button. The device sends this input data to the server. The server performs a full-text search of the text data within the video based on the received keywords. This search uses a search engine such as Elasticsearch. The server extracts the relevant chapter information as search results and sends it to the device. The device displays the received search results in list format on the user interface. The user can select the part they want to view from the displayed search results and play it.

[0121] As a specific example, consider the case where a university student uploads a recorded lecture. When a user uploads a video file of a lecture to the system, the server extracts the audio data and converts it into text data using a speech recognition engine. The server then uses natural language processing technology to extract major topics from the text data and divides the video into chapters based on this. The content of each chapter is summarized and displayed on the device. Users can select the chapters they want to view and review them efficiently. They can also quickly find the information they need by searching for specific keywords. This system provides an effective means for efficiently obtaining the information they need from video content.

[0122] An example prompt is as follows:

[0123] "I want to easily review lecture videos. I would like to know if there is a system that will automatically summarize the lecture content and divide it into easy-to-understand chapters when I upload a video file."

[0124] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0125] Step 1:

[0126] A user logs in to the system using a terminal and uploads a video file by clicking the upload button, selecting the video file, and pressing the send button.

[0127] Input: Video file (e.g. lecture recording file)

[0128] Output: A request to transfer a video file to the server

[0129] Specific operation: The device sends an HTTP POST request to the server and sends request data including the selected video file.

[0130] Step 2:

[0131] The server saves the received video file in the specified directory and registers the destination path, file name, upload date and time, user ID, and other metadata in the database.

[0132] Input: Video file, metadata (file name, upload date and time, user ID, etc.)

[0133] Output: Path where the video file is saved, metadata registered in the database

[0134] Specific operation: The server saves the received video file in the specified directory of the file system and records the save path in the database.

[0135] Step 3:

[0136] The server extracts the audio track from the saved video file using a tool such as FFmpeg.

[0137] Input: Saved video file

[0138] Output: Extracted audio data file

[0139] Specific operation: The server executes FFmpeg commands to extract the audio track from the video file and save it as an audio data file.

[0140] Step 4:

[0141] The server sends the extracted voice data to a voice recognition engine (such as Google Cloud Speech-to-Text or DeepSpeech) and converts it into text data.

[0142] Input: Audio data file

[0143] Output: The resulting text data

[0144] Specific operation: The server calls the API of the speech recognition engine, sends the voice data, receives the resulting text data, and saves it as an intermediate file.

[0145] Step 5:

[0146] The server reads the stored text data and extracts major topics and key points using natural language processing libraries (such as NLTK or spaCy).

[0147] Input: Text data

[0148] Output: A list of topics and keypoints

[0149] What it does: The server reads the text data and uses natural language processing libraries to tokenize the text, tag it with POS tags, recognize entities, and extract key topics and key points.

[0150] Step 6:

[0151] The server divides the video into chapters based on the extracted topics and key points, calculates the start and end times of each chapter, and lists the chapter information.

[0152] Input: List of topics and keypoints

[0153] Output: List of chapter information (start time, end time, content)

[0154] Specific operation: The server divides the video into multiple sections based on the timestamps of the topics extracted from the text data, and lists the start and end times of each section.

[0155] Step 7:

[0156] The server generates a summary of the text data for each chapter using a summarization tool (such as GPT-3 or BERT).

[0157] Input: Text data for each chapter

[0158] Output: Summary text

[0159] Specific operation: The server calls the summary generation API, sends the text data for each chapter, and generates a summary text based on the results.

[0160] Step 8:

[0161] The server stores the generated chapter and summary information in a database.

[0162] Input: List of chapter information, summary text

[0163] Output: Chapter and summary information stored in a database

[0164] Specific operation: The server inserts and saves each chapter information and summary into the database.

[0165] Step 9:

[0166] The server retrieves the generated chapter and summary information from the database, converts it into JSON format, and sends it to the terminal.

[0167] Input: Chapter and summary information stored in the database

[0168] Output: JSON format data

[0169] Specific operation: The server queries the database for the required data, serializes the retrieved data into JSON format, and returns it to the device as an HTTP response.

[0170] Step 10:

[0171] The device then displays information for each chapter on the user interface based on the analysis results received, allowing the user to select the part they want to watch and play it.

[0172] Input: Analysis result data in JSON format

[0173] Output: Chapter information displayed in the user interface

[0174] What it does: The device parses the received JSON data and renders an interface that includes the start time, summary, thumbnails, etc. for each chapter.

[0175] Step 11:

[0176] The user inputs a specific keyword and clicks the search button, and the device sends this input data to the server.

[0177] Input: Keywords (e.g., specific lecture content)

[0178] Output: Search request based on keywords

[0179] Specific operation: The device obtains the keywords entered in the search bar and sends them to the server as an HTTP request.

[0180] Step 12:

[0181] The server performs a full-text search of the text data in the video based on the received keywords (using Elasticsearch, etc.), extracts the relevant chapter information as search results, and sends it to the device.

[0182] Input: Keywords, text data

[0183] Output: Chapter information as search results

[0184] Specific operation: The server queries Elasticsearch for the received keywords, extracts chapter information for matching text, serializes it in JSON format, and returns it to the device.

[0185] Step 13:

[0186] The terminal displays the received search results in a list format on the user interface. The user selects the part they want to watch from the displayed search results and clicks the play button.

[0187] Input: Search results (JSON data)

[0188] Output: Search results displayed in the user interface

[0189] Specific operation: The terminal parses the received JSON data and displays the results in a list format in the user interface.

[0190] Step 14:

[0191] The device will play the video from the specific part of the chapter you clicked.

[0192] Input: User selected chapter information

[0193] Output: Video played from the specified chapter

[0194] Specific operation: The device captures the user's click event and starts playing the video from the start time of the corresponding chapter.

[0195] (Application example 1)

[0196] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0197] In today's world, the amount of video content is increasing, and users are required to quickly obtain the information they need. However, it is not easy to extract the necessary information from long video content and grasp it in a short time. In addition, because the content contains a large amount of information, there is also the problem that viewing and searching cannot be performed efficiently. Therefore, an object of the present invention is to provide a system that efficiently extracts the necessary information from video content and provides it in a short time.

[0198] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0199] In this invention, the server includes a means for receiving video files, a means for extracting audio data from the received video files, and a means for converting the audio data into text data. This allows for efficient extraction of major topics and key points from video content, division of the video into multiple chapters, and generation and display of summaries. The system also includes a means for generating summaries based on input prompts using a generative AI model, and furthermore, enables searching for specific keywords and playback of relevant sections. This allows users to quickly obtain the information they need and efficiently watch and search videos.

[0200] A "video file" is a digital file that contains video and audio data.

[0201] "Audio data" refers to sound data containing human voices extracted from video files.

[0202] "Text data" refers to data obtained by converting voice data into character information.

[0203] "Major topics" are important topics or themes of videos extracted from text data.

[0204] "Key points" are important points or elements related to the main topic.

[0205] "Chapters" are sections of a video file that are divided based on specific topics or key points.

[0206] A "summary" is a text that briefly describes the contents of each chapter.

[0207] A "generative AI model" is an artificial intelligence model that generates and understands text based on large datasets.

[0208] A "prompt" is a text instruction that is input to a generative AI model.

[0209] The "means for receiving" is the part that has the function of uploading and saving video files to the system.

[0210] The "extraction means" is the part that has the function of extracting audio data from a video file.

[0211] The "conversion means" is a part that has the function of converting voice data into text data.

[0212] The "means for generating a summary" is a part that has the function of converting text data into a summary.

[0213] The "means for saving" is a part that has the function of saving the generated chapters and summaries.

[0214] The "display means" is a part that has the function of displaying the saved chapters and summaries on the user interface.

[0215] The "search means" is the part that has the function of searching the text data within the video based on specific keywords and displaying the results.

[0216] "Means for playback" refers to the part that has the function of playing the video of the relevant part of the search results.

[0217] This invention is a system that extracts necessary information from video content in a short time and efficiently provides it to users. The system receives video files, extracts and converts audio data, extracts major topics and key points, generates summaries using a generative AI model, and displays and searches the results.

[0218] First, a user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the server, which receives the video file and saves it in the specified directory. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[0219] Next, the server extracts the audio track from the saved video file using a tool such as FFmpeg. The server then sends the extracted audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert it into text data, which is then saved as an intermediate file.

[0220] The server then reads the text data and uses natural language processing libraries (e.g., spaCy) to extract major topics and key points, clarifying the content of the video and establishing a foundation for the next steps.

[0221] The server divides the video into chapters based on topics and key points. It identifies the start and end times of each chapter and lists the chapter information. The server then generates a summary of each chapter's text data using a summarization tool (e.g., GPT-3). The generated chapters and summaries are stored in a database.

[0222] When a user enters a specific keyword, the server performs a full-text search of the text data within the video based on the received keyword. The server extracts the relevant chapter information as search results and sends it to the device. The device then displays the received search results in a list format on the user interface. The user can then select the part they want to watch from the displayed search results and play it.

[0223] As a concrete example, consider the case where a university student uploads a recorded lecture. When a user uploads a video file of a lecture to the system, the server extracts the audio data and converts it into text data using a speech recognition engine. Then, natural language processing technology is used to extract key topics from the text data and divide the video into chapters based on these. The content of each chapter is summarized and displayed on the device, allowing the user to select the chapter they want to view and review it efficiently.

[0224] An example of a prompt sentence in a generative AI model is "Please summarize the following text in a concise manner: \n\n[text content]", which allows the generative AI model to summarize the input text.

[0225] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0226] Step 1:

[0227] A user logs in to the system using a terminal and uploads a video file. The terminal sends the video file selected by the user to the server. The input is the video file, and the output is the video file received by the server.

[0228] Step 2:

[0229] The server saves the received video file in the specified directory. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database. The input is the video file, and the output is the saved video file and the registration of metadata in the database.

[0230] Step 3:

[0231] The server extracts the audio track from the stored video file using a tool such as FFmpeg. The input is the stored video file, and the output is the extracted audio data.

[0232] Step 4:

[0233] The server sends the extracted voice data to a voice recognition engine (such as Google Cloud Speech-to-Text) and converts it into text data. The input is voice data, and the output is the converted text data.

[0234] Step 5:

[0235] The server reads the text data and extracts the main topics and keypoints using a natural language processing library (such as spaCy). The input is the text data, and the output is the extracted main topics and keypoints.

[0236] Step 6:

[0237] The server divides the video into chapters based on topics and key points, identifies the start and end times of each chapter, and lists the chapter information. The input is the main topics and key points, and the output is the divided chapter information.

[0238] Step 7:

[0239] Next, the server generates a summary of the text data of each chapter using a summarization tool (a generative AI model such as GPT-3). A prompt is used for generation. The input is the text data and prompt for each chapter, and the output is the generated summary. An example of a prompt is "Please summarize the following text in a concise manner: \n\n[text content]".

[0240] Step 8:

[0241] The server stores the generated chapters and summaries in a database. The input is the generated chapters and summaries, and the output is the data stored in the database.

[0242] Step 9:

[0243] When a user inputs a specific keyword, the server performs a full-text search of the text data in the video based on the received keyword. The input is the specific keyword, and the output is the searched chapter information.

[0244] Step 10:

[0245] The server extracts the relevant chapter information as the search result and sends it to the terminal. The input is the chapter information of the search result, and the output is the search result sent to the terminal.

[0246] Step 11:

[0247] The terminal displays the received search results in a list format on the user interface. The user can select the part they want to watch from the displayed search results and play it. The input is the chapter information of the search results, and the output is the search results displayed on the user interface.

[0248] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0249] This invention improves the user experience by combining a system that quickly acquires necessary information from video content with an emotion engine that recognizes the user's emotions. The specific processing and operation of each step are explained below.

[0250] Importing videos

[0251] A user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the system, and the server receives the video file and saves it in its internal storage. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[0252] Video Analysis

[0253] The server extracts the audio track from the stored video file, typically using a tool like FFmpeg. The server then sends the extracted audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech) to convert it into text data, which is then saved as an intermediate file.

[0254] Natural Language Processing

[0255] The server reads the text data stored as intermediate files and uses natural language processing libraries (e.g., NLTK or spaCy) to extract major topics and key points, clarifying the content of the video and generating the foundational data for the next step.

[0256] Chapter generation and summarization

[0257] The server divides the video into multiple chapters based on the extracted topics and key points. It identifies the start and end times of each chapter and lists the chapter information. Then, the server generates a summary for each chapter using a summarization tool (e.g., GPT-3 or BERT) based on each chapter. The generated chapters and summaries are stored in a database.

[0258] Providing a user interface

[0259] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the device in JSON format or similar. The device then displays information for each chapter on the user interface based on the analysis results received. The start time, summary, thumbnail, and other information for each chapter are displayed, allowing the user to select and play the part they want to watch.

[0260] Reverse search function

[0261] The user enters a specific keyword in the search bar and clicks the search button. The server uses Elasticsearch or similar tools to perform a full-text search of the text data within the video based on the received keyword. The server extracts the relevant chapter information as search results and sends it to the device. The device displays the received search results in list format, and for each chapter in the search results, displays the timestamp where the keyword appears and a summary. The user can select the part they want to watch and click the play button to play the video from that specific part of that chapter.

[0262] Emotion engine integration

[0263] The present invention further integrates an emotion engine that recognizes the user's emotions. The emotion engine allows the device to analyze the user's emotional state in real time using the user's facial expressions, voice, keyboard and mouse operation patterns, etc. For example, if the system determines that the user is interested, it will prioritize displaying related chapters and summaries.

[0264] Emotion-based chapter display

[0265] The server receives the emotion data obtained by the emotion engine and adjusts the display order of chapters and summaries based on the user's emotional state, thereby prioritizing the display of content that is likely to interest the user and improving the user experience.

[0266] Filtering search results based on sentiment

[0267] When a user enters a search keyword, the emotional engine provides emotional data to filter search results. For example, if a user is feeling stressed, chapters with relaxing content will be prioritized.

[0268] Specific examples

[0269] For example, when a user watches a recording of a business meeting and uses the system with the emotion engine, the system analyzes the user's emotions and prioritizes displaying important parts or parts that the user is likely to be interested in. This allows the user to efficiently obtain the information they need.

[0270] In this way, the present invention provides a means for efficiently obtaining necessary information from video content, while also providing a more personalized experience by taking into account the user's emotional state.

[0271] The processing flow will be explained below.

[0272] Step 1:

[0273] Uploading videos

[0274] The user logs into the system, selects a video file, and clicks the upload button.

[0275] The terminal transmits the selected video file to the system.

[0276] Step 2:

[0277] Saving video files

[0278] The server saves the received video file in a specified directory in the internal storage.

[0279] The server registers the destination path and related metadata (file name, upload date and time, user ID, etc.) in a database.

[0280] Step 3:

[0281] Extracting audio data

[0282] The server extracts the audio track from the stored video file using a tool such as FFmpeg.

[0283] Step 4:

[0284] Voice Recognition

[0285] The server sends the extracted voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech) and converts it into text data.

[0286] The server stores the converted text data as an intermediate file.

[0287] Step 5:

[0288] Natural Language Processing

[0289] The server reads the text data stored as intermediate files and extracts major topics and key points using natural language processing libraries (e.g., NLTK or spaCy).

[0290] Step 6:

[0291] Chapter Generation

[0292] The server divides the video into multiple chapters based on the extracted topics and key points, identifies the start and end times of each chapter, and lists the chapter information.

[0293] Step 7:

[0294] Generate a summary

[0295] Based on each chapter, the server generates a summary for each chapter using a summarization tool (e.g., GPT-3 or BERT).

[0296] The server stores the generated chapters and summaries in a database.

[0297] Step 8:

[0298] Sending analysis results

[0299] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the terminal in JSON format or similar.

[0300] Step 9:

[0301] View Chapters

[0302] Based on the analysis results, the device displays information for each chapter on the user interface, including the start time, summary, and thumbnail of each chapter.

[0303] Step 10:

[0304] Starting emotion recognition

[0305] The device runs an emotion engine and analyzes the user's facial expressions, voice, and operation patterns, including input from the camera, microphone, mouse, and keyboard.

[0306] The emotion engine analyzes the acquired data in real time to determine the user's emotional state.

[0307] Step 11:

[0308] Emotion-based interface adjustment

[0309] The server adjusts the display order of chapters and the content of summaries based on the emotion data received from the emotion engine.

[0310] For example, if it is determined that the user is interested, related chapters are preferentially displayed.

[0311] Step 12:

[0312] Play video

[0313] The user selects the chapter they wish to view and clicks the play button.

[0314] The device will play the video from the start time of the selected chapter.

[0315] Step 13:

[0316] Keyword search

[0317] The user enters a specific keyword in the search bar and clicks the search button.

[0318] The server performs a full-text search of the text data in the video based on the received keywords (e.g., using Elasticsearch).

[0319] The server extracts the relevant chapter information as the search result and transmits it to the terminal.

[0320] Step 14:

[0321] Filtering search results based on sentiment

[0322] The server uses data from the emotion engine to filter search results according to the user's emotions.

[0323] For example, if you want to relax, a chapter with relaxing content will be displayed.

[0324] Step 15:

[0325] Displaying search results

[0326] The device displays the received search results in a list format, with a timestamp and summary of the chapters in each search result.

[0327] The user selects the portion of the search result that he or she wants to view from the displayed search results and clicks the play button.

[0328] The device will play the video from the specific part of the chapter you clicked.

[0329] Example 2

[0330] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0331] In recent years, the use of video data has increased, creating a need for quickly extracting only the necessary parts from this vast amount of information. However, conventional technologies have been unable to take into account user emotion data when converting audio from video to text, extracting key topics and key points, dividing video into segments, and generating summaries, making it difficult to provide optimal information. Furthermore, the search function was unable to filter content based on user emotion, resulting in a poor user experience.

[0332] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0333] In this invention, the server includes means for receiving video data, means for extracting audio data from the received video data, means for converting the audio data into text data, means for extracting main topics and important points from the text data, means for dividing the video data into segments based on the extracted topics or important points, means for generating summaries for each segment, means for saving the generated segments and summaries, means for displaying the saved segments and summaries, means for inputting a specific keyword for search and displaying search results, means for analyzing user emotion data, and means for filtering the display order and search results based on the emotion data, thereby enabling optimal information provision that takes user emotions into consideration.

[0334] "Video data" refers to data in a file format that includes video and audio.

[0335] "Audio data" refers to data consisting of only the audio track extracted from video data.

[0336] "Text data" refers to data obtained by converting voice data into character information.

[0337] "Major topics" refer to themes or topics that are considered particularly important in the content of a video.

[0338] "Important points" refer to specific points or key information in the video content that are particularly noteworthy.

[0339] "Segmenting" refers to dividing video data into multiple parts and treating each part as an independent chapter.

[0340] A "segment" refers to each part of divided video data.

[0341] "Generating a summary" refers to creating a short description that succinctly summarizes the content of each segment.

[0342] "Storing" refers to recording the generated data in storage such as a database.

[0343] "Displaying" refers to visually showing the generated data in a user interface.

[0344] "Searching" refers to matching information in a database using specific keywords and extracting relevant results.

[0345] "Displaying search results" refers to displaying the information obtained by the search in a user interface.

[0346] "Analyzing user emotional data" refers to evaluating the user's emotional state in real time based on their facial expressions, voice, operation patterns, etc.

[0347] "Filtering the display order or search results based on emotional data" refers to adjusting the order or content of the information to be displayed in accordance with the acquired emotional data.

[0348] This invention is a system that quickly acquires necessary information from video data, and improves the user experience by combining it with an emotion engine that recognizes the user's emotions. Specific operations and embodiments of this system are described below.

[0349] Importing videos

[0350] A user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the server. The server saves the received video file in its internal storage. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[0351] Video Analysis

[0352] The server extracts the audio track from the saved video file using a tool called FFmpeg. The server then sends the extracted audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech), which converts the audio data into text data. The text data is then saved as an intermediate file.

[0353] Natural Language Processing

[0354] The server reads the text data stored as intermediate files and uses natural language processing libraries (e.g., NLTK or spaCy) to extract the main topics and key points, clarifying the content of the video and generating the foundational data for the next step.

[0355] Chapter generation and summarization

[0356] The server divides the video into multiple segments based on the extracted topics and key points. It identifies the start and end times of each segment and lists the segment information. Then, the server generates a summary for each segment using a summarization tool (e.g., GPT-3 or BERT). The generated segments and summaries are stored in a database.

[0357] Providing a user interface

[0358] The server retrieves the generated segment and summary information from the database and sends the analysis results to the device in JSON format or similar. Based on the analysis results received, the device displays information for each segment on the user interface. The start time, summary, thumbnail, and other information for each segment are displayed, allowing the user to select and play the part they want to watch.

[0359] Reverse search function

[0360] The user enters a specific keyword in the search bar and clicks the search button. The server uses Elasticsearch or similar tools to perform a full-text search of the text data within the video based on the received keyword. The server extracts the relevant segment information as search results and sends it to the device. The device displays the received search results in list format, and for each segment in the search results, displays the timestamp where the keyword appears and a summary. The user can select the part they want to watch and click the play button to play the video from that specific part of that segment.

[0361] Emotion engine integration

[0362] The present invention further integrates an emotion engine that recognizes the user's emotions. The emotion engine allows the terminal to analyze the user's emotional state in real time using the user's facial expressions, voice, keyboard and mouse operation patterns, etc. For example, if it is determined that the user is interested, the system will prioritize displaying related segments and summaries.

[0363] Segment display based on emotions

[0364] The server receives the emotion data obtained by the emotion engine and adjusts the display order of the segments and summaries based on the user's emotional state, thereby prioritizing the display of content that is likely to interest the user and improving the user experience.

[0365] Filtering search results based on sentiment

[0366] When a user enters a search keyword, the emotion engine provides emotional data to filter search results. For example, if a user is feeling stressed, segments with relaxing content will be prioritized.

[0367] Specific examples

[0368] For example, when a user watches a recording of a business meeting, the system analyzes the user's emotions and prioritizes displaying important parts or parts likely to interest the user. This allows the user to efficiently obtain the information they need. In this way, the present invention provides a means for efficiently obtaining necessary information from video data, while also providing a more personalized experience by taking the user's emotional state into consideration.

[0369] Prompt Sentence Examples

[0370] "Please extract the main topics and key points from this video."

[0371] "Summarize and segment recordings of business meetings."

[0372] "Rearrange your segments and summaries taking into account the output data of the sentiment engine to prioritize what is likely to interest the user."

[0373] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0374] Step 1: Import video

[0375] After logging in, the user selects and uploads a video file.

[0376] Input: Video file, user information

[0377] The terminal reads the selected video file and transfers it to the server.

[0378] The server receives the video file and saves it to its internal storage. At this time, it registers the destination path and video metadata (file name, upload date and time, user ID, etc.) in a database.

[0379] Output: Video files stored in the internal storage, metadata registered in the database

[0380] Step 2: Analyze the video

[0381] The server extracts the audio track from the stored video file.

[0382] Input: Saved video file

[0383] The server uses FFmpeg to extract the audio data and sends it to a speech recognition engine, specifically, by calling the Google Cloud Speech-to-Text or DeepSpeech API.

[0384] The server converts the voice data into text data and stores the text data as an intermediate file.

[0385] Output: Text data saved as an intermediate file

[0386] Step 3: Natural Language Processing

[0387] The server reads the text data stored as an intermediate file.

[0388] Input: Text data saved as an intermediate file

[0389] The server uses natural language processing libraries (such as NLTK or spaCy) to extract key topics and key points, which involves tokenizing words and parsing sentences.

[0390] Output: Extracted main topics and key points data

[0391] Step 4: Chapter generation and summarization

[0392] The server divides the video into multiple segments based on the extracted topics and key points.

[0393] Input: Data on key topics and key points

[0394] The server identifies and lists the start and end times of each segment, possibly using a video editing library.

[0395] The server uses a summarization tool (such as GPT-3 or BERT) to generate a summary of each segment.

[0396] The generated segments and summaries are stored in a database.

[0397] Output: Generated segments and summaries stored in a database

[0398] Step 5: Providing a User Interface

[0399] The server retrieves the generated segment and summary information from the database, converts it into JSON format, and sends it to the terminal.

[0400] Input: Generated segments and summaries stored in a database

[0401] Based on the analysis results received, the device displays information for each segment on the user interface, including the start time, summary, and thumbnail of each segment.

[0402] Output: Information for each segment displayed in the user interface

[0403] Step 6: Reverse Search Function

[0404] The user enters a specific keyword into the search bar and clicks the search button.

[0405] Input: Search keyword

[0406] Based on the received keywords, the server uses Elasticsearch to perform a full-text search of the text data within the video.

[0407] The server extracts the relevant segment information as a search result and transmits it to the terminal.

[0408] The terminal displays the received search results in a list format, and displays the timestamps at which the keywords appear and a summary for each segment of each search result.

[0409] Output: Search results displayed in list format

[0410] Step 7: Integrating the Emotion Engine

[0411] The emotion engine analyzes the user's facial expressions, voice, keyboard and mouse operation patterns, and other information acquired by the device in real time.

[0412] Input: User's facial expressions, voice, and operation patterns

[0413] The device analyzes this data and estimates the emotional state.

[0414] The server adjusts the order of segments and summaries to be displayed based on the emotion data received from the emotion engine.

[0415] Output: Reordered segments and summaries based on sentiment data

[0416] Step 8: Segment based on sentiment

[0417] The server receives the emotion data obtained from the emotion engine.

[0418] Input: Emotion data

[0419] The server adjusts the display order of the segments and summaries based on the user's emotional state and transmits them again to the terminal.

[0420] The device will prioritize updating the displayed information to improve the user experience.

[0421] Output: Reordered segments and summary information

[0422] Step 9: Filter search results based on sentiment

[0423] When a user enters a search keyword and performs a search, the search results are filtered based on the emotional data provided by the emotion engine.

[0424] Input: Search keywords, emotion data

[0425] The server filters the search results taking into account the emotion data and transmits the results based on that to the terminal.

[0426] The terminal displays the filtered results in a list format.

[0427] Output: Filtered search results

[0428] Prompt Sentence Examples

[0429] "Please extract the main topics and key points from this video."

[0430] "Summarize and segment recordings of business meetings."

[0431] "Rearrange your segments and summaries taking into account the output data of the sentiment engine to prioritize what is likely to interest the user."

[0432] (Application example 2)

[0433] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0434] Systems for quickly obtaining necessary information from video content have the problem of not being able to efficiently provide the parts of a user's interest, resulting in a lack of improvement in the user experience. Furthermore, conventional systems are unable to display content taking into account the user's emotions, making it difficult to provide personalized services.

[0435] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a video file, means for extracting audio data from the received video file, means for converting the audio data into text data, means for extracting major topics and key points from the text data, means for dividing the video file into chapters based on the extracted topics or key points, means for generating summaries for each chapter, means for saving the generated chapters and summaries, means for displaying the saved chapters and summaries, means for inputting a specific keyword to perform a search and displaying search results, emotion recognition means for recognizing the user's emotional state, means for adjusting the display order of chapters and summaries based on the emotional state, and means for filtering search results based on the emotional state. This allows the user to prioritize viewing parts that interest them.

[0436] A "video file" is a file that is stored in digital format and contains video and audio.

[0437] "Means for receiving" refers to hardware or software for acquiring data from the outside and incorporating it into the system.

[0438] "Audio data" is digital sound information extracted from the audio track of a video file.

[0439] "Means for converting voice data into text data" refers to a process of converting voice information into text information using a voice recognition engine.

[0440] "Text data" is character information converted from voice data.

[0441] "Major topics and key points" refer to important themes and key points extracted from text data.

[0442] A "chapter" is a part of a video file that is divided into multiple segments.

[0443] A "summary" is a sentence or paragraph that succinctly summarizes the contents of each chapter.

[0444] "Emotion recognition means" refers to technology or equipment that analyzes the user's emotional state in real time from their facial expressions, voice, operation patterns, etc.

[0445] An "emotional state" refers to the emotion a user is feeling at a particular moment.

[0446] A "means for adjusting the display order" is a process or method for changing the order in which content is displayed based on the user's emotional state.

[0447] A "search result filtering method" is a process or technique that sorts search results based on the user's emotional state and tailors the content displayed to specific conditions.

[0448] The present invention provides a system for improving the user experience by quickly acquiring necessary information from video files and adjusting the display order and search results based on the user's emotions. An embodiment of the system will be described below.

[0449] The system includes means for receiving a video file, means for extracting audio data, means for converting the audio data into text data, means for extracting main topics and key points, means for dividing into chapters, means for generating summaries, means for saving the generated chapters and summaries, means for displaying the saved information, means for displaying search results by inputting specific keywords, means for recognizing a user's emotions, means for adjusting the display order based on the user's emotional state, and means for filtering the search results.

[0450] The server receives video files from user devices and extracts audio data from the received videos using common media processing tools such as FFmpeg. The extracted audio data is converted into text data using a speech recognition engine such as Google Cloud Speech-to-Text or DeepSpeech. The converted text data is then used to extract major topics and key points using natural language processing libraries such as NLTK or spaCy.

[0451] The server then divides the video into chapters based on the extracted topics and key points, and generates summaries for each chapter using a generative AI model such as GPT-3. The generated chapters and summaries are stored in a database and later displayed through a user interface.

[0452] When a user enters a specific keyword to perform a search, the server uses a search engine such as Elasticsearch to perform a full-text search of the text data and extracts the relevant chapter information. Based on the results, the device displays the search results and the user can play that chapter.

[0453] The emotion recognition means analyzes the user's emotional state in real time using the smartphone's camera, microphone, touch operation data, etc. Specifically, it utilizes Microsoft® Azure® Emotion API and Affectiva SDK. Based on this emotion recognition, the server adjusts the order of chapters and summaries displayed and further filters search results. For example, if the user shows interest, relevant chapters and summaries are displayed preferentially.

[0454] An example of a specific input prompt is as follows:

[0455] input_video_path = " / path / to / uploads / your_video.mp4"

[0456] user_emotion = "interested"

[0457] In this way, the invention is a system that provides efficient information acquisition and a personalized experience.

[0458] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0459] Step 1:

[0460] Uploading a video file

[0461] What happens: A user uses a device to upload a video file to the system.

[0462] Input: A video file selected by the user.

[0463] Output: The uploaded video file is sent to the server, and the server receives the video file and saves it in its internal storage.

[0464] Specific operation: The device sends a video file to the server as an HTTP POST request, and the server receives the file and registers the storage path and metadata in the database.

[0465] Step 2:

[0466] Extracting audio data

[0467] Processing content: Extracts audio data from the video file received by the server.

[0468] Input: Saved video file.

[0469] Output: The extracted audio data.

[0470] Specific operation: The server uses the FFmpeg tool to extract the audio track from the video file and generate the audio data.

[0471] Step 3:

[0472] Convert speech to text

[0473] Processing content: Convert the extracted audio data into text data.

[0474] Input: Audio data.

[0475] Output: The converted text data.

[0476] How it works: The server uses a speech recognition engine such as Google Cloud Speech-to-Text or DeepSpeech to analyze the audio data and convert it into text. The converted text data is saved as an intermediate file.

[0477] Step 4:

[0478] Extracting major topics and key points

[0479] What it does: Extract major topics and key points from text data.

[0480] Input: Text data.

[0481] Output: Major topics and key points.

[0482] How it works: The server analyzes the text data using natural language processing libraries (such as NLTK or spaCy) to extract important topics and key points from the text. This information is then stored in a database.

[0483] Step 5:

[0484] Chapter Generation

[0485] What it does: Divide the video into chapters based on extracted topics and key points.

[0486] Input: Major topics and key points.

[0487] Output: Chapter information.

[0488] Specific operation: The server identifies the start and end times of each topic and divides the video into chapters. The chapter information is listed and stored in a database.

[0489] Step 6:

[0490] Generate a summary for each chapter

[0491] Process: Generate a summary for each chapter.

[0492] Input: Chapter information.

[0493] Output: The generated summary.

[0494] How it works: The server uses a generative AI model (e.g., GPT-3) to summarize the content of each chapter and stores the summaries in a database.

[0495] Step 7:

[0496] Displaying analysis results

[0497] What it does: Display the saved chapters and summaries in the user interface.

[0498] Input: Chapter and abstract information stored in the database.

[0499] Output: Chapter and summary information displayed on the device.

[0500] Specific operation: The server retrieves the analysis results from the database and sends them to the device in JSON format. The device then displays the analysis results on the user interface based on the received data.

[0501] Step 8:

[0502] Keyword search

[0503] What happens: A user enters a specific keyword into the search bar and displays search results.

[0504] Input: The keyword entered by the user.

[0505] Output: Chapter information and summary displayed as search results.

[0506] Specific operation: The server performs a full-text search of the text data using Elasticsearch or similar tools to extract the relevant chapter information. The device displays the received search results in a list, and the user can click the play button to watch the relevant section.

[0507] Step 9:

[0508] Emotion recognition and display order adjustment

[0509] What it does: Recognizes the user's emotional state and adjusts the order in which chapters and summaries are displayed based on that information.

[0510] Input: User emotion data.

[0511] Output: Adjusted chapter and summary display order.

[0512] Specific operation: The emotion recognition means analyzes the user's emotions in real time using the device's camera, microphone, and touch operation data. The server adjusts the display order based on the emotion data and prioritizes displaying the parts that are of interest to the user.

[0513] Step 10:

[0514] Sentiment-based search result filtering

[0515] What it does: Filter search results based on emotional state.

[0516] Input: User sentiment data and search keywords.

[0517] Output: The filtered search results.

[0518] How it works: The server adjusts search results based on emotional data. For example, if a user is feeling stressed, chapters with relaxing content will be prioritized.

[0519] In this way, through each step, the user can view the parts that interest him / her preferentially, and can efficiently obtain the necessary information.

[0520] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0521] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0522] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0523] [Second embodiment]

[0524] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0525] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0526] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0527] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0528] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0529] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0530] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0531] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0532] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0533] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0534] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0535] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0536] This invention is a system for acquiring "necessary information" from video content in a "short time." The specific processing and operation of each step will be described below.

[0537] Importing videos

[0538] A user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the system, and the server receives the video file and saves it in the specified directory. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[0539] Video Analysis

[0540] The server extracts the audio track from the stored video file, for example using a tool such as FFmpeg. The server then sends the extracted audio data to a speech recognition engine (for example, Google Cloud Speech-to-Text or DeepSpeech) to convert it into text. The resulting text data is saved as an intermediate file.

[0541] Natural Language Processing

[0542] The server reads the text data and uses natural language processing libraries (e.g., NLTK or spaCy) to extract major topics and key points, clarifying the content of the video and establishing a foundation for the next steps.

[0543] Chapter generation and summarization

[0544] The server divides the video into chapters based on topics and key points, identifies the start and end times of each chapter, and lists the chapter information. Next, the server generates summaries of the text data for each chapter using a summarization tool (e.g., GPT-3 or BERT). The generated chapters and summaries are stored in a database.

[0545] Providing a user interface

[0546] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the device in JSON format or similar. The device then displays information for each chapter on the user interface based on the analysis results received. This information includes the start time, summary, and thumbnail of each chapter. This allows the user to select the part they want to watch and play it.

[0547] Reverse search function

[0548] The user enters a specific keyword in the search bar and clicks the search button. The server performs a full-text search of the text data within the video based on the received keyword (using Elasticsearch, for example). The server extracts the relevant chapter information as search results and sends it to the device. The device displays the received search results in list format on the user interface. The user selects the part they want to view from the displayed search results and clicks the play button. The device then plays the video from the specific part of the chapter they clicked.

[0549] Specific examples

[0550] For example, consider the case where a university student uploads a recorded lecture. When the user uploads the lecture video file to the system, the server extracts the audio data and converts it into text data using a speech recognition engine. Then, natural language processing technology is used to extract key topics from the text data and divide the video into chapters based on this. The content of each chapter is summarized and displayed on the device, allowing the user to select the chapter they want to view and review it efficiently. Users can also quickly find the information they need by searching for specific keywords.

[0551] In this way, the present invention provides an effective means for efficiently obtaining necessary information from video content.

[0552] The processing flow will be explained below.

[0553] Step 1:

[0554] Uploading videos

[0555] The user logs into the system, selects a video file, and clicks the upload button.

[0556] The terminal transmits the selected video file to the system.

[0557] Step 2:

[0558] Saving video files

[0559] The server saves the received video file in the specified directory in the internal storage.

[0560] The server registers the destination path and related metadata (file name, upload date and time, user ID, etc.) in a database.

[0561] Step 3:

[0562] Extracting audio data

[0563] The server extracts the audio track from the stored video file using a tool such as FFmpeg.

[0564] Step 4:

[0565] Voice Recognition

[0566] The server sends the extracted voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech) and converts it into text data.

[0567] The server stores the converted text data as an intermediate file.

[0568] Step 5:

[0569] Natural Language Processing

[0570] The server reads the text data stored as intermediate files and extracts major topics and key points using natural language processing libraries (e.g., NLTK or spaCy).

[0571] Step 6:

[0572] Chapter Generation

[0573] The server divides the video into chapters based on the extracted topics and key points, identifies the start and end times of each chapter, and lists the chapter information.

[0574] Step 7:

[0575] Generate a summary

[0576] Based on each chapter, the server generates a summary for each chapter using a summarization tool (e.g., GPT-3 or BERT).

[0577] The server stores the generated chapters and summaries in a database.

[0578] Step 8:

[0579] Sending analysis results

[0580] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the terminal in JSON format or similar.

[0581] Step 9:

[0582] View Chapters

[0583] Based on the analysis results, the device displays information for each chapter on the user interface, including the start time, summary, and thumbnail of each chapter.

[0584] Step 10:

[0585] Play video

[0586] The user selects the chapter they want to watch and clicks the play button.

[0587] The device will play the video from the start time of the selected chapter using HTML5 Video Player or VLC.

[0588] Step 11:

[0589] Keyword search

[0590] The user enters a specific keyword into the search bar and clicks the search button.

[0591] The server performs a full-text search of the text data in the video based on the received keywords (e.g., using Elasticsearch).

[0592] The server extracts the relevant chapter information as a search result and transmits it to the terminal.

[0593] Step 12:

[0594] Displaying search results

[0595] The device displays the received search results in a list format, with a timestamp and summary of the chapters in each search result.

[0596] The user selects the part they want to watch from the search results and clicks the play button.

[0597] The device will play the video from the specific part of the chapter you clicked.

[0598] Example 1

[0599] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0600] In recent years, the amount of video content has rapidly increased, creating a demand for users to quickly obtain the information they need from videos. However, conventional technologies are inefficient because they require users to watch the entire video to find the information they need. Furthermore, they lack the ability to search for specific content within a video, which means it takes a long time to find the desired information. Furthermore, video content summarization and automatic chapter division are performed manually, which is cumbersome and time-consuming.

[0601] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0602] In this invention, the server includes means for receiving a video file, means for extracting audio data from the video file, means for converting the audio data into text data, means for extracting major topics and key points from the text data, means for dividing the video file into chapters based on the extracted topics or key points, means for generating summaries for each chapter, means for saving the generated chapters and summaries, means for displaying the saved chapters and summaries, means for inputting a specific keyword to perform a search and displaying the search results, and means for extracting relevant chapter information as the search results and playing the content from the first section. This allows users to efficiently obtain the information they need from videos and quickly search for and play specific content.

[0603] The "means for receiving video files" is a mechanism by which a user uploads video files to the system and the server receives the video files.

[0604] A "means for extracting audio data from video files" is a tool or program used to separate the audio track from a saved video file and obtain the audio data.

[0605] The "means for converting voice data into text data" refers to a mechanism that analyzes the extracted voice data and converts it into text data, such as using a voice recognition engine.

[0606] "Means for extracting major topics and key points from text data" refers to natural language processing techniques for analyzing text data and identifying important topics and key points from its content.

[0607] The "means for dividing a video file into chapters based on extracted topics or key points" refers to a mechanism for dividing the content of a video based on the analysis results and identifying each section.

[0608] The "means for generating a summary for each chapter" is a technology for automatically creating a summary that concisely summarizes the contents of each chapter.

[0609] The "means for saving the generated chapters and summaries" is a mechanism for storing the divided chapter information and summaries thereof in a database or the like.

[0610] The "means for displaying saved chapters and summaries" is a mechanism for displaying the analysis results on a terminal, allowing the user to view the information.

[0611] "Means for searching by entering specific keywords and displaying search results" is a mechanism for searching the text data of videos based on keywords entered by the user and displaying matching results.

[0612] "Means for extracting relevant chapter information as a search result and playing from the first section" refers to a mechanism that starts playing the video from the relevant point based on the chapter information obtained by the search.

[0613] This invention is a system that enables a user to quickly and efficiently obtain necessary information from video content. The specific processing and operation of each step will be described in detail below.

[0614] First, the user logs in to the system using a terminal and clicks the upload button to specify a video file. When the user selects a video file and executes upload, the terminal sends the selected video file to the server. The server receives the video file and saves it in the specified directory. At this time, metadata such as the save path and file name, upload date and time, and user ID are registered in the database.

[0615] Next, the server extracts the audio track from the saved video file. This process is performed using a tool such as FFmpeg, specifically by executing FFmpeg commands to separate the audio data. The server then sends the extracted audio data to a speech recognition engine (such as Google Cloud Speech-to-Text or DeepSpeech) for analysis. The speech recognition engine converts the audio data into text data and returns the results to the server. The server then saves the obtained text data as an intermediate file.

[0616] The server reads the stored text data and extracts major topics and key points using natural language processing libraries (e.g., NLTK or spaCy). This process involves tokenizing the text data, tagging it with POS tags, and recognizing entities. The results of this analysis are used in the next step.

[0617] The server divides the video into chapters based on the extracted topics and key points. It calculates the start and end times of each chapter and lists the chapter information. The server then generates a summary of each chapter's text data using a summarization tool (such as GPT-3 or BERT). This allows for a concise summary of the content of each chapter. The generated chapters and summaries are stored in a database.

[0618] The server then retrieves the generated chapter and summary information from the database. The retrieved data is converted to JSON format and sent to the device. The device then displays information for each chapter on the user interface based on the analysis results. The displayed information includes the chapter start time, summary, thumbnail, etc. Through the interface, the user can select the part they want to watch and play it.

[0619] Users can also enter specific keywords into the search bar and click the search button. The device sends this input data to the server. The server performs a full-text search of the text data within the video based on the received keywords. This search uses a search engine such as Elasticsearch. The server extracts the relevant chapter information as search results and sends it to the device. The device displays the received search results in list format on the user interface. The user can select the part they want to view from the displayed search results and play it.

[0620] As a specific example, consider the case where a university student uploads a recorded lecture. When a user uploads a video file of a lecture to the system, the server extracts the audio data and converts it into text data using a speech recognition engine. The server then uses natural language processing technology to extract major topics from the text data and divides the video into chapters based on this. The content of each chapter is summarized and displayed on the device. Users can select the chapters they want to view and review them efficiently. They can also quickly find the information they need by searching for specific keywords. This system provides an effective means for efficiently obtaining the information they need from video content.

[0621] An example prompt is as follows:

[0622] "I want to easily review lecture videos. I would like to know if there is a system that will automatically summarize the lecture content and divide it into easy-to-understand chapters when I upload a video file."

[0623] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0624] Step 1:

[0625] A user logs in to the system using a terminal and uploads a video file by clicking the upload button, selecting the video file, and pressing the send button.

[0626] Input: Video file (e.g. lecture recording file)

[0627] Output: A request to transfer a video file to the server

[0628] Specific operation: The device sends an HTTP POST request to the server and sends request data including the selected video file.

[0629] Step 2:

[0630] The server saves the received video file in the specified directory and registers the destination path, file name, upload date and time, user ID, and other metadata in the database.

[0631] Input: Video file, metadata (file name, upload date and time, user ID, etc.)

[0632] Output: Path where the video file is saved, metadata registered in the database

[0633] Specific operation: The server saves the received video file in the specified directory of the file system and records the save path in the database.

[0634] Step 3:

[0635] The server extracts the audio track from the saved video file using a tool such as FFmpeg.

[0636] Input: Saved video file

[0637] Output: Extracted audio data file

[0638] Specific operation: The server executes FFmpeg commands to extract the audio track from the video file and save it as an audio data file.

[0639] Step 4:

[0640] The server sends the extracted voice data to a voice recognition engine (such as Google Cloud Speech-to-Text or DeepSpeech) and converts it into text data.

[0641] Input: Audio data file

[0642] Output: The resulting text data

[0643] Specific operation: The server calls the API of the speech recognition engine, sends the voice data, receives the resulting text data, and saves it as an intermediate file.

[0644] Step 5:

[0645] The server reads the stored text data and extracts major topics and key points using natural language processing libraries (such as NLTK or spaCy).

[0646] Input: Text data

[0647] Output: A list of topics and keypoints

[0648] What it does: The server reads the text data and uses natural language processing libraries to tokenize the text, tag it with POS tags, recognize entities, and extract key topics and key points.

[0649] Step 6:

[0650] The server divides the video into chapters based on the extracted topics and key points, calculates the start and end times of each chapter, and lists the chapter information.

[0651] Input: List of topics and keypoints

[0652] Output: List of chapter information (start time, end time, content)

[0653] Specific operation: The server divides the video into multiple sections based on the timestamps of the topics extracted from the text data, and lists the start and end times of each section.

[0654] Step 7:

[0655] The server generates a summary of the text data for each chapter using a summarization tool (such as GPT-3 or BERT).

[0656] Input: Text data for each chapter

[0657] Output: Summary text

[0658] Specific operation: The server calls the summary generation API, sends the text data for each chapter, and generates a summary text based on the results.

[0659] Step 8:

[0660] The server stores the generated chapter and summary information in a database.

[0661] Input: List of chapter information, summary text

[0662] Output: Chapter and summary information stored in a database

[0663] Specific operation: The server inserts and saves each chapter information and summary into the database.

[0664] Step 9:

[0665] The server retrieves the generated chapter and summary information from the database, converts it into JSON format, and sends it to the terminal.

[0666] Input: Chapter and summary information stored in the database

[0667] Output: JSON format data

[0668] Specific operation: The server queries the database for the required data, serializes the retrieved data into JSON format, and returns it to the device as an HTTP response.

[0669] Step 10:

[0670] The device then displays information for each chapter on the user interface based on the analysis results received, allowing the user to select the part they want to watch and play it.

[0671] Input: Analysis result data in JSON format

[0672] Output: Chapter information displayed in the user interface

[0673] What it does: The device parses the received JSON data and renders an interface that includes the start time, summary, thumbnails, etc. for each chapter.

[0674] Step 11:

[0675] The user inputs a specific keyword and clicks the search button, and the device sends this input data to the server.

[0676] Input: Keywords (e.g., specific lecture content)

[0677] Output: Search request based on keywords

[0678] Specific operation: The device obtains the keywords entered in the search bar and sends them to the server as an HTTP request.

[0679] Step 12:

[0680] The server performs a full-text search of the text data in the video based on the received keywords (using Elasticsearch, etc.), extracts the relevant chapter information as search results, and sends it to the device.

[0681] Input: Keywords, text data

[0682] Output: Chapter information as search results

[0683] Specific operation: The server queries Elasticsearch for the received keywords, extracts chapter information for matching text, serializes it in JSON format, and returns it to the device.

[0684] Step 13:

[0685] The terminal displays the received search results in a list format on the user interface. The user selects the part they want to watch from the displayed search results and clicks the play button.

[0686] Input: Search results (JSON data)

[0687] Output: Search results displayed in the user interface

[0688] Specific operation: The terminal parses the received JSON data and displays the results in a list format in the user interface.

[0689] Step 14:

[0690] The device will play the video from the specific part of the chapter you clicked.

[0691] Input: User selected chapter information

[0692] Output: Video played from the specified chapter

[0693] Specific operation: The device captures the user's click event and starts playing the video from the start time of the corresponding chapter.

[0694] (Application example 1)

[0695] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0696] In today's world, the amount of video content is increasing, and users are required to quickly obtain the information they need. However, it is not easy to extract the necessary information from long video content and grasp it in a short time. In addition, because the content contains a large amount of information, there is also the problem that viewing and searching cannot be performed efficiently. Therefore, an object of the present invention is to provide a system that efficiently extracts the necessary information from video content and provides it in a short time.

[0697] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0698] In this invention, the server includes a means for receiving video files, a means for extracting audio data from the received video files, and a means for converting the audio data into text data. This allows for efficient extraction of major topics and key points from video content, division of the video into multiple chapters, and generation and display of summaries. The system also includes a means for generating summaries based on input prompts using a generative AI model, and furthermore, enables searching for specific keywords and playback of relevant sections. This allows users to quickly obtain the information they need and efficiently watch and search videos.

[0699] A "video file" is a digital file that contains video and audio data.

[0700] "Audio data" refers to sound data containing human voices extracted from video files.

[0701] "Text data" refers to data obtained by converting voice data into character information.

[0702] "Major topics" are important topics or themes of videos extracted from text data.

[0703] "Key points" are important points or elements related to the main topic.

[0704] "Chapters" are sections of a video file that are divided based on specific topics or key points.

[0705] A "summary" is a text that briefly describes the contents of each chapter.

[0706] A "generative AI model" is an artificial intelligence model that generates and understands text based on large datasets.

[0707] A "prompt" is a text instruction that is input to a generative AI model.

[0708] The "means for receiving" is the part that has the function of uploading and saving video files to the system.

[0709] The "extraction means" is the part that has the function of extracting audio data from a video file.

[0710] The "conversion means" is a part that has the function of converting voice data into text data.

[0711] The "means for generating a summary" is a part that has the function of converting text data into a summary.

[0712] The "means for saving" is a part that has the function of saving the generated chapters and summaries.

[0713] The "display means" is a part that has the function of displaying the saved chapters and summaries on the user interface.

[0714] The "search means" is the part that has the function of searching the text data within the video based on specific keywords and displaying the results.

[0715] "Means for playback" refers to the part that has the function of playing the video of the relevant part of the search results.

[0716] This invention is a system that extracts necessary information from video content in a short time and efficiently provides it to users. The system receives video files, extracts and converts audio data, extracts major topics and key points, generates summaries using a generative AI model, and displays and searches the results.

[0717] First, a user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the server, which receives the video file and saves it in the specified directory. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[0718] Next, the server extracts the audio track from the saved video file using a tool such as FFmpeg. The server then sends the extracted audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert it into text data, which is then saved as an intermediate file.

[0719] The server then reads the text data and uses natural language processing libraries (e.g., spaCy) to extract major topics and key points, clarifying the content of the video and establishing a foundation for the next steps.

[0720] The server divides the video into chapters based on topics and key points. It identifies the start and end times of each chapter and lists the chapter information. The server then generates a summary of each chapter's text data using a summarization tool (e.g., GPT-3). The generated chapters and summaries are stored in a database.

[0721] When a user enters a specific keyword, the server performs a full-text search of the text data within the video based on the received keyword. The server extracts the relevant chapter information as search results and sends it to the device. The device then displays the received search results in a list format on the user interface. The user can then select the part they want to watch from the displayed search results and play it.

[0722] As a concrete example, consider the case where a university student uploads a recorded lecture. When a user uploads a video file of a lecture to the system, the server extracts the audio data and converts it into text data using a speech recognition engine. Then, natural language processing technology is used to extract key topics from the text data and divide the video into chapters based on these. The content of each chapter is summarized and displayed on the device, allowing the user to select the chapter they want to view and review it efficiently.

[0723] An example of a prompt sentence in a generative AI model is "Please summarize the following text in a concise manner: \n\n[text content]", which allows the generative AI model to summarize the input text.

[0724] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0725] Step 1:

[0726] A user logs in to the system using a terminal and uploads a video file. The terminal sends the video file selected by the user to the server. The input is the video file, and the output is the video file received by the server.

[0727] Step 2:

[0728] The server saves the received video file in the specified directory. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database. The input is the video file, and the output is the saved video file and the registration of metadata in the database.

[0729] Step 3:

[0730] The server extracts the audio track from the stored video file using a tool such as FFmpeg. The input is the stored video file, and the output is the extracted audio data.

[0731] Step 4:

[0732] The server sends the extracted voice data to a voice recognition engine (such as Google Cloud Speech-to-Text) and converts it into text data. The input is voice data, and the output is the converted text data.

[0733] Step 5:

[0734] The server reads the text data and extracts the main topics and keypoints using a natural language processing library (such as spaCy). The input is the text data, and the output is the extracted main topics and keypoints.

[0735] Step 6:

[0736] The server divides the video into chapters based on topics and key points, identifies the start and end times of each chapter, and lists the chapter information. The input is the main topics and key points, and the output is the divided chapter information.

[0737] Step 7:

[0738] Next, the server generates a summary of the text data of each chapter using a summarization tool (a generative AI model such as GPT-3). A prompt is used for generation. The input is the text data and prompt for each chapter, and the output is the generated summary. An example of a prompt is "Please summarize the following text in a concise manner: \n\n[text content]".

[0739] Step 8:

[0740] The server stores the generated chapters and summaries in a database. The input is the generated chapters and summaries, and the output is the data stored in the database.

[0741] Step 9:

[0742] When a user inputs a specific keyword, the server performs a full-text search of the text data in the video based on the received keyword. The input is the specific keyword, and the output is the searched chapter information.

[0743] Step 10:

[0744] The server extracts the relevant chapter information as the search result and sends it to the terminal. The input is the chapter information of the search result, and the output is the search result sent to the terminal.

[0745] Step 11:

[0746] The terminal displays the received search results in a list format on the user interface. The user can select the part they want to watch from the displayed search results and play it. The input is the chapter information of the search results, and the output is the search results displayed on the user interface.

[0747] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0748] This invention improves the user experience by combining a system that quickly acquires necessary information from video content with an emotion engine that recognizes the user's emotions. The specific processing and operation of each step are explained below.

[0749] Importing videos

[0750] A user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the system, and the server receives the video file and saves it in its internal storage. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[0751] Video Analysis

[0752] The server extracts the audio track from the stored video file, typically using a tool like FFmpeg. The server then sends the extracted audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech) to convert it into text data, which is then saved as an intermediate file.

[0753] Natural Language Processing

[0754] The server reads the text data stored as intermediate files and uses natural language processing libraries (e.g., NLTK or spaCy) to extract major topics and key points, clarifying the content of the video and generating the foundational data for the next step.

[0755] Chapter generation and summarization

[0756] The server divides the video into multiple chapters based on the extracted topics and key points. It identifies the start and end times of each chapter and lists the chapter information. Then, the server generates a summary for each chapter using a summarization tool (e.g., GPT-3 or BERT) based on each chapter. The generated chapters and summaries are stored in a database.

[0757] Providing a user interface

[0758] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the device in JSON format or similar. The device then displays information for each chapter on the user interface based on the analysis results received. The start time, summary, thumbnail, and other information for each chapter are displayed, allowing the user to select and play the part they want to watch.

[0759] Reverse search function

[0760] The user enters a specific keyword in the search bar and clicks the search button. The server uses Elasticsearch or similar tools to perform a full-text search of the text data within the video based on the received keyword. The server extracts the relevant chapter information as search results and sends it to the device. The device displays the received search results in list format, and for each chapter in the search results, displays the timestamp where the keyword appears and a summary. The user can select the part they want to watch and click the play button to play the video from that specific part of that chapter.

[0761] Emotion engine integration

[0762] The present invention further integrates an emotion engine that recognizes the user's emotions. The emotion engine allows the device to analyze the user's emotional state in real time using the user's facial expressions, voice, keyboard and mouse operation patterns, etc. For example, if the system determines that the user is interested, it will prioritize displaying related chapters and summaries.

[0763] Emotion-based chapter display

[0764] The server receives the emotion data obtained by the emotion engine and adjusts the display order of chapters and summaries based on the user's emotional state, thereby prioritizing the display of content that is likely to interest the user and improving the user experience.

[0765] Filtering search results based on sentiment

[0766] When a user enters a search keyword, the emotional engine provides emotional data to filter search results. For example, if a user is feeling stressed, chapters with relaxing content will be prioritized.

[0767] Specific examples

[0768] For example, when a user watches a recording of a business meeting and uses the system with the emotion engine, the system analyzes the user's emotions and prioritizes displaying important parts or parts that the user is likely to be interested in. This allows the user to efficiently obtain the information they need.

[0769] In this way, the present invention provides a means for efficiently obtaining necessary information from video content, while also providing a more personalized experience by taking into account the user's emotional state.

[0770] The processing flow will be explained below.

[0771] Step 1:

[0772] Uploading videos

[0773] The user logs into the system, selects a video file, and clicks the upload button.

[0774] The terminal transmits the selected video file to the system.

[0775] Step 2:

[0776] Saving video files

[0777] The server saves the received video file in a specified directory in the internal storage.

[0778] The server registers the destination path and related metadata (file name, upload date and time, user ID, etc.) in a database.

[0779] Step 3:

[0780] Extracting audio data

[0781] The server extracts the audio track from the stored video file using a tool such as FFmpeg.

[0782] Step 4:

[0783] Voice Recognition

[0784] The server sends the extracted voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech) and converts it into text data.

[0785] The server stores the converted text data as an intermediate file.

[0786] Step 5:

[0787] Natural Language Processing

[0788] The server reads the text data stored as intermediate files and extracts major topics and key points using natural language processing libraries (e.g., NLTK or spaCy).

[0789] Step 6:

[0790] Chapter Generation

[0791] The server divides the video into multiple chapters based on the extracted topics and key points, identifies the start and end times of each chapter, and lists the chapter information.

[0792] Step 7:

[0793] Generate a summary

[0794] Based on each chapter, the server generates a summary for each chapter using a summarization tool (e.g., GPT-3 or BERT).

[0795] The server stores the generated chapters and summaries in a database.

[0796] Step 8:

[0797] Sending analysis results

[0798] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the terminal in JSON format or similar.

[0799] Step 9:

[0800] View Chapters

[0801] Based on the analysis results, the device displays information for each chapter on the user interface, including the start time, summary, and thumbnail of each chapter.

[0802] Step 10:

[0803] Starting emotion recognition

[0804] The device runs an emotion engine and analyzes the user's facial expressions, voice, and operation patterns, including input from the camera, microphone, mouse, and keyboard.

[0805] The emotion engine analyzes the acquired data in real time to determine the user's emotional state.

[0806] Step 11:

[0807] Emotion-based interface adjustment

[0808] The server adjusts the display order of chapters and the content of summaries based on the emotion data received from the emotion engine.

[0809] For example, if it is determined that the user is interested, related chapters are preferentially displayed.

[0810] Step 12:

[0811] Play video

[0812] The user selects the chapter they wish to view and clicks the play button.

[0813] The device will play the video from the start time of the selected chapter.

[0814] Step 13:

[0815] Keyword search

[0816] The user enters a specific keyword in the search bar and clicks the search button.

[0817] The server performs a full-text search of the text data in the video based on the received keywords (e.g., using Elasticsearch).

[0818] The server extracts the relevant chapter information as the search result and transmits it to the terminal.

[0819] Step 14:

[0820] Filtering search results based on sentiment

[0821] The server uses data from the emotion engine to filter search results according to the user's emotions.

[0822] For example, if you want to relax, a chapter with relaxing content will be displayed.

[0823] Step 15:

[0824] Displaying search results

[0825] The device displays the received search results in a list format, with a timestamp and summary of the chapters in each search result.

[0826] The user selects the portion of the search result that he or she wants to view from the displayed search results and clicks the play button.

[0827] The device will play the video from the specific part of the chapter you clicked.

[0828] Example 2

[0829] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0830] In recent years, the use of video data has increased, creating a need for quickly extracting only the necessary parts from this vast amount of information. However, conventional technologies have been unable to take into account user emotion data when converting audio from video to text, extracting key topics and key points, dividing video into segments, and generating summaries, making it difficult to provide optimal information. Furthermore, the search function was unable to filter content based on user emotion, resulting in a poor user experience.

[0831] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0832] In this invention, the server includes means for receiving video data, means for extracting audio data from the received video data, means for converting the audio data into text data, means for extracting main topics and important points from the text data, means for dividing the video data into segments based on the extracted topics or important points, means for generating summaries for each segment, means for saving the generated segments and summaries, means for displaying the saved segments and summaries, means for inputting a specific keyword for search and displaying search results, means for analyzing user emotion data, and means for filtering the display order and search results based on the emotion data, thereby enabling optimal information provision that takes user emotions into consideration.

[0833] "Video data" refers to data in a file format that includes video and audio.

[0834] "Audio data" refers to data consisting of only the audio track extracted from video data.

[0835] "Text data" refers to data obtained by converting voice data into character information.

[0836] "Major topics" refer to themes or topics that are considered particularly important in the content of a video.

[0837] "Important points" refer to specific points or key information in the video content that are particularly noteworthy.

[0838] "Segmenting" refers to dividing video data into multiple parts and treating each part as an independent chapter.

[0839] A "segment" refers to each part of divided video data.

[0840] "Generating a summary" refers to creating a short description that succinctly summarizes the content of each segment.

[0841] "Storing" refers to recording the generated data in storage such as a database.

[0842] "Displaying" refers to visually showing the generated data in a user interface.

[0843] "Searching" refers to matching information in a database using specific keywords and extracting relevant results.

[0844] "Displaying search results" refers to displaying the information obtained by the search in a user interface.

[0845] "Analyzing user emotional data" refers to evaluating the user's emotional state in real time based on their facial expressions, voice, operation patterns, etc.

[0846] "Filtering the display order or search results based on emotional data" refers to adjusting the order or content of the information to be displayed in accordance with the acquired emotional data.

[0847] This invention is a system that quickly acquires necessary information from video data, and improves the user experience by combining it with an emotion engine that recognizes the user's emotions. Specific operations and embodiments of this system are described below.

[0848] Importing videos

[0849] A user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the server. The server saves the received video file in its internal storage. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[0850] Video Analysis

[0851] The server extracts the audio track from the saved video file using a tool called FFmpeg. The server then sends the extracted audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech), which converts the audio data into text data. The text data is then saved as an intermediate file.

[0852] Natural Language Processing

[0853] The server reads the text data stored as intermediate files and uses natural language processing libraries (e.g., NLTK or spaCy) to extract the main topics and key points, clarifying the content of the video and generating the foundational data for the next step.

[0854] Chapter generation and summarization

[0855] The server divides the video into multiple segments based on the extracted topics and key points. It identifies the start and end times of each segment and lists the segment information. Then, the server generates a summary for each segment using a summarization tool (e.g., GPT-3 or BERT). The generated segments and summaries are stored in a database.

[0856] Providing a user interface

[0857] The server retrieves the generated segment and summary information from the database and sends the analysis results to the device in JSON format or similar. Based on the analysis results received, the device displays information for each segment on the user interface. The start time, summary, thumbnail, and other information for each segment are displayed, allowing the user to select and play the part they want to watch.

[0858] Reverse search function

[0859] The user enters a specific keyword in the search bar and clicks the search button. The server uses Elasticsearch or similar tools to perform a full-text search of the text data within the video based on the received keyword. The server extracts the relevant segment information as search results and sends it to the device. The device displays the received search results in list format, and for each segment in the search results, displays the timestamp where the keyword appears and a summary. The user can select the part they want to watch and click the play button to play the video from that specific part of that segment.

[0860] Emotion engine integration

[0861] The present invention further integrates an emotion engine that recognizes the user's emotions. The emotion engine allows the terminal to analyze the user's emotional state in real time using the user's facial expressions, voice, keyboard and mouse operation patterns, etc. For example, if it is determined that the user is interested, the system will prioritize displaying related segments and summaries.

[0862] Segment display based on emotions

[0863] The server receives the emotion data obtained by the emotion engine and adjusts the display order of the segments and summaries based on the user's emotional state, thereby prioritizing the display of content that is likely to interest the user and improving the user experience.

[0864] Filtering search results based on sentiment

[0865] When a user enters a search keyword, the emotion engine provides emotional data to filter search results. For example, if a user is feeling stressed, segments with relaxing content will be prioritized.

[0866] Specific examples

[0867] For example, when a user watches a recording of a business meeting, the system analyzes the user's emotions and prioritizes displaying important parts or parts likely to interest the user. This allows the user to efficiently obtain the information they need. In this way, the present invention provides a means for efficiently obtaining necessary information from video data, while also providing a more personalized experience by taking the user's emotional state into consideration.

[0868] Prompt Sentence Examples

[0869] "Please extract the main topics and key points from this video."

[0870] "Summarize and segment recordings of business meetings."

[0871] "Rearrange your segments and summaries taking into account the output data of the sentiment engine to prioritize what is likely to interest the user."

[0872] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0873] Step 1: Import video

[0874] After logging in, the user selects and uploads a video file.

[0875] Input: Video file, user information

[0876] The terminal reads the selected video file and transfers it to the server.

[0877] The server receives the video file and saves it to its internal storage. At this time, it registers the destination path and video metadata (file name, upload date and time, user ID, etc.) in a database.

[0878] Output: Video files stored in the internal storage, metadata registered in the database

[0879] Step 2: Analyze the video

[0880] The server extracts the audio track from the stored video file.

[0881] Input: Saved video file

[0882] The server uses FFmpeg to extract the audio data and sends it to a speech recognition engine, specifically, by calling the Google Cloud Speech-to-Text or DeepSpeech API.

[0883] The server converts the voice data into text data and stores the text data as an intermediate file.

[0884] Output: Text data saved as an intermediate file

[0885] Step 3: Natural Language Processing

[0886] The server reads the text data stored as an intermediate file.

[0887] Input: Text data saved as an intermediate file

[0888] The server uses natural language processing libraries (such as NLTK or spaCy) to extract key topics and key points, which involves tokenizing words and parsing sentences.

[0889] Output: Extracted main topics and key points data

[0890] Step 4: Chapter generation and summarization

[0891] The server divides the video into multiple segments based on the extracted topics and key points.

[0892] Input: Data on key topics and key points

[0893] The server identifies and lists the start and end times of each segment, possibly using a video editing library.

[0894] The server uses a summarization tool (such as GPT-3 or BERT) to generate a summary of each segment.

[0895] The generated segments and summaries are stored in a database.

[0896] Output: Generated segments and summaries stored in a database

[0897] Step 5: Providing a User Interface

[0898] The server retrieves the generated segment and summary information from the database, converts it into JSON format, and sends it to the terminal.

[0899] Input: Generated segments and summaries stored in a database

[0900] Based on the analysis results received, the device displays information for each segment on the user interface, including the start time, summary, and thumbnail of each segment.

[0901] Output: Information for each segment displayed in the user interface

[0902] Step 6: Reverse Search Function

[0903] The user enters a specific keyword into the search bar and clicks the search button.

[0904] Input: Search keyword

[0905] Based on the received keywords, the server uses Elasticsearch to perform a full-text search of the text data within the video.

[0906] The server extracts the relevant segment information as a search result and transmits it to the terminal.

[0907] The terminal displays the received search results in a list format, and displays the timestamps at which the keywords appear and a summary for each segment of each search result.

[0908] Output: Search results displayed in list format

[0909] Step 7: Integrating the Emotion Engine

[0910] The emotion engine analyzes the user's facial expressions, voice, keyboard and mouse operation patterns, and other information acquired by the device in real time.

[0911] Input: User's facial expressions, voice, and operation patterns

[0912] The device analyzes this data and estimates the emotional state.

[0913] The server adjusts the order of segments and summaries to be displayed based on the emotion data received from the emotion engine.

[0914] Output: Reordered segments and summaries based on sentiment data

[0915] Step 8: Segment based on sentiment

[0916] The server receives the emotion data obtained from the emotion engine.

[0917] Input: Emotion data

[0918] The server adjusts the display order of the segments and summaries based on the user's emotional state and transmits them again to the terminal.

[0919] The device will prioritize updating the displayed information to improve the user experience.

[0920] Output: Reordered segments and summary information

[0921] Step 9: Filter search results based on sentiment

[0922] When a user enters a search keyword and performs a search, the search results are filtered based on the emotional data provided by the emotion engine.

[0923] Input: Search keywords, emotion data

[0924] The server filters the search results taking into account the emotion data and transmits the results based on that to the terminal.

[0925] The terminal displays the filtered results in a list format.

[0926] Output: Filtered search results

[0927] Prompt Sentence Examples

[0928] "Please extract the main topics and key points from this video."

[0929] "Summarize and segment recordings of business meetings."

[0930] "Rearrange your segments and summaries taking into account the output data of the sentiment engine to prioritize what is likely to interest the user."

[0931] (Application example 2)

[0932] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0933] Systems for quickly obtaining necessary information from video content have the problem of not being able to efficiently provide the parts of a user's interest, resulting in a lack of improvement in the user experience. Furthermore, conventional systems are unable to display content taking into account the user's emotions, making it difficult to provide personalized services.

[0934] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a video file, means for extracting audio data from the received video file, means for converting the audio data into text data, means for extracting major topics and key points from the text data, means for dividing the video file into chapters based on the extracted topics or key points, means for generating summaries for each chapter, means for saving the generated chapters and summaries, means for displaying the saved chapters and summaries, means for inputting a specific keyword to perform a search and displaying search results, emotion recognition means for recognizing the user's emotional state, means for adjusting the display order of chapters and summaries based on the emotional state, and means for filtering search results based on the emotional state. This allows the user to prioritize viewing parts that interest them.

[0935] A "video file" is a file that is stored in digital format and contains video and audio.

[0936] "Means for receiving" refers to hardware or software for acquiring data from the outside and incorporating it into the system.

[0937] "Audio data" is digital sound information extracted from the audio track of a video file.

[0938] "Means for converting voice data into text data" refers to a process of converting voice information into text information using a voice recognition engine.

[0939] "Text data" is character information converted from voice data.

[0940] "Major topics and key points" refer to important themes and key points extracted from text data.

[0941] A "chapter" is a part of a video file that is divided into multiple segments.

[0942] A "summary" is a sentence or paragraph that succinctly summarizes the contents of each chapter.

[0943] "Emotion recognition means" refers to technology or equipment that analyzes the user's emotional state in real time from their facial expressions, voice, operation patterns, etc.

[0944] An "emotional state" refers to the emotion a user is feeling at a particular moment.

[0945] A "means for adjusting the display order" is a process or method for changing the order in which content is displayed based on the user's emotional state.

[0946] A "search result filtering method" is a process or technique that sorts search results based on the user's emotional state and tailors the content displayed to specific conditions.

[0947] The present invention provides a system for improving the user experience by quickly acquiring necessary information from video files and adjusting the display order and search results based on the user's emotions. An embodiment of the system will be described below.

[0948] The system includes means for receiving a video file, means for extracting audio data, means for converting the audio data into text data, means for extracting main topics and key points, means for dividing into chapters, means for generating summaries, means for saving the generated chapters and summaries, means for displaying the saved information, means for displaying search results by inputting specific keywords, means for recognizing a user's emotions, means for adjusting the display order based on the user's emotional state, and means for filtering the search results.

[0949] The server receives video files from user devices and extracts audio data from the received videos using common media processing tools such as FFmpeg. The extracted audio data is converted into text data using a speech recognition engine such as Google Cloud Speech-to-Text or DeepSpeech. The converted text data is then used to extract major topics and key points using natural language processing libraries such as NLTK or spaCy.

[0950] The server then divides the video into chapters based on the extracted topics and key points, and generates summaries for each chapter using a generative AI model such as GPT-3. The generated chapters and summaries are stored in a database and later displayed through a user interface.

[0951] When a user enters a specific keyword to perform a search, the server uses a search engine such as Elasticsearch to perform a full-text search of the text data and extracts the relevant chapter information. Based on the results, the device displays the search results and the user can play that chapter.

[0952] The emotion recognition system analyzes the user's emotional state in real time using the smartphone's camera, microphone, and touch operation data. Specifically, it utilizes Microsoft Azure Emotion API and Affectiva SDK. Based on this emotion recognition, the server adjusts the order of chapters and summaries displayed and further filters search results. For example, if the user shows interest, relevant chapters and summaries will be displayed first.

[0953] An example of a specific input prompt is as follows:

[0954] input_video_path = " / path / to / uploads / your_video.mp4"

[0955] user_emotion = "interested"

[0956] In this way, the invention is a system that provides efficient information acquisition and a personalized experience.

[0957] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0958] Step 1:

[0959] Uploading a video file

[0960] What happens: A user uses a device to upload a video file to the system.

[0961] Input: A video file selected by the user.

[0962] Output: The uploaded video file is sent to the server, and the server receives the video file and saves it in its internal storage.

[0963] Specific operation: The device sends a video file to the server as an HTTP POST request, and the server receives the file and registers the storage path and metadata in the database.

[0964] Step 2:

[0965] Extracting audio data

[0966] Processing content: Extracts audio data from the video file received by the server.

[0967] Input: Saved video file.

[0968] Output: The extracted audio data.

[0969] Specific operation: The server uses the FFmpeg tool to extract the audio track from the video file and generate the audio data.

[0970] Step 3:

[0971] Convert speech to text

[0972] Processing content: Convert the extracted audio data into text data.

[0973] Input: Audio data.

[0974] Output: The converted text data.

[0975] How it works: The server uses a speech recognition engine such as Google Cloud Speech-to-Text or DeepSpeech to analyze the audio data and convert it into text. The converted text data is saved as an intermediate file.

[0976] Step 4:

[0977] Extracting major topics and key points

[0978] What it does: Extract major topics and key points from text data.

[0979] Input: Text data.

[0980] Output: Major topics and key points.

[0981] How it works: The server analyzes the text data using natural language processing libraries (such as NLTK or spaCy) to extract important topics and key points from the text. This information is then stored in a database.

[0982] Step 5:

[0983] Chapter Generation

[0984] What it does: Divide the video into chapters based on extracted topics and key points.

[0985] Input: Major topics and key points.

[0986] Output: Chapter information.

[0987] Specific operation: The server identifies the start and end times of each topic and divides the video into chapters. The chapter information is listed and stored in a database.

[0988] Step 6:

[0989] Generate a summary for each chapter

[0990] Process: Generate a summary for each chapter.

[0991] Input: Chapter information.

[0992] Output: The generated summary.

[0993] How it works: The server uses a generative AI model (e.g., GPT-3) to summarize the content of each chapter and stores the summaries in a database.

[0994] Step 7:

[0995] Displaying analysis results

[0996] What it does: Display the saved chapters and summaries in the user interface.

[0997] Input: Chapter and abstract information stored in the database.

[0998] Output: Chapter and summary information displayed on the device.

[0999] Specific operation: The server retrieves the analysis results from the database and sends them to the device in JSON format. The device then displays the analysis results on the user interface based on the received data.

[1000] Step 8:

[1001] Keyword search

[1002] What happens: A user enters a specific keyword into the search bar and displays search results.

[1003] Input: The keyword entered by the user.

[1004] Output: Chapter information and summary displayed as search results.

[1005] Specific operation: The server performs a full-text search of the text data using Elasticsearch or similar tools to extract the relevant chapter information. The device displays the received search results in a list, and the user can click the play button to watch the relevant section.

[1006] Step 9:

[1007] Emotion recognition and display order adjustment

[1008] What it does: Recognizes the user's emotional state and adjusts the order in which chapters and summaries are displayed based on that information.

[1009] Input: User emotion data.

[1010] Output: Adjusted chapter and summary display order.

[1011] Specific operation: The emotion recognition means analyzes the user's emotions in real time using the device's camera, microphone, and touch operation data. The server adjusts the display order based on the emotion data and prioritizes displaying the parts that are of interest to the user.

[1012] Step 10:

[1013] Sentiment-based search result filtering

[1014] What it does: Filter search results based on emotional state.

[1015] Input: User sentiment data and search keywords.

[1016] Output: The filtered search results.

[1017] How it works: The server adjusts search results based on emotional data. For example, if a user is feeling stressed, chapters with relaxing content will be prioritized.

[1018] In this way, through each step, the user can view the parts that interest him / her preferentially, and can efficiently obtain the necessary information.

[1019] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1020] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1021] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1022] [Third embodiment]

[1023] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1024] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1026] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1027] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1028] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1030] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1031] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1033] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1034] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1035] This invention is a system for acquiring "necessary information" from video content in a "short time." The specific processing and operation of each step will be described below.

[1036] Importing videos

[1037] A user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the system, and the server receives the video file and saves it in the specified directory. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[1038] Video Analysis

[1039] The server extracts the audio track from the stored video file, for example using a tool such as FFmpeg. The server then sends the extracted audio data to a speech recognition engine (for example, Google Cloud Speech-to-Text or DeepSpeech) to convert it into text. The resulting text data is saved as an intermediate file.

[1040] Natural Language Processing

[1041] The server reads the text data and uses natural language processing libraries (e.g., NLTK or spaCy) to extract major topics and key points, clarifying the content of the video and establishing a foundation for the next steps.

[1042] Chapter generation and summarization

[1043] The server divides the video into chapters based on topics and key points, identifies the start and end times of each chapter, and lists the chapter information. Next, the server generates summaries of the text data for each chapter using a summarization tool (e.g., GPT-3 or BERT). The generated chapters and summaries are stored in a database.

[1044] Providing a user interface

[1045] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the device in JSON format or similar. The device then displays information for each chapter on the user interface based on the analysis results received. This information includes the start time, summary, and thumbnail of each chapter. This allows the user to select the part they want to watch and play it.

[1046] Reverse search function

[1047] The user enters a specific keyword in the search bar and clicks the search button. The server performs a full-text search of the text data within the video based on the received keyword (using Elasticsearch, for example). The server extracts the relevant chapter information as search results and sends it to the device. The device displays the received search results in list format on the user interface. The user selects the part they want to view from the displayed search results and clicks the play button. The device then plays the video from the specific part of the chapter they clicked.

[1048] Specific examples

[1049] For example, consider the case where a university student uploads a recorded lecture. When the user uploads the lecture video file to the system, the server extracts the audio data and converts it into text data using a speech recognition engine. Then, natural language processing technology is used to extract key topics from the text data and divide the video into chapters based on this. The content of each chapter is summarized and displayed on the device, allowing the user to select the chapter they want to view and review it efficiently. Users can also quickly find the information they need by searching for specific keywords.

[1050] In this way, the present invention provides an effective means for efficiently obtaining necessary information from video content.

[1051] The processing flow will be explained below.

[1052] Step 1:

[1053] Uploading videos

[1054] The user logs into the system, selects a video file, and clicks the upload button.

[1055] The terminal transmits the selected video file to the system.

[1056] Step 2:

[1057] Saving video files

[1058] The server saves the received video file in the specified directory in the internal storage.

[1059] The server registers the destination path and related metadata (file name, upload date and time, user ID, etc.) in a database.

[1060] Step 3:

[1061] Extracting audio data

[1062] The server extracts the audio track from the stored video file using a tool such as FFmpeg.

[1063] Step 4:

[1064] Voice Recognition

[1065] The server sends the extracted voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech) and converts it into text data.

[1066] The server stores the converted text data as an intermediate file.

[1067] Step 5:

[1068] Natural Language Processing

[1069] The server reads the text data stored as intermediate files and extracts major topics and key points using natural language processing libraries (e.g., NLTK or spaCy).

[1070] Step 6:

[1071] Chapter Generation

[1072] The server divides the video into chapters based on the extracted topics and key points, identifies the start and end times of each chapter, and lists the chapter information.

[1073] Step 7:

[1074] Generate a summary

[1075] Based on each chapter, the server generates a summary for each chapter using a summarization tool (e.g., GPT-3 or BERT).

[1076] The server stores the generated chapters and summaries in a database.

[1077] Step 8:

[1078] Sending analysis results

[1079] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the terminal in JSON format or similar.

[1080] Step 9:

[1081] View Chapters

[1082] Based on the analysis results, the device displays information for each chapter on the user interface, including the start time, summary, and thumbnail of each chapter.

[1083] Step 10:

[1084] Play video

[1085] The user selects the chapter they want to watch and clicks the play button.

[1086] The device will play the video from the start time of the selected chapter using HTML5 Video Player or VLC.

[1087] Step 11:

[1088] Keyword search

[1089] The user enters a specific keyword into the search bar and clicks the search button.

[1090] The server performs a full-text search of the text data in the video based on the received keywords (e.g., using Elasticsearch).

[1091] The server extracts the relevant chapter information as a search result and transmits it to the terminal.

[1092] Step 12:

[1093] Displaying search results

[1094] The device displays the received search results in a list format, with a timestamp and summary of the chapters in each search result.

[1095] The user selects the part they want to watch from the search results and clicks the play button.

[1096] The device will play the video from the specific part of the chapter you clicked.

[1097] Example 1

[1098] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1099] In recent years, the amount of video content has rapidly increased, creating a demand for users to quickly obtain the information they need from videos. However, conventional technologies are inefficient because they require users to watch the entire video to find the information they need. Furthermore, they lack the ability to search for specific content within a video, which means it takes a long time to find the desired information. Furthermore, video content summarization and automatic chapter division are performed manually, which is cumbersome and time-consuming.

[1100] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1101] In this invention, the server includes means for receiving a video file, means for extracting audio data from the video file, means for converting the audio data into text data, means for extracting major topics and key points from the text data, means for dividing the video file into chapters based on the extracted topics or key points, means for generating summaries for each chapter, means for saving the generated chapters and summaries, means for displaying the saved chapters and summaries, means for inputting a specific keyword to perform a search and displaying the search results, and means for extracting relevant chapter information as the search results and playing the content from the first section. This allows users to efficiently obtain the information they need from videos and quickly search for and play specific content.

[1102] The "means for receiving video files" is a mechanism by which a user uploads video files to the system and the server receives the video files.

[1103] A "means for extracting audio data from video files" is a tool or program used to separate the audio track from a saved video file and obtain the audio data.

[1104] The "means for converting voice data into text data" refers to a mechanism that analyzes the extracted voice data and converts it into text data, such as using a voice recognition engine.

[1105] "Means for extracting major topics and key points from text data" refers to natural language processing techniques for analyzing text data and identifying important topics and key points from its content.

[1106] The "means for dividing a video file into chapters based on extracted topics or key points" refers to a mechanism for dividing the content of a video based on the analysis results and identifying each section.

[1107] The "means for generating a summary for each chapter" is a technology for automatically creating a summary that concisely summarizes the contents of each chapter.

[1108] The "means for saving the generated chapters and summaries" is a mechanism for storing the divided chapter information and summaries thereof in a database or the like.

[1109] The "means for displaying saved chapters and summaries" is a mechanism for displaying the analysis results on a terminal, allowing the user to view the information.

[1110] "Means for searching by entering specific keywords and displaying search results" is a mechanism for searching the text data of videos based on keywords entered by the user and displaying matching results.

[1111] "Means for extracting relevant chapter information as a search result and playing from the first section" refers to a mechanism that starts playing the video from the relevant point based on the chapter information obtained by the search.

[1112] This invention is a system that enables a user to quickly and efficiently obtain necessary information from video content. The specific processing and operation of each step will be described in detail below.

[1113] First, the user logs in to the system using a terminal and clicks the upload button to specify a video file. When the user selects a video file and executes upload, the terminal sends the selected video file to the server. The server receives the video file and saves it in the specified directory. At this time, metadata such as the save path and file name, upload date and time, and user ID are registered in the database.

[1114] Next, the server extracts the audio track from the saved video file. This process is performed using a tool such as FFmpeg, specifically by executing FFmpeg commands to separate the audio data. The server then sends the extracted audio data to a speech recognition engine (such as Google Cloud Speech-to-Text or DeepSpeech) for analysis. The speech recognition engine converts the audio data into text data and returns the results to the server. The server then saves the obtained text data as an intermediate file.

[1115] The server reads the stored text data and extracts major topics and key points using natural language processing libraries (e.g., NLTK or spaCy). This process involves tokenizing the text data, tagging it with POS tags, and recognizing entities. The results of this analysis are used in the next step.

[1116] The server divides the video into chapters based on the extracted topics and key points. It calculates the start and end times of each chapter and lists the chapter information. The server then generates a summary of each chapter's text data using a summarization tool (such as GPT-3 or BERT). This allows for a concise summary of the content of each chapter. The generated chapters and summaries are stored in a database.

[1117] The server then retrieves the generated chapter and summary information from the database. The retrieved data is converted to JSON format and sent to the device. The device then displays information for each chapter on the user interface based on the analysis results. The displayed information includes the chapter start time, summary, thumbnail, etc. Through the interface, the user can select the part they want to watch and play it.

[1118] Users can also enter specific keywords into the search bar and click the search button. The device sends this input data to the server. The server performs a full-text search of the text data within the video based on the received keywords. This search uses a search engine such as Elasticsearch. The server extracts the relevant chapter information as search results and sends it to the device. The device displays the received search results in list format on the user interface. The user can select the part they want to view from the displayed search results and play it.

[1119] As a specific example, consider the case where a university student uploads a recorded lecture. When a user uploads a video file of a lecture to the system, the server extracts the audio data and converts it into text data using a speech recognition engine. The server then uses natural language processing technology to extract major topics from the text data and divides the video into chapters based on this. The content of each chapter is summarized and displayed on the device. Users can select the chapters they want to view and review them efficiently. They can also quickly find the information they need by searching for specific keywords. This system provides an effective means for efficiently obtaining the information they need from video content.

[1120] An example prompt is as follows:

[1121] "I want to easily review lecture videos. I would like to know if there is a system that will automatically summarize the lecture content and divide it into easy-to-understand chapters when I upload a video file."

[1122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1123] Step 1:

[1124] A user logs in to the system using a terminal and uploads a video file by clicking the upload button, selecting the video file, and pressing the send button.

[1125] Input: Video file (e.g. lecture recording file)

[1126] Output: A request to transfer a video file to the server

[1127] Specific operation: The device sends an HTTP POST request to the server and sends request data including the selected video file.

[1128] Step 2:

[1129] The server saves the received video file in the specified directory and registers the destination path, file name, upload date and time, user ID, and other metadata in the database.

[1130] Input: Video file, metadata (file name, upload date and time, user ID, etc.)

[1131] Output: Path where the video file is saved, metadata registered in the database

[1132] Specific operation: The server saves the received video file in the specified directory of the file system and records the save path in the database.

[1133] Step 3:

[1134] The server extracts the audio track from the saved video file using a tool such as FFmpeg.

[1135] Input: Saved video file

[1136] Output: Extracted audio data file

[1137] Specific operation: The server executes FFmpeg commands to extract the audio track from the video file and save it as an audio data file.

[1138] Step 4:

[1139] The server sends the extracted voice data to a voice recognition engine (such as Google Cloud Speech-to-Text or DeepSpeech) and converts it into text data.

[1140] Input: Audio data file

[1141] Output: The resulting text data

[1142] Specific operation: The server calls the API of the speech recognition engine, sends the voice data, receives the resulting text data, and saves it as an intermediate file.

[1143] Step 5:

[1144] The server reads the stored text data and extracts major topics and key points using natural language processing libraries (such as NLTK or spaCy).

[1145] Input: Text data

[1146] Output: A list of topics and keypoints

[1147] What it does: The server reads the text data and uses natural language processing libraries to tokenize the text, tag it with POS tags, recognize entities, and extract key topics and key points.

[1148] Step 6:

[1149] The server divides the video into chapters based on the extracted topics and key points, calculates the start and end times of each chapter, and lists the chapter information.

[1150] Input: List of topics and keypoints

[1151] Output: List of chapter information (start time, end time, content)

[1152] Specific operation: The server divides the video into multiple sections based on the timestamps of the topics extracted from the text data, and lists the start and end times of each section.

[1153] Step 7:

[1154] The server generates a summary of the text data for each chapter using a summarization tool (such as GPT-3 or BERT).

[1155] Input: Text data for each chapter

[1156] Output: Summary text

[1157] Specific operation: The server calls the summary generation API, sends the text data for each chapter, and generates a summary text based on the results.

[1158] Step 8:

[1159] The server stores the generated chapter and summary information in a database.

[1160] Input: List of chapter information, summary text

[1161] Output: Chapter and summary information stored in a database

[1162] Specific operation: The server inserts and saves each chapter information and summary into the database.

[1163] Step 9:

[1164] The server retrieves the generated chapter and summary information from the database, converts it into JSON format, and sends it to the terminal.

[1165] Input: Chapter and summary information stored in the database

[1166] Output: JSON format data

[1167] Specific operation: The server queries the database for the required data, serializes the retrieved data into JSON format, and returns it to the device as an HTTP response.

[1168] Step 10:

[1169] The device then displays information for each chapter on the user interface based on the analysis results received, allowing the user to select the part they want to watch and play it.

[1170] Input: Analysis result data in JSON format

[1171] Output: Chapter information displayed in the user interface

[1172] What it does: The device parses the received JSON data and renders an interface that includes the start time, summary, thumbnails, etc. for each chapter.

[1173] Step 11:

[1174] The user inputs a specific keyword and clicks the search button, and the device sends this input data to the server.

[1175] Input: Keywords (e.g., specific lecture content)

[1176] Output: Search request based on keywords

[1177] Specific operation: The device obtains the keywords entered in the search bar and sends them to the server as an HTTP request.

[1178] Step 12:

[1179] The server performs a full-text search of the text data in the video based on the received keywords (using Elasticsearch, etc.), extracts the relevant chapter information as search results, and sends it to the device.

[1180] Input: Keywords, text data

[1181] Output: Chapter information as search results

[1182] Specific operation: The server queries Elasticsearch for the received keywords, extracts chapter information for matching text, serializes it in JSON format, and returns it to the device.

[1183] Step 13:

[1184] The terminal displays the received search results in a list format on the user interface. The user selects the part they want to watch from the displayed search results and clicks the play button.

[1185] Input: Search results (JSON data)

[1186] Output: Search results displayed in the user interface

[1187] Specific operation: The terminal parses the received JSON data and displays the results in a list format in the user interface.

[1188] Step 14:

[1189] The device will play the video from the specific part of the chapter you clicked.

[1190] Input: User selected chapter information

[1191] Output: Video played from the specified chapter

[1192] Specific operation: The device captures the user's click event and starts playing the video from the start time of the corresponding chapter.

[1193] (Application example 1)

[1194] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1195] In today's world, the amount of video content is increasing, and users are required to quickly obtain the information they need. However, it is not easy to extract the necessary information from long video content and grasp it in a short time. In addition, because the content contains a large amount of information, there is also the problem that viewing and searching cannot be performed efficiently. Therefore, an object of the present invention is to provide a system that efficiently extracts the necessary information from video content and provides it in a short time.

[1196] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1197] In this invention, the server includes a means for receiving video files, a means for extracting audio data from the received video files, and a means for converting the audio data into text data. This allows for efficient extraction of major topics and key points from video content, division of the video into multiple chapters, and generation and display of summaries. The system also includes a means for generating summaries based on input prompts using a generative AI model, and furthermore, enables searching for specific keywords and playback of relevant sections. This allows users to quickly obtain the information they need and efficiently watch and search videos.

[1198] A "video file" is a digital file that contains video and audio data.

[1199] "Audio data" refers to sound data containing human voices extracted from video files.

[1200] "Text data" refers to data obtained by converting voice data into character information.

[1201] "Major topics" are important topics or themes of videos extracted from text data.

[1202] "Key points" are important points or elements related to the main topic.

[1203] "Chapters" are sections of a video file that are divided based on specific topics or key points.

[1204] A "summary" is a text that briefly describes the contents of each chapter.

[1205] A "generative AI model" is an artificial intelligence model that generates and understands text based on large datasets.

[1206] A "prompt" is a text instruction that is input to a generative AI model.

[1207] The "means for receiving" is the part that has the function of uploading and saving video files to the system.

[1208] The "extraction means" is the part that has the function of extracting audio data from a video file.

[1209] The "conversion means" is a part that has the function of converting voice data into text data.

[1210] The "means for generating a summary" is a part that has the function of converting text data into a summary.

[1211] The "means for saving" is a part that has the function of saving the generated chapters and summaries.

[1212] The "display means" is a part that has the function of displaying the saved chapters and summaries on the user interface.

[1213] The "search means" is the part that has the function of searching the text data within the video based on specific keywords and displaying the results.

[1214] "Means for playback" refers to the part that has the function of playing the video of the relevant part of the search results.

[1215] This invention is a system that extracts necessary information from video content in a short time and efficiently provides it to users. The system receives video files, extracts and converts audio data, extracts major topics and key points, generates summaries using a generative AI model, and displays and searches the results.

[1216] First, a user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the server, which receives the video file and saves it in the specified directory. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[1217] Next, the server extracts the audio track from the saved video file using a tool such as FFmpeg. The server then sends the extracted audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert it into text data, which is then saved as an intermediate file.

[1218] The server then reads the text data and uses natural language processing libraries (e.g., spaCy) to extract major topics and key points, clarifying the content of the video and establishing a foundation for the next steps.

[1219] The server divides the video into chapters based on topics and key points. It identifies the start and end times of each chapter and lists the chapter information. The server then generates a summary of each chapter's text data using a summarization tool (e.g., GPT-3). The generated chapters and summaries are stored in a database.

[1220] When a user enters a specific keyword, the server performs a full-text search of the text data within the video based on the received keyword. The server extracts the relevant chapter information as search results and sends it to the device. The device then displays the received search results in a list format on the user interface. The user can then select the part they want to watch from the displayed search results and play it.

[1221] As a concrete example, consider the case where a university student uploads a recorded lecture. When a user uploads a video file of a lecture to the system, the server extracts the audio data and converts it into text data using a speech recognition engine. Then, natural language processing technology is used to extract key topics from the text data and divide the video into chapters based on these. The content of each chapter is summarized and displayed on the device, allowing the user to select the chapter they want to view and review it efficiently.

[1222] An example of a prompt sentence in a generative AI model is "Please summarize the following text in a concise manner: \n\n[text content]", which allows the generative AI model to summarize the input text.

[1223] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1224] Step 1:

[1225] A user logs in to the system using a terminal and uploads a video file. The terminal sends the video file selected by the user to the server. The input is the video file, and the output is the video file received by the server.

[1226] Step 2:

[1227] The server saves the received video file in the specified directory. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database. The input is the video file, and the output is the saved video file and the registration of metadata in the database.

[1228] Step 3:

[1229] The server extracts the audio track from the stored video file using a tool such as FFmpeg. The input is the stored video file, and the output is the extracted audio data.

[1230] Step 4:

[1231] The server sends the extracted voice data to a voice recognition engine (such as Google Cloud Speech-to-Text) and converts it into text data. The input is voice data, and the output is the converted text data.

[1232] Step 5:

[1233] The server reads the text data and extracts the main topics and keypoints using a natural language processing library (such as spaCy). The input is the text data, and the output is the extracted main topics and keypoints.

[1234] Step 6:

[1235] The server divides the video into chapters based on topics and key points, identifies the start and end times of each chapter, and lists the chapter information. The input is the main topics and key points, and the output is the divided chapter information.

[1236] Step 7:

[1237] Next, the server generates a summary of the text data of each chapter using a summarization tool (a generative AI model such as GPT-3). A prompt is used for generation. The input is the text data and prompt for each chapter, and the output is the generated summary. An example of a prompt is "Please summarize the following text in a concise manner: \n\n[text content]".

[1238] Step 8:

[1239] The server stores the generated chapters and summaries in a database. The input is the generated chapters and summaries, and the output is the data stored in the database.

[1240] Step 9:

[1241] When a user inputs a specific keyword, the server performs a full-text search of the text data in the video based on the received keyword. The input is the specific keyword, and the output is the searched chapter information.

[1242] Step 10:

[1243] The server extracts the relevant chapter information as the search result and sends it to the terminal. The input is the chapter information of the search result, and the output is the search result sent to the terminal.

[1244] Step 11:

[1245] The terminal displays the received search results in a list format on the user interface. The user can select the part they want to watch from the displayed search results and play it. The input is the chapter information of the search results, and the output is the search results displayed on the user interface.

[1246] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1247] This invention improves the user experience by combining a system that quickly acquires necessary information from video content with an emotion engine that recognizes the user's emotions. The specific processing and operation of each step are explained below.

[1248] Importing videos

[1249] A user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the system, and the server receives the video file and saves it in its internal storage. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[1250] Video Analysis

[1251] The server extracts the audio track from the stored video file, typically using a tool like FFmpeg. The server then sends the extracted audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech) to convert it into text data, which is then saved as an intermediate file.

[1252] Natural Language Processing

[1253] The server reads the text data stored as intermediate files and uses natural language processing libraries (e.g., NLTK or spaCy) to extract major topics and key points, clarifying the content of the video and generating the foundational data for the next step.

[1254] Chapter generation and summarization

[1255] The server divides the video into multiple chapters based on the extracted topics and key points. It identifies the start and end times of each chapter and lists the chapter information. Then, the server generates a summary for each chapter using a summarization tool (e.g., GPT-3 or BERT) based on each chapter. The generated chapters and summaries are stored in a database.

[1256] Providing a user interface

[1257] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the device in JSON format or similar. The device then displays information for each chapter on the user interface based on the analysis results received. The start time, summary, thumbnail, and other information for each chapter are displayed, allowing the user to select and play the part they want to watch.

[1258] Reverse search function

[1259] The user enters a specific keyword in the search bar and clicks the search button. The server uses Elasticsearch or similar tools to perform a full-text search of the text data within the video based on the received keyword. The server extracts the relevant chapter information as search results and sends it to the device. The device displays the received search results in list format, and for each chapter in the search results, displays the timestamp where the keyword appears and a summary. The user can select the part they want to watch and click the play button to play the video from that specific part of that chapter.

[1260] Emotion engine integration

[1261] The present invention further integrates an emotion engine that recognizes the user's emotions. The emotion engine allows the device to analyze the user's emotional state in real time using the user's facial expressions, voice, keyboard and mouse operation patterns, etc. For example, if the system determines that the user is interested, it will prioritize displaying related chapters and summaries.

[1262] Emotion-based chapter display

[1263] The server receives the emotion data obtained by the emotion engine and adjusts the display order of chapters and summaries based on the user's emotional state, thereby prioritizing the display of content that is likely to interest the user and improving the user experience.

[1264] Filtering search results based on sentiment

[1265] When a user enters a search keyword, the emotional engine provides emotional data to filter search results. For example, if a user is feeling stressed, chapters with relaxing content will be prioritized.

[1266] Specific examples

[1267] For example, when a user watches a recording of a business meeting and uses the system with the emotion engine, the system analyzes the user's emotions and prioritizes displaying important parts or parts that the user is likely to be interested in. This allows the user to efficiently obtain the information they need.

[1268] In this way, the present invention provides a means for efficiently obtaining necessary information from video content, while also providing a more personalized experience by taking into account the user's emotional state.

[1269] The processing flow will be explained below.

[1270] Step 1:

[1271] Uploading videos

[1272] The user logs into the system, selects a video file, and clicks the upload button.

[1273] The terminal transmits the selected video file to the system.

[1274] Step 2:

[1275] Saving video files

[1276] The server saves the received video file in a specified directory in the internal storage.

[1277] The server registers the destination path and related metadata (file name, upload date and time, user ID, etc.) in a database.

[1278] Step 3:

[1279] Extracting audio data

[1280] The server extracts the audio track from the stored video file using a tool such as FFmpeg.

[1281] Step 4:

[1282] Voice Recognition

[1283] The server sends the extracted voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech) and converts it into text data.

[1284] The server stores the converted text data as an intermediate file.

[1285] Step 5:

[1286] Natural Language Processing

[1287] The server reads the text data stored as intermediate files and extracts major topics and key points using natural language processing libraries (e.g., NLTK or spaCy).

[1288] Step 6:

[1289] Chapter Generation

[1290] The server divides the video into multiple chapters based on the extracted topics and key points, identifies the start and end times of each chapter, and lists the chapter information.

[1291] Step 7:

[1292] Generate a summary

[1293] Based on each chapter, the server generates a summary for each chapter using a summarization tool (e.g., GPT-3 or BERT).

[1294] The server stores the generated chapters and summaries in a database.

[1295] Step 8:

[1296] Sending analysis results

[1297] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the terminal in JSON format or similar.

[1298] Step 9:

[1299] View Chapters

[1300] Based on the analysis results, the device displays information for each chapter on the user interface, including the start time, summary, and thumbnail of each chapter.

[1301] Step 10:

[1302] Starting emotion recognition

[1303] The device runs an emotion engine and analyzes the user's facial expressions, voice, and operation patterns, including input from the camera, microphone, mouse, and keyboard.

[1304] The emotion engine analyzes the acquired data in real time to determine the user's emotional state.

[1305] Step 11:

[1306] Emotion-based interface adjustment

[1307] The server adjusts the display order of chapters and the content of summaries based on the emotion data received from the emotion engine.

[1308] For example, if it is determined that the user is interested, related chapters are preferentially displayed.

[1309] Step 12:

[1310] Play video

[1311] The user selects the chapter they wish to view and clicks the play button.

[1312] The device will play the video from the start time of the selected chapter.

[1313] Step 13:

[1314] Keyword search

[1315] The user enters a specific keyword in the search bar and clicks the search button.

[1316] The server performs a full-text search of the text data in the video based on the received keywords (e.g., using Elasticsearch).

[1317] The server extracts the relevant chapter information as the search result and transmits it to the terminal.

[1318] Step 14:

[1319] Filtering search results based on sentiment

[1320] The server uses data from the emotion engine to filter search results according to the user's emotions.

[1321] For example, if you want to relax, a chapter with relaxing content will be displayed.

[1322] Step 15:

[1323] Displaying search results

[1324] The device displays the received search results in a list format, with a timestamp and summary of the chapters in each search result.

[1325] The user selects the portion of the search result that he or she wants to view from the displayed search results and clicks the play button.

[1326] The device will play the video from the specific part of the chapter you clicked.

[1327] Example 2

[1328] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1329] In recent years, the use of video data has increased, creating a need for quickly extracting only the necessary parts from this vast amount of information. However, conventional technologies have been unable to take into account user emotion data when converting audio from video to text, extracting key topics and key points, dividing video into segments, and generating summaries, making it difficult to provide optimal information. Furthermore, the search function was unable to filter content based on user emotion, resulting in a poor user experience.

[1330] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1331] In this invention, the server includes means for receiving video data, means for extracting audio data from the received video data, means for converting the audio data into text data, means for extracting main topics and important points from the text data, means for dividing the video data into segments based on the extracted topics or important points, means for generating summaries for each segment, means for saving the generated segments and summaries, means for displaying the saved segments and summaries, means for inputting a specific keyword for search and displaying search results, means for analyzing user emotion data, and means for filtering the display order and search results based on the emotion data, thereby enabling optimal information provision that takes user emotions into consideration.

[1332] "Video data" refers to data in a file format that includes video and audio.

[1333] "Audio data" refers to data consisting of only the audio track extracted from video data.

[1334] "Text data" refers to data obtained by converting voice data into character information.

[1335] "Major topics" refer to themes or topics that are considered particularly important in the content of a video.

[1336] "Important points" refer to specific points or key information in the video content that are particularly noteworthy.

[1337] "Segmenting" refers to dividing video data into multiple parts and treating each part as an independent chapter.

[1338] A "segment" refers to each part of divided video data.

[1339] "Generating a summary" refers to creating a short description that succinctly summarizes the content of each segment.

[1340] "Storing" refers to recording the generated data in storage such as a database.

[1341] "Displaying" refers to visually showing the generated data in a user interface.

[1342] "Searching" refers to matching information in a database using specific keywords and extracting relevant results.

[1343] "Displaying search results" refers to displaying the information obtained by the search in a user interface.

[1344] "Analyzing user emotional data" refers to evaluating the user's emotional state in real time based on their facial expressions, voice, operation patterns, etc.

[1345] "Filtering the display order or search results based on emotional data" refers to adjusting the order or content of the information to be displayed in accordance with the acquired emotional data.

[1346] This invention is a system that quickly acquires necessary information from video data, and improves the user experience by combining it with an emotion engine that recognizes the user's emotions. Specific operations and embodiments of this system are described below.

[1347] Importing videos

[1348] A user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the server. The server saves the received video file in its internal storage. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[1349] Video Analysis

[1350] The server extracts the audio track from the saved video file using a tool called FFmpeg. The server then sends the extracted audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech), which converts the audio data into text data. The text data is then saved as an intermediate file.

[1351] Natural Language Processing

[1352] The server reads the text data stored as intermediate files and uses natural language processing libraries (e.g., NLTK or spaCy) to extract the main topics and key points, clarifying the content of the video and generating the foundational data for the next step.

[1353] Chapter generation and summarization

[1354] The server divides the video into multiple segments based on the extracted topics and key points. It identifies the start and end times of each segment and lists the segment information. Then, the server generates a summary for each segment using a summarization tool (e.g., GPT-3 or BERT). The generated segments and summaries are stored in a database.

[1355] Providing a user interface

[1356] The server retrieves the generated segment and summary information from the database and sends the analysis results to the device in JSON format or similar. Based on the analysis results received, the device displays information for each segment on the user interface. The start time, summary, thumbnail, and other information for each segment are displayed, allowing the user to select and play the part they want to watch.

[1357] Reverse search function

[1358] The user enters a specific keyword in the search bar and clicks the search button. The server uses Elasticsearch or similar tools to perform a full-text search of the text data within the video based on the received keyword. The server extracts the relevant segment information as search results and sends it to the device. The device displays the received search results in list format, and for each segment in the search results, displays the timestamp where the keyword appears and a summary. The user can select the part they want to watch and click the play button to play the video from that specific part of that segment.

[1359] Emotion engine integration

[1360] The present invention further integrates an emotion engine that recognizes the user's emotions. The emotion engine allows the terminal to analyze the user's emotional state in real time using the user's facial expressions, voice, keyboard and mouse operation patterns, etc. For example, if it is determined that the user is interested, the system will prioritize displaying related segments and summaries.

[1361] Segment display based on emotions

[1362] The server receives the emotion data obtained by the emotion engine and adjusts the display order of the segments and summaries based on the user's emotional state, thereby prioritizing the display of content that is likely to interest the user and improving the user experience.

[1363] Filtering search results based on sentiment

[1364] When a user enters a search keyword, the emotion engine provides emotional data to filter search results. For example, if a user is feeling stressed, segments with relaxing content will be prioritized.

[1365] Specific examples

[1366] For example, when a user watches a recording of a business meeting, the system analyzes the user's emotions and prioritizes displaying important parts or parts likely to interest the user. This allows the user to efficiently obtain the information they need. In this way, the present invention provides a means for efficiently obtaining necessary information from video data, while also providing a more personalized experience by taking the user's emotional state into consideration.

[1367] Prompt Sentence Examples

[1368] "Please extract the main topics and key points from this video."

[1369] "Summarize and segment recordings of business meetings."

[1370] "Rearrange your segments and summaries taking into account the output data of the sentiment engine to prioritize what is likely to interest the user."

[1371] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1372] Step 1: Import video

[1373] After logging in, the user selects and uploads a video file.

[1374] Input: Video file, user information

[1375] The terminal reads the selected video file and transfers it to the server.

[1376] The server receives the video file and saves it to its internal storage. At this time, it registers the destination path and video metadata (file name, upload date and time, user ID, etc.) in a database.

[1377] Output: Video files stored in the internal storage, metadata registered in the database

[1378] Step 2: Analyze the video

[1379] The server extracts the audio track from the stored video file.

[1380] Input: Saved video file

[1381] The server uses FFmpeg to extract the audio data and sends it to a speech recognition engine, specifically, by calling the Google Cloud Speech-to-Text or DeepSpeech API.

[1382] The server converts the voice data into text data and stores the text data as an intermediate file.

[1383] Output: Text data saved as an intermediate file

[1384] Step 3: Natural Language Processing

[1385] The server reads the text data stored as an intermediate file.

[1386] Input: Text data saved as an intermediate file

[1387] The server uses natural language processing libraries (such as NLTK or spaCy) to extract key topics and key points, which involves tokenizing words and parsing sentences.

[1388] Output: Extracted main topics and key points data

[1389] Step 4: Chapter generation and summarization

[1390] The server divides the video into multiple segments based on the extracted topics and key points.

[1391] Input: Data on key topics and key points

[1392] The server identifies and lists the start and end times of each segment, possibly using a video editing library.

[1393] The server uses a summarization tool (such as GPT-3 or BERT) to generate a summary of each segment.

[1394] The generated segments and summaries are stored in a database.

[1395] Output: Generated segments and summaries stored in a database

[1396] Step 5: Providing a User Interface

[1397] The server retrieves the generated segment and summary information from the database, converts it into JSON format, and sends it to the terminal.

[1398] Input: Generated segments and summaries stored in a database

[1399] Based on the analysis results received, the device displays information for each segment on the user interface, including the start time, summary, and thumbnail of each segment.

[1400] Output: Information for each segment displayed in the user interface

[1401] Step 6: Reverse Search Function

[1402] The user enters a specific keyword into the search bar and clicks the search button.

[1403] Input: Search keyword

[1404] Based on the received keywords, the server uses Elasticsearch to perform a full-text search of the text data within the video.

[1405] The server extracts the relevant segment information as a search result and transmits it to the terminal.

[1406] The terminal displays the received search results in a list format, and displays the timestamps at which the keywords appear and a summary for each segment of each search result.

[1407] Output: Search results displayed in list format

[1408] Step 7: Integrating the Emotion Engine

[1409] The emotion engine analyzes the user's facial expressions, voice, keyboard and mouse operation patterns, and other information acquired by the device in real time.

[1410] Input: User's facial expressions, voice, and operation patterns

[1411] The device analyzes this data and estimates the emotional state.

[1412] The server adjusts the order of segments and summaries to be displayed based on the emotion data received from the emotion engine.

[1413] Output: Reordered segments and summaries based on sentiment data

[1414] Step 8: Segment based on sentiment

[1415] The server receives the emotion data obtained from the emotion engine.

[1416] Input: Emotion data

[1417] The server adjusts the display order of the segments and summaries based on the user's emotional state and transmits them again to the terminal.

[1418] The device will prioritize updating the displayed information to improve the user experience.

[1419] Output: Reordered segments and summary information

[1420] Step 9: Filter search results based on sentiment

[1421] When a user enters a search keyword and performs a search, the search results are filtered based on the emotional data provided by the emotion engine.

[1422] Input: Search keywords, emotion data

[1423] The server filters the search results taking into account the emotion data and transmits the results based on that to the terminal.

[1424] The terminal displays the filtered results in a list format.

[1425] Output: Filtered search results

[1426] Prompt Sentence Examples

[1427] "Please extract the main topics and key points from this video."

[1428] "Summarize and segment recordings of business meetings."

[1429] "Rearrange your segments and summaries taking into account the output data of the sentiment engine to prioritize what is likely to interest the user."

[1430] (Application example 2)

[1431] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1432] Systems for quickly obtaining necessary information from video content have the problem of not being able to efficiently provide the parts of a user's interest, resulting in a lack of improvement in the user experience. Furthermore, conventional systems are unable to display content taking into account the user's emotions, making it difficult to provide personalized services.

[1433] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a video file, means for extracting audio data from the received video file, means for converting the audio data into text data, means for extracting major topics and key points from the text data, means for dividing the video file into chapters based on the extracted topics or key points, means for generating summaries for each chapter, means for saving the generated chapters and summaries, means for displaying the saved chapters and summaries, means for inputting a specific keyword to perform a search and displaying search results, emotion recognition means for recognizing the user's emotional state, means for adjusting the display order of chapters and summaries based on the emotional state, and means for filtering search results based on the emotional state. This allows the user to prioritize viewing parts that interest them.

[1434] A "video file" is a file that is stored in digital format and contains video and audio.

[1435] "Means for receiving" refers to hardware or software for acquiring data from the outside and incorporating it into the system.

[1436] "Audio data" is digital sound information extracted from the audio track of a video file.

[1437] "Means for converting voice data into text data" refers to a process of converting voice information into text information using a voice recognition engine.

[1438] "Text data" is character information converted from voice data.

[1439] "Major topics and key points" refer to important themes and key points extracted from text data.

[1440] A "chapter" is a part of a video file that is divided into multiple segments.

[1441] A "summary" is a sentence or paragraph that succinctly summarizes the contents of each chapter.

[1442] "Emotion recognition means" refers to technology or equipment that analyzes the user's emotional state in real time from their facial expressions, voice, operation patterns, etc.

[1443] An "emotional state" refers to the emotion a user is feeling at a particular moment.

[1444] A "means for adjusting the display order" is a process or method for changing the order in which content is displayed based on the user's emotional state.

[1445] A "search result filtering method" is a process or technique that sorts search results based on the user's emotional state and tailors the content displayed to specific conditions.

[1446] The present invention provides a system for improving the user experience by quickly acquiring necessary information from video files and adjusting the display order and search results based on the user's emotions. An embodiment of the system will be described below.

[1447] The system includes means for receiving a video file, means for extracting audio data, means for converting the audio data into text data, means for extracting main topics and key points, means for dividing into chapters, means for generating summaries, means for saving the generated chapters and summaries, means for displaying the saved information, means for displaying search results by inputting specific keywords, means for recognizing a user's emotions, means for adjusting the display order based on the user's emotional state, and means for filtering the search results.

[1448] The server receives video files from user devices and extracts audio data from the received videos using common media processing tools such as FFmpeg. The extracted audio data is converted into text data using a speech recognition engine such as Google Cloud Speech-to-Text or DeepSpeech. The converted text data is then used to extract major topics and key points using natural language processing libraries such as NLTK or spaCy.

[1449] The server then divides the video into chapters based on the extracted topics and key points, and generates summaries for each chapter using a generative AI model such as GPT-3. The generated chapters and summaries are stored in a database and later displayed through a user interface.

[1450] When a user enters a specific keyword to perform a search, the server uses a search engine such as Elasticsearch to perform a full-text search of the text data and extracts the relevant chapter information. Based on the results, the device displays the search results and the user can play that chapter.

[1451] The emotion recognition system analyzes the user's emotional state in real time using the smartphone's camera, microphone, and touch operation data. Specifically, it utilizes Microsoft Azure Emotion API and Affectiva SDK. Based on this emotion recognition, the server adjusts the order of chapters and summaries displayed and further filters search results. For example, if the user shows interest, relevant chapters and summaries will be displayed first.

[1452] An example of a specific input prompt is as follows:

[1453] input_video_path = " / path / to / uploads / your_video.mp4"

[1454] user_emotion = "interested"

[1455] In this way, the invention is a system that provides efficient information acquisition and a personalized experience.

[1456] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1457] Step 1:

[1458] Uploading a video file

[1459] What happens: A user uses a device to upload a video file to the system.

[1460] Input: A video file selected by the user.

[1461] Output: The uploaded video file is sent to the server, and the server receives the video file and saves it in its internal storage.

[1462] Specific operation: The device sends a video file to the server as an HTTP POST request, and the server receives the file and registers the storage path and metadata in the database.

[1463] Step 2:

[1464] Extracting audio data

[1465] Processing content: Extracts audio data from the video file received by the server.

[1466] Input: Saved video file.

[1467] Output: The extracted audio data.

[1468] Specific operation: The server uses the FFmpeg tool to extract the audio track from the video file and generate the audio data.

[1469] Step 3:

[1470] Convert speech to text

[1471] Processing content: Convert the extracted audio data into text data.

[1472] Input: Audio data.

[1473] Output: The converted text data.

[1474] How it works: The server uses a speech recognition engine such as Google Cloud Speech-to-Text or DeepSpeech to analyze the audio data and convert it into text. The converted text data is saved as an intermediate file.

[1475] Step 4:

[1476] Extracting major topics and key points

[1477] What it does: Extract major topics and key points from text data.

[1478] Input: Text data.

[1479] Output: Major topics and key points.

[1480] How it works: The server analyzes the text data using natural language processing libraries (such as NLTK or spaCy) to extract important topics and key points from the text. This information is then stored in a database.

[1481] Step 5:

[1482] Chapter Generation

[1483] What it does: Divide the video into chapters based on extracted topics and key points.

[1484] Input: Major topics and key points.

[1485] Output: Chapter information.

[1486] Specific operation: The server identifies the start and end times of each topic and divides the video into chapters. The chapter information is listed and stored in a database.

[1487] Step 6:

[1488] Generate a summary for each chapter

[1489] Process: Generate a summary for each chapter.

[1490] Input: Chapter information.

[1491] Output: The generated summary.

[1492] How it works: The server uses a generative AI model (e.g., GPT-3) to summarize the content of each chapter and stores the summaries in a database.

[1493] Step 7:

[1494] Displaying analysis results

[1495] What it does: Display the saved chapters and summaries in the user interface.

[1496] Input: Chapter and abstract information stored in the database.

[1497] Output: Chapter and summary information displayed on the device.

[1498] Specific operation: The server retrieves the analysis results from the database and sends them to the device in JSON format. The device then displays the analysis results on the user interface based on the received data.

[1499] Step 8:

[1500] Keyword search

[1501] What happens: A user enters a specific keyword into the search bar and displays search results.

[1502] Input: The keyword entered by the user.

[1503] Output: Chapter information and summary displayed as search results.

[1504] Specific operation: The server performs a full-text search of the text data using Elasticsearch or similar tools to extract the relevant chapter information. The device displays the received search results in a list, and the user can click the play button to watch the relevant section.

[1505] Step 9:

[1506] Emotion recognition and display order adjustment

[1507] What it does: Recognizes the user's emotional state and adjusts the order in which chapters and summaries are displayed based on that information.

[1508] Input: User emotion data.

[1509] Output: Adjusted chapter and summary display order.

[1510] Specific operation: The emotion recognition means analyzes the user's emotions in real time using the device's camera, microphone, and touch operation data. The server adjusts the display order based on the emotion data and prioritizes displaying the parts that are of interest to the user.

[1511] Step 10:

[1512] Sentiment-based search result filtering

[1513] What it does: Filter search results based on emotional state.

[1514] Input: User sentiment data and search keywords.

[1515] Output: The filtered search results.

[1516] How it works: The server adjusts search results based on emotional data. For example, if a user is feeling stressed, chapters with relaxing content will be prioritized.

[1517] In this way, through each step, the user can view the parts that interest him / her preferentially, and can efficiently obtain the necessary information.

[1518] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1519] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1520] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1521] [Fourth embodiment]

[1522] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1523] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1524] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1525] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1526] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1527] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1528] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1529] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1530] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1531] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1532] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1533] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1534] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1535] This invention is a system for acquiring "necessary information" from video content in a "short time." The specific processing and operation of each step will be described below.

[1536] Importing videos

[1537] A user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the system, and the server receives the video file and saves it in the specified directory. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[1538] Video Analysis

[1539] The server extracts the audio track from the stored video file, for example using a tool such as FFmpeg. The server then sends the extracted audio data to a speech recognition engine (for example, Google Cloud Speech-to-Text or DeepSpeech) to convert it into text. The resulting text data is saved as an intermediate file.

[1540] Natural Language Processing

[1541] The server reads the text data and uses natural language processing libraries (e.g., NLTK or spaCy) to extract major topics and key points, clarifying the content of the video and establishing a foundation for the next steps.

[1542] Chapter generation and summarization

[1543] The server divides the video into chapters based on topics and key points, identifies the start and end times of each chapter, and lists the chapter information. Next, the server generates summaries of the text data for each chapter using a summarization tool (e.g., GPT-3 or BERT). The generated chapters and summaries are stored in a database.

[1544] Providing a user interface

[1545] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the device in JSON format or similar. The device then displays information for each chapter on the user interface based on the analysis results received. This information includes the start time, summary, and thumbnail of each chapter. This allows the user to select the part they want to watch and play it.

[1546] Reverse search function

[1547] The user enters a specific keyword in the search bar and clicks the search button. The server performs a full-text search of the text data within the video based on the received keyword (using Elasticsearch, for example). The server extracts the relevant chapter information as search results and sends it to the device. The device displays the received search results in list format on the user interface. The user selects the part they want to view from the displayed search results and clicks the play button. The device then plays the video from the specific part of the chapter they clicked.

[1548] Specific examples

[1549] For example, consider the case where a university student uploads a recorded lecture. When the user uploads the lecture video file to the system, the server extracts the audio data and converts it into text data using a speech recognition engine. Then, natural language processing technology is used to extract key topics from the text data and divide the video into chapters based on this. The content of each chapter is summarized and displayed on the device, allowing the user to select the chapter they want to view and review it efficiently. Users can also quickly find the information they need by searching for specific keywords.

[1550] In this way, the present invention provides an effective means for efficiently obtaining necessary information from video content.

[1551] The processing flow will be explained below.

[1552] Step 1:

[1553] Uploading videos

[1554] The user logs into the system, selects a video file, and clicks the upload button.

[1555] The terminal transmits the selected video file to the system.

[1556] Step 2:

[1557] Saving video files

[1558] The server saves the received video file in the specified directory in the internal storage.

[1559] The server registers the destination path and related metadata (file name, upload date and time, user ID, etc.) in a database.

[1560] Step 3:

[1561] Extracting audio data

[1562] The server extracts the audio track from the stored video file using a tool such as FFmpeg.

[1563] Step 4:

[1564] Voice Recognition

[1565] The server sends the extracted voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech) and converts it into text data.

[1566] The server stores the converted text data as an intermediate file.

[1567] Step 5:

[1568] Natural Language Processing

[1569] The server reads the text data stored as intermediate files and extracts major topics and key points using natural language processing libraries (e.g., NLTK or spaCy).

[1570] Step 6:

[1571] Chapter Generation

[1572] The server divides the video into chapters based on the extracted topics and key points, identifies the start and end times of each chapter, and lists the chapter information.

[1573] Step 7:

[1574] Generate a summary

[1575] Based on each chapter, the server generates a summary for each chapter using a summarization tool (e.g., GPT-3 or BERT).

[1576] The server stores the generated chapters and summaries in a database.

[1577] Step 8:

[1578] Sending analysis results

[1579] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the terminal in JSON format or similar.

[1580] Step 9:

[1581] View Chapters

[1582] Based on the analysis results, the device displays information for each chapter on the user interface, including the start time, summary, and thumbnail of each chapter.

[1583] Step 10:

[1584] Play video

[1585] The user selects the chapter they want to watch and clicks the play button.

[1586] The device will play the video from the start time of the selected chapter using HTML5 Video Player or VLC.

[1587] Step 11:

[1588] Keyword search

[1589] The user enters a specific keyword into the search bar and clicks the search button.

[1590] The server performs a full-text search of the text data in the video based on the received keywords (e.g., using Elasticsearch).

[1591] The server extracts the relevant chapter information as a search result and transmits it to the terminal.

[1592] Step 12:

[1593] Displaying search results

[1594] The device displays the received search results in a list format, with a timestamp and summary of the chapters in each search result.

[1595] The user selects the part they want to watch from the search results and clicks the play button.

[1596] The device will play the video from the specific part of the chapter you clicked.

[1597] Example 1

[1598] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1599] In recent years, the amount of video content has rapidly increased, creating a demand for users to quickly obtain the information they need from videos. However, conventional technologies are inefficient because they require users to watch the entire video to find the information they need. Furthermore, they lack the ability to search for specific content within a video, which means it takes a long time to find the desired information. Furthermore, video content summarization and automatic chapter division are performed manually, which is cumbersome and time-consuming.

[1600] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1601] In this invention, the server includes means for receiving a video file, means for extracting audio data from the video file, means for converting the audio data into text data, means for extracting major topics and key points from the text data, means for dividing the video file into chapters based on the extracted topics or key points, means for generating summaries for each chapter, means for saving the generated chapters and summaries, means for displaying the saved chapters and summaries, means for inputting a specific keyword to perform a search and displaying the search results, and means for extracting relevant chapter information as the search results and playing the content from the first section. This allows users to efficiently obtain the information they need from videos and quickly search for and play specific content.

[1602] The "means for receiving video files" is a mechanism by which a user uploads video files to the system and the server receives the video files.

[1603] A "means for extracting audio data from video files" is a tool or program used to separate the audio track from a saved video file and obtain the audio data.

[1604] The "means for converting voice data into text data" refers to a mechanism that analyzes the extracted voice data and converts it into text data, such as using a voice recognition engine.

[1605] "Means for extracting major topics and key points from text data" refers to natural language processing techniques for analyzing text data and identifying important topics and key points from its content.

[1606] The "means for dividing a video file into chapters based on extracted topics or key points" refers to a mechanism for dividing the content of a video based on the analysis results and identifying each section.

[1607] The "means for generating a summary for each chapter" is a technology for automatically creating a summary that concisely summarizes the contents of each chapter.

[1608] The "means for saving the generated chapters and summaries" is a mechanism for storing the divided chapter information and summaries thereof in a database or the like.

[1609] The "means for displaying saved chapters and summaries" is a mechanism for displaying the analysis results on a terminal, allowing the user to view the information.

[1610] "Means for searching by entering specific keywords and displaying search results" is a mechanism for searching the text data of videos based on keywords entered by the user and displaying matching results.

[1611] "Means for extracting relevant chapter information as a search result and playing from the first section" refers to a mechanism that starts playing the video from the relevant point based on the chapter information obtained by the search.

[1612] This invention is a system that enables a user to quickly and efficiently obtain necessary information from video content. The specific processing and operation of each step will be described in detail below.

[1613] First, the user logs in to the system using a terminal and clicks the upload button to specify a video file. When the user selects a video file and executes upload, the terminal sends the selected video file to the server. The server receives the video file and saves it in the specified directory. At this time, metadata such as the save path and file name, upload date and time, and user ID are registered in the database.

[1614] Next, the server extracts the audio track from the saved video file. This process is performed using a tool such as FFmpeg, specifically by executing FFmpeg commands to separate the audio data. The server then sends the extracted audio data to a speech recognition engine (such as Google Cloud Speech-to-Text or DeepSpeech) for analysis. The speech recognition engine converts the audio data into text data and returns the results to the server. The server then saves the obtained text data as an intermediate file.

[1615] The server reads the stored text data and extracts major topics and key points using natural language processing libraries (e.g., NLTK or spaCy). This process involves tokenizing the text data, tagging it with POS tags, and recognizing entities. The results of this analysis are used in the next step.

[1616] The server divides the video into chapters based on the extracted topics and key points. It calculates the start and end times of each chapter and lists the chapter information. The server then generates a summary of each chapter's text data using a summarization tool (such as GPT-3 or BERT). This allows for a concise summary of the content of each chapter. The generated chapters and summaries are stored in a database.

[1617] The server then retrieves the generated chapter and summary information from the database. The retrieved data is converted to JSON format and sent to the device. The device then displays information for each chapter on the user interface based on the analysis results. The displayed information includes the chapter start time, summary, thumbnail, etc. Through the interface, the user can select the part they want to watch and play it.

[1618] Users can also enter specific keywords into the search bar and click the search button. The device sends this input data to the server. The server performs a full-text search of the text data within the video based on the received keywords. This search uses a search engine such as Elasticsearch. The server extracts the relevant chapter information as search results and sends it to the device. The device displays the received search results in list format on the user interface. The user can select the part they want to view from the displayed search results and play it.

[1619] As a specific example, consider the case where a university student uploads a recorded lecture. When a user uploads a video file of a lecture to the system, the server extracts the audio data and converts it into text data using a speech recognition engine. The server then uses natural language processing technology to extract major topics from the text data and divides the video into chapters based on this. The content of each chapter is summarized and displayed on the device. Users can select the chapters they want to view and review them efficiently. They can also quickly find the information they need by searching for specific keywords. This system provides an effective means for efficiently obtaining the information they need from video content.

[1620] An example prompt is as follows:

[1621] "I want to easily review lecture videos. I would like to know if there is a system that will automatically summarize the lecture content and divide it into easy-to-understand chapters when I upload a video file."

[1622] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1623] Step 1:

[1624] A user logs in to the system using a terminal and uploads a video file by clicking the upload button, selecting the video file, and pressing the send button.

[1625] Input: Video file (e.g. lecture recording file)

[1626] Output: A request to transfer a video file to the server

[1627] Specific operation: The device sends an HTTP POST request to the server and sends request data including the selected video file.

[1628] Step 2:

[1629] The server saves the received video file in the specified directory and registers the destination path, file name, upload date and time, user ID, and other metadata in the database.

[1630] Input: Video file, metadata (file name, upload date and time, user ID, etc.)

[1631] Output: Path where the video file is saved, metadata registered in the database

[1632] Specific operation: The server saves the received video file in the specified directory of the file system and records the save path in the database.

[1633] Step 3:

[1634] The server extracts the audio track from the saved video file using a tool such as FFmpeg.

[1635] Input: Saved video file

[1636] Output: Extracted audio data file

[1637] Specific operation: The server executes FFmpeg commands to extract the audio track from the video file and save it as an audio data file.

[1638] Step 4:

[1639] The server sends the extracted voice data to a voice recognition engine (such as Google Cloud Speech-to-Text or DeepSpeech) and converts it into text data.

[1640] Input: Audio data file

[1641] Output: The resulting text data

[1642] Specific operation: The server calls the API of the speech recognition engine, sends the voice data, receives the resulting text data, and saves it as an intermediate file.

[1643] Step 5:

[1644] The server reads the stored text data and extracts major topics and key points using natural language processing libraries (such as NLTK or spaCy).

[1645] Input: Text data

[1646] Output: A list of topics and keypoints

[1647] What it does: The server reads the text data and uses natural language processing libraries to tokenize the text, tag it with POS tags, recognize entities, and extract key topics and key points.

[1648] Step 6:

[1649] The server divides the video into chapters based on the extracted topics and key points, calculates the start and end times of each chapter, and lists the chapter information.

[1650] Input: List of topics and keypoints

[1651] Output: List of chapter information (start time, end time, content)

[1652] Specific operation: The server divides the video into multiple sections based on the timestamps of the topics extracted from the text data, and lists the start and end times of each section.

[1653] Step 7:

[1654] The server generates a summary of the text data for each chapter using a summarization tool (such as GPT-3 or BERT).

[1655] Input: Text data for each chapter

[1656] Output: Summary text

[1657] Specific operation: The server calls the summary generation API, sends the text data for each chapter, and generates a summary text based on the results.

[1658] Step 8:

[1659] The server stores the generated chapter and summary information in a database.

[1660] Input: List of chapter information, summary text

[1661] Output: Chapter and summary information stored in a database

[1662] Specific operation: The server inserts and saves each chapter information and summary into the database.

[1663] Step 9:

[1664] The server retrieves the generated chapter and summary information from the database, converts it into JSON format, and sends it to the terminal.

[1665] Input: Chapter and summary information stored in the database

[1666] Output: JSON format data

[1667] Specific operation: The server queries the database for the required data, serializes the retrieved data into JSON format, and returns it to the device as an HTTP response.

[1668] Step 10:

[1669] The device then displays information for each chapter on the user interface based on the analysis results received, allowing the user to select the part they want to watch and play it.

[1670] Input: Analysis result data in JSON format

[1671] Output: Chapter information displayed in the user interface

[1672] What it does: The device parses the received JSON data and renders an interface that includes the start time, summary, thumbnails, etc. for each chapter.

[1673] Step 11:

[1674] The user inputs a specific keyword and clicks the search button, and the device sends this input data to the server.

[1675] Input: Keywords (e.g., specific lecture content)

[1676] Output: Search request based on keywords

[1677] Specific operation: The device obtains the keywords entered in the search bar and sends them to the server as an HTTP request.

[1678] Step 12:

[1679] The server performs a full-text search of the text data in the video based on the received keywords (using Elasticsearch, etc.), extracts the relevant chapter information as search results, and sends it to the device.

[1680] Input: Keywords, text data

[1681] Output: Chapter information as search results

[1682] Specific operation: The server queries Elasticsearch for the received keywords, extracts chapter information for matching text, serializes it in JSON format, and returns it to the device.

[1683] Step 13:

[1684] The terminal displays the received search results in a list format on the user interface. The user selects the part they want to watch from the displayed search results and clicks the play button.

[1685] Input: Search results (JSON data)

[1686] Output: Search results displayed in the user interface

[1687] Specific operation: The terminal parses the received JSON data and displays the results in a list format in the user interface.

[1688] Step 14:

[1689] The device will play the video from the specific part of the chapter you clicked.

[1690] Input: User selected chapter information

[1691] Output: Video played from the specified chapter

[1692] Specific operation: The device captures the user's click event and starts playing the video from the start time of the corresponding chapter.

[1693] (Application example 1)

[1694] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1695] In today's world, the amount of video content is increasing, and users are required to quickly obtain the information they need. However, it is not easy to extract the necessary information from long video content and grasp it in a short time. In addition, because the content contains a large amount of information, there is also the problem that viewing and searching cannot be performed efficiently. Therefore, an object of the present invention is to provide a system that efficiently extracts the necessary information from video content and provides it in a short time.

[1696] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1697] In this invention, the server includes a means for receiving video files, a means for extracting audio data from the received video files, and a means for converting the audio data into text data. This allows for efficient extraction of major topics and key points from video content, division of the video into multiple chapters, and generation and display of summaries. The system also includes a means for generating summaries based on input prompts using a generative AI model, and furthermore, enables searching for specific keywords and playback of relevant sections. This allows users to quickly obtain the information they need and efficiently watch and search videos.

[1698] A "video file" is a digital file that contains video and audio data.

[1699] "Audio data" refers to sound data containing human voices extracted from video files.

[1700] "Text data" refers to data obtained by converting voice data into character information.

[1701] "Major topics" are important topics or themes of videos extracted from text data.

[1702] "Key points" are important points or elements related to the main topic.

[1703] "Chapters" are sections of a video file that are divided based on specific topics or key points.

[1704] A "summary" is a text that briefly describes the contents of each chapter.

[1705] A "generative AI model" is an artificial intelligence model that generates and understands text based on large datasets.

[1706] A "prompt" is a text instruction that is input to a generative AI model.

[1707] The "means for receiving" is the part that has the function of uploading and saving video files to the system.

[1708] The "extraction means" is the part that has the function of extracting audio data from a video file.

[1709] The "conversion means" is a part that has the function of converting voice data into text data.

[1710] The "means for generating a summary" is a part that has the function of converting text data into a summary.

[1711] The "means for saving" is a part that has the function of saving the generated chapters and summaries.

[1712] The "display means" is a part that has the function of displaying the saved chapters and summaries on the user interface.

[1713] The "search means" is the part that has the function of searching the text data within the video based on specific keywords and displaying the results.

[1714] "Means for playback" refers to the part that has the function of playing the video of the relevant part of the search results.

[1715] This invention is a system that extracts necessary information from video content in a short time and efficiently provides it to users. The system receives video files, extracts and converts audio data, extracts major topics and key points, generates summaries using a generative AI model, and displays and searches the results.

[1716] First, a user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the server, which receives the video file and saves it in the specified directory. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[1717] Next, the server extracts the audio track from the saved video file using a tool such as FFmpeg. The server then sends the extracted audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert it into text data, which is then saved as an intermediate file.

[1718] The server then reads the text data and uses natural language processing libraries (e.g., spaCy) to extract major topics and key points, clarifying the content of the video and establishing a foundation for the next steps.

[1719] The server divides the video into chapters based on topics and key points. It identifies the start and end times of each chapter and lists the chapter information. The server then generates a summary of each chapter's text data using a summarization tool (e.g., GPT-3). The generated chapters and summaries are stored in a database.

[1720] When a user enters a specific keyword, the server performs a full-text search of the text data within the video based on the received keyword. The server extracts the relevant chapter information as search results and sends it to the device. The device then displays the received search results in a list format on the user interface. The user can then select the part they want to watch from the displayed search results and play it.

[1721] As a concrete example, consider the case where a university student uploads a recorded lecture. When a user uploads a video file of a lecture to the system, the server extracts the audio data and converts it into text data using a speech recognition engine. Then, natural language processing technology is used to extract key topics from the text data and divide the video into chapters based on these. The content of each chapter is summarized and displayed on the device, allowing the user to select the chapter they want to view and review it efficiently.

[1722] An example of a prompt sentence in a generative AI model is "Please summarize the following text in a concise manner: \n\n[text content]", which allows the generative AI model to summarize the input text.

[1723] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1724] Step 1:

[1725] A user logs in to the system using a terminal and uploads a video file. The terminal sends the video file selected by the user to the server. The input is the video file, and the output is the video file received by the server.

[1726] Step 2:

[1727] The server saves the received video file in the specified directory. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database. The input is the video file, and the output is the saved video file and the registration of metadata in the database.

[1728] Step 3:

[1729] The server extracts the audio track from the stored video file using a tool such as FFmpeg. The input is the stored video file, and the output is the extracted audio data.

[1730] Step 4:

[1731] The server sends the extracted voice data to a voice recognition engine (such as Google Cloud Speech-to-Text) and converts it into text data. The input is voice data, and the output is the converted text data.

[1732] Step 5:

[1733] The server reads the text data and extracts the main topics and keypoints using a natural language processing library (such as spaCy). The input is the text data, and the output is the extracted main topics and keypoints.

[1734] Step 6:

[1735] The server divides the video into chapters based on topics and key points, identifies the start and end times of each chapter, and lists the chapter information. The input is the main topics and key points, and the output is the divided chapter information.

[1736] Step 7:

[1737] Next, the server generates a summary of the text data of each chapter using a summarization tool (a generative AI model such as GPT-3). A prompt is used for generation. The input is the text data and prompt for each chapter, and the output is the generated summary. An example of a prompt is "Please summarize the following text in a concise manner: \n\n[text content]".

[1738] Step 8:

[1739] The server stores the generated chapters and summaries in a database. The input is the generated chapters and summaries, and the output is the data stored in the database.

[1740] Step 9:

[1741] When a user inputs a specific keyword, the server performs a full-text search of the text data in the video based on the received keyword. The input is the specific keyword, and the output is the searched chapter information.

[1742] Step 10:

[1743] The server extracts the relevant chapter information as the search result and sends it to the terminal. The input is the chapter information of the search result, and the output is the search result sent to the terminal.

[1744] Step 11:

[1745] The terminal displays the received search results in a list format on the user interface. The user can select the part they want to watch from the displayed search results and play it. The input is the chapter information of the search results, and the output is the search results displayed on the user interface.

[1746] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1747] This invention improves the user experience by combining a system that quickly acquires necessary information from video content with an emotion engine that recognizes the user's emotions. The specific processing and operation of each step are explained below.

[1748] Importing videos

[1749] A user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the system, and the server receives the video file and saves it in its internal storage. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[1750] Video Analysis

[1751] The server extracts the audio track from the stored video file, typically using a tool like FFmpeg. The server then sends the extracted audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech) to convert it into text data, which is then saved as an intermediate file.

[1752] Natural Language Processing

[1753] The server reads the text data stored as intermediate files and uses natural language processing libraries (e.g., NLTK or spaCy) to extract major topics and key points, clarifying the content of the video and generating the foundational data for the next step.

[1754] Chapter generation and summarization

[1755] The server divides the video into multiple chapters based on the extracted topics and key points. It identifies the start and end times of each chapter and lists the chapter information. Then, the server generates a summary for each chapter using a summarization tool (e.g., GPT-3 or BERT) based on each chapter. The generated chapters and summaries are stored in a database.

[1756] Providing a user interface

[1757] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the device in JSON format or similar. The device then displays information for each chapter on the user interface based on the analysis results received. The start time, summary, thumbnail, and other information for each chapter are displayed, allowing the user to select and play the part they want to watch.

[1758] Reverse search function

[1759] The user enters a specific keyword in the search bar and clicks the search button. The server uses Elasticsearch or similar tools to perform a full-text search of the text data within the video based on the received keyword. The server extracts the relevant chapter information as search results and sends it to the device. The device displays the received search results in list format, and for each chapter in the search results, displays the timestamp where the keyword appears and a summary. The user can select the part they want to watch and click the play button to play the video from that specific part of that chapter.

[1760] Emotion engine integration

[1761] The present invention further integrates an emotion engine that recognizes the user's emotions. The emotion engine allows the device to analyze the user's emotional state in real time using the user's facial expressions, voice, keyboard and mouse operation patterns, etc. For example, if the system determines that the user is interested, it will prioritize displaying related chapters and summaries.

[1762] Emotion-based chapter display

[1763] The server receives the emotion data obtained by the emotion engine and adjusts the display order of chapters and summaries based on the user's emotional state, thereby prioritizing the display of content that is likely to interest the user and improving the user experience.

[1764] Filtering search results based on sentiment

[1765] When a user enters a search keyword, the emotional engine provides emotional data to filter search results. For example, if a user is feeling stressed, chapters with relaxing content will be prioritized.

[1766] Specific examples

[1767] For example, when a user watches a recording of a business meeting and uses the system with the emotion engine, the system analyzes the user's emotions and prioritizes displaying important parts or parts that the user is likely to be interested in. This allows the user to efficiently obtain the information they need.

[1768] In this way, the present invention provides a means for efficiently obtaining necessary information from video content, while also providing a more personalized experience by taking into account the user's emotional state.

[1769] The processing flow will be explained below.

[1770] Step 1:

[1771] Uploading videos

[1772] The user logs into the system, selects a video file, and clicks the upload button.

[1773] The terminal transmits the selected video file to the system.

[1774] Step 2:

[1775] Saving video files

[1776] The server saves the received video file in a specified directory in the internal storage.

[1777] The server registers the destination path and related metadata (file name, upload date and time, user ID, etc.) in a database.

[1778] Step 3:

[1779] Extracting audio data

[1780] The server extracts the audio track from the stored video file using a tool such as FFmpeg.

[1781] Step 4:

[1782] Voice Recognition

[1783] The server sends the extracted voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech) and converts it into text data.

[1784] The server stores the converted text data as an intermediate file.

[1785] Step 5:

[1786] Natural Language Processing

[1787] The server reads the text data stored as intermediate files and extracts major topics and key points using natural language processing libraries (e.g., NLTK or spaCy).

[1788] Step 6:

[1789] Chapter Generation

[1790] The server divides the video into multiple chapters based on the extracted topics and key points, identifies the start and end times of each chapter, and lists the chapter information.

[1791] Step 7:

[1792] Generate a summary

[1793] Based on each chapter, the server generates a summary for each chapter using a summarization tool (e.g., GPT-3 or BERT).

[1794] The server stores the generated chapters and summaries in a database.

[1795] Step 8:

[1796] Sending analysis results

[1797] The server retrieves the generated chapter and summary information from the database and sends the analysis results to the terminal in JSON format or similar.

[1798] Step 9:

[1799] View Chapters

[1800] Based on the analysis results, the device displays information for each chapter on the user interface, including the start time, summary, and thumbnail of each chapter.

[1801] Step 10:

[1802] Starting emotion recognition

[1803] The device runs an emotion engine and analyzes the user's facial expressions, voice, and operation patterns, including input from the camera, microphone, mouse, and keyboard.

[1804] The emotion engine analyzes the acquired data in real time to determine the user's emotional state.

[1805] Step 11:

[1806] Emotion-based interface adjustment

[1807] The server adjusts the display order of chapters and the content of summaries based on the emotion data received from the emotion engine.

[1808] For example, if it is determined that the user is interested, related chapters are preferentially displayed.

[1809] Step 12:

[1810] Play video

[1811] The user selects the chapter they wish to view and clicks the play button.

[1812] The device will play the video from the start time of the selected chapter.

[1813] Step 13:

[1814] Keyword search

[1815] The user enters a specific keyword in the search bar and clicks the search button.

[1816] The server performs a full-text search of the text data in the video based on the received keywords (e.g., using Elasticsearch).

[1817] The server extracts the relevant chapter information as the search result and transmits it to the terminal.

[1818] Step 14:

[1819] Filtering search results based on sentiment

[1820] The server uses data from the emotion engine to filter search results according to the user's emotions.

[1821] For example, if you want to relax, a chapter with relaxing content will be displayed.

[1822] Step 15:

[1823] Displaying search results

[1824] The device displays the received search results in a list format, with a timestamp and summary of the chapters in each search result.

[1825] The user selects the portion of the search result that he or she wants to view from the displayed search results and clicks the play button.

[1826] The device will play the video from the specific part of the chapter you clicked.

[1827] Example 2

[1828] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1829] In recent years, the use of video data has increased, creating a need for quickly extracting only the necessary parts from this vast amount of information. However, conventional technologies have been unable to take into account user emotion data when converting audio from video to text, extracting key topics and key points, dividing video into segments, and generating summaries, making it difficult to provide optimal information. Furthermore, the search function was unable to filter content based on user emotion, resulting in a poor user experience.

[1830] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1831] In this invention, the server includes means for receiving video data, means for extracting audio data from the received video data, means for converting the audio data into text data, means for extracting main topics and important points from the text data, means for dividing the video data into segments based on the extracted topics or important points, means for generating summaries for each segment, means for saving the generated segments and summaries, means for displaying the saved segments and summaries, means for inputting a specific keyword for search and displaying search results, means for analyzing user emotion data, and means for filtering the display order and search results based on the emotion data, thereby enabling optimal information provision that takes user emotions into consideration.

[1832] "Video data" refers to data in a file format that includes video and audio.

[1833] "Audio data" refers to data consisting of only the audio track extracted from video data.

[1834] "Text data" refers to data obtained by converting voice data into character information.

[1835] "Major topics" refer to themes or topics that are considered particularly important in the content of a video.

[1836] "Important points" refer to specific points or key information in the video content that are particularly noteworthy.

[1837] "Segmenting" refers to dividing video data into multiple parts and treating each part as an independent chapter.

[1838] A "segment" refers to each part of divided video data.

[1839] "Generating a summary" refers to creating a short description that succinctly summarizes the content of each segment.

[1840] "Storing" refers to recording the generated data in storage such as a database.

[1841] "Displaying" refers to visually showing the generated data in a user interface.

[1842] "Searching" refers to matching information in a database using specific keywords and extracting relevant results.

[1843] "Displaying search results" refers to displaying the information obtained by the search in a user interface.

[1844] "Analyzing user emotional data" refers to evaluating the user's emotional state in real time based on their facial expressions, voice, operation patterns, etc.

[1845] "Filtering the display order or search results based on emotional data" refers to adjusting the order or content of the information to be displayed in accordance with the acquired emotional data.

[1846] This invention is a system that quickly acquires necessary information from video data, and improves the user experience by combining it with an emotion engine that recognizes the user's emotions. Specific operations and embodiments of this system are described below.

[1847] Importing videos

[1848] A user logs in to the system using a terminal and uploads a video file. The terminal sends the selected video file to the server. The server saves the received video file in its internal storage. The destination path and video metadata (file name, upload date and time, user ID, etc.) are registered in the database.

[1849] Video Analysis

[1850] The server extracts the audio track from the saved video file using a tool called FFmpeg. The server then sends the extracted audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text or DeepSpeech), which converts the audio data into text data. The text data is then saved as an intermediate file.

[1851] Natural Language Processing

[1852] The server reads the text data stored as intermediate files and uses natural language processing libraries (e.g., NLTK or spaCy) to extract the main topics and key points, clarifying the content of the video and generating the foundational data for the next step.

[1853] Chapter generation and summarization

[1854] The server divides the video into multiple segments based on the extracted topics and key points. It identifies the start and end times of each segment and lists the segment information. Then, the server generates a summary for each segment using a summarization tool (e.g., GPT-3 or BERT). The generated segments and summaries are stored in a database.

[1855] Providing a user interface

[1856] The server retrieves the generated segment and summary information from the database and sends the analysis results to the device in JSON format or similar. Based on the analysis results received, the device displays information for each segment on the user interface. The start time, summary, thumbnail, and other information for each segment are displayed, allowing the user to select and play the part they want to watch.

[1857] Reverse search function

[1858] The user enters a specific keyword in the search bar and clicks the search button. The server uses Elasticsearch or similar tools to perform a full-text search of the text data within the video based on the received keyword. The server extracts the relevant segment information as search results and sends it to the device. The device displays the received search results in list format, and for each segment in the search results, displays the timestamp where the keyword appears and a summary. The user can select the part they want to watch and click the play button to play the video from that specific part of that segment.

[1859] Emotion engine integration

[1860] The present invention further integrates an emotion engine that recognizes the user's emotions. The emotion engine allows the terminal to analyze the user's emotional state in real time using the user's facial expressions, voice, keyboard and mouse operation patterns, etc. For example, if it is determined that the user is interested, the system will prioritize displaying related segments and summaries.

[1861] Segment display based on emotions

[1862] The server receives the emotion data obtained by the emotion engine and adjusts the display order of the segments and summaries based on the user's emotional state, thereby prioritizing the display of content that is likely to interest the user and improving the user experience.

[1863] Filtering search results based on sentiment

[1864] When a user enters a search keyword, the emotion engine provides emotional data to filter search results. For example, if a user is feeling stressed, segments with relaxing content will be prioritized.

[1865] Specific examples

[1866] For example, when a user watches a recording of a business meeting, the system analyzes the user's emotions and prioritizes displaying important parts or parts likely to interest the user. This allows the user to efficiently obtain the information they need. In this way, the present invention provides a means for efficiently obtaining necessary information from video data, while also providing a more personalized experience by taking the user's emotional state into consideration.

[1867] Prompt Sentence Examples

[1868] "Please extract the main topics and key points from this video."

[1869] "Summarize and segment recordings of business meetings."

[1870] "Rearrange your segments and summaries taking into account the output data of the sentiment engine to prioritize what is likely to interest the user."

[1871] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1872] Step 1: Import video

[1873] After logging in, the user selects and uploads a video file.

[1874] Input: Video file, user information

[1875] The terminal reads the selected video file and transfers it to the server.

[1876] The server receives the video file and saves it to its internal storage. At this time, it registers the destination path and video metadata (file name, upload date and time, user ID, etc.) in a database.

[1877] Output: Video files stored in the internal storage, metadata registered in the database

[1878] Step 2: Analyze the video

[1879] The server extracts the audio track from the stored video file.

[1880] Input: Saved video file

[1881] The server uses FFmpeg to extract the audio data and sends it to a speech recognition engine, specifically, by calling the Google Cloud Speech-to-Text or DeepSpeech API.

[1882] The server converts the voice data into text data and stores the text data as an intermediate file.

[1883] Output: Text data saved as an intermediate file

[1884] Step 3: Natural Language Processing

[1885] The server reads the text data stored as an intermediate file.

[1886] Input: Text data saved as an intermediate file

[1887] The server uses natural language processing libraries (such as NLTK or spaCy) to extract key topics and key points, which involves tokenizing words and parsing sentences.

[1888] Output: Extracted main topics and key points data

[1889] Step 4: Chapter generation and summarization

[1890] The server divides the video into multiple segments based on the extracted topics and key points.

[1891] Input: Data on key topics and key points

[1892] The server identifies and lists the start and end times of each segment, possibly using a video editing library.

[1893] The server uses a summarization tool (such as GPT-3 or BERT) to generate a summary of each segment.

[1894] The generated segments and summaries are stored in a database.

[1895] Output: Generated segments and summaries stored in a database

[1896] Step 5: Providing a User Interface

[1897] The server retrieves the generated segment and summary information from the database, converts it into JSON format, and sends it to the terminal.

[1898] Input: Generated segments and summaries stored in a database

[1899] Based on the analysis results received, the device displays information for each segment on the user interface, including the start time, summary, and thumbnail of each segment.

[1900] Output: Information for each segment displayed in the user interface

[1901] Step 6: Reverse Search Function

[1902] The user enters a specific keyword into the search bar and clicks the search button.

[1903] Input: Search keyword

[1904] Based on the received keywords, the server uses Elasticsearch to perform a full-text search of the text data within the video.

[1905] The server extracts the relevant segment information as a search result and transmits it to the terminal.

[1906] The terminal displays the received search results in a list format, and displays the timestamps at which the keywords appear and a summary for each segment of each search result.

[1907] Output: Search results displayed in list format

[1908] Step 7: Integrating the Emotion Engine

[1909] The emotion engine analyzes the user's facial expressions, voice, keyboard and mouse operation patterns, and other information acquired by the device in real time.

[1910] Input: User's facial expressions, voice, and operation patterns

[1911] The device analyzes this data and estimates the emotional state.

[1912] The server adjusts the order of segments and summaries to be displayed based on the emotion data received from the emotion engine.

[1913] Output: Reordered segments and summaries based on sentiment data

[1914] Step 8: Segment based on sentiment

[1915] The server receives the emotion data obtained from the emotion engine.

[1916] Input: Emotion data

[1917] The server adjusts the display order of the segments and summaries based on the user's emotional state and transmits them again to the terminal.

[1918] The device will prioritize updating the displayed information to improve the user experience.

[1919] Output: Reordered segments and summary information

[1920] Step 9: Filter search results based on sentiment

[1921] When a user enters a search keyword and performs a search, the search results are filtered based on the emotional data provided by the emotion engine.

[1922] Input: Search keywords, emotion data

[1923] The server filters the search results taking into account the emotion data and transmits the results based on that to the terminal.

[1924] The terminal displays the filtered results in a list format.

[1925] Output: Filtered search results

[1926] Prompt Sentence Examples

[1927] "Please extract the main topics and key points from this video."

[1928] "Summarize and segment recordings of business meetings."

[1929] "Rearrange your segments and summaries taking into account the output data of the sentiment engine to prioritize what is likely to interest the user."

[1930] (Application example 2)

[1931] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1932] Systems for quickly obtaining necessary information from video content have the problem of not being able to efficiently provide the parts of a user's interest, resulting in a lack of improvement in the user experience. Furthermore, conventional systems are unable to display content taking into account the user's emotions, making it difficult to provide personalized services.

[1933] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a video file, means for extracting audio data from the received video file, means for converting the audio data into text data, means for extracting major topics and key points from the text data, means for dividing the video file into chapters based on the extracted topics or key points, means for generating summaries for each chapter, means for saving the generated chapters and summaries, means for displaying the saved chapters and summaries, means for inputting a specific keyword to perform a search and displaying search results, emotion recognition means for recognizing the user's emotional state, means for adjusting the display order of chapters and summaries based on the emotional state, and means for filtering search results based on the emotional state. This allows the user to prioritize viewing parts that interest them.

[1934] A "video file" is a file that is stored in digital format and contains video and audio.

[1935] "Means for receiving" refers to hardware or software for acquiring data from the outside and incorporating it into the system.

[1936] "Audio data" is digital sound information extracted from the audio track of a video file.

[1937] "Means for converting voice data into text data" refers to a process of converting voice information into text information using a voice recognition engine.

[1938] "Text data" is character information converted from voice data.

[1939] "Major topics and key points" refer to important themes and key points extracted from text data.

[1940] A "chapter" is a part of a video file that is divided into multiple segments.

[1941] A "summary" is a sentence or paragraph that succinctly summarizes the contents of each chapter.

[1942] "Emotion recognition means" refers to technology or equipment that analyzes the user's emotional state in real time from their facial expressions, voice, operation patterns, etc.

[1943] An "emotional state" refers to the emotion a user is feeling at a particular moment.

[1944] A "means for adjusting the display order" is a process or method for changing the order in which content is displayed based on the user's emotional state.

[1945] A "search result filtering method" is a process or technique that sorts search results based on the user's emotional state and tailors the content displayed to specific conditions.

[1946] The present invention provides a system for improving the user experience by quickly acquiring necessary information from video files and adjusting the display order and search results based on the user's emotions. An embodiment of the system will be described below.

[1947] The system includes means for receiving a video file, means for extracting audio data, means for converting the audio data into text data, means for extracting main topics and key points, means for dividing into chapters, means for generating summaries, means for saving the generated chapters and summaries, means for displaying the saved information, means for displaying search results by inputting specific keywords, means for recognizing a user's emotions, means for adjusting the display order based on the user's emotional state, and means for filtering the search results.

[1948] The server receives video files from user devices and extracts audio data from the received videos using common media processing tools such as FFmpeg. The extracted audio data is converted into text data using a speech recognition engine such as Google Cloud Speech-to-Text or DeepSpeech. The converted text data is then used to extract major topics and key points using natural language processing libraries such as NLTK or spaCy.

[1949] The server then divides the video into chapters based on the extracted topics and key points, and generates summaries for each chapter using a generative AI model such as GPT-3. The generated chapters and summaries are stored in a database and later displayed through a user interface.

[1950] When a user enters a specific keyword to perform a search, the server uses a search engine such as Elasticsearch to perform a full-text search of the text data and extracts the relevant chapter information. Based on the results, the device displays the search results and the user can play that chapter.

[1951] The emotion recognition system analyzes the user's emotional state in real time using the smartphone's camera, microphone, and touch operation data. Specifically, it utilizes Microsoft Azure Emotion API and Affectiva SDK. Based on this emotion recognition, the server adjusts the order of chapters and summaries displayed and further filters search results. For example, if the user shows interest, relevant chapters and summaries will be displayed first.

[1952] An example of a specific input prompt is as follows:

[1953] input_video_path = " / path / to / uploads / your_video.mp4"

[1954] user_emotion = "interested"

[1955] In this way, the invention is a system that provides efficient information acquisition and a personalized experience.

[1956] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1957] Step 1:

[1958] Uploading a video file

[1959] What happens: A user uses a device to upload a video file to the system.

[1960] Input: A video file selected by the user.

[1961] Output: The uploaded video file is sent to the server, and the server receives the video file and saves it in its internal storage.

[1962] Specific operation: The device sends a video file to the server as an HTTP POST request, and the server receives the file and registers the storage path and metadata in the database.

[1963] Step 2:

[1964] Extracting audio data

[1965] Processing content: Extracts audio data from the video file received by the server.

[1966] Input: Saved video file.

[1967] Output: The extracted audio data.

[1968] Specific operation: The server uses the FFmpeg tool to extract the audio track from the video file and generate the audio data.

[1969] Step 3:

[1970] Convert speech to text

[1971] Processing content: Convert the extracted audio data into text data.

[1972] Input: Audio data.

[1973] Output: The converted text data.

[1974] How it works: The server uses a speech recognition engine such as Google Cloud Speech-to-Text or DeepSpeech to analyze the audio data and convert it into text. The converted text data is saved as an intermediate file.

[1975] Step 4:

[1976] Extracting major topics and key points

[1977] What it does: Extract major topics and key points from text data.

[1978] Input: Text data.

[1979] Output: Major topics and key points.

[1980] How it works: The server analyzes the text data using natural language processing libraries (such as NLTK or spaCy) to extract important topics and key points from the text. This information is then stored in a database.

[1981] Step 5:

[1982] Chapter Generation

[1983] What it does: Divide the video into chapters based on extracted topics and key points.

[1984] Input: Major topics and key points.

[1985] Output: Chapter information.

[1986] Specific operation: The server identifies the start and end times of each topic and divides the video into chapters. The chapter information is listed and stored in a database.

[1987] Step 6:

[1988] Generate a summary for each chapter

[1989] Process: Generate a summary for each chapter.

[1990] Input: Chapter information.

[1991] Output: The generated summary.

[1992] How it works: The server uses a generative AI model (e.g., GPT-3) to summarize the content of each chapter and stores the summaries in a database.

[1993] Step 7:

[1994] Displaying analysis results

[1995] What it does: Display the saved chapters and summaries in the user interface.

[1996] Input: Chapter and abstract information stored in the database.

[1997] Output: Chapter and summary information displayed on the device.

[1998] Specific operation: The server retrieves the analysis results from the database and sends them to the device in JSON format. The device then displays the analysis results on the user interface based on the received data.

[1999] Step 8:

[2000] Keyword search

[2001] What happens: A user enters a specific keyword into the search bar and displays search results.

[2002] Input: The keyword entered by the user.

[2003] Output: Chapter information and summary displayed as search results.

[2004] Specific operation: The server performs a full-text search of the text data using Elasticsearch or similar tools to extract the relevant chapter information. The device displays the received search results in a list, and the user can click the play button to watch the relevant section.

[2005] Step 9:

[2006] Emotion recognition and display order adjustment

[2007] What it does: Recognizes the user's emotional state and adjusts the order in which chapters and summaries are displayed based on that information.

[2008] Input: User emotion data.

[2009] Output: Adjusted chapter and summary display order.

[2010] Specific operation: The emotion recognition means analyzes the user's emotions in real time using the device's camera, microphone, and touch operation data. The server adjusts the display order based on the emotion data and prioritizes displaying the parts that are of interest to the user.

[2011] Step 10:

[2012] Sentiment-based search result filtering

[2013] What it does: Filter search results based on emotional state.

[2014] Input: User sentiment data and search keywords.

[2015] Output: The filtered search results.

[2016] How it works: The server adjusts search results based on emotional data. For example, if a user is feeling stressed, chapters with relaxing content will be prioritized.

[2017] In this way, through each step, the user can view the parts that interest him / her preferentially, and can efficiently obtain the necessary information.

[2018] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2019] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2020] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2021] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2022] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2023] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2024] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2025] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2026] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2027] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2028] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure m...

Claims

1. means for receiving a video file; means for extracting audio data from the received video file; means for converting voice data into text data; means for extracting major topics and key points from text data; means for dividing the video file into chapters based on the extracted topics or key points; a means for generating chapter-by-chapter summaries; means for storing the generated chapters and summaries; a means for displaying the saved chapters and summaries; A means to search by entering specific keywords and display search results; A system including:

2. 10. The system of claim 1, wherein the means for generating summaries for each chapter uses natural language processing techniques to generate summaries.

3. 2. The system according to claim 1, further comprising means for extracting relevant chapter information as a search result and reproducing the relevant portion.

Images