System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting log data of a user; means for analyzing the collected log data and extracting feature quantities related to the personality of the user; means for extracting scene cuts and thumbnails from video content based on the feature quantities; means for combining the feature quantities and the extracted scene cuts and thumbnails to generate candidates for a video summary and thumbnails suitable for the personality of the user; and means for selecting an optimal video summary and thumbnail from among the candidates and presenting the selected video summary and thumbnail to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional video distribution platforms, social media, online education platforms, and other platforms face the challenge of providing content that fully takes into account each user's individual preferences and interests. Generic thumbnails and summaries struggle to attract users' attention and make it difficult for them to discover new content. This leads to a poor user experience and a decrease in the efficiency of content consumption. To address this challenge, the present invention aims to develop a system that semi-automatically generates attractive and unique video summaries and thumbnails tailored to each user's individuality and provides them to users. [Means for solving the problem]

[0005] The present invention provides a system including means for collecting user log data, means for analyzing the collected log data and extracting features related to the user's personality, means for extracting scene cuts and thumbnails from video content based on the features, means for generating candidate video summaries and thumbnails tailored to the user's personality by combining the features with the extracted scene cuts and thumbnails, and means for selecting the most suitable video summaries and thumbnails from the candidates and presenting them to the user, thereby enabling the system to quickly and accurately provide content tailored to each user's preferences and improve the user experience.

[0006] "User log data" refers to data that records a series of operations and actions that a user has taken on a digital platform, such as a user's past behavioral data, viewing history, search history, and click history.

[0007] "Collection means" means a software or hardware mechanism for extracting and retrieving user log data from a database or log file.

[0008] "Means of analyzing and extracting features related to the user's personality" refers to software and algorithms that identify the user's interests and concerns based on collected log data and express them in numbers, categories, etc.

[0009] "Video content" refers to video data stored in digital format, and is media containing images and audio that can be viewed by users.

[0010] "Feature-based scene cut and thumbnail extraction means" refers to software and algorithms that analyze specific scenes or frames within video content, extract them, and generate thumbnail images or video segments tailored to the user's interests.

[0011] A "video summary" is a video that extracts important scenes from a long video content and allows the viewer to understand the main points in a short amount of time.

[0012] A "thumbnail" is a still image that represents video content and is a small image used as a cue for users to click on the content.

[0013] "Embedding similarity calculation" is a method for calculating the similarity between different data using a numerical vector (embedding) to measure the degree of relevance or similarity between the data.

[0014] "Means for selecting and presenting optimal video summaries and thumbnails to a user" means software and algorithms for determining from among multiple candidates the video summaries and thumbnails that best match the user's preferences and displaying them to the user. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system that provides video summaries and thumbnails that take into account the user's individuality. In this system, the server collects and analyzes user log data, and generates and provides video summaries and thumbnails that are tailored to the user's individuality. The specific program processing flow and its explanation are described below.

[0037] System Overview

[0038] This system analyzes user behavior data to generate summaries and thumbnails of video content. User log data includes viewing history, search history, click history, etc. This log data is used to identify user interests and preferences.

[0039] What the program does

[0040] Collecting and analyzing user log data

[0041] The server collects log data from the database, including the user's past behavior data, such as the types of videos the user watched, the keywords they searched for, and the thumbnails they clicked on.The collected log data is analyzed and natural language processing (NLP) and machine learning models are used to extract features related to the user's personality.

[0042] Example: Extract the features "sports" and "entertainment" from user A's past viewing history.

[0043] Video content analysis

[0044] The server analyzes the video content frame by frame to detect key scenes and key points, using image recognition technology and scene detection algorithms, and extracts specific cuts and frames from the video based on the user's personality features.

[0045] Example: Extracting goal scenes and player highlights from videos of sports matches.

[0046] Video summary and thumbnail generation

[0047] The server combines the user's personality features with key scenes from the video to generate video summary and thumbnail candidates. The candidates are then evaluated using embedding similarity calculations and the most suitable one is selected.

[0048] Example: For User A, automatically generate a two-minute summary of a sports match highlights with thumbnails capturing the exciting moments.

[0049] Optimal summary and thumbnail presentation

[0050] The device will present users with optimally selected video summaries and thumbnails, allowing them to quickly get an overview of the content they are watching.

[0051] Specific examples

[0052] When User A wants to watch a new sports game video, the system works as follows:

[0053] 1. The server collects user A's past log data and identifies that he has sports-related preferences.

[0054] 2. The server analyzes frames from new match videos and extracts key moments such as goals and player highlights.

[0055] 3. The server combines these scenes to generate a two-minute highlight summary and visually interesting thumbnails.

[0056] 4. The device presents these summaries and thumbnails to User A.

[0057] This allows User A to quickly check scenes that interest him / her, improving his / her viewing experience and promoting the discovery of new content.

[0058] The above is a specific embodiment for carrying out the present invention.

[0059] The processing flow will be explained below.

[0060] Step 1:

[0061] The server collects the user's past behavior data, including viewing history, search history, click history, etc. It accesses the database to extract the target user's log data and stores the collected data in memory or cache for later analysis.

[0062] Step 2:

[0063] The server analyzes the collected log data and identifies user interests and preferences using natural language processing (NLP) and machine learning models. Specifically, it analyzes keywords and viewing patterns in the log data and extracts features such as "sports preferences" and "entertainment preferences."

[0064] Step 3:

[0065] The server analyzes the video content, extracts each frame, and detects key scenes and key moments. This analysis uses image recognition technology to detect scene changes and specific events (e.g., goal scenes, scenes featuring key characters). These scenes are then extracted as important cuts based on the user's personality features.

[0066] Step 4:

[0067] The server combines the extracted key scenes and points with the user's personality features to generate video summary and thumbnail candidates. At this stage, we use embedding similarity calculation to evaluate whether the generated candidates are interesting to the user, and then select the most relevant summary and thumbnail.

[0068] Step 5:

[0069] The device receives the optimal video summary and thumbnail sent from the server and then presents them to the user, for example, displaying a two-minute summary video of a sports highlight along with a visually appealing thumbnail image.

[0070] Step 6:

[0071] Users can view the video summary and thumbnails provided and watch the video if they are interested, a process that allows users to efficiently discover content that interests them.

[0072] This allows users to quickly view scenes that are of particular interest to them, improving their viewing experience and making it easier to discover new content.

[0073] Example 1

[0074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0075] The wide variety of modern video content and the large amount of data available make it difficult for users to quickly find content that matches their interests and preferences. In particular, the selection of video summaries and thumbnails does not provide appropriate content that takes into account the individual needs of users, creating a demand for an improved viewing experience. The present invention aims to solve this problem by providing a system for generating video summaries and thumbnails that take into account the individual needs of users.

[0076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0077] In this invention, the server includes: means for collecting user log data; means for analyzing the collected log data using a natural language processing model and a machine learning model to extract features related to the user's personality; means for analyzing video content frame by frame and extracting key scenes and thumbnails based on the features; means for combining the features with the extracted key scenes and thumbnails to generate video summaries and thumbnail candidates tailored to the user's personality; and means for evaluating the generated candidates using embedding similarity calculation, selecting optimal video summaries and thumbnails, and presenting them to the user. This enables the generation and presentation of video summaries and thumbnails that take the user's personality and preferences into consideration.

[0078] "User log data" refers to data including a user's past behavioral data, viewing history, search history, and click history.

[0079] A "natural language processing model" is a computational algorithm for analyzing and understanding human language, and is used to extract topics and categories from users' search keywords and viewing history.

[0080] A "machine learning model" is an algorithm that analyzes large amounts of data to learn patterns and rules and make predictions and classifications.

[0081] A "feature" is an element of extracted data that indicates a user's interests and preferences.

[0082] "Video content" refers to all video data viewed by users, including specific videos, movies, and programs.

[0083] A "key scene" is a significant event or point in the video content.

[0084] A "thumbnail" is a small image that represents the video content and serves to visually attract the user's attention.

[0085] "Embedding similarity calculation" is a method of calculating similarity by converting feature values ​​and scene information into vector space.

[0086] A "video summary" is a video created by combining key scenes from video content into a shortened version.

[0087] "Candidates" refer to the generated video summaries and thumbnail options.

[0088] The present invention provides a system for providing video summaries and thumbnails that take into account the user's individuality. In this system, a server collects and analyzes user log data, generates video summaries and thumbnails that are tailored to the user's individuality, and provides them to the user. Specific embodiments of the present invention are described below.

[0089] The server collects log data from a database, including users' past behavioral data. This data includes the types of videos the user watched, the keywords they searched for, and the thumbnails they clicked. The collected log data is analyzed using natural language processing (NLP) and machine learning models. As a result of the analysis, features related to the user's personality are extracted. For example, features such as "sports" and "entertainment" are extracted from User A's past viewing history.

[0090] The server then analyzes the video content frame by frame, using image recognition technology and scene detection algorithms. As a result of the analysis, key scenes and points in the video are detected, and specific cuts and frames are extracted based on the user's personality features. For example, goal scenes and player highlights can be extracted from a video of a sports game.

[0091] The server then generates video summary and thumbnail candidates by combining the user's personality features and key scenes from the video. The candidates are evaluated using embedding similarity calculations and the most suitable one is selected. For example, for user A, a two-minute summary of a sports game highlights and thumbnails capturing the most exciting moments are generated.

[0092] Finally, the device presents the optimally selected video summary and thumbnail to the user. By checking the summary and thumbnail, the user can quickly get an overview of the content they are watching. As a concrete example, when user A watches a new sports game video, the system operates as follows:

[0093] 1. The server collects user A's past log data and identifies that he has sports-related preferences.

[0094] 2. The server analyzes new match videos frame by frame and extracts key moments such as goals and player highlights.

[0095] 3. The server combines these scenes to generate a two-minute highlight summary and visually interesting thumbnails.

[0096] 4. The device presents these summaries and thumbnails to User A.

[0097] An example prompt is, "For a sports fan, please create a two-minute summary of sports game highlights based on their past viewing history and create an engaging thumbnail." Using this prompt, the generative AI model can generate optimal content based on the user's personality.

[0098] This invention enables the generation and presentation of video summaries and thumbnails that take into account the user's individuality and preferences, which is expected to improve the viewing experience and promote the discovery of new content.

[0099] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0100] Step 1: Collect user log data

[0101] The server connects to the database to collect user log data. This log data includes the user's viewing history, search history, click history, etc. Specifically, it issues SQL queries to the database to extract the target user's behavioral data. The input is the user behavior data in the database, and the output is the extracted log data.

[0102] Specific behavior:

[0103] The server establishes a database connection and executes a query based on the user's ID.

[0104] The data acquired by the server is temporarily stored in memory.

[0105] Step 2: Analyze the log data

[0106] The server analyzes the collected log data using natural language processing (NLP) and machine learning models. This analysis extracts features related to the user's personality. The input is the collected log data, and the output is the extracted features.

[0107] Specific behavior:

[0108] The server applies NLP models to extract topics and categories from search keywords and viewing history.

[0109] The server uses machine learning models to analyze user behavior patterns and identify categories of interest.

[0110] Step 3: Frame analysis of video content

[0111] The server analyzes the video content the user wants to watch, frame by frame, and uses image recognition technology and scene detection algorithms to detect key scenes and key points. The input is the video file, and the output is key scene information.

[0112] Specific behavior:

[0113] The server loads the video file and splits it into frames.

[0114] The server uses image recognition technology to analyze the frames and evaluate the importance of the scenes.

[0115] The extracted main scene information is linked to the features.

[0116] Step 4: Generate video summary and thumbnails

[0117] The server combines the user's personality features with the main scenes of the video to generate video summaries and thumbnail candidates. The input is the features and main scene information, and the output is video summaries and thumbnail candidates.

[0118] Specific behavior:

[0119] The server sequentially connects the identified key scenes to generate a short summary.

[0120] The server selects visually interesting frames and creates thumbnail candidates.

[0121] Step 5: Evaluate and select summaries and thumbnails

[0122] The server evaluates the generated summaries and thumbnail candidates and selects the best one using embedding similarity calculation. The input is the video summary and thumbnail candidates, and the output is the selected summary and thumbnail.

[0123] Specific behavior:

[0124] The server uses embedding technology to calculate the similarity of each candidate.

[0125] The most similar summaries and thumbnails are selected and prepared for delivery to the user.

[0126] Step 6: Present to the user

[0127] The terminal presents the optimally selected video summary and thumbnail to the user. The input is the selected video summary and thumbnail, and the output is the content displayed in the user interface.

[0128] Specific behavior:

[0129] The server sends the selected summary and thumbnail to the terminal.

[0130] The data received by the terminal is displayed in the user interface.

[0131] The user checks the presented summary and thumbnails to quickly get an overview of the content to be viewed.

[0132] (Application example 1)

[0133] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] Conventional content distribution services lack the technology to automatically generate video summaries and thumbnails that match users' personalities and preferences. This makes it difficult for users to quickly identify scenes that interest or are important to them, resulting in reduced viewing efficiency. Furthermore, the lack of appropriate thumbnails makes it difficult for users to discover new content.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0136] In this invention, the server includes means for collecting user log data, means for analyzing the collected log data and extracting features related to the user's personality, means for extracting scene cuts and thumbnails from video content based on the features, means for generating candidate video summaries and thumbnails tailored to the user's personality by combining the features with the extracted scene cuts and thumbnails, means for selecting optimal video summaries and thumbnails from the candidates and presenting them to the user, means for collecting real-time data, means for using a framework for presenting content to a personal mobile device, and means for generating video summaries and thumbnails using a generative AI model. This allows for the rapid provision of video summaries and thumbnails tailored to the user's personality, improving viewing efficiency and promoting the discovery of new content.

[0137] "User log data" refers to various behavioral data generated when a user uses the system, such as viewing history, search history, and click history.

[0138] "Features" are characteristic data extracted from log data to indicate a user's personality and preferences.

[0139] A "scene cut" is a video fragment cut out from a specific portion of video content.

[0140] A "thumbnail" is a still image that represents a portion of video content and is used to provide a visual overview.

[0141] A "video summary" is a video that extracts the main scenes of a video content and compresses them so that they can be viewed in a short time.

[0142] "Candidates" is a collection of multiple generated video summaries and thumbnails before selecting the best video summaries and thumbnails.

[0143] "Best suited video summaries and thumbnails" are those video summaries and thumbnails that are determined to be most suitable to the user's personality and interests and are presented to the user.

[0144] "Real-time data" is data that is collected immediately and generated on the spot when users use the system.

[0145] A "personal mobile device" is an information processing device that is individually owned and portable by a user, such as a smartphone or tablet.

[0146] A "framework" is a basic structure or framework for efficient software development, and is particularly used to present content.

[0147] A "generative AI model" is an algorithm or program that uses artificial intelligence technology to generate and process information.

[0148] The present invention provides a system for providing personalized video summaries and thumbnails to users, and the specific implementation method thereof will be described below. The system is configured using a server and a user's personal mobile terminal.

[0149] System Overview

[0150] The server is configured using the following hardware and software.

[0151] Hardware: Server computers, database servers

[0152] Software: Apache Kafka, MySQL, BERT model, machine learning models (random forest, SVM), OpenCV, YOLO, React Native

[0153] The server first collects user log data, including viewing history, search history, click history, etc. Apache Kafka is used to collect data in real time, and the collected data is stored in a MySQL database.

[0154] Next, the server analyzes the user's log data using a BERT model or machine learning model to extract personality features. For example, it identifies areas of interest such as "sports" or "entertainment" from the user's viewing history.

[0155] The server then analyzes the video content frame by frame, using image recognition techniques such as OpenCV and YOLO to detect key scenes and key moments, such as goal scenes and player highlights in sports videos.

[0156] The server then uses a generative AI model to combine the user's personality features with the detected scenes to generate a video summary and thumbnails, which are then evaluated using a similarity calculation to select the most suitable one.

[0157] The selected video summaries and thumbnails are then presented to the user's personal mobile device through an app developed using React Native, allowing the user to quickly get an overview of the content they are about to watch.

[0158] Specific examples

[0159] When user A wants to watch a new sports game video, the following happens:

[0160] 1. The server collects user A's past log data and identifies that he has sports-related preferences.

[0161] 2. The server analyzes frames from new match videos and extracts key moments such as goals and player highlights.

[0162] 3. The server combines these scenes to generate a two-minute highlight summary and visually interesting thumbnails.

[0163] 4. The personal mobile device presents these summaries and thumbnails to User A.

[0164] Prompt Sentence Examples

[0165] Examples of prompts for generative AI models include:

[0166] Based on the user's viewing history and interests, extract key scenes from the following video and generate a 2-minute summary and thumbnail.

[0167] Viewing history features: Sports, Entertainment

[0168] Video: "Football match video"

[0169] Scenes to be extracted: Goal scenes, player highlights

[0170] Generated summary length: 2 minutes

[0171] result:

[0172] 1. 2-minute video summary

[0173] 2. Thumbnails capturing goal scenes

[0174] In this way, personalized video summaries and thumbnails can be efficiently generated to provide an optimal viewing experience.

[0175] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0176] Step 1:

[0177] The server collects user log data. Specifically, data such as user viewing history, search history, and click history is collected in real time using Apache Kafka and stored in a MySQL database. This collected data serves as basic information for detailed analysis of user behavior patterns. The input is user behavior data, and the output is log data stored in the MySQL database.

[0178] Step 2:

[0179] The server analyzes the collected log data and extracts features related to the user's personality. Specifically, it uses the BERT model and machine learning models (random forest and SVM) to analyze the user's log data and extract features such as "sports" and "entertainment" from their viewing history and search history. The input for this process is the log data in the MySQL database, and the output is analyzed data containing personality features.

[0180] Step 3:

[0181] The server analyzes the video content frame by frame. Specifically, it uses image recognition technologies such as OpenCV and YOLO to analyze each frame of the video and detect key scenes and key points. This process identifies, for example, goal scenes and player highlights. The input is the original video content, and the output is scene data containing key scenes.

[0182] Step 4:

[0183] The server combines the user's personality features with the extracted scenes to generate video summaries and thumbnail candidates. Specifically, a generative AI model is used to automatically generate video summaries and thumbnails based on the personality features and scene data. In this process, multiple candidates are generated, and the best one is selected. The input is the personality features and scene data, and the output is video summaries and thumbnail candidates.

[0184] Step 5:

[0185] The server selects the best video summary and thumbnail from the generated candidates and presents them to the user. Specifically, it performs a similarity calculation to select the most suitable summary and thumbnail. The selected video summary and thumbnail are then displayed on the user's personal mobile device through an app developed using React Native. The input of this process is the video summary and thumbnail candidates, and the output is the best summary and thumbnail presented to the user.

[0186] Step 6:

[0187] The user checks the presented video summary and thumbnails. Specifically, the user watches the summary and thumbnails using a React Native-based application on a personal mobile device such as a smartphone or tablet. This step is the user interaction phase, where the input is the optimal summary and thumbnail presented to the user, and the output is the user's viewing behavior.

[0188] Through these steps, users can quickly view personalized video summaries and thumbnails, improving their viewing experience.

[0189] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0190] This invention is a system for providing video summaries and thumbnails based on a user's personality and emotional state. In this system, a server collects and analyzes user log data, generates video summaries and thumbnails based on the user's personality features and emotional state, and provides them to the user. The specific program processing flow and its explanation are described below.

[0191] System Overview

[0192] The system analyzes user behavioral and emotional data to identify their personality and emotional state, and then generates summaries and thumbnails of video content. User log data includes viewing history, search history, click history, and emotional data from an emotion engine. Based on this data, the system identifies the user's interests, preferences, and emotional state.

[0193] What the program does

[0194] Collecting and analyzing user log data and emotional data

[0195] The server collects log data from the database, including users' past behavioral data and emotional data. This data includes the types of videos users watched, keywords they searched for, thumbnails they clicked, and the emotional state users displayed while watching a particular video. The collected log data and emotional data are analyzed and natural language processing (NLP) models and emotion analysis models are used to extract features related to the user's personality and emotional state.

[0196] Example: Extract emotional features such as "comedy preference" and "frequent smiling while watching" from user B's past viewing history and facial expressions while watching.

[0197] Video content analysis

[0198] The server analyzes the video content, extracts each frame, and detects key scenes and key moments. Image recognition technology is used for the analysis to detect scene changes and specific events (e.g., interesting or moving scenes). These scenes are extracted as important cuts based on the user's personality and emotional features.

[0199] Example: Extracting funny and touching scenes from a comedy show video.

[0200] Video summary and thumbnail generation

[0201] The server combines the user's personality features with the extracted key scenes from the video to generate candidate video summaries and thumbnails. At this stage, we use embedding similarity calculations to evaluate whether the generated candidates are interesting to the user. By taking the user's emotional state into account, we generate the most visually and emotionally appealing summaries and thumbnails.

[0202] Example: For user B, generate a two-minute video summary highlighting funny moments from a comedy, along with thumbnails capturing smiling moments.

[0203] Optimal summary and thumbnail presentation

[0204] The device receives the optimal video summary and thumbnail sent from the server and presents them to the user, allowing the user to instantly get an overview of the content they are watching.

[0205] Specific examples

[0206] If User B wants to watch a new comedy video, the system works as follows:

[0207] 1. The server collects user B's past log data and emotional data while watching, identifies that user B has a comedy-related preference, and confirms that user B smiles frequently while watching.

[0208] 2. The server analyzes frames from a new comedy video and extracts funny and touching moments.

[0209] 3. The server combines these scenes to generate a two-minute comedy summary video and visually interesting smiley thumbnails.

[0210] 4. The device presents these summaries and thumbnails to User B.

[0211] This allows User B to quickly check scenes that interest them, improving their viewing experience and making it easier for them to discover new content. Emotional data is also taken into account, which tends to increase satisfaction while viewing.

[0212] The processing flow will be explained below.

[0213] Step 1:

[0214] The server collects users' past behavioral data and emotional data. Specifically, it accesses a database to obtain log data including viewing history, search history, click history, and emotional data during viewing. This data is stored in memory or cache as it is required for later analysis steps.

[0215] Step 2:

[0216] The server analyzes the collected log data and emotion data. First, it uses natural language processing (NLP) models and machine learning models to extract features that indicate the user's interests and preferences from the log data. Next, it uses an emotion analysis model to analyze the user's emotional fluctuations and identify the emotional state (e.g., joy, surprise, excitement) shown while watching.

[0217] Step 3:

[0218] The server analyzes the video content frame by frame, and uses image recognition technology and scene detection algorithms to detect key scenes and events (e.g., humorous or moving scenes) in each frame. From the detected scenes, important cuts and thumbnail images are extracted based on the user's personality and emotional features.

[0219] Step 4:

[0220] The server generates candidate video summaries and thumbnails based on the extracted scene cuts and thumbnails. Using embedding similarity calculations, the server evaluates and selects candidates based on their match with the user's interests and emotions. For example, it prioritizes video summaries that contain scenes that the user expressed high enjoyment in while watching.

[0221] Step 5:

[0222] The device receives the optimal video summary and thumbnail sent from the server, passes the received data to the display component, and presents it visually to the user, allowing the user to get a quick overview of the content they are watching.

[0223] Step 6:

[0224] Users can view the video summary and thumbnails provided and choose to watch the video if they are interested. Emotional data is also collected during viewing and used for future analysis.

[0225] This improves the viewing experience by allowing users to quickly check scenes they are particularly interested in. Emotional data is also taken into account, resulting in higher viewing satisfaction and content tailored to the user's preferences.

[0226] Example 2

[0227] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0228] Currently, existing systems that provide optimized video summaries and thumbnails based on a user's personality and emotional state are inadequate. In particular, technologies for selecting and presenting content that reflects a user's preferences and emotions are immature, and user satisfaction cannot be fully enhanced. Therefore, there is a need for a system that can efficiently generate and present video summaries and thumbnails based on a user's personality and emotional state.

[0229] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0230] In this invention, the server includes means for collecting user history data, means for analyzing the collected history data and extracting features related to the user's personality, means for extracting scene cuts and representative images from video content based on the features, means for combining the features with the extracted scene cuts and representative images to generate candidates for video summaries and representative images tailored to the user's personality, and means for selecting the most suitable video summaries and representative images from the candidates and presenting them to the user, thereby making it possible to provide content summaries and thumbnails that accurately reflect the user's interests and emotional state.

[0231] "History data" refers to data about a user's past activities, including viewing history, search history, click history, etc.

[0232] "Personality-related features" are data features that indicate a user's interests, preferences, behavioral patterns, emotional state, etc.

[0233] "Video content" is information content in digital form that includes video and audio data.

[0234] A "scene cut" is data that represents a specific scene or frame extracted from video content.

[0235] A "representative image" is an image that represents the video content, and is often used as a thumbnail.

[0236] "Candidate generation means" refers to methods or techniques for generating multiple options.

[0237] The "optimal video summary and representative image" are the summarized video and representative image that best fit the user's interests, preferences, and emotional state.

[0238] The present invention relates to a system for providing video summaries and thumbnails based on a user's personality and emotional state. In this system, a server collects and analyzes user log data, generates video summaries and thumbnails based on the user's personality features and emotional state, and provides them to the user.

[0239] System Overview

[0240] The system analyzes user behavioral and emotional data to generate summaries and representative images of video content. User history data includes viewing history, search history, click history, and even emotional data from an emotional engine. Based on this data, the system identifies the user's interests, preferences, and emotional state.

[0241] Collection and analysis of user history and emotion data

[0242] The server collects users' historical behavioral data and emotional data from the database. This data includes the types of videos users watched, the keywords they searched for, the representative images they clicked on, and the emotional state they displayed while watching a particular video. The collected historical data and emotional data are analyzed and natural language processing (NLP) models and emotion analysis models are used to extract user personality features and emotional states.

[0243] Video content analysis

[0244] The server analyzes the video content, extracts each frame, and detects key scenes and key moments. Image recognition technology is used for the analysis to detect scene changes and specific events (e.g., interesting or moving scenes). These scenes are extracted as important cuts based on the user's personality and emotional features.

[0245] Video summarization and representative image generation

[0246] The server combines the user's personality features with the extracted key scenes from the video to generate candidate video summaries and representative images. At this stage, we use embedding similarity calculations to evaluate whether the generated candidates are of interest to the user. By also taking the user's emotional state into consideration, we generate the most visually and emotionally appealing summary and representative images.

[0247] Optimal summary and representative image presentation

[0248] The device receives the optimal video summary and representative image sent from the server, and then presents them to the user, allowing the user to instantly grasp the overview of the content being viewed.

[0249] Specific examples

[0250] If User B wants to watch a new comedy video, the system works as follows:

[0251] 1. The server collects user B's past history data and emotional data while watching, identifies that user B has a comedy-related preference, and confirms that user B smiles frequently while watching.

[0252] 2. The server analyzes frames from a new comedy video and extracts funny and touching moments.

[0253] 3. The server combines these scenes to generate a two-minute comedy summary video and a visually interesting representative image of a smiling face.

[0254] 4. The device presents these summaries and representative images to User B.

[0255] This allows User B to quickly check scenes that he or she is particularly interested in, improving the viewing experience. In addition, since emotional data is taken into consideration, satisfaction during viewing is likely to increase.

[0256] Prompt Sentence Examples

[0257] Example prompt for a generative AI model:

[0258] "Please explain the process of a system that extracts personality features and emotional features based on user log data and emotional data during viewing, automatically selects important scenes from video content, and generates the most appropriate video summary and thumbnail for the user."

[0259] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0260] Step 1:

[0261] The server collects user history data. The inputs are viewing history, search history, click history, and emotion data. The output is a set of collected history data.

[0262] Specific behavior:

[0263] The server queries the database to retrieve data on User B's past viewing content, search keywords, representative images clicked, and emotional states (e.g., facial expressions, voice) expressed while watching the video.

[0264] Step 2:

[0265] The server analyzes the collected historical data and extracts features related to the user's personality. The input is a set of historical data. The output is a set of features related to the personality.

[0266] Specific behavior:

[0267] The server uses natural language processing (NLP) and sentiment analysis models to extract features such as "comedy preference" or "frequent smiling while watching" from viewing history. For example, it uses Python's TENSORFLOW® library to run machine learning models to extract features from complex datasets.

[0268] Step 3:

[0269] The server analyzes the video content and detects key scenes and key moments. The input is the video data to be analyzed and the user's personality features. The output is a set of candidate scene cuts and representative images.

[0270] Specific behavior:

[0271] The server uses OpenCV or similar image recognition technology to analyze video frames to identify scenes with a lot of laughter and smiling faces, as well as the rate of change and audio data in each frame to detect important events.

[0272] Step 4:

[0273] The server generates a video summary and candidate representative images based on the extracted scene cuts and representative images. The input is the set of scene cuts and representative images from the analysis results and the user's personality features. The output is a video summary and a set of candidate representative images.

[0274] Specific behavior:

[0275] The server creates a two-minute video summary focusing on interesting scenes, and uses embedding similarity calculations to evaluate whether it matches the user's preferences and select the most appealing summary and representative image.

[0276] Step 5:

[0277] The terminal receives the optimal video summary and representative image from the server and presents them to the user. The input is the video summary and representative image sent from the server. The output is the visual content to be presented to the user.

[0278] Specific behavior:

[0279] The device displays the summary video and representative image on the application's home screen or notification area. When User B opens the app, the summary video and representative image that is most likely to interest them are presented. When the user clicks on it, a two-minute summary video plays and the related representative image is also displayed.

[0280] (Application example 2)

[0281] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0282] In recent years, with the increasing volume of video content, it has become increasingly difficult for viewers to quickly find content that best suits their interests and emotional state. Furthermore, conventional systems cannot generate video summaries and thumbnails that fully take into account the user's personality and emotional state, resulting in a poor user experience.

[0283] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting user log data, means for analyzing the collected log data and extracting features related to the user's personality, means for extracting scene cuts and thumbnails from video content based on the features, means for collecting and analyzing user emotional data and identifying the user's emotional state, means for evaluating optimal video summary and thumbnail candidates based on the emotional state, and means for selecting optimal video summaries and thumbnails from the candidates and presenting them to the user. This makes it possible to provide video summaries and thumbnails that are tailored to the user's personality and emotional state.

[0284] "User log data" refers to data about a user's past behavior, such as their viewing history, search history, and click history.

[0285] "Features related to individuality" are characteristic information that indicates the user's interests and preferences.

[0286] A "scene cut" is an extracted section of an important scene or point in video content.

[0287] A "thumbnail" is a small image that visually represents a portion of the video content.

[0288] A "video summary" is content that extracts the main points and important scenes from video content so that they can be viewed in a short amount of time.

[0289] "Emotional data" refers to data that indicates the emotional state of a user analyzed from their facial expressions and behavior while watching a video.

[0290] The "emotional state" indicates the emotional state that the user feels while watching a video, and includes, for example, emotions such as joy, anger, sadness, and pleasure.

[0291] An "Embedding model" is a machine learning model that converts natural language and image data into vector format and calculates their meaning and characteristics.

[0292] An "evaluation method" is a method for determining how well a candidate video summary or thumbnail suits a user's interests and emotional state.

[0293] A "presenting means" is a means including a user interface function for displaying the selected optimal video summary or thumbnail to a user.

[0294] The present invention provides a system for providing optimal video summaries and thumbnails based on the user's personality and emotional state. The system comprises a plurality of processing means, and a server and a terminal operate in unison.

[0295] System Configuration

[0296] The server comprises the following processing means:

[0297] 1. Means of collecting user log data

[0298] 2. A method for analyzing log data and extracting features related to user personality

[0299] 3. A method for extracting scene cuts and thumbnails from video content based on features

[0300] 4. A means of collecting and analyzing user emotional data to identify the user's emotional state

[0301] 5. A means of evaluating candidate video summaries and thumbnails based on emotional state

[0302] 6. A means of selecting and presenting the best video summaries and thumbnails to users

[0303] Specific processing flow

[0304] 1. The data collection server collects behavioral data such as user viewing history, search history, click history, etc. While viewing, a camera is used to capture the user's facial expressions and collect them as emotional data.

[0305] 2. Using an emotion analysis engine (e.g., OpenFace or Facial Emotion Recognition Software), analyze the collected facial expression data and identify the user's emotional state (joy, anger, sadness, or happiness).

[0306] 3. The server uses a natural language processing (NLP) model (e.g., BERT, GPT-3 (registered trademark)) based on the collected log data to extract features that indicate the user's interests and preferences.

[0307] 4. Use image recognition techniques (e.g., YOLO, ResNet) to detect key scenes and key points (e.g., funny or moving scenes) from video content.

[0308] 5. Using the embedding model, we evaluate the generated video summaries and thumbnail candidates based on the user's personality features and emotional state. The evaluation is performed using the embedding similarity calculation.

[0309] 6. The terminal presents the user with the most appropriate video summaries and thumbnails sent from the server, allowing the user to quickly view content that matches their interests and emotional state.

[0310] Specific hardware and software used

[0311] Data collection servers: servers and cloud storage services

[0312] Emotion analysis engine: OpenFace, Facial Emotion Recognition Software

[0313] Image recognition technology: YOLO, ResNet

[0314] Natural Language Processing models: BERT, GPT-3

[0315] Embedding model: (used in conjunction with BERT and GPT-3)

[0316] Specific examples

[0317] For example, if User B wants to watch a new comedy video, the process goes like this:

[0318] 1. The data collection server collects user B's past viewing history and facial expression data while viewing.

[0319] 2. The emotion analysis engine analyzes the facial expression data and identifies User B's emotions while watching.

[0320] 3. The server analyzes past log data and determines that User B has a strong preference for comedy and smiles a lot while watching.

[0321] 4. Use image recognition technology to analyze new comedy videos and extract funny and touching scenes.

[0322] 5. The embedding model evaluates these scenes and generates the best summary video and thumbnail for user B.

[0323] 6. The device presents these summaries and thumbnails to User B.

[0324] Prompt Sentence Examples

[0325] Here are some example prompts to use as input to a generative AI model:

[0326] Based on the user's viewing history and sentiment data, generate a summary video and thumbnail that meets the following criteria:

[0327] 1. Video Summary: Comedy genre, 2 minutes, mainly funny scenes.

[0328] 2. Thumbnail: A smiling scene while the user is watching.

[0329] This allows the AI ​​model to generate optimal content that matches the user's preferences and emotions based on the specified conditions.

[0330] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0331] Step 1:

[0332] The server collects the user's viewing history, search history, click history, and emotional data while viewing. This data is stored in a database and used as the basis for processing. Specifically, it records the user's behavioral data as a log and uses a camera to capture their facial expressions while viewing. The input is the user's past behavioral data and facial expression data while viewing. The output is the log data and emotional data stored in the database.

[0333] Step 2:

[0334] The server analyzes the collected log data and emotion data. In this step, it uses a natural language processing (NLP) model (e.g., BERT, GPT-3) to extract features related to the user's personality (interests and preferences) from the log data. It also uses an emotion analysis engine (e.g., OpenFace) to extract the user's emotional state from facial expression data. The input is the log data and emotion data stored in the database. The output is the features of the personality and emotional state.

[0335] Step 3:

[0336] The server extracts scene cuts from the video content frame by frame and detects events of particular interest (e.g., funny or emotional scenes). In this step, image recognition techniques (e.g., YOLO, ResNet) are used to analyze the video frames. The input is the video content. The output is scene cuts related to interests and emotions.

[0337] Step 4:

[0338] The server combines the extracted scene cuts with the user's personality features and emotional state to generate candidate video summaries and thumbnails. In this step, an embedding model (e.g., BERT, GPT-3) is used to perform similarity calculations to select the video summaries and thumbnails that best fit the user's interests and emotional state. The input is scene cuts and user features. The output is candidate video summaries and thumbnails.

[0339] Step 5:

[0340] The server evaluates the generated video summaries and thumbnail candidates and selects the most suitable candidate. This evaluation is performed by scoring them based on the user's interests and emotional state. The input is the video summaries and thumbnail candidates. The output is the optimal video summary and thumbnail.

[0341] Step 6:

[0342] The terminal presents the optimal video summary and thumbnail sent from the server to the user, allowing the user to visually grasp and quickly select content that matches their interests and emotional state. The input is the optimal video summary and thumbnail, and the output is the content displayed in the user interface.

[0343] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0344] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0345] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0346] [Second embodiment]

[0347] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0348] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0349] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0350] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0351] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0352] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0353] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0354] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0355] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0356] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0357] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0358] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0359] This invention is a system that provides video summaries and thumbnails that take into account the user's individuality. In this system, the server collects and analyzes user log data, and generates and provides video summaries and thumbnails that are tailored to the user's individuality. The specific program processing flow and its explanation are described below.

[0360] System Overview

[0361] This system analyzes user behavior data to generate summaries and thumbnails of video content. User log data includes viewing history, search history, click history, etc. This log data is used to identify user interests and preferences.

[0362] What the program does

[0363] Collecting and analyzing user log data

[0364] The server collects log data from the database, including the user's past behavior data, such as the types of videos the user watched, the keywords they searched for, and the thumbnails they clicked on.The collected log data is analyzed and natural language processing (NLP) and machine learning models are used to extract features related to the user's personality.

[0365] Example: Extract the features "sports" and "entertainment" from user A's past viewing history.

[0366] Video content analysis

[0367] The server analyzes the video content frame by frame to detect key scenes and key points, using image recognition technology and scene detection algorithms, and extracts specific cuts and frames from the video based on the user's personality features.

[0368] Example: Extracting goal scenes and player highlights from videos of sports matches.

[0369] Video summary and thumbnail generation

[0370] The server combines the user's personality features with key scenes from the video to generate video summary and thumbnail candidates. The candidates are then evaluated using embedding similarity calculations and the most suitable one is selected.

[0371] Example: For User A, automatically generate a two-minute summary of a sports match highlights with thumbnails capturing the exciting moments.

[0372] Optimal summary and thumbnail presentation

[0373] The device will present users with optimally selected video summaries and thumbnails, allowing them to quickly get an overview of the content they are watching.

[0374] Specific examples

[0375] When User A wants to watch a new sports game video, the system works as follows:

[0376] 1. The server collects user A's past log data and identifies that he has sports-related preferences.

[0377] 2. The server analyzes frames from new match videos and extracts key moments such as goals and player highlights.

[0378] 3. The server combines these scenes to generate a two-minute highlight summary and visually interesting thumbnails.

[0379] 4. The device presents these summaries and thumbnails to User A.

[0380] This allows User A to quickly check scenes that interest him / her, improving his / her viewing experience and promoting the discovery of new content.

[0381] The above is a specific embodiment for carrying out the present invention.

[0382] The processing flow will be explained below.

[0383] Step 1:

[0384] The server collects the user's past behavior data, including viewing history, search history, click history, etc. It accesses the database to extract the target user's log data and stores the collected data in memory or cache for later analysis.

[0385] Step 2:

[0386] The server analyzes the collected log data and identifies user interests and preferences using natural language processing (NLP) and machine learning models. Specifically, it analyzes keywords and viewing patterns in the log data and extracts features such as "sports preferences" and "entertainment preferences."

[0387] Step 3:

[0388] The server analyzes the video content, extracts each frame, and detects key scenes and key moments. This analysis uses image recognition technology to detect scene changes and specific events (e.g., goal scenes, scenes featuring key characters). These scenes are then extracted as important cuts based on the user's personality features.

[0389] Step 4:

[0390] The server combines the extracted key scenes and points with the user's personality features to generate video summary and thumbnail candidates. At this stage, we use embedding similarity calculation to evaluate whether the generated candidates are interesting to the user, and then select the most relevant summary and thumbnail.

[0391] Step 5:

[0392] The device receives the optimal video summary and thumbnail sent from the server and then presents them to the user, for example, displaying a two-minute summary video of a sports highlight along with a visually appealing thumbnail image.

[0393] Step 6:

[0394] Users can view the video summary and thumbnails provided and watch the video if they are interested, a process that allows users to efficiently discover content that interests them.

[0395] This allows users to quickly view scenes that are of particular interest to them, improving their viewing experience and making it easier to discover new content.

[0396] Example 1

[0397] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0398] The wide variety of modern video content and the large amount of data available make it difficult for users to quickly find content that matches their interests and preferences. In particular, the selection of video summaries and thumbnails does not provide appropriate content that takes into account the individual needs of users, creating a demand for an improved viewing experience. The present invention aims to solve this problem by providing a system for generating video summaries and thumbnails that take into account the individual needs of users.

[0399] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0400] In this invention, the server includes: means for collecting user log data; means for analyzing the collected log data using a natural language processing model and a machine learning model to extract features related to the user's personality; means for analyzing video content frame by frame and extracting key scenes and thumbnails based on the features; means for combining the features with the extracted key scenes and thumbnails to generate video summaries and thumbnail candidates tailored to the user's personality; and means for evaluating the generated candidates using embedding similarity calculation, selecting optimal video summaries and thumbnails, and presenting them to the user. This enables the generation and presentation of video summaries and thumbnails that take the user's personality and preferences into consideration.

[0401] "User log data" refers to data including a user's past behavioral data, viewing history, search history, and click history.

[0402] A "natural language processing model" is a computational algorithm for analyzing and understanding human language, and is used to extract topics and categories from users' search keywords and viewing history.

[0403] A "machine learning model" is an algorithm that analyzes large amounts of data to learn patterns and rules and make predictions and classifications.

[0404] A "feature" is an element of extracted data that indicates a user's interests and preferences.

[0405] "Video content" refers to all video data viewed by users, including specific videos, movies, and programs.

[0406] A "key scene" is a significant event or point in the video content.

[0407] A "thumbnail" is a small image that represents the video content and serves to visually attract the user's attention.

[0408] "Embedding similarity calculation" is a method of calculating similarity by converting feature values ​​and scene information into vector space.

[0409] A "video summary" is a video created by combining key scenes from video content into a shortened version.

[0410] "Candidates" refer to the generated video summaries and thumbnail options.

[0411] The present invention provides a system for providing video summaries and thumbnails that take into account the user's individuality. In this system, a server collects and analyzes user log data, generates video summaries and thumbnails that are tailored to the user's individuality, and provides them to the user. Specific embodiments of the present invention are described below.

[0412] The server collects log data from a database, including users' past behavioral data. This data includes the types of videos the user watched, the keywords they searched for, and the thumbnails they clicked. The collected log data is analyzed using natural language processing (NLP) and machine learning models. As a result of the analysis, features related to the user's personality are extracted. For example, features such as "sports" and "entertainment" are extracted from User A's past viewing history.

[0413] The server then analyzes the video content frame by frame, using image recognition technology and scene detection algorithms. As a result of the analysis, key scenes and points in the video are detected, and specific cuts and frames are extracted based on the user's personality features. For example, goal scenes and player highlights can be extracted from a video of a sports game.

[0414] The server then generates video summary and thumbnail candidates by combining the user's personality features and key scenes from the video. The candidates are evaluated using embedding similarity calculations and the most suitable one is selected. For example, for user A, a two-minute summary of a sports game highlights and thumbnails capturing the most exciting moments are generated.

[0415] Finally, the device presents the optimally selected video summary and thumbnail to the user. By checking the summary and thumbnail, the user can quickly get an overview of the content they are watching. As a concrete example, when user A watches a new sports game video, the system operates as follows:

[0416] 1. The server collects user A's past log data and identifies that he has sports-related preferences.

[0417] 2. The server analyzes new match videos frame by frame and extracts key moments such as goals and player highlights.

[0418] 3. The server combines these scenes to generate a two-minute highlight summary and visually interesting thumbnails.

[0419] 4. The device presents these summaries and thumbnails to User A.

[0420] An example prompt is, "For a sports fan, please create a two-minute summary of sports game highlights based on their past viewing history and create an engaging thumbnail." Using this prompt, the generative AI model can generate optimal content based on the user's personality.

[0421] This invention enables the generation and presentation of video summaries and thumbnails that take into account the user's individuality and preferences, which is expected to improve the viewing experience and promote the discovery of new content.

[0422] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0423] Step 1: Collect user log data

[0424] The server connects to the database to collect user log data. This log data includes the user's viewing history, search history, click history, etc. Specifically, it issues SQL queries to the database to extract the target user's behavioral data. The input is the user behavior data in the database, and the output is the extracted log data.

[0425] Specific behavior:

[0426] The server establishes a database connection and executes a query based on the user's ID.

[0427] The data acquired by the server is temporarily stored in memory.

[0428] Step 2: Analyze the log data

[0429] The server analyzes the collected log data using natural language processing (NLP) and machine learning models. This analysis extracts features related to the user's personality. The input is the collected log data, and the output is the extracted features.

[0430] Specific behavior:

[0431] The server applies NLP models to extract topics and categories from search keywords and viewing history.

[0432] The server uses machine learning models to analyze user behavior patterns and identify categories of interest.

[0433] Step 3: Frame analysis of video content

[0434] The server analyzes the video content the user wants to watch, frame by frame, and uses image recognition technology and scene detection algorithms to detect key scenes and key points. The input is the video file, and the output is key scene information.

[0435] Specific behavior:

[0436] The server loads the video file and splits it into frames.

[0437] The server uses image recognition technology to analyze the frames and evaluate the importance of the scenes.

[0438] The extracted main scene information is linked to the features.

[0439] Step 4: Generate video summary and thumbnails

[0440] The server combines the user's personality features with the main scenes of the video to generate video summaries and thumbnail candidates. The input is the features and main scene information, and the output is video summaries and thumbnail candidates.

[0441] Specific behavior:

[0442] The server sequentially connects the identified key scenes to generate a short summary.

[0443] The server selects visually interesting frames and creates thumbnail candidates.

[0444] Step 5: Evaluate and select summaries and thumbnails

[0445] The server evaluates the generated summaries and thumbnail candidates and selects the best one using embedding similarity calculation. The input is the video summary and thumbnail candidates, and the output is the selected summary and thumbnail.

[0446] Specific behavior:

[0447] The server uses embedding technology to calculate the similarity of each candidate.

[0448] The most similar summaries and thumbnails are selected and prepared for delivery to the user.

[0449] Step 6: Present to the user

[0450] The terminal presents the optimally selected video summary and thumbnail to the user. The input is the selected video summary and thumbnail, and the output is the content displayed in the user interface.

[0451] Specific behavior:

[0452] The server sends the selected summary and thumbnail to the terminal.

[0453] The data received by the terminal is displayed in the user interface.

[0454] The user checks the presented summary and thumbnails to quickly get an overview of the content to be viewed.

[0455] (Application example 1)

[0456] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0457] Conventional content distribution services lack the technology to automatically generate video summaries and thumbnails that match users' personalities and preferences. This makes it difficult for users to quickly identify scenes that interest or are important to them, resulting in reduced viewing efficiency. Furthermore, the lack of appropriate thumbnails makes it difficult for users to discover new content.

[0458] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0459] In this invention, the server includes means for collecting user log data, means for analyzing the collected log data and extracting features related to the user's personality, means for extracting scene cuts and thumbnails from video content based on the features, means for generating candidate video summaries and thumbnails tailored to the user's personality by combining the features with the extracted scene cuts and thumbnails, means for selecting optimal video summaries and thumbnails from the candidates and presenting them to the user, means for collecting real-time data, means for using a framework for presenting content to a personal mobile device, and means for generating video summaries and thumbnails using a generative AI model. This allows for the rapid provision of video summaries and thumbnails tailored to the user's personality, improving viewing efficiency and promoting the discovery of new content.

[0460] "User log data" refers to various behavioral data generated when a user uses the system, such as viewing history, search history, and click history.

[0461] "Features" are characteristic data extracted from log data to indicate a user's personality and preferences.

[0462] A "scene cut" is a video fragment cut out from a specific portion of video content.

[0463] A "thumbnail" is a still image that represents a portion of video content and is used to provide a visual overview.

[0464] A "video summary" is a video that extracts the main scenes of a video content and compresses them so that they can be viewed in a short time.

[0465] "Candidates" is a collection of multiple generated video summaries and thumbnails before selecting the best video summaries and thumbnails.

[0466] "Best suited video summaries and thumbnails" are those video summaries and thumbnails that are determined to be most suitable to the user's personality and interests and are presented to the user.

[0467] "Real-time data" is data that is collected immediately and generated on the spot when users use the system.

[0468] A "personal mobile device" is an information processing device that is individually owned and portable by a user, such as a smartphone or tablet.

[0469] A "framework" is a basic structure or framework for efficient software development, and is particularly used to present content.

[0470] A "generative AI model" is an algorithm or program that uses artificial intelligence technology to generate and process information.

[0471] The present invention provides a system for providing personalized video summaries and thumbnails to users, and the specific implementation method thereof will be described below. The system is configured using a server and a user's personal mobile terminal.

[0472] System Overview

[0473] The server is configured using the following hardware and software.

[0474] Hardware: Server computers, database servers

[0475] Software: Apache Kafka, MySQL, BERT model, machine learning models (random forest, SVM), OpenCV, YOLO, React Native

[0476] The server first collects user log data, including viewing history, search history, click history, etc. Apache Kafka is used to collect data in real time, and the collected data is stored in a MySQL database.

[0477] Next, the server analyzes the user's log data using a BERT model or machine learning model to extract personality features. For example, it identifies areas of interest such as "sports" or "entertainment" from the user's viewing history.

[0478] The server then analyzes the video content frame by frame, using image recognition techniques such as OpenCV and YOLO to detect key scenes and key moments, such as goal scenes and player highlights in sports videos.

[0479] The server then uses a generative AI model to combine the user's personality features with the detected scenes to generate a video summary and thumbnails, which are then evaluated using a similarity calculation to select the most suitable one.

[0480] The selected video summaries and thumbnails are then presented to the user's personal mobile device through an app developed using React Native, allowing the user to quickly get an overview of the content they are about to watch.

[0481] Specific examples

[0482] When user A wants to watch a new sports game video, the following happens:

[0483] 1. The server collects user A's past log data and identifies that he has sports-related preferences.

[0484] 2. The server analyzes frames from new match videos and extracts key moments such as goals and player highlights.

[0485] 3. The server combines these scenes to generate a two-minute highlight summary and visually interesting thumbnails.

[0486] 4. The personal mobile device presents these summaries and thumbnails to User A.

[0487] Prompt Sentence Examples

[0488] Examples of prompts for generative AI models include:

[0489] Based on the user's viewing history and interests, extract key scenes from the following video and generate a 2-minute summary and thumbnail.

[0490] Viewing history features: Sports, Entertainment

[0491] Video: "Football match video"

[0492] Scenes to be extracted: Goal scenes, player highlights

[0493] Generated summary length: 2 minutes

[0494] result:

[0495] 1. 2-minute video summary

[0496] 2. Thumbnails capturing goal scenes

[0497] In this way, personalized video summaries and thumbnails can be efficiently generated to provide an optimal viewing experience.

[0498] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0499] Step 1:

[0500] The server collects user log data. Specifically, data such as user viewing history, search history, and click history is collected in real time using Apache Kafka and stored in a MySQL database. This collected data serves as basic information for detailed analysis of user behavior patterns. The input is user behavior data, and the output is log data stored in the MySQL database.

[0501] Step 2:

[0502] The server analyzes the collected log data and extracts features related to the user's personality. Specifically, it uses the BERT model and machine learning models (random forest and SVM) to analyze the user's log data and extract features such as "sports" and "entertainment" from their viewing history and search history. The input for this process is the log data in the MySQL database, and the output is analyzed data containing personality features.

[0503] Step 3:

[0504] The server analyzes the video content frame by frame. Specifically, it uses image recognition technologies such as OpenCV and YOLO to analyze each frame of the video and detect key scenes and key points. This process identifies, for example, goal scenes and player highlights. The input is the original video content, and the output is scene data containing key scenes.

[0505] Step 4:

[0506] The server combines the user's personality features with the extracted scenes to generate video summaries and thumbnail candidates. Specifically, a generative AI model is used to automatically generate video summaries and thumbnails based on the personality features and scene data. In this process, multiple candidates are generated, and the best one is selected. The input is the personality features and scene data, and the output is video summaries and thumbnail candidates.

[0507] Step 5:

[0508] The server selects the best video summary and thumbnail from the generated candidates and presents them to the user. Specifically, it performs a similarity calculation to select the most suitable summary and thumbnail. The selected video summary and thumbnail are then displayed on the user's personal mobile device through an app developed using React Native. The input of this process is the video summary and thumbnail candidates, and the output is the best summary and thumbnail presented to the user.

[0509] Step 6:

[0510] The user checks the presented video summary and thumbnails. Specifically, the user watches the summary and thumbnails using a React Native-based application on a personal mobile device such as a smartphone or tablet. This step is the user interaction phase, where the input is the optimal summary and thumbnail presented to the user, and the output is the user's viewing behavior.

[0511] Through these steps, users can quickly view personalized video summaries and thumbnails, improving their viewing experience.

[0512] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0513] This invention is a system for providing video summaries and thumbnails based on a user's personality and emotional state. In this system, a server collects and analyzes user log data, generates video summaries and thumbnails based on the user's personality features and emotional state, and provides them to the user. The specific program processing flow and its explanation are described below.

[0514] System Overview

[0515] The system analyzes user behavioral and emotional data to identify their personality and emotional state, and then generates summaries and thumbnails of video content. User log data includes viewing history, search history, click history, and emotional data from an emotion engine. Based on this data, the system identifies the user's interests, preferences, and emotional state.

[0516] What the program does

[0517] Collecting and analyzing user log data and emotional data

[0518] The server collects log data from the database, including users' past behavioral data and emotional data. This data includes the types of videos users watched, keywords they searched for, thumbnails they clicked, and the emotional state users displayed while watching a particular video. The collected log data and emotional data are analyzed and natural language processing (NLP) models and emotion analysis models are used to extract features related to the user's personality and emotional state.

[0519] Example: Extract emotional features such as "comedy preference" and "frequent smiling while watching" from user B's past viewing history and facial expressions while watching.

[0520] Video content analysis

[0521] The server analyzes the video content, extracts each frame, and detects key scenes and key moments. Image recognition technology is used for the analysis to detect scene changes and specific events (e.g., interesting or moving scenes). These scenes are extracted as important cuts based on the user's personality and emotional features.

[0522] Example: Extracting funny and touching scenes from a comedy show video.

[0523] Video summary and thumbnail generation

[0524] The server combines the user's personality features with the extracted key scenes from the video to generate candidate video summaries and thumbnails. At this stage, we use embedding similarity calculations to evaluate whether the generated candidates are interesting to the user. By taking the user's emotional state into account, we generate the most visually and emotionally appealing summaries and thumbnails.

[0525] Example: For user B, generate a two-minute video summary highlighting funny moments from a comedy, along with thumbnails capturing smiling moments.

[0526] Optimal summary and thumbnail presentation

[0527] The device receives the optimal video summary and thumbnail sent from the server and presents them to the user, allowing the user to instantly get an overview of the content they are watching.

[0528] Specific examples

[0529] If User B wants to watch a new comedy video, the system works as follows:

[0530] 1. The server collects user B's past log data and emotional data while watching, identifies that user B has a comedy-related preference, and confirms that user B smiles frequently while watching.

[0531] 2. The server analyzes frames from a new comedy video and extracts funny and touching moments.

[0532] 3. The server combines these scenes to generate a two-minute comedy summary video and visually interesting smiley thumbnails.

[0533] 4. The device presents these summaries and thumbnails to User B.

[0534] This allows User B to quickly check scenes that interest them, improving their viewing experience and making it easier for them to discover new content. Emotional data is also taken into account, which tends to increase satisfaction while viewing.

[0535] The processing flow will be explained below.

[0536] Step 1:

[0537] The server collects users' past behavioral data and emotional data. Specifically, it accesses a database to obtain log data including viewing history, search history, click history, and emotional data during viewing. This data is stored in memory or cache as it is required for later analysis steps.

[0538] Step 2:

[0539] The server analyzes the collected log data and emotion data. First, it uses natural language processing (NLP) models and machine learning models to extract features that indicate the user's interests and preferences from the log data. Next, it uses an emotion analysis model to analyze the user's emotional fluctuations and identify the emotional state (e.g., joy, surprise, excitement) shown while watching.

[0540] Step 3:

[0541] The server analyzes the video content frame by frame, and uses image recognition technology and scene detection algorithms to detect key scenes and events (e.g., humorous or moving scenes) in each frame. From the detected scenes, important cuts and thumbnail images are extracted based on the user's personality and emotional features.

[0542] Step 4:

[0543] The server generates candidate video summaries and thumbnails based on the extracted scene cuts and thumbnails. Using embedding similarity calculations, the server evaluates and selects candidates based on their match with the user's interests and emotions. For example, it prioritizes video summaries that contain scenes that the user expressed high enjoyment in while watching.

[0544] Step 5:

[0545] The device receives the optimal video summary and thumbnail sent from the server, passes the received data to the display component, and presents it visually to the user, allowing the user to get a quick overview of the content they are watching.

[0546] Step 6:

[0547] Users can view the video summary and thumbnails provided and choose to watch the video if they are interested. Emotional data is also collected during viewing and used for future analysis.

[0548] This improves the viewing experience by allowing users to quickly check scenes they are particularly interested in. Emotional data is also taken into account, resulting in higher viewing satisfaction and content tailored to the user's preferences.

[0549] Example 2

[0550] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0551] Currently, existing systems that provide optimized video summaries and thumbnails based on a user's personality and emotional state are inadequate. In particular, technologies for selecting and presenting content that reflects a user's preferences and emotions are immature, and user satisfaction cannot be fully enhanced. Therefore, there is a need for a system that can efficiently generate and present video summaries and thumbnails based on a user's personality and emotional state.

[0552] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0553] In this invention, the server includes means for collecting user history data, means for analyzing the collected history data and extracting features related to the user's personality, means for extracting scene cuts and representative images from video content based on the features, means for combining the features with the extracted scene cuts and representative images to generate candidates for video summaries and representative images tailored to the user's personality, and means for selecting the most suitable video summaries and representative images from the candidates and presenting them to the user, thereby making it possible to provide content summaries and thumbnails that accurately reflect the user's interests and emotional state.

[0554] "History data" refers to data about a user's past activities, including viewing history, search history, click history, etc.

[0555] "Personality-related features" are data features that indicate a user's interests, preferences, behavioral patterns, emotional state, etc.

[0556] "Video content" is information content in digital form that includes video and audio data.

[0557] A "scene cut" is data that represents a specific scene or frame extracted from video content.

[0558] A "representative image" is an image that represents the video content, and is often used as a thumbnail.

[0559] "Candidate generation means" refers to methods or techniques for generating multiple options.

[0560] The "optimal video summary and representative image" are the summarized video and representative image that best fit the user's interests, preferences, and emotional state.

[0561] The present invention relates to a system for providing video summaries and thumbnails based on a user's personality and emotional state. In this system, a server collects and analyzes user log data, generates video summaries and thumbnails based on the user's personality features and emotional state, and provides them to the user.

[0562] System Overview

[0563] The system analyzes user behavioral and emotional data to generate summaries and representative images of video content. User history data includes viewing history, search history, click history, and even emotional data from an emotional engine. Based on this data, the system identifies the user's interests, preferences, and emotional state.

[0564] Collection and analysis of user history and emotion data

[0565] The server collects users' historical behavioral data and emotional data from the database. This data includes the types of videos users watched, the keywords they searched for, the representative images they clicked on, and the emotional state they displayed while watching a particular video. The collected historical data and emotional data are analyzed and natural language processing (NLP) models and emotion analysis models are used to extract user personality features and emotional states.

[0566] Video content analysis

[0567] The server analyzes the video content, extracts each frame, and detects key scenes and key moments. Image recognition technology is used for the analysis to detect scene changes and specific events (e.g., interesting or moving scenes). These scenes are extracted as important cuts based on the user's personality and emotional features.

[0568] Video summarization and representative image generation

[0569] The server combines the user's personality features with the extracted key scenes from the video to generate candidate video summaries and representative images. At this stage, we use embedding similarity calculations to evaluate whether the generated candidates are of interest to the user. By also taking the user's emotional state into consideration, we generate the most visually and emotionally appealing summary and representative images.

[0570] Optimal summary and representative image presentation

[0571] The device receives the optimal video summary and representative image sent from the server, and then presents them to the user, allowing the user to instantly grasp the overview of the content being viewed.

[0572] Specific examples

[0573] If User B wants to watch a new comedy video, the system works as follows:

[0574] 1. The server collects user B's past history data and emotional data while watching, identifies that user B has a comedy-related preference, and confirms that user B smiles frequently while watching.

[0575] 2. The server analyzes frames from a new comedy video and extracts funny and touching moments.

[0576] 3. The server combines these scenes to generate a two-minute comedy summary video and a visually interesting representative image of a smiling face.

[0577] 4. The device presents these summaries and representative images to User B.

[0578] This allows User B to quickly check scenes that he or she is particularly interested in, improving the viewing experience. In addition, since emotional data is taken into consideration, satisfaction during viewing is likely to increase.

[0579] Prompt Sentence Examples

[0580] Example prompt for a generative AI model:

[0581] "Please explain the process of a system that extracts personality features and emotional features based on user log data and emotional data during viewing, automatically selects important scenes from video content, and generates the most appropriate video summary and thumbnail for the user."

[0582] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0583] Step 1:

[0584] The server collects user history data. The inputs are viewing history, search history, click history, and emotion data. The output is a set of collected history data.

[0585] Specific behavior:

[0586] The server queries the database to retrieve data on User B's past viewing content, search keywords, representative images clicked, and emotional states (e.g., facial expressions, voice) expressed while watching the video.

[0587] Step 2:

[0588] The server analyzes the collected historical data and extracts features related to the user's personality. The input is a set of historical data. The output is a set of features related to the personality.

[0589] Specific behavior:

[0590] The server uses natural language processing (NLP) and sentiment analysis models to extract features from viewing history, such as "comedy preference" or "frequent smiling while watching." For example, it uses Python's TensorFlow library to run machine learning models to extract features from complex datasets.

[0591] Step 3:

[0592] The server analyzes the video content and detects key scenes and key moments. The input is the video data to be analyzed and the user's personality features. The output is a set of candidate scene cuts and representative images.

[0593] Specific behavior:

[0594] The server uses OpenCV or similar image recognition technology to analyze video frames to identify scenes with a lot of laughter and smiling faces, as well as the rate of change and audio data in each frame to detect important events.

[0595] Step 4:

[0596] The server generates a video summary and candidate representative images based on the extracted scene cuts and representative images. The input is the set of scene cuts and representative images from the analysis results and the user's personality features. The output is a video summary and a set of candidate representative images.

[0597] Specific behavior:

[0598] The server creates a two-minute video summary focusing on interesting scenes, and uses embedding similarity calculations to evaluate whether it matches the user's preferences and select the most appealing summary and representative image.

[0599] Step 5:

[0600] The terminal receives the optimal video summary and representative image from the server and presents them to the user. The input is the video summary and representative image sent from the server. The output is the visual content to be presented to the user.

[0601] Specific behavior:

[0602] The device displays the summary video and representative image on the application's home screen or notification area. When User B opens the app, the summary video and representative image that is most likely to interest them are presented. When the user clicks on it, a two-minute summary video plays and the related representative image is also displayed.

[0603] (Application example 2)

[0604] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0605] In recent years, with the increasing volume of video content, it has become increasingly difficult for viewers to quickly find content that best suits their interests and emotional state. Furthermore, conventional systems cannot generate video summaries and thumbnails that fully take into account the user's personality and emotional state, resulting in a poor user experience.

[0606] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting user log data, means for analyzing the collected log data and extracting features related to the user's personality, means for extracting scene cuts and thumbnails from video content based on the features, means for collecting and analyzing user emotional data and identifying the user's emotional state, means for evaluating optimal video summary and thumbnail candidates based on the emotional state, and means for selecting optimal video summaries and thumbnails from the candidates and presenting them to the user. This makes it possible to provide video summaries and thumbnails that are tailored to the user's personality and emotional state.

[0607] "User log data" refers to data about a user's past behavior, such as their viewing history, search history, and click history.

[0608] "Features related to individuality" are characteristic information that indicates the user's interests and preferences.

[0609] A "scene cut" is an extracted section of an important scene or point in video content.

[0610] A "thumbnail" is a small image that visually represents a portion of the video content.

[0611] A "video summary" is content that extracts the main points and important scenes from video content so that they can be viewed in a short amount of time.

[0612] "Emotional data" refers to data that indicates the emotional state of a user analyzed from their facial expressions and behavior while watching a video.

[0613] The "emotional state" indicates the emotional state that the user feels while watching a video, and includes, for example, emotions such as joy, anger, sadness, and pleasure.

[0614] An "Embedding model" is a machine learning model that converts natural language and image data into vector format and calculates their meaning and characteristics.

[0615] An "evaluation method" is a method for determining how well a candidate video summary or thumbnail suits a user's interests and emotional state.

[0616] A "presenting means" is a means including a user interface function for displaying the selected optimal video summary or thumbnail to a user.

[0617] The present invention provides a system for providing optimal video summaries and thumbnails based on the user's personality and emotional state. The system comprises a plurality of processing means, and a server and a terminal operate in unison.

[0618] System Configuration

[0619] The server comprises the following processing means:

[0620] 1. Means of collecting user log data

[0621] 2. A method for analyzing log data and extracting features related to user personality

[0622] 3. A method for extracting scene cuts and thumbnails from video content based on features

[0623] 4. A means of collecting and analyzing user emotional data to identify the user's emotional state

[0624] 5. A means of evaluating candidate video summaries and thumbnails based on emotional state

[0625] 6. A means of selecting and presenting the best video summaries and thumbnails to users

[0626] Specific processing flow

[0627] 1. The data collection server collects behavioral data such as user viewing history, search history, click history, etc. While viewing, a camera is used to capture the user's facial expressions and collect them as emotional data.

[0628] 2. Using an emotion analysis engine (e.g., OpenFace or Facial Emotion Recognition Software), analyze the collected facial expression data and identify the user's emotional state (joy, anger, sadness, or happiness).

[0629] 3. The server uses a natural language processing (NLP) model (e.g., BERT, GPT-3) based on the collected log data to extract features that indicate the user's interests and preferences.

[0630] 4. Use image recognition techniques (e.g., YOLO, ResNet) to detect key scenes and key points (e.g., funny or moving scenes) from video content.

[0631] 5. Using the embedding model, we evaluate the generated video summaries and thumbnail candidates based on the user's personality features and emotional state. The evaluation is performed using the embedding similarity calculation.

[0632] 6. The terminal presents the user with the most appropriate video summaries and thumbnails sent from the server, allowing the user to quickly view content that matches their interests and emotional state.

[0633] Specific hardware and software used

[0634] Data collection servers: servers and cloud storage services

[0635] Emotion analysis engine: OpenFace, Facial Emotion Recognition Software

[0636] Image recognition technology: YOLO, ResNet

[0637] Natural Language Processing models: BERT, GPT-3

[0638] Embedding model: (used in conjunction with BERT and GPT-3)

[0639] Specific examples

[0640] For example, if User B wants to watch a new comedy video, the process goes like this:

[0641] 1. The data collection server collects user B's past viewing history and facial expression data while viewing.

[0642] 2. The emotion analysis engine analyzes the facial expression data and identifies User B's emotions while watching.

[0643] 3. The server analyzes past log data and determines that User B has a strong preference for comedy and smiles a lot while watching.

[0644] 4. Use image recognition technology to analyze new comedy videos and extract funny and touching scenes.

[0645] 5. The embedding model evaluates these scenes and generates the best summary video and thumbnail for user B.

[0646] 6. The device presents these summaries and thumbnails to User B.

[0647] Prompt Sentence Examples

[0648] Here are some example prompts to use as input to a generative AI model:

[0649] Based on the user's viewing history and sentiment data, generate a summary video and thumbnail that meets the following criteria:

[0650] 1. Video Summary: Comedy genre, 2 minutes, mainly funny scenes.

[0651] 2. Thumbnail: A smiling scene while the user is watching.

[0652] This allows the AI ​​model to generate optimal content that matches the user's preferences and emotions based on the specified conditions.

[0653] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0654] Step 1:

[0655] The server collects the user's viewing history, search history, click history, and emotional data while viewing. This data is stored in a database and used as the basis for processing. Specifically, it records the user's behavioral data as a log and uses a camera to capture their facial expressions while viewing. The input is the user's past behavioral data and facial expression data while viewing. The output is the log data and emotional data stored in the database.

[0656] Step 2:

[0657] The server analyzes the collected log data and emotion data. In this step, it uses a natural language processing (NLP) model (e.g., BERT, GPT-3) to extract features related to the user's personality (interests and preferences) from the log data. It also uses an emotion analysis engine (e.g., OpenFace) to extract the user's emotional state from facial expression data. The input is the log data and emotion data stored in the database. The output is the features of the personality and emotional state.

[0658] Step 3:

[0659] The server extracts scene cuts from the video content frame by frame and detects events of particular interest (e.g., funny or emotional scenes). In this step, image recognition techniques (e.g., YOLO, ResNet) are used to analyze the video frames. The input is the video content. The output is scene cuts related to interests and emotions.

[0660] Step 4:

[0661] The server combines the extracted scene cuts with the user's personality features and emotional state to generate candidate video summaries and thumbnails. In this step, an embedding model (e.g., BERT, GPT-3) is used to perform similarity calculations to select the video summaries and thumbnails that best fit the user's interests and emotional state. The input is scene cuts and user features. The output is candidate video summaries and thumbnails.

[0662] Step 5:

[0663] The server evaluates the generated video summaries and thumbnail candidates and selects the most suitable candidate. This evaluation is performed by scoring them based on the user's interests and emotional state. The input is the video summaries and thumbnail candidates. The output is the optimal video summary and thumbnail.

[0664] Step 6:

[0665] The terminal presents the optimal video summary and thumbnail sent from the server to the user, allowing the user to visually grasp and quickly select content that matches their interests and emotional state. The input is the optimal video summary and thumbnail, and the output is the content displayed in the user interface.

[0666] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0667] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0668] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0669] [Third embodiment]

[0670] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0671] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0672] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0673] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0674] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0675] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0676] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0677] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0678] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0679] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0680] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0681] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0682] This invention is a system that provides video summaries and thumbnails that take into account the user's individuality. In this system, the server collects and analyzes user log data, and generates and provides video summaries and thumbnails that are tailored to the user's individuality. The specific program processing flow and its explanation are described below.

[0683] System Overview

[0684] This system analyzes user behavior data to generate summaries and thumbnails of video content. User log data includes viewing history, search history, click history, etc. This log data is used to identify user interests and preferences.

[0685] What the program does

[0686] Collecting and analyzing user log data

[0687] The server collects log data from the database, including the user's past behavior data, such as the types of videos the user watched, the keywords they searched for, and the thumbnails they clicked on.The collected log data is analyzed and natural language processing (NLP) and machine learning models are used to extract features related to the user's personality.

[0688] Example: Extract the features "sports" and "entertainment" from user A's past viewing history.

[0689] Video content analysis

[0690] The server analyzes the video content frame by frame to detect key scenes and key points, using image recognition technology and scene detection algorithms, and extracts specific cuts and frames from the video based on the user's personality features.

[0691] Example: Extracting goal scenes and player highlights from videos of sports matches.

[0692] Video summary and thumbnail generation

[0693] The server combines the user's personality features with key scenes from the video to generate video summary and thumbnail candidates. The candidates are then evaluated using embedding similarity calculations and the most suitable one is selected.

[0694] Example: For User A, automatically generate a two-minute summary of a sports match highlights with thumbnails capturing the exciting moments.

[0695] Optimal summary and thumbnail presentation

[0696] The device will present users with optimally selected video summaries and thumbnails, allowing them to quickly get an overview of the content they are watching.

[0697] Specific examples

[0698] When User A wants to watch a new sports game video, the system works as follows:

[0699] 1. The server collects user A's past log data and identifies that he has sports-related preferences.

[0700] 2. The server analyzes frames from new match videos and extracts key moments such as goals and player highlights.

[0701] 3. The server combines these scenes to generate a two-minute highlight summary and visually interesting thumbnails.

[0702] 4. The device presents these summaries and thumbnails to User A.

[0703] This allows User A to quickly check scenes that interest him / her, improving his / her viewing experience and promoting the discovery of new content.

[0704] The above is a specific embodiment for carrying out the present invention.

[0705] The processing flow will be explained below.

[0706] Step 1:

[0707] The server collects the user's past behavior data, including viewing history, search history, click history, etc. It accesses the database to extract the target user's log data and stores the collected data in memory or cache for later analysis.

[0708] Step 2:

[0709] The server analyzes the collected log data and identifies user interests and preferences using natural language processing (NLP) and machine learning models. Specifically, it analyzes keywords and viewing patterns in the log data and extracts features such as "sports preferences" and "entertainment preferences."

[0710] Step 3:

[0711] The server analyzes the video content, extracts each frame, and detects key scenes and key moments. This analysis uses image recognition technology to detect scene changes and specific events (e.g., goal scenes, scenes featuring key characters). These scenes are then extracted as important cuts based on the user's personality features.

[0712] Step 4:

[0713] The server combines the extracted key scenes and points with the user's personality features to generate video summary and thumbnail candidates. At this stage, we use embedding similarity calculation to evaluate whether the generated candidates are interesting to the user, and then select the most relevant summary and thumbnail.

[0714] Step 5:

[0715] The device receives the optimal video summary and thumbnail sent from the server and then presents them to the user, for example, displaying a two-minute summary video of a sports highlight along with a visually appealing thumbnail image.

[0716] Step 6:

[0717] Users can view the video summary and thumbnails provided and watch the video if they are interested, a process that allows users to efficiently discover content that interests them.

[0718] This allows users to quickly view scenes that are of particular interest to them, improving their viewing experience and making it easier to discover new content.

[0719] Example 1

[0720] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0721] The wide variety of modern video content and the large amount of data available make it difficult for users to quickly find content that matches their interests and preferences. In particular, the selection of video summaries and thumbnails does not provide appropriate content that takes into account the individual needs of users, creating a demand for an improved viewing experience. The present invention aims to solve this problem by providing a system for generating video summaries and thumbnails that take into account the individual needs of users.

[0722] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0723] In this invention, the server includes: means for collecting user log data; means for analyzing the collected log data using a natural language processing model and a machine learning model to extract features related to the user's personality; means for analyzing video content frame by frame and extracting key scenes and thumbnails based on the features; means for combining the features with the extracted key scenes and thumbnails to generate video summaries and thumbnail candidates tailored to the user's personality; and means for evaluating the generated candidates using embedding similarity calculation, selecting optimal video summaries and thumbnails, and presenting them to the user. This enables the generation and presentation of video summaries and thumbnails that take the user's personality and preferences into consideration.

[0724] "User log data" refers to data including a user's past behavioral data, viewing history, search history, and click history.

[0725] A "natural language processing model" is a computational algorithm for analyzing and understanding human language, and is used to extract topics and categories from users' search keywords and viewing history.

[0726] A "machine learning model" is an algorithm that analyzes large amounts of data to learn patterns and rules and make predictions and classifications.

[0727] A "feature" is an element of extracted data that indicates a user's interests and preferences.

[0728] "Video content" refers to all video data viewed by users, including specific videos, movies, and programs.

[0729] A "key scene" is a significant event or point in the video content.

[0730] A "thumbnail" is a small image that represents the video content and serves to visually attract the user's attention.

[0731] "Embedding similarity calculation" is a method of calculating similarity by converting feature values ​​and scene information into vector space.

[0732] A "video summary" is a video created by combining key scenes from video content into a shortened version.

[0733] "Candidates" refer to the generated video summaries and thumbnail options.

[0734] The present invention provides a system for providing video summaries and thumbnails that take into account the user's individuality. In this system, a server collects and analyzes user log data, generates video summaries and thumbnails that are tailored to the user's individuality, and provides them to the user. Specific embodiments of the present invention are described below.

[0735] The server collects log data from a database, including users' past behavioral data. This data includes the types of videos the user watched, the keywords they searched for, and the thumbnails they clicked. The collected log data is analyzed using natural language processing (NLP) and machine learning models. As a result of the analysis, features related to the user's personality are extracted. For example, features such as "sports" and "entertainment" are extracted from User A's past viewing history.

[0736] The server then analyzes the video content frame by frame, using image recognition technology and scene detection algorithms. As a result of the analysis, key scenes and points in the video are detected, and specific cuts and frames are extracted based on the user's personality features. For example, goal scenes and player highlights can be extracted from a video of a sports game.

[0737] The server then generates video summary and thumbnail candidates by combining the user's personality features and key scenes from the video. The candidates are evaluated using embedding similarity calculations and the most suitable one is selected. For example, for user A, a two-minute summary of a sports game highlights and thumbnails capturing the most exciting moments are generated.

[0738] Finally, the device presents the optimally selected video summary and thumbnail to the user. By checking the summary and thumbnail, the user can quickly get an overview of the content they are watching. As a concrete example, when user A watches a new sports game video, the system operates as follows:

[0739] 1. The server collects user A's past log data and identifies that he has sports-related preferences.

[0740] 2. The server analyzes new match videos frame by frame and extracts key moments such as goals and player highlights.

[0741] 3. The server combines these scenes to generate a two-minute highlight summary and visually interesting thumbnails.

[0742] 4. The device presents these summaries and thumbnails to User A.

[0743] An example prompt is, "For a sports fan, please create a two-minute summary of sports game highlights based on their past viewing history and create an engaging thumbnail." Using this prompt, the generative AI model can generate optimal content based on the user's personality.

[0744] This invention enables the generation and presentation of video summaries and thumbnails that take into account the user's individuality and preferences, which is expected to improve the viewing experience and promote the discovery of new content.

[0745] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0746] Step 1: Collect user log data

[0747] The server connects to the database to collect user log data. This log data includes the user's viewing history, search history, click history, etc. Specifically, it issues SQL queries to the database to extract the target user's behavioral data. The input is the user behavior data in the database, and the output is the extracted log data.

[0748] Specific behavior:

[0749] The server establishes a database connection and executes a query based on the user's ID.

[0750] The data acquired by the server is temporarily stored in memory.

[0751] Step 2: Analyze the log data

[0752] The server analyzes the collected log data using natural language processing (NLP) and machine learning models. This analysis extracts features related to the user's personality. The input is the collected log data, and the output is the extracted features.

[0753] Specific behavior:

[0754] The server applies NLP models to extract topics and categories from search keywords and viewing history.

[0755] The server uses machine learning models to analyze user behavior patterns and identify categories of interest.

[0756] Step 3: Frame analysis of video content

[0757] The server analyzes the video content the user wants to watch, frame by frame, and uses image recognition technology and scene detection algorithms to detect key scenes and key points. The input is the video file, and the output is key scene information.

[0758] Specific behavior:

[0759] The server loads the video file and splits it into frames.

[0760] The server uses image recognition technology to analyze the frames and evaluate the importance of the scenes.

[0761] The extracted main scene information is linked to the features.

[0762] Step 4: Generate video summary and thumbnails

[0763] The server combines the user's personality features with the main scenes of the video to generate video summaries and thumbnail candidates. The input is the features and main scene information, and the output is video summaries and thumbnail candidates.

[0764] Specific behavior:

[0765] The server sequentially connects the identified key scenes to generate a short summary.

[0766] The server selects visually interesting frames and creates thumbnail candidates.

[0767] Step 5: Evaluate and select summaries and thumbnails

[0768] The server evaluates the generated summaries and thumbnail candidates and selects the best one using embedding similarity calculation. The input is the video summary and thumbnail candidates, and the output is the selected summary and thumbnail.

[0769] Specific behavior:

[0770] The server uses embedding technology to calculate the similarity of each candidate.

[0771] The most similar summaries and thumbnails are selected and prepared for delivery to the user.

[0772] Step 6: Present to the user

[0773] The terminal presents the optimally selected video summary and thumbnail to the user. The input is the selected video summary and thumbnail, and the output is the content displayed in the user interface.

[0774] Specific behavior:

[0775] The server sends the selected summary and thumbnail to the terminal.

[0776] The data received by the terminal is displayed in the user interface.

[0777] The user checks the presented summary and thumbnails to quickly get an overview of the content to be viewed.

[0778] (Application example 1)

[0779] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0780] Conventional content distribution services lack the technology to automatically generate video summaries and thumbnails that match users' personalities and preferences. This makes it difficult for users to quickly identify scenes that interest or are important to them, resulting in reduced viewing efficiency. Furthermore, the lack of appropriate thumbnails makes it difficult for users to discover new content.

[0781] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0782] In this invention, the server includes means for collecting user log data, means for analyzing the collected log data and extracting features related to the user's personality, means for extracting scene cuts and thumbnails from video content based on the features, means for generating candidate video summaries and thumbnails tailored to the user's personality by combining the features with the extracted scene cuts and thumbnails, means for selecting optimal video summaries and thumbnails from the candidates and presenting them to the user, means for collecting real-time data, means for using a framework for presenting content to a personal mobile device, and means for generating video summaries and thumbnails using a generative AI model. This allows for the rapid provision of video summaries and thumbnails tailored to the user's personality, improving viewing efficiency and promoting the discovery of new content.

[0783] "User log data" refers to various behavioral data generated when a user uses the system, such as viewing history, search history, and click history.

[0784] "Features" are characteristic data extracted from log data to indicate a user's personality and preferences.

[0785] A "scene cut" is a video fragment cut out from a specific portion of video content.

[0786] A "thumbnail" is a still image that represents a portion of video content and is used to provide a visual overview.

[0787] A "video summary" is a video that extracts the main scenes of a video content and compresses them so that they can be viewed in a short time.

[0788] "Candidates" is a collection of multiple generated video summaries and thumbnails before selecting the best video summaries and thumbnails.

[0789] "Best suited video summaries and thumbnails" are those video summaries and thumbnails that are determined to be most suitable to the user's personality and interests and are presented to the user.

[0790] "Real-time data" is data that is collected immediately and generated on the spot when users use the system.

[0791] A "personal mobile device" is an information processing device that is individually owned and portable by a user, such as a smartphone or tablet.

[0792] A "framework" is a basic structure or framework for efficient software development, and is particularly used to present content.

[0793] A "generative AI model" is an algorithm or program that uses artificial intelligence technology to generate and process information.

[0794] The present invention provides a system for providing personalized video summaries and thumbnails to users, and the specific implementation method thereof will be described below. The system is configured using a server and a user's personal mobile terminal.

[0795] System Overview

[0796] The server is configured using the following hardware and software.

[0797] Hardware: Server computers, database servers

[0798] Software: Apache Kafka, MySQL, BERT model, machine learning models (random forest, SVM), OpenCV, YOLO, React Native

[0799] The server first collects user log data, including viewing history, search history, click history, etc. Apache Kafka is used to collect data in real time, and the collected data is stored in a MySQL database.

[0800] Next, the server analyzes the user's log data using a BERT model or machine learning model to extract personality features. For example, it identifies areas of interest such as "sports" or "entertainment" from the user's viewing history.

[0801] The server then analyzes the video content frame by frame, using image recognition techniques such as OpenCV and YOLO to detect key scenes and key moments, such as goal scenes and player highlights in sports videos.

[0802] The server then uses a generative AI model to combine the user's personality features with the detected scenes to generate a video summary and thumbnails, which are then evaluated using a similarity calculation to select the most suitable one.

[0803] The selected video summaries and thumbnails are then presented to the user's personal mobile device through an app developed using React Native, allowing the user to quickly get an overview of the content they are about to watch.

[0804] Specific examples

[0805] When user A wants to watch a new sports game video, the following happens:

[0806] 1. The server collects user A's past log data and identifies that he has sports-related preferences.

[0807] 2. The server analyzes frames from new match videos and extracts key moments such as goals and player highlights.

[0808] 3. The server combines these scenes to generate a two-minute highlight summary and visually interesting thumbnails.

[0809] 4. The personal mobile device presents these summaries and thumbnails to User A.

[0810] Prompt Sentence Examples

[0811] Examples of prompts for generative AI models include:

[0812] Based on the user's viewing history and interests, extract key scenes from the following video and generate a 2-minute summary and thumbnail.

[0813] Viewing history features: Sports, Entertainment

[0814] Video: "Football match video"

[0815] Scenes to be extracted: Goal scenes, player highlights

[0816] Generated summary length: 2 minutes

[0817] result:

[0818] 1. 2-minute video summary

[0819] 2. Thumbnails capturing goal scenes

[0820] In this way, personalized video summaries and thumbnails can be efficiently generated to provide an optimal viewing experience.

[0821] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0822] Step 1:

[0823] The server collects user log data. Specifically, data such as user viewing history, search history, and click history is collected in real time using Apache Kafka and stored in a MySQL database. This collected data serves as basic information for detailed analysis of user behavior patterns. The input is user behavior data, and the output is log data stored in the MySQL database.

[0824] Step 2:

[0825] The server analyzes the collected log data and extracts features related to the user's personality. Specifically, it uses the BERT model and machine learning models (random forest and SVM) to analyze the user's log data and extract features such as "sports" and "entertainment" from their viewing history and search history. The input for this process is the log data in the MySQL database, and the output is analyzed data containing personality features.

[0826] Step 3:

[0827] The server analyzes the video content frame by frame. Specifically, it uses image recognition technologies such as OpenCV and YOLO to analyze each frame of the video and detect key scenes and key points. This process identifies, for example, goal scenes and player highlights. The input is the original video content, and the output is scene data containing key scenes.

[0828] Step 4:

[0829] The server combines the user's personality features with the extracted scenes to generate video summaries and thumbnail candidates. Specifically, a generative AI model is used to automatically generate video summaries and thumbnails based on the personality features and scene data. In this process, multiple candidates are generated, and the best one is selected. The input is the personality features and scene data, and the output is video summaries and thumbnail candidates.

[0830] Step 5:

[0831] The server selects the best video summary and thumbnail from the generated candidates and presents them to the user. Specifically, it performs a similarity calculation to select the most suitable summary and thumbnail. The selected video summary and thumbnail are then displayed on the user's personal mobile device through an app developed using React Native. The input of this process is the video summary and thumbnail candidates, and the output is the best summary and thumbnail presented to the user.

[0832] Step 6:

[0833] The user checks the presented video summary and thumbnails. Specifically, the user watches the summary and thumbnails using a React Native-based application on a personal mobile device such as a smartphone or tablet. This step is the user interaction phase, where the input is the optimal summary and thumbnail presented to the user, and the output is the user's viewing behavior.

[0834] Through these steps, users can quickly view personalized video summaries and thumbnails, improving their viewing experience.

[0835] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0836] This invention is a system for providing video summaries and thumbnails based on a user's personality and emotional state. In this system, a server collects and analyzes user log data, generates video summaries and thumbnails based on the user's personality features and emotional state, and provides them to the user. The specific program processing flow and its explanation are described below.

[0837] System Overview

[0838] The system analyzes user behavioral and emotional data to identify their personality and emotional state, and then generates summaries and thumbnails of video content. User log data includes viewing history, search history, click history, and emotional data from an emotion engine. Based on this data, the system identifies the user's interests, preferences, and emotional state.

[0839] What the program does

[0840] Collecting and analyzing user log data and emotional data

[0841] The server collects log data from the database, including users' past behavioral data and emotional data. This data includes the types of videos users watched, keywords they searched for, thumbnails they clicked, and the emotional state users displayed while watching a particular video. The collected log data and emotional data are analyzed and natural language processing (NLP) models and emotion analysis models are used to extract features related to the user's personality and emotional state.

[0842] Example: Extract emotional features such as "comedy preference" and "frequent smiling while watching" from user B's past viewing history and facial expressions while watching.

[0843] Video content analysis

[0844] The server analyzes the video content, extracts each frame, and detects key scenes and key moments. Image recognition technology is used for the analysis to detect scene changes and specific events (e.g., interesting or moving scenes). These scenes are extracted as important cuts based on the user's personality and emotional features.

[0845] Example: Extracting funny and touching scenes from a comedy show video.

[0846] Video summary and thumbnail generation

[0847] The server combines the user's personality features with the extracted key scenes from the video to generate candidate video summaries and thumbnails. At this stage, we use embedding similarity calculations to evaluate whether the generated candidates are interesting to the user. By taking the user's emotional state into account, we generate the most visually and emotionally appealing summaries and thumbnails.

[0848] Example: For user B, generate a two-minute video summary highlighting funny moments from a comedy, along with thumbnails capturing smiling moments.

[0849] Optimal summary and thumbnail presentation

[0850] The device receives the optimal video summary and thumbnail sent from the server and presents them to the user, allowing the user to instantly get an overview of the content they are watching.

[0851] Specific examples

[0852] If User B wants to watch a new comedy video, the system works as follows:

[0853] 1. The server collects user B's past log data and emotional data while watching, identifies that user B has a comedy-related preference, and confirms that user B smiles frequently while watching.

[0854] 2. The server analyzes frames from a new comedy video and extracts funny and touching moments.

[0855] 3. The server combines these scenes to generate a two-minute comedy summary video and visually interesting smiley thumbnails.

[0856] 4. The device presents these summaries and thumbnails to User B.

[0857] This allows User B to quickly check scenes that interest them, improving their viewing experience and making it easier for them to discover new content. Emotional data is also taken into account, which tends to increase satisfaction while viewing.

[0858] The processing flow will be explained below.

[0859] Step 1:

[0860] The server collects users' past behavioral data and emotional data. Specifically, it accesses a database to obtain log data including viewing history, search history, click history, and emotional data during viewing. This data is stored in memory or cache as it is required for later analysis steps.

[0861] Step 2:

[0862] The server analyzes the collected log data and emotion data. First, it uses natural language processing (NLP) models and machine learning models to extract features that indicate the user's interests and preferences from the log data. Next, it uses an emotion analysis model to analyze the user's emotional fluctuations and identify the emotional state (e.g., joy, surprise, excitement) shown while watching.

[0863] Step 3:

[0864] The server analyzes the video content frame by frame, and uses image recognition technology and scene detection algorithms to detect key scenes and events (e.g., humorous or moving scenes) in each frame. From the detected scenes, important cuts and thumbnail images are extracted based on the user's personality and emotional features.

[0865] Step 4:

[0866] The server generates candidate video summaries and thumbnails based on the extracted scene cuts and thumbnails. Using embedding similarity calculations, the server evaluates and selects candidates based on their match with the user's interests and emotions. For example, it prioritizes video summaries that contain scenes that the user expressed high enjoyment in while watching.

[0867] Step 5:

[0868] The device receives the optimal video summary and thumbnail sent from the server, passes the received data to the display component, and presents it visually to the user, allowing the user to get a quick overview of the content they are watching.

[0869] Step 6:

[0870] Users can view the video summary and thumbnails provided and choose to watch the video if they are interested. Emotional data is also collected during viewing and used for future analysis.

[0871] This improves the viewing experience by allowing users to quickly check scenes they are particularly interested in. Emotional data is also taken into account, resulting in higher viewing satisfaction and content tailored to the user's preferences.

[0872] Example 2

[0873] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0874] Currently, existing systems that provide optimized video summaries and thumbnails based on a user's personality and emotional state are inadequate. In particular, technologies for selecting and presenting content that reflects a user's preferences and emotions are immature, and user satisfaction cannot be fully enhanced. Therefore, there is a need for a system that can efficiently generate and present video summaries and thumbnails based on a user's personality and emotional state.

[0875] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0876] In this invention, the server includes means for collecting user history data, means for analyzing the collected history data and extracting features related to the user's personality, means for extracting scene cuts and representative images from video content based on the features, means for combining the features with the extracted scene cuts and representative images to generate candidates for video summaries and representative images tailored to the user's personality, and means for selecting the most suitable video summaries and representative images from the candidates and presenting them to the user, thereby making it possible to provide content summaries and thumbnails that accurately reflect the user's interests and emotional state.

[0877] "History data" refers to data about a user's past activities, including viewing history, search history, click history, etc.

[0878] "Personality-related features" are data features that indicate a user's interests, preferences, behavioral patterns, emotional state, etc.

[0879] "Video content" is information content in digital form that includes video and audio data.

[0880] A "scene cut" is data that represents a specific scene or frame extracted from video content.

[0881] A "representative image" is an image that represents the video content, and is often used as a thumbnail.

[0882] "Candidate generation means" refers to methods or techniques for generating multiple options.

[0883] The "optimal video summary and representative image" are the summarized video and representative image that best fit the user's interests, preferences, and emotional state.

[0884] The present invention relates to a system for providing video summaries and thumbnails based on a user's personality and emotional state. In this system, a server collects and analyzes user log data, generates video summaries and thumbnails based on the user's personality features and emotional state, and provides them to the user.

[0885] System Overview

[0886] The system analyzes user behavioral and emotional data to generate summaries and representative images of video content. User history data includes viewing history, search history, click history, and even emotional data from an emotional engine. Based on this data, the system identifies the user's interests, preferences, and emotional state.

[0887] Collection and analysis of user history and emotion data

[0888] The server collects users' historical behavioral data and emotional data from the database. This data includes the types of videos users watched, the keywords they searched for, the representative images they clicked on, and the emotional state they displayed while watching a particular video. The collected historical data and emotional data are analyzed and natural language processing (NLP) models and emotion analysis models are used to extract user personality features and emotional states.

[0889] Video content analysis

[0890] The server analyzes the video content, extracts each frame, and detects key scenes and key moments. Image recognition technology is used for the analysis to detect scene changes and specific events (e.g., interesting or moving scenes). These scenes are extracted as important cuts based on the user's personality and emotional features.

[0891] Video summarization and representative image generation

[0892] The server combines the user's personality features with the extracted key scenes from the video to generate candidate video summaries and representative images. At this stage, we use embedding similarity calculations to evaluate whether the generated candidates are of interest to the user. By also taking the user's emotional state into consideration, we generate the most visually and emotionally appealing summary and representative images.

[0893] Optimal summary and representative image presentation

[0894] The device receives the optimal video summary and representative image sent from the server, and then presents them to the user, allowing the user to instantly grasp the overview of the content being viewed.

[0895] Specific examples

[0896] If User B wants to watch a new comedy video, the system works as follows:

[0897] 1. The server collects user B's past history data and emotional data while watching, identifies that user B has a comedy-related preference, and confirms that user B smiles frequently while watching.

[0898] 2. The server analyzes frames from a new comedy video and extracts funny and touching moments.

[0899] 3. The server combines these scenes to generate a two-minute comedy summary video and a visually interesting representative image of a smiling face.

[0900] 4. The device presents these summaries and representative images to User B.

[0901] This allows User B to quickly check scenes that he or she is particularly interested in, improving the viewing experience. In addition, since emotional data is taken into consideration, satisfaction during viewing is likely to increase.

[0902] Prompt Sentence Examples

[0903] Example prompt for a generative AI model:

[0904] "Please explain the process of a system that extracts personality features and emotional features based on user log data and emotional data during viewing, automatically selects important scenes from video content, and generates the most appropriate video summary and thumbnail for the user."

[0905] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0906] Step 1:

[0907] The server collects user history data. The inputs are viewing history, search history, click history, and emotion data. The output is a set of collected history data.

[0908] Specific behavior:

[0909] The server queries the database to retrieve data on User B's past viewing content, search keywords, representative images clicked, and emotional states (e.g., facial expressions, voice) expressed while watching the video.

[0910] Step 2:

[0911] The server analyzes the collected historical data and extracts features related to the user's personality. The input is a set of historical data. The output is a set of features related to the personality.

[0912] Specific behavior:

[0913] The server uses natural language processing (NLP) and sentiment analysis models to extract features from viewing history, such as "comedy preference" or "frequent smiling while watching." For example, it uses Python's TensorFlow library to run machine learning models to extract features from complex datasets.

[0914] Step 3:

[0915] The server analyzes the video content and detects key scenes and key moments. The input is the video data to be analyzed and the user's personality features. The output is a set of candidate scene cuts and representative images.

[0916] Specific behavior:

[0917] The server uses OpenCV or similar image recognition technology to analyze video frames to identify scenes with a lot of laughter and smiling faces, as well as the rate of change and audio data in each frame to detect important events.

[0918] Step 4:

[0919] The server generates a video summary and candidate representative images based on the extracted scene cuts and representative images. The input is the set of scene cuts and representative images from the analysis results and the user's personality features. The output is a video summary and a set of candidate representative images.

[0920] Specific behavior:

[0921] The server creates a two-minute video summary focusing on interesting scenes, and uses embedding similarity calculations to evaluate whether it matches the user's preferences and select the most appealing summary and representative image.

[0922] Step 5:

[0923] The terminal receives the optimal video summary and representative image from the server and presents them to the user. The input is the video summary and representative image sent from the server. The output is the visual content to be presented to the user.

[0924] Specific behavior:

[0925] The device displays the summary video and representative image on the application's home screen or notification area. When User B opens the app, the summary video and representative image that is most likely to interest them are presented. When the user clicks on it, a two-minute summary video plays and the related representative image is also displayed.

[0926] (Application example 2)

[0927] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0928] In recent years, with the increasing volume of video content, it has become increasingly difficult for viewers to quickly find content that best suits their interests and emotional state. Furthermore, conventional systems cannot generate video summaries and thumbnails that fully take into account the user's personality and emotional state, resulting in a poor user experience.

[0929] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting user log data, means for analyzing the collected log data and extracting features related to the user's personality, means for extracting scene cuts and thumbnails from video content based on the features, means for collecting and analyzing user emotional data and identifying the user's emotional state, means for evaluating optimal video summary and thumbnail candidates based on the emotional state, and means for selecting optimal video summaries and thumbnails from the candidates and presenting them to the user. This makes it possible to provide video summaries and thumbnails that are tailored to the user's personality and emotional state.

[0930] "User log data" refers to data about a user's past behavior, such as their viewing history, search history, and click history.

[0931] "Features related to individuality" are characteristic information that indicates the user's interests and preferences.

[0932] A "scene cut" is an extracted section of an important scene or point in video content.

[0933] A "thumbnail" is a small image that visually represents a portion of the video content.

[0934] A "video summary" is content that extracts the main points and important scenes from video content so that they can be viewed in a short amount of time.

[0935] "Emotional data" refers to data that indicates the emotional state of a user analyzed from their facial expressions and behavior while watching a video.

[0936] The "emotional state" indicates the emotional state that the user feels while watching a video, and includes, for example, emotions such as joy, anger, sadness, and pleasure.

[0937] An "Embedding model" is a machine learning model that converts natural language and image data into vector format and calculates their meaning and characteristics.

[0938] An "evaluation method" is a method for determining how well a candidate video summary or thumbnail suits a user's interests and emotional state.

[0939] A "presenting means" is a means including a user interface function for displaying the selected optimal video summary or thumbnail to a user.

[0940] The present invention provides a system for providing optimal video summaries and thumbnails based on the user's personality and emotional state. The system comprises a plurality of processing means, and a server and a terminal operate in unison.

[0941] System Configuration

[0942] The server comprises the following processing means:

[0943] 1. Means of collecting user log data

[0944] 2. A method for analyzing log data and extracting features related to user personality

[0945] 3. A method for extracting scene cuts and thumbnails from video content based on features

[0946] 4. A means of collecting and analyzing user emotional data to identify the user's emotional state

[0947] 5. A means of evaluating candidate video summaries and thumbnails based on emotional state

[0948] 6. A means of selecting and presenting the best video summaries and thumbnails to users

[0949] Specific processing flow

[0950] 1. The data collection server collects behavioral data such as user viewing history, search history, click history, etc. While viewing, a camera is used to capture the user's facial expressions and collect them as emotional data.

[0951] 2. Using an emotion analysis engine (e.g., OpenFace or Facial Emotion Recognition Software), analyze the collected facial expression data and identify the user's emotional state (joy, anger, sadness, or happiness).

[0952] 3. The server uses a natural language processing (NLP) model (e.g., BERT, GPT-3) based on the collected log data to extract features that indicate the user's interests and preferences.

[0953] 4. Use image recognition techniques (e.g., YOLO, ResNet) to detect key scenes and key points (e.g., funny or moving scenes) from video content.

[0954] 5. Using the embedding model, we evaluate the generated video summaries and thumbnail candidates based on the user's personality features and emotional state. The evaluation is performed using the embedding similarity calculation.

[0955] 6. The terminal presents the user with the most appropriate video summaries and thumbnails sent from the server, allowing the user to quickly view content that matches their interests and emotional state.

[0956] Specific hardware and software used

[0957] Data collection servers: servers and cloud storage services

[0958] Emotion analysis engine: OpenFace, Facial Emotion Recognition Software

[0959] Image recognition technology: YOLO, ResNet

[0960] Natural Language Processing models: BERT, GPT-3

[0961] Embedding model: (used in conjunction with BERT and GPT-3)

[0962] Specific examples

[0963] For example, if User B wants to watch a new comedy video, the process goes like this:

[0964] 1. The data collection server collects user B's past viewing history and facial expression data while viewing.

[0965] 2. The emotion analysis engine analyzes the facial expression data and identifies User B's emotions while watching.

[0966] 3. The server analyzes past log data and determines that User B has a strong preference for comedy and smiles a lot while watching.

[0967] 4. Use image recognition technology to analyze new comedy videos and extract funny and touching scenes.

[0968] 5. The embedding model evaluates these scenes and generates the best summary video and thumbnail for user B.

[0969] 6. The device presents these summaries and thumbnails to User B.

[0970] Prompt Sentence Examples

[0971] Here are some example prompts to use as input to a generative AI model:

[0972] Based on the user's viewing history and sentiment data, generate a summary video and thumbnail that meets the following criteria:

[0973] 1. Video Summary: Comedy genre, 2 minutes, mainly funny scenes.

[0974] 2. Thumbnail: A smiling scene while the user is watching.

[0975] This allows the AI ​​model to generate optimal content that matches the user's preferences and emotions based on the specified conditions.

[0976] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0977] Step 1:

[0978] The server collects the user's viewing history, search history, click history, and emotional data while viewing. This data is stored in a database and used as the basis for processing. Specifically, it records the user's behavioral data as a log and uses a camera to capture their facial expressions while viewing. The input is the user's past behavioral data and facial expression data while viewing. The output is the log data and emotional data stored in the database.

[0979] Step 2:

[0980] The server analyzes the collected log data and emotion data. In this step, it uses a natural language processing (NLP) model (e.g., BERT, GPT-3) to extract features related to the user's personality (interests and preferences) from the log data. It also uses an emotion analysis engine (e.g., OpenFace) to extract the user's emotional state from facial expression data. The input is the log data and emotion data stored in the database. The output is the features of the personality and emotional state.

[0981] Step 3:

[0982] The server extracts scene cuts from the video content frame by frame and detects events of particular interest (e.g., funny or emotional scenes). In this step, image recognition techniques (e.g., YOLO, ResNet) are used to analyze the video frames. The input is the video content. The output is scene cuts related to interests and emotions.

[0983] Step 4:

[0984] The server combines the extracted scene cuts with the user's personality features and emotional state to generate candidate video summaries and thumbnails. In this step, an embedding model (e.g., BERT, GPT-3) is used to perform similarity calculations to select the video summaries and thumbnails that best fit the user's interests and emotional state. The input is scene cuts and user features. The output is candidate video summaries and thumbnails.

[0985] Step 5:

[0986] The server evaluates the generated video summaries and thumbnail candidates and selects the most suitable candidate. This evaluation is performed by scoring them based on the user's interests and emotional state. The input is the video summaries and thumbnail candidates. The output is the optimal video summary and thumbnail.

[0987] Step 6:

[0988] The terminal presents the optimal video summary and thumbnail sent from the server to the user, allowing the user to visually grasp and quickly select content that matches their interests and emotional state. The input is the optimal video summary and thumbnail, and the output is the content displayed in the user interface.

[0989] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0990] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0991] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0992] [Fourth embodiment]

[0993] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0994] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0995] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0996] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0997] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0998] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0999] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1000] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1001] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1002] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1003] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1004] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1005] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1006] This invention is a system that provides video summaries and thumbnails that take into account the user's individuality. In this system, the server collects and analyzes user log data, and generates and provides video summaries and thumbnails that are tailored to the user's individuality. The specific program processing flow and its explanation are described below.

[1007] System Overview

[1008] This system analyzes user behavior data to generate summaries and thumbnails of video content. User log data includes viewing history, search history, click history, etc. This log data is used to identify user interests and preferences.

[1009] What the program does

[1010] Collecting and analyzing user log data

[1011] The server collects log data from the database, including the user's past behavior data, such as the types of videos the user watched, the keywords they searched for, and the thumbnails they clicked on.The collected log data is analyzed and natural language processing (NLP) and machine learning models are used to extract features related to the user's personality.

[1012] Example: Extract the features "sports" and "entertainment" from user A's past viewing history.

[1013] Video content analysis

[1014] The server analyzes the video content frame by frame to detect key scenes and key points, using image recognition technology and scene detection algorithms, and extracts specific cuts and frames from the video based on the user's personality features.

[1015] Example: Extracting goal scenes and player highlights from videos of sports matches.

[1016] Video summary and thumbnail generation

[1017] The server combines the user's personality features with key scenes from the video to generate video summary and thumbnail candidates. The candidates are then evaluated using embedding similarity calculations and the most suitable one is selected.

[1018] Example: For User A, automatically generate a two-minute summary of a sports match highlights with thumbnails capturing the exciting moments.

[1019] Optimal summary and thumbnail presentation

[1020] The device will present users with optimally selected video summaries and thumbnails, allowing them to quickly get an overview of the content they are watching.

[1021] Specific examples

[1022] When User A wants to watch a new sports game video, the system works as follows:

[1023] 1. The server collects user A's past log data and identifies that he has sports-related preferences.

[1024] 2. The server analyzes frames from new match videos and extracts key moments such as goals and player highlights.

[1025] 3. The server combines these scenes to generate a two-minute highlight summary and visually interesting thumbnails.

[1026] 4. The device presents these summaries and thumbnails to User A.

[1027] This allows User A to quickly check scenes that interest him / her, improving his / her viewing experience and promoting the discovery of new content.

[1028] The above is a specific embodiment for carrying out the present invention.

[1029] The processing flow will be explained below.

[1030] Step 1:

[1031] The server collects the user's past behavior data, including viewing history, search history, click history, etc. It accesses the database to extract the target user's log data and stores the collected data in memory or cache for later analysis.

[1032] Step 2:

[1033] The server analyzes the collected log data and identifies user interests and preferences using natural language processing (NLP) and machine learning models. Specifically, it analyzes keywords and viewing patterns in the log data and extracts features such as "sports preferences" and "entertainment preferences."

[1034] Step 3:

[1035] The server analyzes the video content, extracts each frame, and detects key scenes and key moments. This analysis uses image recognition technology to detect scene changes and specific events (e.g., goal scenes, scenes featuring key characters). These scenes are then extracted as important cuts based on the user's personality features.

[1036] Step 4:

[1037] The server combines the extracted key scenes and points with the user's personality features to generate video summary and thumbnail candidates. At this stage, we use embedding similarity calculation to evaluate whether the generated candidates are interesting to the user, and then select the most relevant summary and thumbnail.

[1038] Step 5:

[1039] The device receives the optimal video summary and thumbnail sent from the server and then presents them to the user, for example, displaying a two-minute summary video of a sports highlight along with a visually appealing thumbnail image.

[1040] Step 6:

[1041] Users can view the video summary and thumbnails provided and watch the video if they are interested, a process that allows users to efficiently discover content that interests them.

[1042] This allows users to quickly view scenes that are of particular interest to them, improving their viewing experience and making it easier to discover new content.

[1043] Example 1

[1044] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1045] The wide variety of modern video content and the large amount of data available make it difficult for users to quickly find content that matches their interests and preferences. In particular, the selection of video summaries and thumbnails does not provide appropriate content that takes into account the individual needs of users, creating a demand for an improved viewing experience. The present invention aims to solve this problem by providing a system for generating video summaries and thumbnails that take into account the individual needs of users.

[1046] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1047] In this invention, the server includes: means for collecting user log data; means for analyzing the collected log data using a natural language processing model and a machine learning model to extract features related to the user's personality; means for analyzing video content frame by frame and extracting key scenes and thumbnails based on the features; means for combining the features with the extracted key scenes and thumbnails to generate video summaries and thumbnail candidates tailored to the user's personality; and means for evaluating the generated candidates using embedding similarity calculation, selecting optimal video summaries and thumbnails, and presenting them to the user. This enables the generation and presentation of video summaries and thumbnails that take the user's personality and preferences into consideration.

[1048] "User log data" refers to data including a user's past behavioral data, viewing history, search history, and click history.

[1049] A "natural language processing model" is a computational algorithm for analyzing and understanding human language, and is used to extract topics and categories from users' search keywords and viewing history.

[1050] A "machine learning model" is an algorithm that analyzes large amounts of data to learn patterns and rules and make predictions and classifications.

[1051] A "feature" is an element of extracted data that indicates a user's interests and preferences.

[1052] "Video content" refers to all video data viewed by users, including specific videos, movies, and programs.

[1053] A "key scene" is a significant event or point in the video content.

[1054] A "thumbnail" is a small image that represents the video content and serves to visually attract the user's attention.

[1055] "Embedding similarity calculation" is a method of calculating similarity by converting feature values ​​and scene information into vector space.

[1056] A "video summary" is a video created by combining key scenes from video content into a shortened version.

[1057] "Candidates" refer to the generated video summaries and thumbnail options.

[1058] The present invention provides a system for providing video summaries and thumbnails that take into account the user's individuality. In this system, a server collects and analyzes user log data, generates video summaries and thumbnails that are tailored to the user's individuality, and provides them to the user. Specific embodiments of the present invention are described below.

[1059] The server collects log data from a database, including users' past behavioral data. This data includes the types of videos the user watched, the keywords they searched for, and the thumbnails they clicked. The collected log data is analyzed using natural language processing (NLP) and machine learning models. As a result of the analysis, features related to the user's personality are extracted. For example, features such as "sports" and "entertainment" are extracted from User A's past viewing history.

[1060] The server then analyzes the video content frame by frame, using image recognition technology and scene detection algorithms. As a result of the analysis, key scenes and points in the video are detected, and specific cuts and frames are extracted based on the user's personality features. For example, goal scenes and player highlights can be extracted from a video of a sports game.

[1061] The server then generates video summary and thumbnail candidates by combining the user's personality features and key scenes from the video. The candidates are evaluated using embedding similarity calculations and the most suitable one is selected. For example, for user A, a two-minute summary of a sports game highlights and thumbnails capturing the most exciting moments are generated.

[1062] Finally, the device presents the optimally selected video summary and thumbnail to the user. By checking the summary and thumbnail, the user can quickly get an overview of the content they are watching. As a concrete example, when user A watches a new sports game video, the system operates as follows:

[1063] 1. The server collects user A's past log data and identifies that he has sports-related preferences.

[1064] 2. The server analyzes new match videos frame by frame and extracts key moments such as goals and player highlights.

[1065] 3. The server combines these scenes to generate a two-minute highlight summary and visually interesting thumbnails.

[1066] 4. The device presents these summaries and thumbnails to User A.

[1067] An example prompt is, "For a sports fan, please create a two-minute summary of sports game highlights based on their past viewing history and create an engaging thumbnail." Using this prompt, the generative AI model can generate optimal content based on the user's personality.

[1068] This invention enables the generation and presentation of video summaries and thumbnails that take into account the user's individuality and preferences, which is expected to improve the viewing experience and promote the discovery of new content.

[1069] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1070] Step 1: Collect user log data

[1071] The server connects to the database to collect user log data. This log data includes the user's viewing history, search history, click history, etc. Specifically, it issues SQL queries to the database to extract the target user's behavioral data. The input is the user behavior data in the database, and the output is the extracted log data.

[1072] Specific behavior:

[1073] The server establishes a database connection and executes a query based on the user's ID.

[1074] The data acquired by the server is temporarily stored in memory.

[1075] Step 2: Analyze the log data

[1076] The server analyzes the collected log data using natural language processing (NLP) and machine learning models. This analysis extracts features related to the user's personality. The input is the collected log data, and the output is the extracted features.

[1077] Specific behavior:

[1078] The server applies NLP models to extract topics and categories from search keywords and viewing history.

[1079] The server uses machine learning models to analyze user behavior patterns and identify categories of interest.

[1080] Step 3: Frame analysis of video content

[1081] The server analyzes the video content the user wants to watch, frame by frame, and uses image recognition technology and scene detection algorithms to detect key scenes and key points. The input is the video file, and the output is key scene information.

[1082] Specific behavior:

[1083] The server loads the video file and splits it into frames.

[1084] The server uses image recognition technology to analyze the frames and evaluate the importance of the scenes.

[1085] The extracted main scene information is linked to the features.

[1086] Step 4: Generate video summary and thumbnails

[1087] The server combines the user's personality features with the main scenes of the video to generate video summaries and thumbnail candidates. The input is the features and main scene information, and the output is video summaries and thumbnail candidates.

[1088] Specific behavior:

[1089] The server sequentially connects the identified key scenes to generate a short summary.

[1090] The server selects visually interesting frames and creates thumbnail candidates.

[1091] Step 5: Evaluate and select summaries and thumbnails

[1092] The server evaluates the generated summaries and thumbnail candidates and selects the best one using embedding similarity calculation. The input is the video summary and thumbnail candidates, and the output is the selected summary and thumbnail.

[1093] Specific behavior:

[1094] The server uses embedding technology to calculate the similarity of each candidate.

[1095] The most similar summaries and thumbnails are selected and prepared for delivery to the user.

[1096] Step 6: Present to the user

[1097] The terminal presents the optimally selected video summary and thumbnail to the user. The input is the selected video summary and thumbnail, and the output is the content displayed in the user interface.

[1098] Specific behavior:

[1099] The server sends the selected summary and thumbnail to the terminal.

[1100] The data received by the terminal is displayed in the user interface.

[1101] The user checks the presented summary and thumbnails to quickly get an overview of the content to be viewed.

[1102] (Application example 1)

[1103] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1104] Conventional content distribution services lack the technology to automatically generate video summaries and thumbnails that match users' personalities and preferences. This makes it difficult for users to quickly identify scenes that interest or are important to them, resulting in reduced viewing efficiency. Furthermore, the lack of appropriate thumbnails makes it difficult for users to discover new content.

[1105] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1106] In this invention, the server includes means for collecting user log data, means for analyzing the collected log data and extracting features related to the user's personality, means for extracting scene cuts and thumbnails from video content based on the features, means for generating candidate video summaries and thumbnails tailored to the user's personality by combining the features with the extracted scene cuts and thumbnails, means for selecting optimal video summaries and thumbnails from the candidates and presenting them to the user, means for collecting real-time data, means for using a framework for presenting content to a personal mobile device, and means for generating video summaries and thumbnails using a generative AI model. This allows for the rapid provision of video summaries and thumbnails tailored to the user's personality, improving viewing efficiency and promoting the discovery of new content.

[1107] "User log data" refers to various behavioral data generated when a user uses the system, such as viewing history, search history, and click history.

[1108] "Features" are characteristic data extracted from log data to indicate a user's personality and preferences.

[1109] A "scene cut" is a video fragment cut out from a specific portion of video content.

[1110] A "thumbnail" is a still image that represents a portion of video content and is used to provide a visual overview.

[1111] A "video summary" is a video that extracts the main scenes of a video content and compresses them so that they can be viewed in a short time.

[1112] "Candidates" is a collection of multiple generated video summaries and thumbnails before selecting the best video summaries and thumbnails.

[1113] "Best suited video summaries and thumbnails" are those video summaries and thumbnails that are determined to be most suitable to the user's personality and interests and are presented to the user.

[1114] "Real-time data" is data that is collected immediately and generated on the spot when users use the system.

[1115] A "personal mobile device" is an information processing device that is individually owned and portable by a user, such as a smartphone or tablet.

[1116] A "framework" is a basic structure or framework for efficient software development, and is particularly used to present content.

[1117] A "generative AI model" is an algorithm or program that uses artificial intelligence technology to generate and process information.

[1118] The present invention provides a system for providing personalized video summaries and thumbnails to users, and the specific implementation method thereof will be described below. The system is configured using a server and a user's personal mobile terminal.

[1119] System Overview

[1120] The server is configured using the following hardware and software.

[1121] Hardware: Server computers, database servers

[1122] Software: Apache Kafka, MySQL, BERT model, machine learning models (random forest, SVM), OpenCV, YOLO, React Native

[1123] The server first collects user log data, including viewing history, search history, click history, etc. Apache Kafka is used to collect data in real time, and the collected data is stored in a MySQL database.

[1124] Next, the server analyzes the user's log data using a BERT model or machine learning model to extract personality features. For example, it identifies areas of interest such as "sports" or "entertainment" from the user's viewing history.

[1125] The server then analyzes the video content frame by frame, using image recognition techniques such as OpenCV and YOLO to detect key scenes and key moments, such as goal scenes and player highlights in sports videos.

[1126] The server then uses a generative AI model to combine the user's personality features with the detected scenes to generate a video summary and thumbnails, which are then evaluated using a similarity calculation to select the most suitable one.

[1127] The selected video summaries and thumbnails are then presented to the user's personal mobile device through an app developed using React Native, allowing the user to quickly get an overview of the content they are about to watch.

[1128] Specific examples

[1129] When user A wants to watch a new sports game video, the following happens:

[1130] 1. The server collects user A's past log data and identifies that he has sports-related preferences.

[1131] 2. The server analyzes frames from new match videos and extracts key moments such as goals and player highlights.

[1132] 3. The server combines these scenes to generate a two-minute highlight summary and visually interesting thumbnails.

[1133] 4. The personal mobile device presents these summaries and thumbnails to User A.

[1134] Prompt Sentence Examples

[1135] Examples of prompts for generative AI models include:

[1136] Based on the user's viewing history and interests, extract key scenes from the following video and generate a 2-minute summary and thumbnail.

[1137] Viewing history features: Sports, Entertainment

[1138] Video: "Football match video"

[1139] Scenes to be extracted: Goal scenes, player highlights

[1140] Generated summary length: 2 minutes

[1141] result:

[1142] 1. 2-minute video summary

[1143] 2. Thumbnails capturing goal scenes

[1144] In this way, personalized video summaries and thumbnails can be efficiently generated to provide an optimal viewing experience.

[1145] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1146] Step 1:

[1147] The server collects user log data. Specifically, data such as user viewing history, search history, and click history is collected in real time using Apache Kafka and stored in a MySQL database. This collected data serves as basic information for detailed analysis of user behavior patterns. The input is user behavior data, and the output is log data stored in the MySQL database.

[1148] Step 2:

[1149] The server analyzes the collected log data and extracts features related to the user's personality. Specifically, it uses the BERT model and machine learning models (random forest and SVM) to analyze the user's log data and extract features such as "sports" and "entertainment" from their viewing history and search history. The input for this process is the log data in the MySQL database, and the output is analyzed data containing personality features.

[1150] Step 3:

[1151] The server analyzes the video content frame by frame. Specifically, it uses image recognition technologies such as OpenCV and YOLO to analyze each frame of the video and detect key scenes and key points. This process identifies, for example, goal scenes and player highlights. The input is the original video content, and the output is scene data containing key scenes.

[1152] Step 4:

[1153] The server combines the user's personality features with the extracted scenes to generate video summaries and thumbnail candidates. Specifically, a generative AI model is used to automatically generate video summaries and thumbnails based on the personality features and scene data. In this process, multiple candidates are generated, and the best one is selected. The input is the personality features and scene data, and the output is video summaries and thumbnail candidates.

[1154] Step 5:

[1155] The server selects the best video summary and thumbnail from the generated candidates and presents them to the user. Specifically, it performs a similarity calculation to select the most suitable summary and thumbnail. The selected video summary and thumbnail are then displayed on the user's personal mobile device through an app developed using React Native. The input of this process is the video summary and thumbnail candidates, and the output is the best summary and thumbnail presented to the user.

[1156] Step 6:

[1157] The user checks the presented video summary and thumbnails. Specifically, the user watches the summary and thumbnails using a React Native-based application on a personal mobile device such as a smartphone or tablet. This step is the user interaction phase, where the input is the optimal summary and thumbnail presented to the user, and the output is the user's viewing behavior.

[1158] Through these steps, users can quickly view personalized video summaries and thumbnails, improving their viewing experience.

[1159] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1160] This invention is a system for providing video summaries and thumbnails based on a user's personality and emotional state. In this system, a server collects and analyzes user log data, generates video summaries and thumbnails based on the user's personality features and emotional state, and provides them to the user. The specific program processing flow and its explanation are described below.

[1161] System Overview

[1162] The system analyzes user behavioral and emotional data to identify their personality and emotional state, and then generates summaries and thumbnails of video content. User log data includes viewing history, search history, click history, and emotional data from an emotion engine. Based on this data, the system identifies the user's interests, preferences, and emotional state.

[1163] What the program does

[1164] Collecting and analyzing user log data and emotional data

[1165] The server collects log data from the database, including users' past behavioral data and emotional data. This data includes the types of videos users watched, keywords they searched for, thumbnails they clicked, and the emotional state users displayed while watching a particular video. The collected log data and emotional data are analyzed and natural language processing (NLP) models and emotion analysis models are used to extract features related to the user's personality and emotional state.

[1166] Example: Extract emotional features such as "comedy preference" and "frequent smiling while watching" from user B's past viewing history and facial expressions while watching.

[1167] Video content analysis

[1168] The server analyzes the video content, extracts each frame, and detects key scenes and key moments. Image recognition technology is used for the analysis to detect scene changes and specific events (e.g., interesting or moving scenes). These scenes are extracted as important cuts based on the user's personality and emotional features.

[1169] Example: Extracting funny and touching scenes from a comedy show video.

[1170] Video summary and thumbnail generation

[1171] The server combines the user's personality features with the extracted key scenes from the video to generate candidate video summaries and thumbnails. At this stage, we use embedding similarity calculations to evaluate whether the generated candidates are interesting to the user. By taking the user's emotional state into account, we generate the most visually and emotionally appealing summaries and thumbnails.

[1172] Example: For user B, generate a two-minute video summary highlighting funny moments from a comedy, along with thumbnails capturing smiling moments.

[1173] Optimal summary and thumbnail presentation

[1174] The device receives the optimal video summary and thumbnail sent from the server and presents them to the user, allowing the user to instantly get an overview of the content they are watching.

[1175] Specific examples

[1176] If User B wants to watch a new comedy video, the system works as follows:

[1177] 1. The server collects user B's past log data and emotional data while watching, identifies that user B has a comedy-related preference, and confirms that user B smiles frequently while watching.

[1178] 2. The server analyzes frames from a new comedy video and extracts funny and touching moments.

[1179] 3. The server combines these scenes to generate a two-minute comedy summary video and visually interesting smiley thumbnails.

[1180] 4. The device presents these summaries and thumbnails to User B.

[1181] This allows User B to quickly check scenes that interest them, improving their viewing experience and making it easier for them to discover new content. Emotional data is also taken into account, which tends to increase satisfaction while viewing.

[1182] The processing flow will be explained below.

[1183] Step 1:

[1184] The server collects users' past behavioral data and emotional data. Specifically, it accesses a database to obtain log data including viewing history, search history, click history, and emotional data during viewing. This data is stored in memory or cache as it is required for later analysis steps.

[1185] Step 2:

[1186] The server analyzes the collected log data and emotion data. First, it uses natural language processing (NLP) models and machine learning models to extract features that indicate the user's interests and preferences from the log data. Next, it uses an emotion analysis model to analyze the user's emotional fluctuations and identify the emotional state (e.g., joy, surprise, excitement) shown while watching.

[1187] Step 3:

[1188] The server analyzes the video content frame by frame, and uses image recognition technology and scene detection algorithms to detect key scenes and events (e.g., humorous or moving scenes) in each frame. From the detected scenes, important cuts and thumbnail images are extracted based on the user's personality and emotional features.

[1189] Step 4:

[1190] The server generates candidate video summaries and thumbnails based on the extracted scene cuts and thumbnails. Using embedding similarity calculations, the server evaluates and selects candidates based on their match with the user's interests and emotions. For example, it prioritizes video summaries that contain scenes that the user expressed high enjoyment in while watching.

[1191] Step 5:

[1192] The device receives the optimal video summary and thumbnail sent from the server, passes the received data to the display component, and presents it visually to the user, allowing the user to get a quick overview of the content they are watching.

[1193] Step 6:

[1194] Users can view the video summary and thumbnails provided and choose to watch the video if they are interested. Emotional data is also collected during viewing and used for future analysis.

[1195] This improves the viewing experience by allowing users to quickly check scenes they are particularly interested in. Emotional data is also taken into account, resulting in higher viewing satisfaction and content tailored to the user's preferences.

[1196] Example 2

[1197] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1198] Currently, existing systems that provide optimized video summaries and thumbnails based on a user's personality and emotional state are inadequate. In particular, technologies for selecting and presenting content that reflects a user's preferences and emotions are immature, and user satisfaction cannot be fully enhanced. Therefore, there is a need for a system that can efficiently generate and present video summaries and thumbnails based on a user's personality and emotional state.

[1199] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1200] In this invention, the server includes means for collecting user history data, means for analyzing the collected history data and extracting features related to the user's personality, means for extracting scene cuts and representative images from video content based on the features, means for combining the features with the extracted scene cuts and representative images to generate candidates for video summaries and representative images tailored to the user's personality, and means for selecting the most suitable video summaries and representative images from the candidates and presenting them to the user, thereby making it possible to provide content summaries and thumbnails that accurately reflect the user's interests and emotional state.

[1201] "History data" refers to data about a user's past activities, including viewing history, search history, click history, etc.

[1202] "Personality-related features" are data features that indicate a user's interests, preferences, behavioral patterns, emotional state, etc.

[1203] "Video content" is information content in digital form that includes video and audio data.

[1204] A "scene cut" is data that represents a specific scene or frame extracted from video content.

[1205] A "representative image" is an image that represents the video content, and is often used as a thumbnail.

[1206] "Candidate generation means" refers to methods or techniques for generating multiple options.

[1207] The "optimal video summary and representative image" are the summarized video and representative image that best fit the user's interests, preferences, and emotional state.

[1208] The present invention relates to a system for providing video summaries and thumbnails based on a user's personality and emotional state. In this system, a server collects and analyzes user log data, generates video summaries and thumbnails based on the user's personality features and emotional state, and provides them to the user.

[1209] System Overview

[1210] The system analyzes user behavioral and emotional data to generate summaries and representative images of video content. User history data includes viewing history, search history, click history, and even emotional data from an emotional engine. Based on this data, the system identifies the user's interests, preferences, and emotional state.

[1211] Collection and analysis of user history and emotion data

[1212] The server collects users' historical behavioral data and emotional data from the database. This data includes the types of videos users watched, the keywords they searched for, the representative images they clicked on, and the emotional state they displayed while watching a particular video. The collected historical data and emotional data are analyzed and natural language processing (NLP) models and emotion analysis models are used to extract user personality features and emotional states.

[1213] Video content analysis

[1214] The server analyzes the video content, extracts each frame, and detects key scenes and key moments. Image recognition technology is used for the analysis to detect scene changes and specific events (e.g., interesting or moving scenes). These scenes are extracted as important cuts based on the user's personality and emotional features.

[1215] Video summarization and representative image generation

[1216] The server combines the user's personality features with the extracted key scenes from the video to generate candidate video summaries and representative images. At this stage, we use embedding similarity calculations to evaluate whether the generated candidates are of interest to the user. By also taking the user's emotional state into consideration, we generate the most visually and emotionally appealing summary and representative images.

[1217] Optimal summary and representative image presentation

[1218] The device receives the optimal video summary and representative image sent from the server, and then presents them to the user, allowing the user to instantly grasp the overview of the content being viewed.

[1219] Specific examples

[1220] If User B wants to watch a new comedy video, the system works as follows:

[1221] 1. The server collects user B's past history data and emotional data while watching, identifies that user B has a comedy-related preference, and confirms that user B smiles frequently while watching.

[1222] 2. The server analyzes frames from a new comedy video and extracts funny and touching moments.

[1223] 3. The server combines these scenes to generate a two-minute comedy summary video and a visually interesting representative image of a smiling face.

[1224] 4. The device presents these summaries and representative images to User B.

[1225] This allows User B to quickly check scenes that he or she is particularly interested in, improving the viewing experience. In addition, since emotional data is taken into consideration, satisfaction during viewing is likely to increase.

[1226] Prompt Sentence Examples

[1227] Example prompt for a generative AI model:

[1228] "Please explain the process of a system that extracts personality features and emotional features based on user log data and emotional data during viewing, automatically selects important scenes from video content, and generates the most appropriate video summary and thumbnail for the user."

[1229] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1230] Step 1:

[1231] The server collects user history data. The inputs are viewing history, search history, click history, and emotion data. The output is a set of collected history data.

[1232] Specific behavior:

[1233] The server queries the database to retrieve data on User B's past viewing content, search keywords, representative images clicked, and emotional states (e.g., facial expressions, voice) expressed while watching the video.

[1234] Step 2:

[1235] The server analyzes the collected historical data and extracts features related to the user's personality. The input is a set of historical data. The output is a set of features related to the personality.

[1236] Specific behavior:

[1237] The server uses natural language processing (NLP) and sentiment analysis models to extract features from viewing history, such as "comedy preference" or "frequent smiling while watching." For example, it uses Python's TensorFlow library to run machine learning models to extract features from complex datasets.

[1238] Step 3:

[1239] The server analyzes the video content and detects key scenes and key moments. The input is the video data to be analyzed and the user's personality features. The output is a set of candidate scene cuts and representative images.

[1240] Specific behavior:

[1241] The server uses OpenCV or similar image recognition technology to analyze video frames to identify scenes with a lot of laughter and smiling faces, as well as the rate of change and audio data in each frame to detect important events.

[1242] Step 4:

[1243] The server generates a video summary and candidate representative images based on the extracted scene cuts and representative images. The input is the set of scene cuts and representative images from the analysis results and the user's personality features. The output is a video summary and a set of candidate representative images.

[1244] Specific behavior:

[1245] The server creates a two-minute video summary focusing on interesting scenes, and uses embedding similarity calculations to evaluate whether it matches the user's preferences and select the most appealing summary and representative image.

[1246] Step 5:

[1247] The terminal receives the optimal video summary and representative image from the server and presents them to the user. The input is the video summary and representative image sent from the server. The output is the visual content to be presented to the user.

[1248] Specific behavior:

[1249] The device displays the summary video and representative image on the application's home screen or notification area. When User B opens the app, the summary video and representative image that is most likely to interest them are presented. When the user clicks on it, a two-minute summary video plays and the related representative image is also displayed.

[1250] (Application example 2)

[1251] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1252] In recent years, with the increasing volume of video content, it has become increasingly difficult for viewers to quickly find content that best suits their interests and emotional state. Furthermore, conventional systems cannot generate video summaries and thumbnails that fully take into account the user's personality and emotional state, resulting in a poor user experience.

[1253] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting user log data, means for analyzing the collected log data and extracting features related to the user's personality, means for extracting scene cuts and thumbnails from video content based on the features, means for collecting and analyzing user emotional data and identifying the user's emotional state, means for evaluating optimal video summary and thumbnail candidates based on the emotional state, and means for selecting optimal video summaries and thumbnails from the candidates and presenting them to the user. This makes it possible to provide video summaries and thumbnails that are tailored to the user's personality and emotional state.

[1254] "User log data" refers to data about a user's past behavior, such as their viewing history, search history, and click history.

[1255] "Features related to individuality" are characteristic information that indicates the user's interests and preferences.

[1256] A "scene cut" is an extracted section of an important scene or point in video content.

[1257] A "thumbnail" is a small image that visually represents a portion of the video content.

[1258] A "video summary" is content that extracts the main points and important scenes from video content so that they can be viewed in a short amount of time.

[1259] "Emotional data" refers to data that indicates the emotional state of a user analyzed from their facial expressions and behavior while watching a video.

[1260] The "emotional state" indicates the emotional state that the user feels while watching a video, and includes, for example, emotions such as joy, anger, sadness, and pleasure.

[1261] An "Embedding model" is a machine learning model that converts natural language and image data into vector format and calculates their meaning and characteristics.

[1262] An "evaluation method" is a method for determining how well a candidate video summary or thumbnail suits a user's interests and emotional state.

[1263] A "presenting means" is a means including a user interface function for displaying the selected optimal video summary or thumbnail to a user.

[1264] The present invention provides a system for providing optimal video summaries and thumbnails based on the user's personality and emotional state. The system comprises a plurality of processing means, and a server and a terminal operate in unison.

[1265] System Configuration

[1266] The server comprises the following processing means:

[1267] 1. Means of collecting user log data

[1268] 2. A method for analyzing log data and extracting features related to user personality

[1269] 3. A method for extracting scene cuts and thumbnails from video content based on features

[1270] 4. A means of collecting and analyzing user emotional data to identify the user's emotional state

[1271] 5. A means of evaluating candidate video summaries and thumbnails based on emotional state

[1272] 6. A means of selecting and presenting the best video summaries and thumbnails to users

[1273] Specific processing flow

[1274] 1. The data collection server collects behavioral data such as user viewing history, search history, click history, etc. While viewing, a camera is used to capture the user's facial expressions and collect them as emotional data.

[1275] 2. Using an emotion analysis engine (e.g., OpenFace or Facial Emotion Recognition Software), analyze the collected facial expression data and identify the user's emotional state (joy, anger, sadness, or happiness).

[1276] 3. The server uses a natural language processing (NLP) model (e.g., BERT, GPT-3) based on the collected log data to extract features that indicate the user's interests and preferences.

[1277] 4. Use image recognition techniques (e.g., YOLO, ResNet) to detect key scenes and key points (e.g., funny or moving scenes) from video content.

[1278] 5. Using the embedding model, we evaluate the generated video summaries and thumbnail candidates based on the user's personality features and emotional state. The evaluation is performed using the embedding similarity calculation.

[1279] 6. The terminal presents the user with the most appropriate video summaries and thumbnails sent from the server, allowing the user to quickly view content that matches their interests and emotional state.

[1280] Specific hardware and software used

[1281] Data collection servers: servers and cloud storage services

[1282] Emotion analysis engine: OpenFace, Facial Emotion Recognition Software

[1283] Image recognition technology: YOLO, ResNet

[1284] Natural Language Processing models: BERT, GPT-3

[1285] Embedding model: (used in conjunction with BERT and GPT-3)

[1286] Specific examples

[1287] For example, if User B wants to watch a new comedy video, the process goes like this:

[1288] 1. The data collection server collects user B's past viewing history and facial expression data while viewing.

[1289] 2. The emotion analysis engine analyzes the facial expression data and identifies User B's emotions while watching.

[1290] 3. The server analyzes past log data and determines that User B has a strong preference for comedy and smiles a lot while watching.

[1291] 4. Use image recognition technology to analyze new comedy videos and extract funny and touching scenes.

[1292] 5. The embedding model evaluates these scenes and generates the best summary video and thumbnail for user B.

[1293] 6. The device presents these summaries and thumbnails to User B.

[1294] Prompt Sentence Examples

[1295] Here are some example prompts to use as input to a generative AI model:

[1296] Based on the user's viewing history and sentiment data, generate a summary video and thumbnail that meets the following criteria:

[1297] 1. Video Summary: Comedy genre, 2 minutes, mainly funny scenes.

[1298] 2. Thumbnail: A smiling scene while the user is watching.

[1299] This allows the AI ​​model to generate optimal content that matches the user's preferences and emotions based on the specified conditions.

[1300] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1301] Step 1:

[1302] The server collects the user's viewing history, search history, click history, and emotional data while viewing. This data is stored in a database and used as the basis for processing. Specifically, it records the user's behavioral data as a log and uses a camera to capture their facial expressions while viewing. The input is the user's past behavioral data and facial expression data while viewing. The output is the log data and emotional data stored in the database.

[1303] Step 2:

[1304] The server analyzes the collected log data and emotion data. In this step, it uses a natural language processing (NLP) model (e.g., BERT, GPT-3) to extract features related to the user's personality (interests and preferences) from the log data. It also uses an emotion analysis engine (e.g., OpenFace) to extract the user's emotional state from facial expression data. The input is the log data and emotion data stored in the database. The output is the features of the personality and emotional state.

[1305] Step 3:

[1306] The server extracts scene cuts from the video content frame by frame and detects events of particular interest (e.g., funny or emotional scenes). In this step, image recognition techniques (e.g., YOLO, ResNet) are used to analyze the video frames. The input is the video content. The output is scene cuts related to interests and emotions.

[1307] Step 4:

[1308] The server combines the extracted scene cuts with the user's personality features and emotional state to generate candidate video summaries and thumbnails. In this step, an embedding model (e.g., BERT, GPT-3) is used to perform similarity calculations to select the video summaries and thumbnails that best fit the user's interests and emotional state. The input is scene cuts and user features. The output is candidate video summaries and thumbnails.

[1309] Step 5:

[1310] The server evaluates the generated video summaries and thumbnail candidates and selects the most suitable candidate. This evaluation is performed by scoring them based on the user's interests and emotional state. The input is the video summaries and thumbnail candidates. The output is the optimal video summary and thumbnail.

[1311] Step 6:

[1312] The terminal presents the optimal video summary and thumbnail sent from the server to the user, allowing the user to visually grasp and quickly select content that matches their interests and emotional state. The input is the optimal video summary and thumbnail, and the output is the content displayed in the user interface.

[1313] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1314] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1315] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1316] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1317] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1318] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1319] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1320] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1321] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1322] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1323] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1324] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1325] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1326] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1327] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1328] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1329] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1330] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1331] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1332] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1333] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1334] The following is further disclosed regarding the above embodiment.

[1335] (Claim 1)

[1336] A means of collecting user log data;

[1337] A means for analyzing the collected log data and extracting features related to the user's personality;

[1338] means for extracting scene cuts and thumbnails from video content based on the feature amounts;

[1339] means for combining the features with the extracted scene cuts and thumbnails to generate candidate video summaries and thumbnails tailored to the user's preferences;

[1340] means for selecting the most suitable video summary and thumbnail from the candidates and presenting them to the user;

[1341] A system including:

[1342] (Claim 2)

[1343] 10. The system of claim 1, further comprising means for collecting log data including user past behavior data, viewing history, search history, and click history.

[1344] (Claim 3)

[1345] 2. The system according to claim 1, further comprising means for analyzing log data using a natural language processing model and extracting features indicative of user interests.

[1346] (Claim 4)

[1347] 10. The system of claim 1, further comprising means for analyzing video frames and detecting key scenes or points in each frame.

[1348] (Claim 5)

[1349] The system of claim 1, further comprising: means for combining user features with video scene cut and thumbnail features to select the best candidate using an embedding similarity calculation.

[1350] "Example 1"

[1351] (Claim 1)

[1352] A means of collecting user log data;

[1353] A means for analyzing the collected log data using a natural language processing model and a machine learning model to extract features related to the user's personality;

[1354] means for analyzing the video content frame by frame and extracting key scenes and thumbnails based on the feature amounts;

[1355] a means for generating candidate video summaries and thumbnails that are tailored to the user's individuality by combining the features with the extracted main scenes and thumbnails;

[1356] A means for evaluating the generated candidates using an embedding similarity calculation, selecting the most suitable video summary and thumbnail, and presenting them to the user;

[1357] A system including:

[1358] (Claim 2)

[1359] 10. The system of claim 1, further comprising means for collecting log data including user past behavior data, viewing history, search history, and click history.

[1360] (Claim 3)

[1361] The system according to claim 1, further comprising means for analyzing log data using a natural language processing model and a machine learning model, and extracting features indicative of user interests.

[1362] "Application Example 1"

[1363] (Claim 1)

[1364] A means of collecting user log data;

[1365] A means for analyzing the collected log data and extracting features related to the user's personality;

[1366] means for extracting scene cuts and thumbnails from video content based on the feature amounts;

[1367] means for combining the features with the extracted scene cuts and thumbnails to generate candidate video summaries and thumbnails tailored to the user's preferences;

[1368] means for selecting the most suitable video summary and thumbnail from the candidates and presenting them to the user;

[1369] a means for collecting real-time data;

[1370] means for using the framework to present content to a personal mobile device;

[1371] means for generating video summaries and thumbnails using a generative AI model;

[1372] A system including:

[1373] (Claim 2)

[1374] 10. The system of claim 1, further comprising means for collecting log data including user past behavior data, viewing history, search history, and click history.

[1375] (Claim 3)

[1376] 2. The system according to claim 1, further comprising means for analyzing log data using a natural language processing model and extracting features indicative of user interests.

[1377] "Example 2: Combining Emotion Engines"

[1378] (Claim 1)

[1379] a means for collecting user history data;

[1380] A means for analyzing the collected historical data and extracting characteristics related to the user's personality;

[1381] means for extracting scene cuts and representative images from video content based on the features;

[1382] means for combining the features with the extracted scene cuts and representative images to generate candidate video summaries and representative images that are personalized to the user;

[1383] means for selecting the best video summary and representative image from the candidates and presenting them to a user;

[1384] A system including:

[1385] (Claim 2)

[1386] 10. The system of claim 1, further comprising means for collecting historical data including user past activity data, viewing history, search history, and click history.

[1387] (Claim 3)

[1388] 10. The system of claim 1, further comprising means for analyzing the historical data using a natural language processing model to extract features indicative of the user's interests.

[1389] "Application example 2 when combining emotion engines"

[1390] (Claim 1)

[1391] A means of collecting user log data;

[1392] A means for analyzing the collected log data and extracting features related to the user's personality;

[1393] means for extracting scene cuts and thumbnails from video content based on the feature amounts;

[1394] means for combining the features with the extracted scene cuts and thumbnails to generate candidate video summaries and thumbnails tailored to the user's preferences;

[1395] A means for collecting and analyzing user emotional data to identify the user's emotional state;

[1396] means for evaluating optimal video summary and thumbnail candidates based on said emotional state;

[1397] means for selecting the most suitable video summary and thumbnail from the candidates and presenting them to the user;

[1398] A system including:

[1399] (Claim 2)

[1400] The system of claim 1, further comprising: means for collecting log data including a user's past behavior data, viewing history, search history, and click history; and means for collecting emotion data during viewing.

[1401] (Claim 3)

[1402] The system of claim 1, further comprising: means for analyzing log data using a natural language processing model and extracting features indicative of a user's interests and concerns; and means for analyzing emotion data using an emotion analysis model and extracting the user's emotional state. [Explanation of symbols]

[1403] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting user log data; A means for analyzing the collected log data and extracting features related to the user's personality; means for extracting scene cuts and thumbnails from video content based on the feature amounts; means for generating candidate video summaries and thumbnails tailored to a user's individuality by combining the features with the extracted scene cuts and thumbnails; means for selecting the most suitable video summary and thumbnail from the candidates and presenting them to the user; A system including:

2. The system of claim 1 , further comprising means for collecting log data including user's past behavior data, viewing history, search history, and click history.

3. The system according to claim 1 , further comprising: means for analyzing the log data using a natural language processing model and extracting features indicative of the user's interests and concerns.

4. 10. The system of claim 1, further comprising means for analyzing video frames and detecting key scenes or points in each frame.

5. The system according to claim 1 , further comprising means for selecting the best candidate by combining the user's features with the features of the video scene cuts and thumbnails and using an embedding similarity calculation.

Images