System

Abstract

A system is provided.SOLUTION: The system includes a means for acquiring moving image data, a means for analyzing the acquired moving image data and specifying an important part, a means for generating a summary version of the moving image based on the specified important part, and a means for distributing the generated summary version moving image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Editing and summarizing videos of product presentations held both inside and outside the company requires the same amount of time and effort as a real-time presentation, resulting in a significant time and effort. Furthermore, while it is important to predict audience reactions after the presentation and plan for the future, doing this manually is extremely difficult. The objective of the present invention is to solve these problems and provide a system for efficient and effective video editing and reaction prediction. [Means for solving the problem]

[0005] The present invention provides a system including a means for acquiring video data, a means for analyzing the acquired video data to identify important portions, a means for generating a video summary based on the identified important portions, and a means for distributing the generated summary video. The system also includes a means for analyzing audio data and summarizing key content, enabling a concise summary of the key content of a briefing. Furthermore, the system includes a means for collecting and analyzing viewing data and a means for predicting viewer reactions based on the collected data, enabling viewer reactions to be predicted in advance and plans to be made based on the predictions.

[0006] "Video Data" means digital files containing visual and audio information such as information sessions and other presentations.

[0007] "Means for acquiring video data" refers to the processes and technical equipment for collecting videos of information sessions, etc. and importing them into the system.

[0008] "Means for analysis" refers to software or a set of algorithms that process input video or audio data and identify key points or information.

[0009] "Key parts" are parts of video or audio that are particularly useful to the viewer and aid comprehension.

[0010] A "summarization method" is a technique or process for extracting key parts and creating a new video in a shorter format than the original video.

[0011] "Means of distribution" refers to the mechanism by which the generated video and summary text are provided to viewers and specific devices via the Internet or other communication means.

[0012] "Audio data" refers to files that store audio information spoken during briefings or presentations in digital format.

[0013] "Means for analyzing audio data" refers to the technology and algorithms used to convert recorded audio data into text and understand its content.

[0014] "Viewing data" is information about how viewers viewed the briefing video, including viewing time, number of views, viewer attributes, etc.

[0015] "Means of collecting viewing data" refers to the technologies and processes used to track viewers' viewing behavior and viewing history and store this as data.

[0016] A "means for predicting viewer responses" is an algorithm that statistically estimates what actions viewers will take in the future (e.g., inquiries or applications) based on collected viewing data. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] MODE FOR CARRYING OUT THE INVENTION

[0039] The present invention provides a system for efficiently summarizing video data from a product presentation and predicting viewer reactions. The system is composed of a server, a terminal, and a user. Each processing step is described in detail below.

[0040] System configuration

[0041] 1. The server provides an upload interface for acquiring video data. Users upload the video of the product presentation to the server through this interface.

[0042] 2. The server splits the uploaded video data into frames and processes them as individual image data, making each frame in the video ready for analysis.

[0043] 3. The server analyzes the video and audio data using generative AI and speech recognition technology. As a result of the analysis, important parts are scored and identified. These important parts are then used to generate a condensed version of the video.

[0044] 4. The server creates a video summary based on the identified key parts. It generates a shortened video by stitching together key scenes for easy user comprehension. This process also includes adding transition effects and subtitles.

[0045] 5. The server delivers the generated summary video to the device. This delivery is done via the Internet, and the user can watch or download the summary video by clicking on a link.

[0046] Creating a text summary

[0047] 1. A user can request a summary of the briefing video. If the request is accepted, the server extracts the audio data from the video.

[0048] 2. The server converts the audio data into text using speech recognition software, which is then processed with natural language processing techniques to remove noise and extract key points from the audio data.

[0049] 3. The server uses generative AI to summarize the text data and generate a text that concisely expresses the main content. This summary text can be used as a report or marketing material.

[0050] Predicting audience reactions

[0051] 1. The server collects viewing data, including viewer attributes, viewing time, number of plays, number of clicks, etc. This data is used to analyze viewer behavior patterns.

[0052] 2. The server uses generative AI to analyze the collected viewing data and learn viewer behavior patterns and trends. By comparing and analyzing past and current data, a predictive model of viewer reactions is generated.

[0053] 3. The server uses the predictive model to predict future inquiries and applications, and displays this forecast data in a user-accessible dashboard.

[0054] Specific examples

[0055] 1. The user uploads a video of a new product briefing to the server.

[0056] 2. The server analyzes the video data, identifies important parts, and generates a 5-minute video summary that is then delivered to the user.

[0057] 3. The user clicks on the download link to watch the generated summary video.

[0058] 4. When the user requests a summary, the server converts the audio data into text, summarizes the key points, and provides the text in a format that the user can download.

[0059] 5. The server collects viewing data and analyzes viewer trends. Based on this, it predicts the number of inquiries and applications and notifies users.

[0060] In this way, the present invention realizes efficient and effective video editing of product presentations and prediction of audience reactions.

[0061] The processing flow will be explained below.

[0062] Creating a summary video

[0063] Step 1:

[0064] The user uploads the video file of the briefing to the server by selecting the video file through the user interface and uploading it to the specified folder.

[0065] Step 2:

[0066] The server divides the uploaded video data into frames, dividing the video file into frames at regular intervals and generating individual image data.

[0067] Step 3:

[0068] The server uses generative AI to analyze the video data. It uses an AI algorithm to score important scenes from each frame, particularly by analyzing the speaker's facial expressions, movements, and vocal rhythm to determine their importance.

[0069] Step 4:

[0070] The server generates a summary of the video based on the scored frames, and then extracts and stitches together the frames with the highest scores to create a short summary video.

[0071] Step 5:

[0072] The server delivers the abridged video to the device, saves the completed video file, and provides a download link to the user's device, which the user can click to watch or download the abridged video.

[0073] Creating a summary of the content

[0074] Step 1:

[0075] A user requests the generation of a summary text for an uploaded video by selecting the summary generation option through a dedicated interface.

[0076] Step 2:

[0077] The server extracts the audio data from the briefing, separates it from the video, and passes it to the speech recognition engine.

[0078] Step 3:

[0079] The server uses speech recognition software to convert speech to text, converts the speech data into text data, and uses natural language processing techniques to remove noise.

[0080] Step 4:

[0081] The server uses generative AI to summarize the generated text data, extracting key keywords and phrases and summarizing them into easy-to-understand sentences.

[0082] Step 5:

[0083] The server delivers the summary text to the user's device, formats the summary results, and displays them on the user's dashboard or provides a downloadable link.

[0084] Predicting audience reactions

[0085] Step 1:

[0086] The server collects viewing data, such as the viewer's access history, viewing time, number of plays, and number of clicks, and stores this data in real time.

[0087] Step 2:

[0088] The server analyzes the collected viewing data, using statistical methods and machine learning algorithms to analyze viewer behavior patterns.

[0089] Step 3:

[0090] The server predicts viewer reactions based on viewing data and past reaction data, for example by building a model to predict the number of inquiries and applications.

[0091] Step 4:

[0092] The server notifies the user of the prediction results. The generated prediction data is displayed visually and provided to the user's dashboard. The user can then use this information to plan their next strategy.

[0093] As described above, the system of the present invention efficiently generates a summary video and text of a briefing session and predicts viewer reactions, thereby significantly improving the efficiency of users' work.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] Traditional video data from product presentations is long and consumes viewers' time, often burying important information. Furthermore, there is a lack of systematic methods for predicting viewer reactions, making it difficult to improve engagement. Furthermore, there is a need for an efficient method for quickly and accurately generating abridged video and summary text.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes means for acquiring video data, means for dividing the acquired video data into frames, means for analyzing the divided frames and identifying important portions, means for converting audio data into text and extracting important content, means for generating a summary of the video based on the identified important portions and the extracted important content, means for adding transfer effects and subtitles to the generated summary video, means for distributing the generated summary video, means for collecting and analyzing viewing data, and means for predicting viewer responses based on the collected data. This makes it possible to quickly extract important information from long videos and efficiently generate summary videos and summary text. Furthermore, predicting viewer responses can improve engagement and optimize marketing strategies.

[0099] "Video data" refers to video content such as product presentations, and is data that includes visual and audio information.

[0100] The "means for acquiring" is a function including an interface and a protocol for transmitting video data from a user to a server.

[0101] A "frame" is an individual still image that makes up a video.

[0102] The "dividing means" is a function that performs processing to cut out video data in time units and save them as individual frames.

[0103] The "analysis means" refers to a technique for extracting important information from divided frames or audio data, and performing evaluation and scoring.

[0104] "Audio data" refers to audio information contained in a video, including narration, conversation, music, and the like.

[0105] "Means for converting to text" refers to speech recognition technology for recognizing voice data as character information and converting it into text format.

[0106] "Extraction means" refers to techniques for finding important content or scenes from the analyzed text or video frames.

[0107] The "means for generating a summary version" is a function that connects the extracted important parts together to create a video that is shorter and easier to understand.

[0108] "Transfer effect" is a technique for adding visual effects when switching scenes.

[0109] "Subtitles" are the display of dialogue and explanations in a video as text.

[0110] The "distribution means" refers to a technique for providing the generated summarized video to users via a communication means such as the Internet.

[0111] "Viewing data" is information relating to the viewer's attributes, viewing time, number of plays, number of clicks, and other behavior.

[0112] "Means of collection" refers to tracking technologies and analytical tools used to automatically collect viewing data.

[0113] "Viewer response" refers to the behavioral patterns and emotional reactions of viewers when they watch a video.

[0114] "Predictive methods" are analytical techniques that use collected viewing data to predict future viewer behavior and reactions.

[0115] The present invention relates to a system for efficiently summarizing video data from a product presentation and predicting viewer reactions. The system is composed of a server, terminals, and users.

[0116] The server provides an interface for users to obtain video data. When a user uploads a video file to the server, the server divides the video data into frames. This process uses video analysis tools such as FFmpeg and OpenCV.

[0117] The server analyzes the segmented frames using generative AI models and speech recognition technology. For example, it uses the Google® Cloud Video Intelligence API or Amazon Transcribe to convert the content and audio data of the video frames into text. Based on the analysis, important parts of the video are identified. An example of a prompt is, "What important events are contained in this video frame?"

[0118] The server then generates a condensed version of the video based on the identified key parts. During this process, video editing software such as Adobe Premiere Pro or iMovie is used to add transition effects and subtitles to the shortened video. The condensed version is then distributed to devices via the Internet, and users can watch or download the condensed version by clicking a link.

[0119] When a user requests a text summary, the server converts the audio data into text using the Google Speech-to-Text API or Amazon Transcribe. It then uses a generative AI model (e.g., GPT-4 (registered trademark)) to summarize the key content. An example prompt at this stage is "Please summarize the following text: [extracted text]."

[0120] In addition, the server collects viewing data (viewer attributes, viewing time, number of views, number of clicks, etc.) and analyzes viewer behavior patterns using tools such as Google Analytics and Mixpanel. A generative AI model is used to analyze the data and predict viewer responses. This allows users to predict future inquiries and applications, and the results can be viewed on a dashboard.

[0121] As a concrete example, a user uploads a video of a new product briefing to a server, which analyzes the video data to generate a 5-minute summary of the video and delivers it to the user. The user clicks a link to watch the video and then requests a summary text. The server converts the audio data into text and generates a summary. Finally, the server collects viewing data and predicts the viewer's reaction.

[0122] In this way, the present invention realizes efficient and effective video editing of product presentations and prediction of audience reactions.

[0123] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0124] System program processing flow

[0125] Step 1: Upload video data

[0126] The server provides an interface for obtaining video data from the user. The user uses this interface to upload video data from their device to the server. The input in this step is a video file selected by the user, and the output is a video file stored on the server.

[0127] Specifically, the server stores the video file in cloud storage and registers the file's metadata (file size, format, resolution, etc.) in a database.

[0128] Step 2: Split the video data into frames

[0129] The server splits the uploaded video data into frames. The input of this step is the video file stored on the server, and the output is the individual frame image files.

[0130] Specifically, the server uses FFmpeg to split the video into fixed frame intervals and save each frame as an image file, generating still images that can be analyzed.

[0131] Step 3: Analyze video and audio data

[0132] The server analyzes the video and audio data using generative AI models and speech recognition technology. The input for this step is the frame image files and audio data, and the output is data identifying key parts.

[0133] Specifically, the server uses the Google Cloud Video Intelligence API to analyze the content of each frame (e.g., objects, actions, scene changes), and then uses Amazon Transcribe to convert the audio data into text. For example, a prompt sentence like "What significant events does this video frame contain?" is input to the generative AI model.

[0134] Step 4: Generate a condensed video

[0135] The server generates a digest version of the video based on the identified important parts. The input of this step is the data of the important parts, and the output is a digest version of the video file.

[0136] Specifically, the server uses Adobe Premiere Pro scripting to stitch together key frames, add transfer effects and subtitles, and generate a condensed video, which is then stored on the server.

[0137] Step 5: Publish a condensed version of your video

[0138] The server then distributes the generated digested video to the user's device via the Internet. The user then clicks on the provided link to watch or download the video. The input of this step is the digested video file, and the output is a link that the user can watch.

[0139] Specifically, the server uploads the summarized video to a cloud storage service and sends the user an email notification with a download link.

[0140] Step 6: Creating a text summary

[0141] When a user requests a text summary, the server converts the audio data into text and uses a generative AI model to create the summary. The input for this step is the audio data, and the output is the summarized text.

[0142] Specifically, the server sends the voice data to the Google Speech-to-Text API, converts it into text, and then inputs the prompt "Please summarize the following text: [Extracted text]" into the generative AI model to generate a summary text.

[0143] Step 7: Predict audience reaction

[0144] The server collects viewing data and uses a generative AI model to predict viewer reactions. The input of this step is the viewing data, and the output is the predicted results.

[0145] Specifically, the server collects viewing data (viewer attributes, viewing time, number of plays, number of clicks, etc.) using Google Analytics, inputs it into a generative AI model (for example, a custom model using TENSORFLOW (registered trademark)) and has it learn the viewer's behavioral patterns. Based on this, the number of inquiries and applications is predicted and notified to the user.

[0146] In this way, the system efficiently and effectively edits videos of product presentations and predicts audience reactions.

[0147] (Application example 1)

[0148] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0149] In the modern advertising industry, generating effective abridged advertising videos that attract viewers' attention is a key challenge. However, long advertising videos have difficulty sustaining viewers' attention, leading to lower viewer ratings. Furthermore, to maximize the effectiveness of advertising, it is necessary to predict viewer responses and select the optimal delivery timing, but this is extremely difficult to do manually.

[0150] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0151] In this invention, the server includes means for acquiring video data, means for analyzing the acquired video data and identifying important parts, means for generating a summary of the video based on the identified important parts, means for distributing the generated summary video, means for analyzing the advertising video received from the user and extracting important parts to generate a summary advertising video, and means for predicting viewer reactions based on data processing and proposing optimal distribution timing, thereby enabling the generation of effective summary advertising videos and optimal advertising distribution taking viewer reactions into consideration.

[0152] The "means for acquiring video data" is a function for users to upload multimedia data such as advertising videos and product explanation videos to the server.

[0153] "Means for analyzing video data and identifying important parts" refers to a function in which the server analyzes uploaded video data frame by frame and automatically recognizes scenes that are visually and audibly important.

[0154] The "means for generating a summary version of a video based on identified important parts" is a function that automatically generates a summary version of a video that conveys the main content in a short amount of time by linking identified important scenes.

[0155] The "means for distributing the generated summarized video" is a function for distributing the generated summarized video to users and viewers via the Internet.

[0156] "Means for analyzing advertising videos received from users, extracting important parts, and generating summarized advertising videos" refers to a function that analyzes advertising videos provided by advertisers, extracts important visual and auditory scenes, and generates effective summarized advertising videos.

[0157] "Means of predicting viewer reactions based on data processing and proposing optimal distribution timing" refers to a function that collects and analyzes viewing data, predicts future viewer reactions based on viewer behavior patterns, and proposes optimal advertising distribution timing.

[0158] MODE FOR CARRYING OUT THE INVENTION

[0159] The present invention relates to a system for efficiently summarizing video data and predicting viewer reactions. DETAILED DESCRIPTION OF THE INVENTION The present invention will be described below with reference to specific examples.

[0160] Overall system configuration

[0161] This system consists of a server that acquires and analyzes video data, a terminal that distributes the generated summarized video, and a user who uploads the video data. The main elements are as follows:

[0162] 1. How to obtain video data

[0163] Users use an interface to upload video data to the server. For example, they can upload advertising videos or product explanation videos via a web browser or smartphone application.

[0164] 2. A method for analyzing video data and identifying important parts

[0165] The server analyzes the uploaded video data. Specifically, it uses OpenCV to divide the video into frames and identify important visual and auditory scenes. It also uses speech recognition technology to extract important parts from the audio data.

[0166] 3. A method for generating a summary of the video based on the identified key parts

[0167] The server then concatenates key scenes based on the analysis results and generates a condensed video using MoviePy, which can be enhanced with visual effects to make it more visually appealing.

[0168] 4. Means of distributing the generated summarized video

[0169] The summarized video generated by the server is delivered to the terminal via the Internet, and the user can view or download the summarized video from the provided link.

[0170] 5. Means for analyzing the advertisement video received from the user, extracting important parts, and generating a summary advertisement video

[0171] The server particularly analyzes the advertisement video received from the advertiser, extracts important parts that will attract the viewer's attention, and generates an effective summary advertisement video.

[0172] 6. A method for predicting viewer reactions based on data processing and proposing optimal distribution timing

[0173] The server collects viewing data, analyzes viewer behavior patterns based on the collected data, and uses a generative AI model to predict these patterns and suggest optimal distribution timing and channels.

[0174] Hardware and Software

[0175] Hardware: General PCs, servers, smartphones

[0176] Software: Python, OpenCV, MoviePy, transformers, OpenAI® API

[0177] Operational procedures and examples

[0178] 1. Upload a video

[0179] Users log in to the system from their smartphones or PCs and upload their own advertising videos. For example, advertisers may upload promotional videos for new products.

[0180] 2. Video analysis and summary generation

[0181] The server analyzes the uploaded video and identifies important scenes using frame analysis with OpenCV and speech recognition technology, then concatenates the identified important parts and generates a summarized video using MoviePy.

[0182] 3. Summary Text Generation

[0183] The server also analyzes the audio data from the video and converts it into text using speech recognition technology, then generates a summary text using a generative AI model.

[0184] 4. Predictions and Recommendations

[0185] The server collects viewing data and uses a generative AI model to predict viewer reactions, and then suggests optimal ad delivery timing based on this.

[0186] Example prompts for generative AI models

[0187] Extract key parts from promotional videos for new products and generate visually impactful summary videos. Convert the video audio into text and summarize the key points. Analyze viewing data to predict the effectiveness of the next advertising campaign.

[0188] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0189] Step 1:

[0190] Users log in to the system using a smartphone or PC and upload their own advertising video to the server. The advertising video file is provided as input. The server receives this file and stores it in a database. As output, a message indicating that the file has been saved is generated. Specifically, a file selection dialog is displayed and the selected video file is sent to the server.

[0191] Step 2:

[0192] The server uses OpenCV to split the uploaded video file into frames. The video file saved in step 1 is provided as input. Data processing involves splitting the video into frames based on timestamps and saving each frame image as a list. The output is a set of frame images in list format. Specifically, the server reads the video file and analyzes each frame sequentially.

[0193] Step 3:

[0194] The server converts the video's audio data into text using speech recognition technology (for example, the speech recognition model from the transformers package). The video file saved in step 1 is provided as input. Data calculations involve extracting the audio data and converting it into text. The output is text data generated from the audio data. Specifically, the audio portion of the video file is analyzed and transcribed using a speech recognition engine.

[0195] Step 4:

[0196] The server uses the generative AI model to summarize the text data and extract key points. The text data generated in step 3 is provided as input. The data is calculated by summarizing the text data based on natural language processing technology. The output is the summarized text data. Specifically, the text data is input into the generative AI model, and a summary is generated.

[0197] Step 5:

[0198] The server identifies important scenes based on the image data and audio analysis results for each frame. The input is the frame image data from step 2 and the audio analysis results from step 3. The data calculation involves scoring and selecting important scenes based on visual and auditory features. The output is a list of important scenes. Specifically, the analysis results for each frame are evaluated using a scoring algorithm.

[0199] Step 6:

[0200] The server concatenates key scenes to generate a condensed video. The input is the list of key scenes identified in step 5. The data is processed by concatenating these scenes in order and combining them into a single short video. The output is a condensed video file. Specifically, MoviePy is used to edit the key scenes and combine them into a single video file.

[0201] Step 7:

[0202] The server distributes the generated summarized video to users and viewers. The input is the summarized video file generated in step 6. The data calculation involves uploading the video file and generating a link. The output is the generation of a viewing link, which is notified to the user. Specifically, the operation involves uploading the video using a video hosting service and providing a viewing link.

[0203] Step 8:

[0204] The server collects viewing data and analyzes viewer behavior patterns. Viewing data such as the number of clicks on viewing links and viewing time is provided as input. Data calculation involves compiling and analyzing the viewing data. The output is analysis results showing viewer behavior patterns. Specifically, the server uses a viewing tracking system to collect data and perform statistical analysis.

[0205] Step 9:

[0206] A generative AI model is used to predict viewer reactions based on viewing data, and the optimal timing for ad delivery is proposed. The input is the viewing data analysis results from step 8. Data calculation involves analyzing behavioral patterns and applying a predictive model. The output is a proposal for the optimal timing for ad delivery. Specifically, the predictive algorithm from the generative AI model is applied to generate the proposed results.

[0207] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0208] MODE FOR CARRYING OUT THE INVENTION

[0209] The present invention is a system for efficiently summarizing video data from product presentations and predicting viewer reactions. This system consists of a server, a terminal, and a user. By combining it with an emotion engine, the system can recognize viewer emotions and identify the importance of content with high accuracy based on those emotions. Each processing step is explained in detail below.

[0210] System configuration

[0211] 1. The server provides an upload interface for acquiring video data. Users upload the video of the product presentation to the server through this interface.

[0212] 2. The server splits the uploaded video data into frames and processes them as individual image data, making each frame in the video ready for analysis.

[0213] 3. The server analyzes the video and audio data using generative AI and speech recognition technology. As a result of the analysis, important parts are scored and identified. These important parts are then used to generate a condensed version of the video.

[0214] 4. The server uses an emotion engine to analyze the viewer's emotional state. Based on the viewer's facial expression and voice data collected during viewing, the server recognizes the viewer's emotions and reflects this emotional data in the video analysis. For example, scenes that the viewer is particularly interested in are given a higher importance rating.

[0215] 5. The server generates a video summary based on the identified key parts and emotional data. It stitches together key scenes to make it easier for users to understand, and generates a shortened video that reflects the viewer's emotional state. This process also includes adding transition effects and subtitles.

[0216] 6. The server delivers the generated summary video to the device. This delivery is done via the Internet, and the user can watch or download the summary video by clicking on a link.

[0217] Creating a text summary

[0218] 1. A user can request a summary of the briefing video. If the request is accepted, the server extracts the audio data from the video.

[0219] 2. The server converts the audio data into text using speech recognition software, which is then processed with natural language processing techniques to remove noise and extract key points from the audio data.

[0220] 3. The server uses generative AI to summarize the text data and generate a text that concisely expresses the main content. This summary text can be used as a report or marketing material.

[0221] Predicting audience reactions

[0222] 1. The server collects viewing data, including viewer attributes, viewing time, number of plays, number of clicks, emotional data, etc. This data is used to analyze viewer behavior patterns.

[0223] 2. The server uses generative AI to analyze the collected viewing data and learn viewer behavior patterns and trends. By comparing and analyzing past and current data, a predictive model of viewer reactions is generated.

[0224] 3. The server uses the predictive model to predict future inquiries and applications, and displays this forecast data in a user-accessible dashboard.

[0225] Specific examples

[0226] 1. The user uploads a video of a new product briefing to the server.

[0227] 2. The server analyzes the video data and uses an emotion engine to analyze the viewer's emotional state. Scenes that the viewer particularly enjoyed are identified as important parts with particularly high scores.

[0228] 3. The server generates a five-minute video summary based on these key scenes, edits it to maximize viewer interest, and delivers the video to the user.

[0229] 4. The user clicks on the download link to watch the generated summary video.

[0230] 5. When the user requests a summary, the server converts the audio data into text, summarizes the key points, and provides the text in a format that the user can download.

[0231] 6. The server collects viewing data and analyzes the viewer's behavioral patterns and emotional state. Based on this, it predicts the number of inquiries and applications and notifies the user.

[0232] In this way, the present invention efficiently generates summary video and text of a briefing session and predicts viewer reactions, greatly improving the efficiency of users' work. Furthermore, by combining it with an emotion engine, it becomes possible to provide content that reflects the viewer's emotional state, thereby drawing out their interest with greater accuracy.

[0233] The processing flow will be explained below.

[0234] Creating a summary video

[0235] Step 1:

[0236] The user uploads the video file of the briefing to the server by selecting the video file through the user interface and uploading it to the specified folder.

[0237] Step 2:

[0238] The server divides the uploaded video data into frames, dividing the video file into frames at regular intervals and generating individual image data.

[0239] Step 3:

[0240] The server uses generative AI to analyze the video data. It uses an AI algorithm to score important scenes from each frame, particularly by analyzing the speaker's facial expressions, movements, and vocal rhythm to determine their importance.

[0241] Step 4:

[0242] The server uses an emotion engine to analyze the viewer's emotional state, acquiring facial expression and voice data and recognizing emotions in real time.

[0243] Step 5:

[0244] Based on the results of the emotion engine, the server re-scores frames that viewers showed particular interest in as high scores, allowing the emotion data to be reflected in the selection of important scenes in the video.

[0245] Step 6:

[0246] The server generates a video summary based on the scored frames. High-scoring frames are extracted and stitched together to create a short video summary.

[0247] Step 7:

[0248] The server delivers the abridged video to the device, saves the completed video file, and provides a download link to the user's device, which the user can click to watch or download the abridged video.

[0249] Creating a summary of the content

[0250] Step 1:

[0251] A user requests the generation of a summary text for an uploaded video by selecting the summary generation option through a dedicated interface.

[0252] Step 2:

[0253] The server extracts the audio data from the briefing, separates it from the video, and passes it to the speech recognition engine.

[0254] Step 3:

[0255] The server uses speech recognition software to convert speech to text, converts the speech data into text data, and uses natural language processing techniques to remove noise.

[0256] Step 4:

[0257] The server uses generative AI to summarize the generated text data, extracting key keywords and phrases and summarizing them into easy-to-understand sentences.

[0258] Step 5:

[0259] The server delivers the summary text to the user's device, formats the summary results, and displays them on the user's dashboard or provides a downloadable link.

[0260] Predicting audience reactions

[0261] Step 1:

[0262] The server collects viewing data, such as the viewer's access history, viewing time, number of plays, number of clicks, and emotional data, and stores this data in real time.

[0263] Step 2:

[0264] The server analyzes the collected viewing data, using statistical methods and machine learning algorithms to analyze viewer behavior patterns.

[0265] Step 3:

[0266] The server predicts viewer reactions based on viewing data and past reaction data, for example by building a model to predict the number of inquiries and applications.

[0267] Step 4:

[0268] The server notifies the user of the prediction results. The generated prediction data is displayed visually and provided to the user's dashboard. The user can then use this information to plan their next strategy.

[0269] As described above, the system of the present invention efficiently generates summary videos and texts of briefing sessions and predicts viewer reactions, greatly improving the efficiency of users' work. Furthermore, by combining it with an emotion engine, it becomes possible to provide content that reflects the viewer's emotional state, thereby more accurately attracting viewer interest.

[0270] Example 2

[0271] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0272] Conventional video summarization systems have difficulty accurately identifying important parts of a video, and have been unable to generate summaries that take into account the viewer's emotions. It has also been difficult to accurately predict viewer reactions. This has led to issues such as users being unable to provide content that piques the viewer's interest, making it difficult to carry out effective marketing and presentations.

[0273] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0274] In this invention, the server includes means for acquiring video data, means for dividing the acquired video data into frames, means for analyzing the divided frame data and identifying important portions, means for collecting and analyzing viewer emotion data, means for generating a summary of the video based on the identified important portions and the viewer emotion data, means for adding transfer effects and subtitles to the generated summary video, and means for delivering the generated summary video. This makes it possible to automatically extract important scenes that reflect the viewer's emotional state, generate highly accurate video summaries, and predict viewer reactions.

[0275] "Video data" refers to data in a file format that continuously records visual information and is played back on a computer or electronic device.

[0276] A "frame" is an individual still image that constitutes video data, and is recognized as a video when it is displayed continuously at regular intervals of seconds.

[0277] "Analysis" is the process of analyzing input data in detail and extracting useful information based on that analysis.

[0278] An "important portion" is a scene or element that is considered to be particularly important within the video data, and is a portion that contains information that is highly valuable to the viewer.

[0279] "Viewer emotional data" refers to data that indicates the emotional state of a viewer that can be obtained from facial expressions, voice, behavior, etc. that the viewer displays while watching a video.

[0280] A "summary video" is a video that has been reconstructed in a shortened format by extracting important parts from the original video data.

[0281] A "transition effect" is a visual effect that creates a smooth transition between different scenes or frames.

[0282] "Subtitles" are textual information displayed to supplement the visual content of a video, providing explanations and translations to viewers.

[0283] "Distribution" is the process of transmitting created content to users over the Internet or a communication network, making it accessible to them.

[0284] "Voice data" refers to data in a file format that records a voice signal, and is the subject of voice recognition.

[0285] "Viewing data" refers to data such as the viewer's behavior and attributes when viewing video content, viewing time, number of plays, and number of clicks.

[0286] A "generative AI model" is an artificial intelligence algorithm or software that learns from large amounts of data and generates appropriate outputs based on the input data.

[0287] This invention is a system for efficiently summarizing video data, such as product presentations, and predicting viewer reactions. This system is composed of a server, terminals, and users, and by combining an emotion engine, it is possible to recognize viewer emotions and identify the importance of content based on those emotions.

[0288] Server Configuration

[0289] The server has the following main functions:

[0290] 1. Acquiring video data:

[0291] The server provides an interface for users to upload video data. Specifically, files can be uploaded using a web page or an application.

[0292] 2. Video data frame division:

[0293] The server uses video processing software such as FFmpeg to split the uploaded video data into frames, which makes later analysis easier.

[0294] 3. Video and audio data analysis:

[0295] The server analyzes the video and audio data using a generative AI model and speech recognition technology. In this step, a generative AI model (e.g., OpenAI's GPT model) is used to understand the text and images, and speech recognition technology such as Google Speech-to-Text is used to convert the speech to text.

[0296] 4. Identify the important parts:

[0297] The server identifies important parts of the video based on the analysis results, for example, by using information scored by a generative AI model to extract particularly important scenes.

[0298] 5. Emotional Data Analysis:

[0299] The server collects viewers' viewing data (facial expressions and voice data) and analyzes it using an emotion engine (e.g., Amazon Rekognition or Microsoft® Azure® emotion recognition API). Scenes that viewers find particularly interesting are given high marks.

[0300] 6. Generate a condensed video:

[0301] The server generates a condensed version of the video based on the identified key parts and viewer sentiment data. It uses Adobe Premiere Pro APIs and other tools to stitch together key scenes to create a shortened video. It also adds transition effects and subtitles.

[0302] 7. Video Distribution:

[0303] The server generates a URL link to deliver the digested video, and users can view the digest by clicking on this link.

[0304] Specific examples

[0305] Uploading videos

[0306] The user uploads a video of a new product briefing to the server.

[0307] Video and audio analysis

[0308] The server divides the uploaded video data into frames using Ffmpeg and analyzes the video and audio data using a generative AI model and voice recognition technology.

[0309] Key points and sentiment analysis

[0310] The server identifies important parts of the video based on the analysis results and collects and analyzes viewer emotion data.

[0311] Summary video generation and delivery

[0312] The server uses the Adobe Premiere Pro API to generate a condensed version of the video based on the identified key parts and viewer sentiment data, and delivers it to the user via a URL link.

[0313] Prompt Sentence Examples

[0314] I've uploaded a video of a new product briefing. I want you to analyze the viewer's emotional state and generate a 5-minute summary version of the video.

[0315] This invention automatically extracts important scenes that reflect the viewer's emotional state, generates highly accurate video summaries, and predicts viewer reactions, allowing users to maximize viewer interest and conduct effective marketing and presentations.

[0316] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0317] Step 1:

[0318] Uploading videos

[0319] The server provides an interface for users to upload video data, which may include platforms such as web pages or applications.

[0320] The user presses the file selection button in the browser or app, selects the briefing video file, and sends the video to the server.

[0321] Input: Product presentation video file

[0322] Output: Video file stored on the server

[0323] Step 2:

[0324] Video data frame division

[0325] The server uses video processing software such as FFmpeg to split the uploaded video data into frames.

[0326] Using FFmpeg, the video data is split into still images at 30 frames per second, and each frame is saved as individual image data.

[0327] Input: Video file

[0328] Output: Split frame image data

[0329] Step 3:

[0330] Video and audio data analysis

[0331] The server analyzes the video and audio data using generative AI models and voice recognition technology.

[0332] A generative AI model (e.g., GPT model) is used to analyze the frame images and score important parts. Google Speech-to-Text technology is used to convert the audio data into text.

[0333] Input: Frame image data and audio data

[0334] Output: Text data with scores and frame analysis results

[0335] Step 4:

[0336] Identifying the important parts

[0337] The server lists important scenes based on the analysis results and identifies particularly important parts.

[0338] The scored data is used to extract scenes that are likely to be of particular interest to viewers.

[0339] Input: Text data with scores and frame analysis results

[0340] Output: Important Scene List

[0341] Step 5:

[0342] Emotional Data Analysis

[0343] The server collects viewers' viewing data and analyzes it using an emotion engine.

[0344] The viewer's facial expression and voice data are input into an emotion engine to analyze the viewer's emotional state and evaluate their emotional response to a particular scene.

[0345] Input: Viewing data (facial expression and voice data)

[0346] Output: Emotion analysis results

[0347] Step 6:

[0348] Generate a summary video

[0349] The server generates a summary version of the video based on the identified important parts and viewer emotion data.

[0350] Use video editing software such as Adobe Premiere Pro API to stitch together key scenes to create a short video, add transfer effects and subtitles.

[0351] Input: Important scene list and sentiment analysis results

[0352] Output: Abridged video

[0353] Step 7:

[0354] Video distribution

[0355] The server generates a URL link to distribute the generated summarized video and sends it to the user.

[0356] Users click on the link they receive to watch or download the abridged video.

[0357] Input: Abridged video

[0358] Output: URL link of the summary video

[0359] Prompt Sentence Examples

[0360] I've uploaded a video of a new product briefing. I want you to analyze the viewer's emotional state and generate a 5-minute summary version of the video.

[0361] The above is the specific flow and operation of each processing step of the present invention.

[0362] (Application example 2)

[0363] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0364] Current video advertising distribution technology is unable to analyze viewer emotions and interests in real time and summarize and distribute videos based on that information, making it difficult to maximize viewer interest. It is also difficult to effectively provide personalized advertising based on viewing data. This reduces advertising effectiveness and creates challenges in maintaining viewer interest.

[0365] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0366] In this invention, the server includes means for acquiring video data, means for analyzing the acquired video data and identifying important portions, means for generating a video digest based on the identified important portions, means for distributing the generated digest video, means for analyzing viewer emotions, means for generating a digest advertisement based on the viewer's interests, and means for providing personalized advertisements based on viewing data. This enables the generation and distribution of effective digest advertisements based on the viewer's emotions and interests, maximizing advertising effectiveness and maintaining viewer interest.

[0367] "Video data" refers to data that contains visual video information, and is usually composed of a series of frames that have been photographed and recorded by a device such as a camera or video recorder.

[0368] The "important portion" refers to a portion of the video data that contains particularly meaningful scenes or information and is useful or interesting to the viewer.

[0369] "Abridged video" refers to a video that can be played visually and audibly in a shortened format by extracting important parts from the original video data.

[0370] "Distribution" refers to the act of sending the generated summarized video to the viewer's terminal using a communication network such as the Internet.

[0371] "Viewer emotion" refers to the emotional reaction that a viewer shows while watching a video, and refers to a mental state such as joy, surprise, interest, or dissatisfaction.

[0372] "Viewer interest" refers to a state in which a viewer shows particular interest in a particular scene or piece of information within a video.

[0373] "Personalized advertising" means advertising content that is customized based on a viewer's individual emotional state and interests.

[0374] "Viewing data" refers to detailed data that records the viewer's behavior and reactions while watching a video, including viewing time, number of clicks, number of plays, changes in emotions, and the like.

[0375] The present invention provides a system for generating a digest version of an advertisement based on a viewer's emotions and interests to provide personalized advertisements in order to improve the viewing experience of advertisement videos. The system is composed of a server, a terminal, and a user.

[0376] The server first has a means for acquiring video data. When a user uploads an advertising video to the server, the server analyzes the acquired video data and identifies important parts. This analysis includes frame analysis using OpenCV and emotion analysis using EmotionRecognizer. Once important parts are identified, a means for generating a summarized version of the video based on the identified parts is activated. Here, Moviepy is used to connect important frames and create a summarized version of the video.

[0377] EmotionRecognizer is also used as a means of analyzing viewer emotions. It collects emotional and viewing data from viewers while they are watching, and generates abridged advertisements based on their interests during viewing. It then analyzes the viewing data to provide personalized advertisements based on the viewer's interests. This process involves data analysis based on the collected viewing data. Data points such as viewing time, number of clicks, number of views, and emotional state are utilized in the analysis.

[0378] Specifically, the system operates in the following scenario: While User A is watching an advertising video using a device, the server analyzes the video in real time to identify the viewer's emotions and interests. The server extracts scenes that the viewer particularly finds interesting and creates a shortened, summarized version of the advertisement. Next, a personalized advertisement is provided to User A based on User A's viewing data. The generated summarized advertisement is then distributed via the Internet for the user to view.

[0379] Examples of prompts include:

[0380] "Your goal is to build a system that analyzes the real-time emotional state of users watching a video explaining a new product, and summarizes and displays scenes that piqued their interest. This system will analyze video data uploaded to a server and maximize the viewer's interest."

[0381] In this way, the present invention realizes summarization and personalization of advertising videos based on the viewer's emotions and interests, thereby increasing advertising effectiveness and improving the viewer experience.

[0382] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0383] Step 1:

[0384] The server acquires the advertising video data uploaded by the user. The input is the video file provided by the user, which is saved in the server's data storage. The output is the acquired video file saved on the server.

[0385] Step 2:

[0386] The server divides the acquired video data into frames and makes them available for analysis as individual image data. The input is the uploaded video file, and OpenCV is used for data processing to break the video down into frames. The output is that each frame of the video is saved as image data on the server.

[0387] Step 3:

[0388] The server analyzes the viewer's emotions for each frame. The input is the image data of each frame obtained in step 2, and performs emotion analysis using EmotionRecognizer. The output is emotional state data for each frame.

[0389] Step 4:

[0390] The server aggregates the emotional state data and identifies important segments that indicate viewer interest. The input is the emotional state data obtained in step 3, and the data is calculated by evaluating the emotional score and marking segments that exceed a certain interest score as important. The output is a list of frames that indicate important segments.

[0391] Step 5:

[0392] The server generates a digest video based on the identified key segments. The input is the frame list of key segments obtained in step 4, and Moviepy is used to concatenate these frames to create a digest video. The output is the generated digest video file.

[0393] Step 6:

[0394] The server delivers the generated summarized video to the terminal. The input is the summarized video file generated in step 5, and a link that the user can access via the Internet is generated. The output is the status that the video is available for viewing on the user's terminal.

[0395] Step 7:

[0396] The server collects and analyzes viewing data. The input is the viewing data (viewing time, clicks, plays, emotional state, etc.) generated when users watch the abridged video, and the viewing database is used for data analysis. The output is the analyzed viewing data.

[0397] Step 8:

[0398] The server predicts each user's interests based on the aggregated viewing data and provides personalized advertisements. The input is the viewing data obtained in step 7, and the generative AI model is used to predict the user's interests. The output is personalized advertising content.

[0399] In this way, a system is realized that performs a series of processes from acquiring video data to distributing it, analyzing viewing data, and providing personalized advertisements through each step between the server, terminal, and user.

[0400] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0401] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0402] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0403] [Second embodiment]

[0404] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0405] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0406] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0407] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0408] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0409] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0410] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0411] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0412] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0413] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0414] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0415] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0416] MODE FOR CARRYING OUT THE INVENTION

[0417] The present invention provides a system for efficiently summarizing video data from a product presentation and predicting viewer reactions. The system is composed of a server, a terminal, and a user. Each processing step is described in detail below.

[0418] System configuration

[0419] 1. The server provides an upload interface for acquiring video data. Users upload the video of the product presentation to the server through this interface.

[0420] 2. The server splits the uploaded video data into frames and processes them as individual image data, making each frame in the video ready for analysis.

[0421] 3. The server analyzes the video and audio data using generative AI and speech recognition technology. As a result of the analysis, important parts are scored and identified. These important parts are then used to generate a condensed version of the video.

[0422] 4. The server creates a video summary based on the identified key parts. It generates a shortened video by stitching together key scenes for easy user comprehension. This process also includes adding transition effects and subtitles.

[0423] 5. The server delivers the generated summary video to the device. This delivery is done via the Internet, and the user can watch or download the summary video by clicking on a link.

[0424] Creating a text summary

[0425] 1. A user can request a summary of the briefing video. If the request is accepted, the server extracts the audio data from the video.

[0426] 2. The server converts the audio data into text using speech recognition software, which is then processed with natural language processing techniques to remove noise and extract key points from the audio data.

[0427] 3. The server uses generative AI to summarize the text data and generate a text that concisely expresses the main content. This summary text can be used as a report or marketing material.

[0428] Predicting audience reactions

[0429] 1. The server collects viewing data, including viewer attributes, viewing time, number of plays, number of clicks, etc. This data is used to analyze viewer behavior patterns.

[0430] 2. The server uses generative AI to analyze the collected viewing data and learn viewer behavior patterns and trends. By comparing and analyzing past and current data, a predictive model of viewer reactions is generated.

[0431] 3. The server uses the predictive model to predict future inquiries and applications, and displays this forecast data in a user-accessible dashboard.

[0432] Specific examples

[0433] 1. The user uploads a video of a new product briefing to the server.

[0434] 2. The server analyzes the video data, identifies important parts, and generates a 5-minute video summary that is then delivered to the user.

[0435] 3. The user clicks on the download link to watch the generated summary video.

[0436] 4. When the user requests a summary, the server converts the audio data into text, summarizes the key points, and provides the text in a format that the user can download.

[0437] 5. The server collects viewing data and analyzes viewer trends. Based on this, it predicts the number of inquiries and applications and notifies users.

[0438] In this way, the present invention realizes efficient and effective video editing of product presentations and prediction of audience reactions.

[0439] The processing flow will be explained below.

[0440] Creating a summary video

[0441] Step 1:

[0442] The user uploads the video file of the briefing to the server by selecting the video file through the user interface and uploading it to the specified folder.

[0443] Step 2:

[0444] The server divides the uploaded video data into frames, dividing the video file into frames at regular intervals and generating individual image data.

[0445] Step 3:

[0446] The server uses generative AI to analyze the video data. It uses an AI algorithm to score important scenes from each frame, particularly by analyzing the speaker's facial expressions, movements, and vocal rhythm to determine their importance.

[0447] Step 4:

[0448] The server generates a summary of the video based on the scored frames, and then extracts and stitches together the frames with the highest scores to create a short summary video.

[0449] Step 5:

[0450] The server delivers the abridged video to the device, saves the completed video file, and provides a download link to the user's device, which the user can click to watch or download the abridged video.

[0451] Creating a summary of the content

[0452] Step 1:

[0453] A user requests the generation of a summary text for an uploaded video by selecting the summary generation option through a dedicated interface.

[0454] Step 2:

[0455] The server extracts the audio data from the briefing, separates it from the video, and passes it to the speech recognition engine.

[0456] Step 3:

[0457] The server uses speech recognition software to convert speech to text, converts the speech data into text data, and uses natural language processing techniques to remove noise.

[0458] Step 4:

[0459] The server uses generative AI to summarize the generated text data, extracting key keywords and phrases and summarizing them into easy-to-understand sentences.

[0460] Step 5:

[0461] The server delivers the summary text to the user's device, formats the summary results, and displays them on the user's dashboard or provides a downloadable link.

[0462] Predicting audience reactions

[0463] Step 1:

[0464] The server collects viewing data, such as the viewer's access history, viewing time, number of plays, and number of clicks, and stores this data in real time.

[0465] Step 2:

[0466] The server analyzes the collected viewing data, using statistical methods and machine learning algorithms to analyze viewer behavior patterns.

[0467] Step 3:

[0468] The server predicts viewer reactions based on viewing data and past reaction data, for example by building a model to predict the number of inquiries and applications.

[0469] Step 4:

[0470] The server notifies the user of the prediction results. The generated prediction data is displayed visually and provided to the user's dashboard. The user can then use this information to plan their next strategy.

[0471] As described above, the system of the present invention efficiently generates a summary video and text of a briefing session and predicts viewer reactions, thereby significantly improving the efficiency of users' work.

[0472] Example 1

[0473] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0474] Traditional video data from product presentations is long and consumes viewers' time, often burying important information. Furthermore, there is a lack of systematic methods for predicting viewer reactions, making it difficult to improve engagement. Furthermore, there is a need for an efficient method for quickly and accurately generating abridged video and summary text.

[0475] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0476] In this invention, the server includes means for acquiring video data, means for dividing the acquired video data into frames, means for analyzing the divided frames and identifying important portions, means for converting audio data into text and extracting important content, means for generating a summary of the video based on the identified important portions and the extracted important content, means for adding transfer effects and subtitles to the generated summary video, means for distributing the generated summary video, means for collecting and analyzing viewing data, and means for predicting viewer responses based on the collected data. This makes it possible to quickly extract important information from long videos and efficiently generate summary videos and summary text. Furthermore, predicting viewer responses can improve engagement and optimize marketing strategies.

[0477] "Video data" refers to video content such as product presentations, and is data that includes visual and audio information.

[0478] The "means for acquiring" is a function including an interface and a protocol for transmitting video data from a user to a server.

[0479] A "frame" is an individual still image that makes up a video.

[0480] The "dividing means" is a function that performs processing to cut out video data in time units and save them as individual frames.

[0481] The "analysis means" refers to a technique for extracting important information from divided frames or audio data, and performing evaluation and scoring.

[0482] "Audio data" refers to audio information contained in a video, including narration, conversation, music, and the like.

[0483] "Means for converting to text" refers to speech recognition technology for recognizing voice data as character information and converting it into text format.

[0484] "Extraction means" refers to techniques for finding important content or scenes from the analyzed text or video frames.

[0485] The "means for generating a summary version" is a function that connects the extracted important parts together to create a video that is shorter and easier to understand.

[0486] "Transfer effect" is a technique for adding visual effects when switching scenes.

[0487] "Subtitles" are the display of dialogue and explanations in a video as text.

[0488] The "distribution means" refers to a technique for providing the generated summarized video to users via a communication means such as the Internet.

[0489] "Viewing data" is information relating to the viewer's attributes, viewing time, number of plays, number of clicks, and other behavior.

[0490] "Means of collection" refers to tracking technologies and analytical tools used to automatically collect viewing data.

[0491] "Viewer response" refers to the behavioral patterns and emotional reactions of viewers when they watch a video.

[0492] "Predictive methods" are analytical techniques that use collected viewing data to predict future viewer behavior and reactions.

[0493] The present invention relates to a system for efficiently summarizing video data from a product presentation and predicting viewer reactions. The system is composed of a server, terminals, and users.

[0494] The server provides an interface for users to obtain video data. When a user uploads a video file to the server, the server divides the video data into frames. This process uses video analysis tools such as FFmpeg and OpenCV.

[0495] The server analyzes the separated frames using generative AI models and speech recognition technology. For example, it uses Google Cloud Video Intelligence API or Amazon Transcribe to convert the content and audio data of the video frames into text. Based on the analysis, important parts of the video are identified. An example of a prompt is, "What important events are contained in this video frame?"

[0496] The server then generates a condensed version of the video based on the identified key parts. During this process, video editing software such as Adobe Premiere Pro or iMovie is used to add transition effects and subtitles to the shortened video. The condensed version is then distributed to devices via the Internet, and users can watch or download the condensed version by clicking a link.

[0497] When a user requests a text summary, the server converts the audio data into text using the Google Speech-to-Text API or Amazon Transcribe. It then uses a generative AI model (e.g., GPT-4) to summarize the key content. An example prompt at this stage is "Please summarize the following text: [Extracted text]."

[0498] In addition, the server collects viewing data (viewer attributes, viewing time, number of views, number of clicks, etc.) and analyzes viewer behavior patterns using tools such as Google Analytics and Mixpanel. A generative AI model is used to analyze the data and predict viewer responses. This allows users to predict future inquiries and applications, and the results can be viewed on a dashboard.

[0499] As a concrete example, a user uploads a video of a new product briefing to a server, which analyzes the video data to generate a 5-minute summary of the video and delivers it to the user. The user clicks a link to watch the video and then requests a summary text. The server converts the audio data into text and generates a summary. Finally, the server collects viewing data and predicts the viewer's reaction.

[0500] In this way, the present invention realizes efficient and effective video editing of product presentations and prediction of audience reactions.

[0501] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0502] System program processing flow

[0503] Step 1: Upload video data

[0504] The server provides an interface for obtaining video data from the user. The user uses this interface to upload video data from their device to the server. The input in this step is a video file selected by the user, and the output is a video file stored on the server.

[0505] Specifically, the server stores the video file in cloud storage and registers the file's metadata (file size, format, resolution, etc.) in a database.

[0506] Step 2: Split the video data into frames

[0507] The server splits the uploaded video data into frames. The input of this step is the video file stored on the server, and the output is the individual frame image files.

[0508] Specifically, the server uses FFmpeg to split the video into fixed frame intervals and save each frame as an image file, generating still images that can be analyzed.

[0509] Step 3: Analyze video and audio data

[0510] The server analyzes the video and audio data using generative AI models and speech recognition technology. The input for this step is the frame image files and audio data, and the output is data identifying key parts.

[0511] Specifically, the server uses the Google Cloud Video Intelligence API to analyze the content of each frame (e.g., objects, actions, scene changes), and then uses Amazon Transcribe to convert the audio data into text. For example, a prompt sentence like "What significant events does this video frame contain?" is input to the generative AI model.

[0512] Step 4: Generate a condensed video

[0513] The server generates a digest version of the video based on the identified important parts. The input of this step is the data of the important parts, and the output is a digest version of the video file.

[0514] Specifically, the server uses Adobe Premiere Pro scripting to stitch together key frames, add transfer effects and subtitles, and generate a condensed video, which is then stored on the server.

[0515] Step 5: Publish a condensed version of your video

[0516] The server then distributes the generated digested video to the user's device via the Internet. The user then clicks on the provided link to watch or download the video. The input of this step is the digested video file, and the output is a link that the user can watch.

[0517] Specifically, the server uploads the summarized video to a cloud storage service and sends the user an email notification with a download link.

[0518] Step 6: Creating a text summary

[0519] When a user requests a text summary, the server converts the audio data into text and uses a generative AI model to create the summary. The input for this step is the audio data, and the output is the summarized text.

[0520] Specifically, the server sends the voice data to the Google Speech-to-Text API, converts it into text, and then inputs the prompt "Please summarize the following text: [Extracted text]" into the generative AI model to generate a summary text.

[0521] Step 7: Predict audience reaction

[0522] The server collects viewing data and uses a generative AI model to predict viewer reactions. The input of this step is the viewing data, and the output is the predicted results.

[0523] Specifically, the server collects viewing data (viewer attributes, viewing time, number of plays, number of clicks, etc.) using Google Analytics, inputs it into a generative AI model (for example, a custom model using TensorFlow) to learn viewer behavior patterns, and based on that, predicts the number of inquiries and applications and notifies users.

[0524] In this way, the system efficiently and effectively edits videos of product presentations and predicts audience reactions.

[0525] (Application example 1)

[0526] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0527] In the modern advertising industry, generating effective abridged advertising videos that attract viewers' attention is a key challenge. However, long advertising videos have difficulty sustaining viewers' attention, leading to lower viewer ratings. Furthermore, to maximize the effectiveness of advertising, it is necessary to predict viewer responses and select the optimal delivery timing, but this is extremely difficult to do manually.

[0528] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0529] In this invention, the server includes means for acquiring video data, means for analyzing the acquired video data and identifying important parts, means for generating a summary of the video based on the identified important parts, means for distributing the generated summary video, means for analyzing the advertising video received from the user and extracting important parts to generate a summary advertising video, and means for predicting viewer reactions based on data processing and proposing optimal distribution timing, thereby enabling the generation of effective summary advertising videos and optimal advertising distribution taking viewer reactions into consideration.

[0530] The "means for acquiring video data" is a function for users to upload multimedia data such as advertising videos and product explanation videos to the server.

[0531] "Means for analyzing video data and identifying important parts" refers to a function in which the server analyzes uploaded video data frame by frame and automatically recognizes scenes that are visually and audibly important.

[0532] The "means for generating a summary version of a video based on identified important parts" is a function that automatically generates a summary version of a video that conveys the main content in a short amount of time by linking identified important scenes.

[0533] The "means for distributing the generated summarized video" is a function for distributing the generated summarized video to users and viewers via the Internet.

[0534] "Means for analyzing advertising videos received from users, extracting important parts, and generating summarized advertising videos" refers to a function that analyzes advertising videos provided by advertisers, extracts important visual and auditory scenes, and generates effective summarized advertising videos.

[0535] "Means of predicting viewer reactions based on data processing and proposing optimal distribution timing" refers to a function that collects and analyzes viewing data, predicts future viewer reactions based on viewer behavior patterns, and proposes optimal advertising distribution timing.

[0536] MODE FOR CARRYING OUT THE INVENTION

[0537] The present invention relates to a system for efficiently summarizing video data and predicting viewer reactions. DETAILED DESCRIPTION OF THE INVENTION The present invention will be described below with reference to specific examples.

[0538] Overall system configuration

[0539] This system consists of a server that acquires and analyzes video data, a terminal that distributes the generated summarized video, and a user who uploads the video data. The main elements are as follows:

[0540] 1. How to obtain video data

[0541] Users use an interface to upload video data to the server. For example, they can upload advertising videos or product explanation videos via a web browser or smartphone application.

[0542] 2. A method for analyzing video data and identifying important parts

[0543] The server analyzes the uploaded video data. Specifically, it uses OpenCV to divide the video into frames and identify important visual and auditory scenes. It also uses speech recognition technology to extract important parts from the audio data.

[0544] 3. A method for generating a summary of the video based on the identified key parts

[0545] The server then concatenates key scenes based on the analysis results and generates a condensed video using MoviePy, which can be enhanced with visual effects to make it more visually appealing.

[0546] 4. Means of distributing the generated summarized video

[0547] The summarized video generated by the server is delivered to the terminal via the Internet, and the user can view or download the summarized video from the provided link.

[0548] 5. Means for analyzing the advertisement video received from the user, extracting important parts, and generating a summary advertisement video

[0549] The server particularly analyzes the advertisement video received from the advertiser, extracts important parts that will attract the viewer's attention, and generates an effective summary advertisement video.

[0550] 6. A method for predicting viewer reactions based on data processing and proposing optimal distribution timing

[0551] The server collects viewing data, analyzes viewer behavior patterns based on the collected data, and uses a generative AI model to predict these patterns and suggest optimal distribution timing and channels.

[0552] Hardware and Software

[0553] Hardware: General PCs, servers, smartphones

[0554] Software: Python, OpenCV, MoviePy, transformers, OpenAI API

[0555] Operational procedures and examples

[0556] 1. Upload a video

[0557] Users log in to the system from their smartphones or PCs and upload their own advertising videos. For example, advertisers may upload promotional videos for new products.

[0558] 2. Video analysis and summary generation

[0559] The server analyzes the uploaded video and identifies important scenes using frame analysis with OpenCV and speech recognition technology, then concatenates the identified important parts and generates a summarized video using MoviePy.

[0560] 3. Summary Text Generation

[0561] The server also analyzes the audio data from the video and converts it into text using speech recognition technology, then generates a summary text using a generative AI model.

[0562] 4. Predictions and Recommendations

[0563] The server collects viewing data and uses a generative AI model to predict viewer reactions, and then suggests optimal ad delivery timing based on this.

[0564] Example prompts for generative AI models

[0565] Extract key parts from promotional videos for new products and generate visually impactful summary videos. Convert the video audio into text and summarize the key points. Analyze viewing data to predict the effectiveness of the next advertising campaign.

[0566] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0567] Step 1:

[0568] Users log in to the system using a smartphone or PC and upload their own advertising video to the server. The advertising video file is provided as input. The server receives this file and stores it in a database. As output, a message indicating that the file has been saved is generated. Specifically, a file selection dialog is displayed and the selected video file is sent to the server.

[0569] Step 2:

[0570] The server uses OpenCV to split the uploaded video file into frames. The video file saved in step 1 is provided as input. Data processing involves splitting the video into frames based on timestamps and saving each frame image as a list. The output is a set of frame images in list format. Specifically, the server reads the video file and analyzes each frame sequentially.

[0571] Step 3:

[0572] The server converts the video's audio data into text using speech recognition technology (for example, the speech recognition model from the transformers package). The video file saved in step 1 is provided as input. Data calculations involve extracting the audio data and converting it into text. The output is text data generated from the audio data. Specifically, the audio portion of the video file is analyzed and transcribed using a speech recognition engine.

[0573] Step 4:

[0574] The server uses the generative AI model to summarize the text data and extract key points. The text data generated in step 3 is provided as input. The data is calculated by summarizing the text data based on natural language processing technology. The output is the summarized text data. Specifically, the text data is input into the generative AI model, and a summary is generated.

[0575] Step 5:

[0576] The server identifies important scenes based on the image data and audio analysis results for each frame. The input is the frame image data from step 2 and the audio analysis results from step 3. The data calculation involves scoring and selecting important scenes based on visual and auditory features. The output is a list of important scenes. Specifically, the analysis results for each frame are evaluated using a scoring algorithm.

[0577] Step 6:

[0578] The server concatenates key scenes to generate a condensed video. The input is the list of key scenes identified in step 5. The data is processed by concatenating these scenes in order and combining them into a single short video. The output is a condensed video file. Specifically, MoviePy is used to edit the key scenes and combine them into a single video file.

[0579] Step 7:

[0580] The server distributes the generated summarized video to users and viewers. The input is the summarized video file generated in step 6. The data calculation involves uploading the video file and generating a link. The output is the generation of a viewing link, which is notified to the user. Specifically, the operation involves uploading the video using a video hosting service and providing a viewing link.

[0581] Step 8:

[0582] The server collects viewing data and analyzes viewer behavior patterns. Viewing data such as the number of clicks on viewing links and viewing time is provided as input. Data calculation involves compiling and analyzing the viewing data. The output is analysis results showing viewer behavior patterns. Specifically, the server uses a viewing tracking system to collect data and perform statistical analysis.

[0583] Step 9:

[0584] A generative AI model is used to predict viewer reactions based on viewing data, and the optimal timing for ad delivery is proposed. The input is the viewing data analysis results from step 8. Data calculation involves analyzing behavioral patterns and applying a predictive model. The output is a proposal for the optimal timing for ad delivery. Specifically, the predictive algorithm from the generative AI model is applied to generate the proposed results.

[0585] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0586] MODE FOR CARRYING OUT THE INVENTION

[0587] The present invention is a system for efficiently summarizing video data from product presentations and predicting viewer reactions. This system consists of a server, a terminal, and a user. By combining it with an emotion engine, the system can recognize viewer emotions and identify the importance of content with high accuracy based on those emotions. Each processing step is explained in detail below.

[0588] System configuration

[0589] 1. The server provides an upload interface for acquiring video data. Users upload the video of the product presentation to the server through this interface.

[0590] 2. The server splits the uploaded video data into frames and processes them as individual image data, making each frame in the video ready for analysis.

[0591] 3. The server analyzes the video and audio data using generative AI and speech recognition technology. As a result of the analysis, important parts are scored and identified. These important parts are then used to generate a condensed version of the video.

[0592] 4. The server uses an emotion engine to analyze the viewer's emotional state. Based on the viewer's facial expression and voice data collected during viewing, the server recognizes the viewer's emotions and reflects this emotional data in the video analysis. For example, scenes that the viewer is particularly interested in are given a higher importance rating.

[0593] 5. The server generates a video summary based on the identified key parts and emotional data. It stitches together key scenes to make it easier for users to understand, and generates a shortened video that reflects the viewer's emotional state. This process also includes adding transition effects and subtitles.

[0594] 6. The server delivers the generated summary video to the device. This delivery is done via the Internet, and the user can watch or download the summary video by clicking on a link.

[0595] Creating a text summary

[0596] 1. A user can request a summary of the briefing video. If the request is accepted, the server extracts the audio data from the video.

[0597] 2. The server converts the audio data into text using speech recognition software, which is then processed with natural language processing techniques to remove noise and extract key points from the audio data.

[0598] 3. The server uses generative AI to summarize the text data and generate a text that concisely expresses the main content. This summary text can be used as a report or marketing material.

[0599] Predicting audience reactions

[0600] 1. The server collects viewing data, including viewer attributes, viewing time, number of plays, number of clicks, emotional data, etc. This data is used to analyze viewer behavior patterns.

[0601] 2. The server uses generative AI to analyze the collected viewing data and learn viewer behavior patterns and trends. By comparing and analyzing past and current data, a predictive model of viewer reactions is generated.

[0602] 3. The server uses the predictive model to predict future inquiries and applications, and displays this forecast data in a user-accessible dashboard.

[0603] Specific examples

[0604] 1. The user uploads a video of a new product briefing to the server.

[0605] 2. The server analyzes the video data and uses an emotion engine to analyze the viewer's emotional state. Scenes that the viewer particularly enjoyed are identified as important parts with particularly high scores.

[0606] 3. The server generates a five-minute video summary based on these key scenes, edits it to maximize viewer interest, and delivers the video to the user.

[0607] 4. The user clicks on the download link to watch the generated summary video.

[0608] 5. When the user requests a summary, the server converts the audio data into text, summarizes the key points, and provides the text in a format that the user can download.

[0609] 6. The server collects viewing data and analyzes the viewer's behavioral patterns and emotional state. Based on this, it predicts the number of inquiries and applications and notifies the user.

[0610] In this way, the present invention efficiently generates summary video and text of a briefing session and predicts viewer reactions, greatly improving the efficiency of users' work. Furthermore, by combining it with an emotion engine, it becomes possible to provide content that reflects the viewer's emotional state, thereby drawing out their interest with greater accuracy.

[0611] The processing flow will be explained below.

[0612] Creating a summary video

[0613] Step 1:

[0614] The user uploads the video file of the briefing to the server by selecting the video file through the user interface and uploading it to the specified folder.

[0615] Step 2:

[0616] The server divides the uploaded video data into frames, dividing the video file into frames at regular intervals and generating individual image data.

[0617] Step 3:

[0618] The server uses generative AI to analyze the video data. It uses an AI algorithm to score important scenes from each frame, particularly by analyzing the speaker's facial expressions, movements, and vocal rhythm to determine their importance.

[0619] Step 4:

[0620] The server uses an emotion engine to analyze the viewer's emotional state, acquiring facial expression and voice data and recognizing emotions in real time.

[0621] Step 5:

[0622] Based on the results of the emotion engine, the server re-scores frames that viewers showed particular interest in as high scores, allowing the emotion data to be reflected in the selection of important scenes in the video.

[0623] Step 6:

[0624] The server generates a video summary based on the scored frames. High-scoring frames are extracted and stitched together to create a short video summary.

[0625] Step 7:

[0626] The server delivers the abridged video to the device, saves the completed video file, and provides a download link to the user's device, which the user can click to watch or download the abridged video.

[0627] Creating a summary of the content

[0628] Step 1:

[0629] A user requests the generation of a summary text for an uploaded video by selecting the summary generation option through a dedicated interface.

[0630] Step 2:

[0631] The server extracts the audio data from the briefing, separates it from the video, and passes it to the speech recognition engine.

[0632] Step 3:

[0633] The server uses speech recognition software to convert speech to text, converts the speech data into text data, and uses natural language processing techniques to remove noise.

[0634] Step 4:

[0635] The server uses generative AI to summarize the generated text data, extracting key keywords and phrases and summarizing them into easy-to-understand sentences.

[0636] Step 5:

[0637] The server delivers the summary text to the user's device, formats the summary results, and displays them on the user's dashboard or provides a downloadable link.

[0638] Predicting audience reactions

[0639] Step 1:

[0640] The server collects viewing data, such as the viewer's access history, viewing time, number of plays, number of clicks, and emotional data, and stores this data in real time.

[0641] Step 2:

[0642] The server analyzes the collected viewing data, using statistical methods and machine learning algorithms to analyze viewer behavior patterns.

[0643] Step 3:

[0644] The server predicts viewer reactions based on viewing data and past reaction data, for example by building a model to predict the number of inquiries and applications.

[0645] Step 4:

[0646] The server notifies the user of the prediction results. The generated prediction data is displayed visually and provided to the user's dashboard. The user can then use this information to plan their next strategy.

[0647] As described above, the system of the present invention efficiently generates summary videos and texts of briefing sessions and predicts viewer reactions, greatly improving the efficiency of users' work. Furthermore, by combining it with an emotion engine, it becomes possible to provide content that reflects the viewer's emotional state, thereby more accurately attracting viewer interest.

[0648] Example 2

[0649] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0650] Conventional video summarization systems have difficulty accurately identifying important parts of a video, and have been unable to generate summaries that take into account the viewer's emotions. It has also been difficult to accurately predict viewer reactions. This has led to issues such as users being unable to provide content that piques the viewer's interest, making it difficult to carry out effective marketing and presentations.

[0651] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0652] In this invention, the server includes means for acquiring video data, means for dividing the acquired video data into frames, means for analyzing the divided frame data and identifying important portions, means for collecting and analyzing viewer emotion data, means for generating a summary of the video based on the identified important portions and the viewer emotion data, means for adding transfer effects and subtitles to the generated summary video, and means for delivering the generated summary video. This makes it possible to automatically extract important scenes that reflect the viewer's emotional state, generate highly accurate video summaries, and predict viewer reactions.

[0653] "Video data" refers to data in a file format that continuously records visual information and is played back on a computer or electronic device.

[0654] A "frame" is an individual still image that constitutes video data, and is recognized as a video when it is displayed continuously at regular intervals of seconds.

[0655] "Analysis" is the process of analyzing input data in detail and extracting useful information based on that analysis.

[0656] An "important portion" is a scene or element that is considered to be particularly important within the video data, and is a portion that contains information that is highly valuable to the viewer.

[0657] "Viewer emotional data" refers to data that indicates the emotional state of a viewer that can be obtained from facial expressions, voice, behavior, etc. that the viewer displays while watching a video.

[0658] A "summary video" is a video that has been reconstructed in a shortened format by extracting important parts from the original video data.

[0659] A "transition effect" is a visual effect that creates a smooth transition between different scenes or frames.

[0660] "Subtitles" are textual information displayed to supplement the visual content of a video, providing explanations and translations to viewers.

[0661] "Distribution" is the process of transmitting created content to users over the Internet or a communication network, making it accessible to them.

[0662] "Voice data" refers to data in a file format that records a voice signal, and is the subject of voice recognition.

[0663] "Viewing data" refers to data such as the viewer's behavior and attributes when viewing video content, viewing time, number of plays, and number of clicks.

[0664] A "generative AI model" is an artificial intelligence algorithm or software that learns from large amounts of data and generates appropriate outputs based on the input data.

[0665] This invention is a system for efficiently summarizing video data, such as product presentations, and predicting viewer reactions. This system is composed of a server, terminals, and users, and by combining an emotion engine, it is possible to recognize viewer emotions and identify the importance of content based on those emotions.

[0666] Server Configuration

[0667] The server has the following main functions:

[0668] 1. Acquiring video data:

[0669] The server provides an interface for users to upload video data. Specifically, files can be uploaded using a web page or an application.

[0670] 2. Video data frame division:

[0671] The server uses video processing software such as FFmpeg to split the uploaded video data into frames, which makes later analysis easier.

[0672] 3. Video and audio data analysis:

[0673] The server analyzes the video and audio data using a generative AI model and speech recognition technology. In this step, a generative AI model (e.g., OpenAI's GPT model) is used to understand the text and images, and speech recognition technology such as Google Speech-to-Text is used to convert the speech to text.

[0674] 4. Identify the important parts:

[0675] The server identifies important parts of the video based on the analysis results, for example, by using information scored by a generative AI model to extract particularly important scenes.

[0676] 5. Emotional Data Analysis:

[0677] The server collects viewers' viewing data (facial expressions and voice data) and analyzes it using an emotion engine (e.g., Amazon Rekognition or Microsoft Azure's emotion recognition API). Scenes that viewers find particularly interesting are given high marks.

[0678] 6. Generate a condensed video:

[0679] The server generates a condensed version of the video based on the identified key parts and viewer sentiment data. It uses Adobe Premiere Pro APIs and other tools to stitch together key scenes to create a shortened video. It also adds transition effects and subtitles.

[0680] 7. Video Distribution:

[0681] The server generates a URL link to deliver the digested video, and users can view the digest by clicking on this link.

[0682] Specific examples

[0683] Uploading videos

[0684] The user uploads a video of a new product briefing to the server.

[0685] Video and audio analysis

[0686] The server divides the uploaded video data into frames using Ffmpeg and analyzes the video and audio data using a generative AI model and voice recognition technology.

[0687] Key points and sentiment analysis

[0688] The server identifies important parts of the video based on the analysis results and collects and analyzes viewer emotion data.

[0689] Summary video generation and delivery

[0690] The server uses the Adobe Premiere Pro API to generate a condensed version of the video based on the identified key parts and viewer sentiment data, and delivers it to the user via a URL link.

[0691] Prompt Sentence Examples

[0692] I've uploaded a video of a new product briefing. I want you to analyze the viewer's emotional state and generate a 5-minute summary version of the video.

[0693] This invention automatically extracts important scenes that reflect the viewer's emotional state, generates highly accurate video summaries, and predicts viewer reactions, allowing users to maximize viewer interest and conduct effective marketing and presentations.

[0694] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0695] Step 1:

[0696] Uploading videos

[0697] The server provides an interface for users to upload video data, which may include platforms such as web pages or applications.

[0698] The user presses the file selection button in the browser or app, selects the briefing video file, and sends the video to the server.

[0699] Input: Product presentation video file

[0700] Output: Video file stored on the server

[0701] Step 2:

[0702] Video data frame division

[0703] The server uses video processing software such as FFmpeg to split the uploaded video data into frames.

[0704] Using FFmpeg, the video data is split into still images at 30 frames per second, and each frame is saved as individual image data.

[0705] Input: Video file

[0706] Output: Split frame image data

[0707] Step 3:

[0708] Video and audio data analysis

[0709] The server analyzes the video and audio data using generative AI models and voice recognition technology.

[0710] A generative AI model (e.g., GPT model) is used to analyze the frame images and score important parts. Google Speech-to-Text technology is used to convert the audio data into text.

[0711] Input: Frame image data and audio data

[0712] Output: Text data with scores and frame analysis results

[0713] Step 4:

[0714] Identifying the important parts

[0715] The server lists important scenes based on the analysis results and identifies particularly important parts.

[0716] The scored data is used to extract scenes that are likely to be of particular interest to viewers.

[0717] Input: Text data with scores and frame analysis results

[0718] Output: Important Scene List

[0719] Step 5:

[0720] Emotional Data Analysis

[0721] The server collects viewers' viewing data and analyzes it using an emotion engine.

[0722] The viewer's facial expression and voice data are input into an emotion engine to analyze the viewer's emotional state and evaluate their emotional response to a particular scene.

[0723] Input: Viewing data (facial expression and voice data)

[0724] Output: Emotion analysis results

[0725] Step 6:

[0726] Generate a summary video

[0727] The server generates a summary version of the video based on the identified important parts and viewer emotion data.

[0728] Use video editing software such as Adobe Premiere Pro API to stitch together key scenes to create a short video, add transfer effects and subtitles.

[0729] Input: Important scene list and sentiment analysis results

[0730] Output: Abridged video

[0731] Step 7:

[0732] Video distribution

[0733] The server generates a URL link to distribute the generated summarized video and sends it to the user.

[0734] Users click on the link they receive to watch or download the abridged video.

[0735] Input: Abridged video

[0736] Output: URL link of the summary video

[0737] Prompt Sentence Examples

[0738] I've uploaded a video of a new product briefing. I want you to analyze the viewer's emotional state and generate a 5-minute summary version of the video.

[0739] The above is the specific flow and operation of each processing step of the present invention.

[0740] (Application example 2)

[0741] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0742] Current video advertising distribution technology is unable to analyze viewer emotions and interests in real time and summarize and distribute videos based on that information, making it difficult to maximize viewer interest. It is also difficult to effectively provide personalized advertising based on viewing data. This reduces advertising effectiveness and creates challenges in maintaining viewer interest.

[0743] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0744] In this invention, the server includes means for acquiring video data, means for analyzing the acquired video data and identifying important portions, means for generating a video digest based on the identified important portions, means for distributing the generated digest video, means for analyzing viewer emotions, means for generating a digest advertisement based on the viewer's interests, and means for providing personalized advertisements based on viewing data. This enables the generation and distribution of effective digest advertisements based on the viewer's emotions and interests, maximizing advertising effectiveness and maintaining viewer interest.

[0745] "Video data" refers to data that contains visual video information, and is usually composed of a series of frames that have been photographed and recorded by a device such as a camera or video recorder.

[0746] The "important portion" refers to a portion of the video data that contains particularly meaningful scenes or information and is useful or interesting to the viewer.

[0747] "Abridged video" refers to a video that can be played visually and audibly in a shortened format by extracting important parts from the original video data.

[0748] "Distribution" refers to the act of sending the generated summarized video to the viewer's terminal using a communication network such as the Internet.

[0749] "Viewer emotion" refers to the emotional reaction that a viewer shows while watching a video, and refers to a mental state such as joy, surprise, interest, or dissatisfaction.

[0750] "Viewer interest" refers to a state in which a viewer shows particular interest in a particular scene or piece of information within a video.

[0751] "Personalized advertising" means advertising content that is customized based on a viewer's individual emotional state and interests.

[0752] "Viewing data" refers to detailed data that records the viewer's behavior and reactions while watching a video, including viewing time, number of clicks, number of plays, changes in emotions, and the like.

[0753] The present invention provides a system for generating a digest version of an advertisement based on a viewer's emotions and interests to provide personalized advertisements in order to improve the viewing experience of advertisement videos. The system is composed of a server, a terminal, and a user.

[0754] The server first has a means for acquiring video data. When a user uploads an advertising video to the server, the server analyzes the acquired video data and identifies important parts. This analysis includes frame analysis using OpenCV and emotion analysis using EmotionRecognizer. Once important parts are identified, a means for generating a summarized version of the video based on the identified parts is activated. Here, Moviepy is used to connect important frames and create a summarized version of the video.

[0755] EmotionRecognizer is also used as a means of analyzing viewer emotions. It collects emotional and viewing data from viewers while they are watching, and generates abridged advertisements based on their interests during viewing. It then analyzes the viewing data to provide personalized advertisements based on the viewer's interests. This process involves data analysis based on the collected viewing data. Data points such as viewing time, number of clicks, number of views, and emotional state are utilized in the analysis.

[0756] Specifically, the system operates in the following scenario: While User A is watching an advertising video using a device, the server analyzes the video in real time to identify the viewer's emotions and interests. The server extracts scenes that the viewer particularly finds interesting and creates a shortened, summarized version of the advertisement. Next, a personalized advertisement is provided to User A based on User A's viewing data. The generated summarized advertisement is then distributed via the Internet for the user to view.

[0757] Examples of prompts include:

[0758] "Your goal is to build a system that analyzes the real-time emotional state of users watching a video explaining a new product, and summarizes and displays scenes that piqued their interest. This system will analyze video data uploaded to a server and maximize the viewer's interest."

[0759] In this way, the present invention realizes summarization and personalization of advertising videos based on the viewer's emotions and interests, thereby increasing advertising effectiveness and improving the viewer experience.

[0760] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0761] Step 1:

[0762] The server acquires the advertising video data uploaded by the user. The input is the video file provided by the user, which is saved in the server's data storage. The output is the acquired video file saved on the server.

[0763] Step 2:

[0764] The server divides the acquired video data into frames and makes them available for analysis as individual image data. The input is the uploaded video file, and OpenCV is used for data processing to break the video down into frames. The output is that each frame of the video is saved as image data on the server.

[0765] Step 3:

[0766] The server analyzes the viewer's emotions for each frame. The input is the image data of each frame obtained in step 2, and performs emotion analysis using EmotionRecognizer. The output is emotional state data for each frame.

[0767] Step 4:

[0768] The server aggregates the emotional state data and identifies important segments that indicate viewer interest. The input is the emotional state data obtained in step 3, and the data is calculated by evaluating the emotional score and marking segments that exceed a certain interest score as important. The output is a list of frames that indicate important segments.

[0769] Step 5:

[0770] The server generates a digest video based on the identified key segments. The input is the frame list of key segments obtained in step 4, and Moviepy is used to concatenate these frames to create a digest video. The output is the generated digest video file.

[0771] Step 6:

[0772] The server delivers the generated summarized video to the terminal. The input is the summarized video file generated in step 5, and a link that the user can access via the Internet is generated. The output is the status that the video is available for viewing on the user's terminal.

[0773] Step 7:

[0774] The server collects and analyzes viewing data. The input is the viewing data (viewing time, clicks, plays, emotional state, etc.) generated when users watch the abridged video, and the viewing database is used for data analysis. The output is the analyzed viewing data.

[0775] Step 8:

[0776] The server predicts each user's interests based on the aggregated viewing data and provides personalized advertisements. The input is the viewing data obtained in step 7, and the generative AI model is used to predict the user's interests. The output is personalized advertising content.

[0777] In this way, a system is realized that performs a series of processes from acquiring video data to distributing it, analyzing viewing data, and providing personalized advertisements through each step between the server, terminal, and user.

[0778] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0779] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0780] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0781] [Third embodiment]

[0782] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0783] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0784] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0785] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0786] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0787] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0788] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0789] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0790] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0791] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0792] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0793] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0794] MODE FOR CARRYING OUT THE INVENTION

[0795] The present invention provides a system for efficiently summarizing video data from a product presentation and predicting viewer reactions. The system is composed of a server, a terminal, and a user. Each processing step is described in detail below.

[0796] System configuration

[0797] 1. The server provides an upload interface for acquiring video data. Users upload the video of the product presentation to the server through this interface.

[0798] 2. The server splits the uploaded video data into frames and processes them as individual image data, making each frame in the video ready for analysis.

[0799] 3. The server analyzes the video and audio data using generative AI and speech recognition technology. As a result of the analysis, important parts are scored and identified. These important parts are then used to generate a condensed version of the video.

[0800] 4. The server creates a video summary based on the identified key parts. It generates a shortened video by stitching together key scenes for easy user comprehension. This process also includes adding transition effects and subtitles.

[0801] 5. The server delivers the generated summary video to the device. This delivery is done via the Internet, and the user can watch or download the summary video by clicking on a link.

[0802] Creating a text summary

[0803] 1. A user can request a summary of the briefing video. If the request is accepted, the server extracts the audio data from the video.

[0804] 2. The server converts the audio data into text using speech recognition software, which is then processed with natural language processing techniques to remove noise and extract key points from the audio data.

[0805] 3. The server uses generative AI to summarize the text data and generate a text that concisely expresses the main content. This summary text can be used as a report or marketing material.

[0806] Predicting audience reactions

[0807] 1. The server collects viewing data, including viewer attributes, viewing time, number of plays, number of clicks, etc. This data is used to analyze viewer behavior patterns.

[0808] 2. The server uses generative AI to analyze the collected viewing data and learn viewer behavior patterns and trends. By comparing and analyzing past and current data, a predictive model of viewer reactions is generated.

[0809] 3. The server uses the predictive model to predict future inquiries and applications, and displays this forecast data in a user-accessible dashboard.

[0810] Specific examples

[0811] 1. The user uploads a video of a new product briefing to the server.

[0812] 2. The server analyzes the video data, identifies important parts, and generates a 5-minute video summary that is then delivered to the user.

[0813] 3. The user clicks on the download link to watch the generated summary video.

[0814] 4. When the user requests a summary, the server converts the audio data into text, summarizes the key points, and provides the text in a format that the user can download.

[0815] 5. The server collects viewing data and analyzes viewer trends. Based on this, it predicts the number of inquiries and applications and notifies users.

[0816] In this way, the present invention realizes efficient and effective video editing of product presentations and prediction of audience reactions.

[0817] The processing flow will be explained below.

[0818] Creating a summary video

[0819] Step 1:

[0820] The user uploads the video file of the briefing to the server by selecting the video file through the user interface and uploading it to the specified folder.

[0821] Step 2:

[0822] The server divides the uploaded video data into frames, dividing the video file into frames at regular intervals and generating individual image data.

[0823] Step 3:

[0824] The server uses generative AI to analyze the video data. It uses an AI algorithm to score important scenes from each frame, particularly by analyzing the speaker's facial expressions, movements, and vocal rhythm to determine their importance.

[0825] Step 4:

[0826] The server generates a summary of the video based on the scored frames, and then extracts and stitches together the frames with the highest scores to create a short summary video.

[0827] Step 5:

[0828] The server delivers the abridged video to the device, saves the completed video file, and provides a download link to the user's device, which the user can click to watch or download the abridged video.

[0829] Creating a summary of the content

[0830] Step 1:

[0831] A user requests the generation of a summary text for an uploaded video by selecting the summary generation option through a dedicated interface.

[0832] Step 2:

[0833] The server extracts the audio data from the briefing, separates it from the video, and passes it to the speech recognition engine.

[0834] Step 3:

[0835] The server uses speech recognition software to convert speech to text, converts the speech data into text data, and uses natural language processing techniques to remove noise.

[0836] Step 4:

[0837] The server uses generative AI to summarize the generated text data, extracting key keywords and phrases and summarizing them into easy-to-understand sentences.

[0838] Step 5:

[0839] The server delivers the summary text to the user's device, formats the summary results, and displays them on the user's dashboard or provides a downloadable link.

[0840] Predicting audience reactions

[0841] Step 1:

[0842] The server collects viewing data, such as the viewer's access history, viewing time, number of plays, and number of clicks, and stores this data in real time.

[0843] Step 2:

[0844] The server analyzes the collected viewing data, using statistical methods and machine learning algorithms to analyze viewer behavior patterns.

[0845] Step 3:

[0846] The server predicts viewer reactions based on viewing data and past reaction data, for example by building a model to predict the number of inquiries and applications.

[0847] Step 4:

[0848] The server notifies the user of the prediction results. The generated prediction data is displayed visually and provided to the user's dashboard. The user can then use this information to plan their next strategy.

[0849] As described above, the system of the present invention efficiently generates a summary video and text of a briefing session and predicts viewer reactions, thereby significantly improving the efficiency of users' work.

[0850] Example 1

[0851] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0852] Traditional video data from product presentations is long and consumes viewers' time, often burying important information. Furthermore, there is a lack of systematic methods for predicting viewer reactions, making it difficult to improve engagement. Furthermore, there is a need for an efficient method for quickly and accurately generating abridged video and summary text.

[0853] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0854] In this invention, the server includes means for acquiring video data, means for dividing the acquired video data into frames, means for analyzing the divided frames and identifying important portions, means for converting audio data into text and extracting important content, means for generating a summary of the video based on the identified important portions and the extracted important content, means for adding transfer effects and subtitles to the generated summary video, means for distributing the generated summary video, means for collecting and analyzing viewing data, and means for predicting viewer responses based on the collected data. This makes it possible to quickly extract important information from long videos and efficiently generate summary videos and summary text. Furthermore, predicting viewer responses can improve engagement and optimize marketing strategies.

[0855] "Video data" refers to video content such as product presentations, and is data that includes visual and audio information.

[0856] The "means for acquiring" is a function including an interface and a protocol for transmitting video data from a user to a server.

[0857] A "frame" is an individual still image that makes up a video.

[0858] The "dividing means" is a function that performs processing to cut out video data in time units and save them as individual frames.

[0859] The "analysis means" refers to a technique for extracting important information from divided frames or audio data, and performing evaluation and scoring.

[0860] "Audio data" refers to audio information contained in a video, including narration, conversation, music, and the like.

[0861] "Means for converting to text" refers to speech recognition technology for recognizing voice data as character information and converting it into text format.

[0862] "Extraction means" refers to techniques for finding important content or scenes from the analyzed text or video frames.

[0863] The "means for generating a summary version" is a function that connects the extracted important parts together to create a video that is shorter and easier to understand.

[0864] "Transfer effect" is a technique for adding visual effects when switching scenes.

[0865] "Subtitles" are the display of dialogue and explanations in a video as text.

[0866] The "distribution means" refers to a technique for providing the generated summarized video to users via a communication means such as the Internet.

[0867] "Viewing data" is information relating to the viewer's attributes, viewing time, number of plays, number of clicks, and other behavior.

[0868] "Means of collection" refers to tracking technologies and analytical tools used to automatically collect viewing data.

[0869] "Viewer response" refers to the behavioral patterns and emotional reactions of viewers when they watch a video.

[0870] "Predictive methods" are analytical techniques that use collected viewing data to predict future viewer behavior and reactions.

[0871] The present invention relates to a system for efficiently summarizing video data from a product presentation and predicting viewer reactions. The system is composed of a server, terminals, and users.

[0872] The server provides an interface for users to obtain video data. When a user uploads a video file to the server, the server divides the video data into frames. This process uses video analysis tools such as FFmpeg and OpenCV.

[0873] The server analyzes the separated frames using generative AI models and speech recognition technology. For example, it uses Google Cloud Video Intelligence API or Amazon Transcribe to convert the content and audio data of the video frames into text. Based on the analysis, important parts of the video are identified. An example of a prompt is, "What important events are contained in this video frame?"

[0874] The server then generates a condensed version of the video based on the identified key parts. During this process, video editing software such as Adobe Premiere Pro or iMovie is used to add transition effects and subtitles to the shortened video. The condensed version is then distributed to devices via the Internet, and users can watch or download the condensed version by clicking a link.

[0875] When a user requests a text summary, the server converts the audio data into text using the Google Speech-to-Text API or Amazon Transcribe. It then uses a generative AI model (e.g., GPT-4) to summarize the key content. An example prompt at this stage is "Please summarize the following text: [Extracted text]."

[0876] In addition, the server collects viewing data (viewer attributes, viewing time, number of views, number of clicks, etc.) and analyzes viewer behavior patterns using tools such as Google Analytics and Mixpanel. A generative AI model is used to analyze the data and predict viewer responses. This allows users to predict future inquiries and applications, and the results can be viewed on a dashboard.

[0877] As a concrete example, a user uploads a video of a new product briefing to a server, which analyzes the video data to generate a 5-minute summary of the video and delivers it to the user. The user clicks a link to watch the video and then requests a summary text. The server converts the audio data into text and generates a summary. Finally, the server collects viewing data and predicts the viewer's reaction.

[0878] In this way, the present invention realizes efficient and effective video editing of product presentations and prediction of audience reactions.

[0879] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0880] System program processing flow

[0881] Step 1: Upload video data

[0882] The server provides an interface for obtaining video data from the user. The user uses this interface to upload video data from their device to the server. The input in this step is a video file selected by the user, and the output is a video file stored on the server.

[0883] Specifically, the server stores the video file in cloud storage and registers the file's metadata (file size, format, resolution, etc.) in a database.

[0884] Step 2: Split the video data into frames

[0885] The server splits the uploaded video data into frames. The input of this step is the video file stored on the server, and the output is the individual frame image files.

[0886] Specifically, the server uses FFmpeg to split the video into fixed frame intervals and save each frame as an image file, generating still images that can be analyzed.

[0887] Step 3: Analyze video and audio data

[0888] The server analyzes the video and audio data using generative AI models and speech recognition technology. The input for this step is the frame image files and audio data, and the output is data identifying key parts.

[0889] Specifically, the server uses the Google Cloud Video Intelligence API to analyze the content of each frame (e.g., objects, actions, scene changes), and then uses Amazon Transcribe to convert the audio data into text. For example, a prompt sentence like "What significant events does this video frame contain?" is input to the generative AI model.

[0890] Step 4: Generate a condensed video

[0891] The server generates a digest version of the video based on the identified important parts. The input of this step is the data of the important parts, and the output is a digest version of the video file.

[0892] Specifically, the server uses Adobe Premiere Pro scripting to stitch together key frames, add transfer effects and subtitles, and generate a condensed video, which is then stored on the server.

[0893] Step 5: Publish a condensed version of your video

[0894] The server then distributes the generated digested video to the user's device via the Internet. The user then clicks on the provided link to watch or download the video. The input of this step is the digested video file, and the output is a link that the user can watch.

[0895] Specifically, the server uploads the summarized video to a cloud storage service and sends the user an email notification with a download link.

[0896] Step 6: Creating a text summary

[0897] When a user requests a text summary, the server converts the audio data into text and uses a generative AI model to create the summary. The input for this step is the audio data, and the output is the summarized text.

[0898] Specifically, the server sends the voice data to the Google Speech-to-Text API, converts it into text, and then inputs the prompt "Please summarize the following text: [Extracted text]" into the generative AI model to generate a summary text.

[0899] Step 7: Predict audience reaction

[0900] The server collects viewing data and uses a generative AI model to predict viewer reactions. The input of this step is the viewing data, and the output is the predicted results.

[0901] Specifically, the server collects viewing data (viewer attributes, viewing time, number of plays, number of clicks, etc.) using Google Analytics, inputs it into a generative AI model (for example, a custom model using TensorFlow) to learn viewer behavior patterns, and based on that, predicts the number of inquiries and applications and notifies users.

[0902] In this way, the system efficiently and effectively edits videos of product presentations and predicts audience reactions.

[0903] (Application example 1)

[0904] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0905] In the modern advertising industry, generating effective abridged advertising videos that attract viewers' attention is a key challenge. However, long advertising videos have difficulty sustaining viewers' attention, leading to lower viewer ratings. Furthermore, to maximize the effectiveness of advertising, it is necessary to predict viewer responses and select the optimal delivery timing, but this is extremely difficult to do manually.

[0906] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0907] In this invention, the server includes means for acquiring video data, means for analyzing the acquired video data and identifying important parts, means for generating a summary of the video based on the identified important parts, means for distributing the generated summary video, means for analyzing the advertising video received from the user and extracting important parts to generate a summary advertising video, and means for predicting viewer reactions based on data processing and proposing optimal distribution timing, thereby enabling the generation of effective summary advertising videos and optimal advertising distribution taking viewer reactions into consideration.

[0908] The "means for acquiring video data" is a function for users to upload multimedia data such as advertising videos and product explanation videos to the server.

[0909] "Means for analyzing video data and identifying important parts" refers to a function in which the server analyzes uploaded video data frame by frame and automatically recognizes scenes that are visually and audibly important.

[0910] The "means for generating a summary version of a video based on identified important parts" is a function that automatically generates a summary version of a video that conveys the main content in a short amount of time by linking identified important scenes.

[0911] The "means for distributing the generated summarized video" is a function for distributing the generated summarized video to users and viewers via the Internet.

[0912] "Means for analyzing advertising videos received from users, extracting important parts, and generating summarized advertising videos" refers to a function that analyzes advertising videos provided by advertisers, extracts important visual and auditory scenes, and generates effective summarized advertising videos.

[0913] "Means of predicting viewer reactions based on data processing and proposing optimal distribution timing" refers to a function that collects and analyzes viewing data, predicts future viewer reactions based on viewer behavior patterns, and proposes optimal advertising distribution timing.

[0914] MODE FOR CARRYING OUT THE INVENTION

[0915] The present invention relates to a system for efficiently summarizing video data and predicting viewer reactions. DETAILED DESCRIPTION OF THE INVENTION The present invention will be described below with reference to specific examples.

[0916] Overall system configuration

[0917] This system consists of a server that acquires and analyzes video data, a terminal that distributes the generated summarized video, and a user who uploads the video data. The main elements are as follows:

[0918] 1. How to obtain video data

[0919] Users use an interface to upload video data to the server. For example, they can upload advertising videos or product explanation videos via a web browser or smartphone application.

[0920] 2. A method for analyzing video data and identifying important parts

[0921] The server analyzes the uploaded video data. Specifically, it uses OpenCV to divide the video into frames and identify important visual and auditory scenes. It also uses speech recognition technology to extract important parts from the audio data.

[0922] 3. A method for generating a summary of the video based on the identified key parts

[0923] The server then concatenates key scenes based on the analysis results and generates a condensed video using MoviePy, which can be enhanced with visual effects to make it more visually appealing.

[0924] 4. Means of distributing the generated summarized video

[0925] The summarized video generated by the server is delivered to the terminal via the Internet, and the user can view or download the summarized video from the provided link.

[0926] 5. Means for analyzing the advertisement video received from the user, extracting important parts, and generating a summary advertisement video

[0927] The server particularly analyzes the advertisement video received from the advertiser, extracts important parts that will attract the viewer's attention, and generates an effective summary advertisement video.

[0928] 6. A method for predicting viewer reactions based on data processing and proposing optimal distribution timing

[0929] The server collects viewing data, analyzes viewer behavior patterns based on the collected data, and uses a generative AI model to predict these patterns and suggest optimal distribution timing and channels.

[0930] Hardware and Software

[0931] Hardware: General PCs, servers, smartphones

[0932] Software: Python, OpenCV, MoviePy, transformers, OpenAI API

[0933] Operational procedures and examples

[0934] 1. Upload a video

[0935] Users log in to the system from their smartphones or PCs and upload their own advertising videos. For example, advertisers may upload promotional videos for new products.

[0936] 2. Video analysis and summary generation

[0937] The server analyzes the uploaded video and identifies important scenes using frame analysis with OpenCV and speech recognition technology, then concatenates the identified important parts and generates a summarized video using MoviePy.

[0938] 3. Summary Text Generation

[0939] The server also analyzes the audio data from the video and converts it into text using speech recognition technology, then generates a summary text using a generative AI model.

[0940] 4. Predictions and Recommendations

[0941] The server collects viewing data and uses a generative AI model to predict viewer reactions, and then suggests optimal ad delivery timing based on this.

[0942] Example prompts for generative AI models

[0943] Extract key parts from promotional videos for new products and generate visually impactful summary videos. Convert the video audio into text and summarize the key points. Analyze viewing data to predict the effectiveness of the next advertising campaign.

[0944] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0945] Step 1:

[0946] Users log in to the system using a smartphone or PC and upload their own advertising video to the server. The advertising video file is provided as input. The server receives this file and stores it in a database. As output, a message indicating that the file has been saved is generated. Specifically, a file selection dialog is displayed and the selected video file is sent to the server.

[0947] Step 2:

[0948] The server uses OpenCV to split the uploaded video file into frames. The video file saved in step 1 is provided as input. Data processing involves splitting the video into frames based on timestamps and saving each frame image as a list. The output is a set of frame images in list format. Specifically, the server reads the video file and analyzes each frame sequentially.

[0949] Step 3:

[0950] The server converts the video's audio data into text using speech recognition technology (for example, the speech recognition model from the transformers package). The video file saved in step 1 is provided as input. Data calculations involve extracting the audio data and converting it into text. The output is text data generated from the audio data. Specifically, the audio portion of the video file is analyzed and transcribed using a speech recognition engine.

[0951] Step 4:

[0952] The server uses the generative AI model to summarize the text data and extract key points. The text data generated in step 3 is provided as input. The data is calculated by summarizing the text data based on natural language processing technology. The output is the summarized text data. Specifically, the text data is input into the generative AI model, and a summary is generated.

[0953] Step 5:

[0954] The server identifies important scenes based on the image data and audio analysis results for each frame. The input is the frame image data from step 2 and the audio analysis results from step 3. The data calculation involves scoring and selecting important scenes based on visual and auditory features. The output is a list of important scenes. Specifically, the analysis results for each frame are evaluated using a scoring algorithm.

[0955] Step 6:

[0956] The server concatenates key scenes to generate a condensed video. The input is the list of key scenes identified in step 5. The data is processed by concatenating these scenes in order and combining them into a single short video. The output is a condensed video file. Specifically, MoviePy is used to edit the key scenes and combine them into a single video file.

[0957] Step 7:

[0958] The server distributes the generated summarized video to users and viewers. The input is the summarized video file generated in step 6. The data calculation involves uploading the video file and generating a link. The output is the generation of a viewing link, which is notified to the user. Specifically, the operation involves uploading the video using a video hosting service and providing a viewing link.

[0959] Step 8:

[0960] The server collects viewing data and analyzes viewer behavior patterns. Viewing data such as the number of clicks on viewing links and viewing time is provided as input. Data calculation involves compiling and analyzing the viewing data. The output is analysis results showing viewer behavior patterns. Specifically, the server uses a viewing tracking system to collect data and perform statistical analysis.

[0961] Step 9:

[0962] A generative AI model is used to predict viewer reactions based on viewing data, and the optimal timing for ad delivery is proposed. The input is the viewing data analysis results from step 8. Data calculation involves analyzing behavioral patterns and applying a predictive model. The output is a proposal for the optimal timing for ad delivery. Specifically, the predictive algorithm from the generative AI model is applied to generate the proposed results.

[0963] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0964] MODE FOR CARRYING OUT THE INVENTION

[0965] The present invention is a system for efficiently summarizing video data from product presentations and predicting viewer reactions. This system consists of a server, a terminal, and a user. By combining it with an emotion engine, the system can recognize viewer emotions and identify the importance of content with high accuracy based on those emotions. Each processing step is explained in detail below.

[0966] System configuration

[0967] 1. The server provides an upload interface for acquiring video data. Users upload the video of the product presentation to the server through this interface.

[0968] 2. The server splits the uploaded video data into frames and processes them as individual image data, making each frame in the video ready for analysis.

[0969] 3. The server analyzes the video and audio data using generative AI and speech recognition technology. As a result of the analysis, important parts are scored and identified. These important parts are then used to generate a condensed version of the video.

[0970] 4. The server uses an emotion engine to analyze the viewer's emotional state. Based on the viewer's facial expression and voice data collected during viewing, the server recognizes the viewer's emotions and reflects this emotional data in the video analysis. For example, scenes that the viewer is particularly interested in are given a higher importance rating.

[0971] 5. The server generates a video summary based on the identified key parts and emotional data. It stitches together key scenes to make it easier for users to understand, and generates a shortened video that reflects the viewer's emotional state. This process also includes adding transition effects and subtitles.

[0972] 6. The server delivers the generated summary video to the device. This delivery is done via the Internet, and the user can watch or download the summary video by clicking on a link.

[0973] Creating a text summary

[0974] 1. A user can request a summary of the briefing video. If the request is accepted, the server extracts the audio data from the video.

[0975] 2. The server converts the audio data into text using speech recognition software, which is then processed with natural language processing techniques to remove noise and extract key points from the audio data.

[0976] 3. The server uses generative AI to summarize the text data and generate a text that concisely expresses the main content. This summary text can be used as a report or marketing material.

[0977] Predicting audience reactions

[0978] 1. The server collects viewing data, including viewer attributes, viewing time, number of plays, number of clicks, emotional data, etc. This data is used to analyze viewer behavior patterns.

[0979] 2. The server uses generative AI to analyze the collected viewing data and learn viewer behavior patterns and trends. By comparing and analyzing past and current data, a predictive model of viewer reactions is generated.

[0980] 3. The server uses the predictive model to predict future inquiries and applications, and displays this forecast data in a user-accessible dashboard.

[0981] Specific examples

[0982] 1. The user uploads a video of a new product briefing to the server.

[0983] 2. The server analyzes the video data and uses an emotion engine to analyze the viewer's emotional state. Scenes that the viewer particularly enjoyed are identified as important parts with particularly high scores.

[0984] 3. The server generates a five-minute video summary based on these key scenes, edits it to maximize viewer interest, and delivers the video to the user.

[0985] 4. The user clicks on the download link to watch the generated summary video.

[0986] 5. When the user requests a summary, the server converts the audio data into text, summarizes the key points, and provides the text in a format that the user can download.

[0987] 6. The server collects viewing data and analyzes the viewer's behavioral patterns and emotional state. Based on this, it predicts the number of inquiries and applications and notifies the user.

[0988] In this way, the present invention efficiently generates summary video and text of a briefing session and predicts viewer reactions, greatly improving the efficiency of users' work. Furthermore, by combining it with an emotion engine, it becomes possible to provide content that reflects the viewer's emotional state, thereby drawing out their interest with greater accuracy.

[0989] The processing flow will be explained below.

[0990] Creating a summary video

[0991] Step 1:

[0992] The user uploads the video file of the briefing to the server by selecting the video file through the user interface and uploading it to the specified folder.

[0993] Step 2:

[0994] The server divides the uploaded video data into frames, dividing the video file into frames at regular intervals and generating individual image data.

[0995] Step 3:

[0996] The server uses generative AI to analyze the video data. It uses an AI algorithm to score important scenes from each frame, particularly by analyzing the speaker's facial expressions, movements, and vocal rhythm to determine their importance.

[0997] Step 4:

[0998] The server uses an emotion engine to analyze the viewer's emotional state, acquiring facial expression and voice data and recognizing emotions in real time.

[0999] Step 5:

[1000] Based on the results of the emotion engine, the server re-scores frames that viewers showed particular interest in as high scores, allowing the emotion data to be reflected in the selection of important scenes in the video.

[1001] Step 6:

[1002] The server generates a video summary based on the scored frames. High-scoring frames are extracted and stitched together to create a short video summary.

[1003] Step 7:

[1004] The server delivers the abridged video to the device, saves the completed video file, and provides a download link to the user's device, which the user can click to watch or download the abridged video.

[1005] Creating a summary of the content

[1006] Step 1:

[1007] A user requests the generation of a summary text for an uploaded video by selecting the summary generation option through a dedicated interface.

[1008] Step 2:

[1009] The server extracts the audio data from the briefing, separates it from the video, and passes it to the speech recognition engine.

[1010] Step 3:

[1011] The server uses speech recognition software to convert speech to text, converts the speech data into text data, and uses natural language processing techniques to remove noise.

[1012] Step 4:

[1013] The server uses generative AI to summarize the generated text data, extracting key keywords and phrases and summarizing them into easy-to-understand sentences.

[1014] Step 5:

[1015] The server delivers the summary text to the user's device, formats the summary results, and displays them on the user's dashboard or provides a downloadable link.

[1016] Predicting audience reactions

[1017] Step 1:

[1018] The server collects viewing data, such as the viewer's access history, viewing time, number of plays, number of clicks, and emotional data, and stores this data in real time.

[1019] Step 2:

[1020] The server analyzes the collected viewing data, using statistical methods and machine learning algorithms to analyze viewer behavior patterns.

[1021] Step 3:

[1022] The server predicts viewer reactions based on viewing data and past reaction data, for example by building a model to predict the number of inquiries and applications.

[1023] Step 4:

[1024] The server notifies the user of the prediction results. The generated prediction data is displayed visually and provided to the user's dashboard. The user can then use this information to plan their next strategy.

[1025] As described above, the system of the present invention efficiently generates summary videos and texts of briefing sessions and predicts viewer reactions, greatly improving the efficiency of users' work. Furthermore, by combining it with an emotion engine, it becomes possible to provide content that reflects the viewer's emotional state, thereby more accurately attracting viewer interest.

[1026] Example 2

[1027] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1028] Conventional video summarization systems have difficulty accurately identifying important parts of a video, and have been unable to generate summaries that take into account the viewer's emotions. It has also been difficult to accurately predict viewer reactions. This has led to issues such as users being unable to provide content that piques the viewer's interest, making it difficult to carry out effective marketing and presentations.

[1029] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1030] In this invention, the server includes means for acquiring video data, means for dividing the acquired video data into frames, means for analyzing the divided frame data and identifying important portions, means for collecting and analyzing viewer emotion data, means for generating a summary of the video based on the identified important portions and the viewer emotion data, means for adding transfer effects and subtitles to the generated summary video, and means for delivering the generated summary video. This makes it possible to automatically extract important scenes that reflect the viewer's emotional state, generate highly accurate video summaries, and predict viewer reactions.

[1031] "Video data" refers to data in a file format that continuously records visual information and is played back on a computer or electronic device.

[1032] A "frame" is an individual still image that constitutes video data, and is recognized as a video when it is displayed continuously at regular intervals of seconds.

[1033] "Analysis" is the process of analyzing input data in detail and extracting useful information based on that analysis.

[1034] An "important portion" is a scene or element that is considered to be particularly important within the video data, and is a portion that contains information that is highly valuable to the viewer.

[1035] "Viewer emotional data" refers to data that indicates the emotional state of a viewer that can be obtained from facial expressions, voice, behavior, etc. that the viewer displays while watching a video.

[1036] A "summary video" is a video that has been reconstructed in a shortened format by extracting important parts from the original video data.

[1037] A "transition effect" is a visual effect that creates a smooth transition between different scenes or frames.

[1038] "Subtitles" are textual information displayed to supplement the visual content of a video, providing explanations and translations to viewers.

[1039] "Distribution" is the process of transmitting created content to users over the Internet or a communication network, making it accessible to them.

[1040] "Voice data" refers to data in a file format that records a voice signal, and is the subject of voice recognition.

[1041] "Viewing data" refers to data such as the viewer's behavior and attributes when viewing video content, viewing time, number of plays, and number of clicks.

[1042] A "generative AI model" is an artificial intelligence algorithm or software that learns from large amounts of data and generates appropriate outputs based on the input data.

[1043] This invention is a system for efficiently summarizing video data, such as product presentations, and predicting viewer reactions. This system is composed of a server, terminals, and users, and by combining an emotion engine, it is possible to recognize viewer emotions and identify the importance of content based on those emotions.

[1044] Server Configuration

[1045] The server has the following main functions:

[1046] 1. Acquiring video data:

[1047] The server provides an interface for users to upload video data. Specifically, files can be uploaded using a web page or an application.

[1048] 2. Video data frame division:

[1049] The server uses video processing software such as FFmpeg to split the uploaded video data into frames, which makes later analysis easier.

[1050] 3. Video and audio data analysis:

[1051] The server analyzes the video and audio data using a generative AI model and speech recognition technology. In this step, a generative AI model (e.g., OpenAI's GPT model) is used to understand the text and images, and speech recognition technology such as Google Speech-to-Text is used to convert the speech to text.

[1052] 4. Identify the important parts:

[1053] The server identifies important parts of the video based on the analysis results, for example, by using information scored by a generative AI model to extract particularly important scenes.

[1054] 5. Emotional Data Analysis:

[1055] The server collects viewers' viewing data (facial expressions and voice data) and analyzes it using an emotion engine (e.g., Amazon Rekognition or Microsoft Azure's emotion recognition API). Scenes that viewers find particularly interesting are given high marks.

[1056] 6. Generate a condensed video:

[1057] The server generates a condensed version of the video based on the identified key parts and viewer sentiment data. It uses Adobe Premiere Pro APIs and other tools to stitch together key scenes to create a shortened video. It also adds transition effects and subtitles.

[1058] 7. Video Distribution:

[1059] The server generates a URL link to deliver the digested video, and users can view the digest by clicking on this link.

[1060] Specific examples

[1061] Uploading videos

[1062] The user uploads a video of a new product briefing to the server.

[1063] Video and audio analysis

[1064] The server divides the uploaded video data into frames using Ffmpeg and analyzes the video and audio data using a generative AI model and voice recognition technology.

[1065] Key points and sentiment analysis

[1066] The server identifies important parts of the video based on the analysis results and collects and analyzes viewer emotion data.

[1067] Summary video generation and delivery

[1068] The server uses the Adobe Premiere Pro API to generate a condensed version of the video based on the identified key parts and viewer sentiment data, and delivers it to the user via a URL link.

[1069] Prompt Sentence Examples

[1070] I've uploaded a video of a new product briefing. I want you to analyze the viewer's emotional state and generate a 5-minute summary version of the video.

[1071] This invention automatically extracts important scenes that reflect the viewer's emotional state, generates highly accurate video summaries, and predicts viewer reactions, allowing users to maximize viewer interest and conduct effective marketing and presentations.

[1072] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1073] Step 1:

[1074] Uploading videos

[1075] The server provides an interface for users to upload video data, which may include platforms such as web pages or applications.

[1076] The user presses the file selection button in the browser or app, selects the briefing video file, and sends the video to the server.

[1077] Input: Product presentation video file

[1078] Output: Video file stored on the server

[1079] Step 2:

[1080] Video data frame division

[1081] The server uses video processing software such as FFmpeg to split the uploaded video data into frames.

[1082] Using FFmpeg, the video data is split into still images at 30 frames per second, and each frame is saved as individual image data.

[1083] Input: Video file

[1084] Output: Split frame image data

[1085] Step 3:

[1086] Video and audio data analysis

[1087] The server analyzes the video and audio data using generative AI models and voice recognition technology.

[1088] A generative AI model (e.g., GPT model) is used to analyze the frame images and score important parts. Google Speech-to-Text technology is used to convert the audio data into text.

[1089] Input: Frame image data and audio data

[1090] Output: Text data with scores and frame analysis results

[1091] Step 4:

[1092] Identifying the important parts

[1093] The server lists important scenes based on the analysis results and identifies particularly important parts.

[1094] The scored data is used to extract scenes that are likely to be of particular interest to viewers.

[1095] Input: Text data with scores and frame analysis results

[1096] Output: Important Scene List

[1097] Step 5:

[1098] Emotional Data Analysis

[1099] The server collects viewers' viewing data and analyzes it using an emotion engine.

[1100] The viewer's facial expression and voice data are input into an emotion engine to analyze the viewer's emotional state and evaluate their emotional response to a particular scene.

[1101] Input: Viewing data (facial expression and voice data)

[1102] Output: Emotion analysis results

[1103] Step 6:

[1104] Generate a summary video

[1105] The server generates a summary version of the video based on the identified important parts and viewer emotion data.

[1106] Use video editing software such as Adobe Premiere Pro API to stitch together key scenes to create a short video, add transfer effects and subtitles.

[1107] Input: Important scene list and sentiment analysis results

[1108] Output: Abridged video

[1109] Step 7:

[1110] Video distribution

[1111] The server generates a URL link to distribute the generated summarized video and sends it to the user.

[1112] Users click on the link they receive to watch or download the abridged video.

[1113] Input: Abridged video

[1114] Output: URL link of the summary video

[1115] Prompt Sentence Examples

[1116] I've uploaded a video of a new product briefing. I want you to analyze the viewer's emotional state and generate a 5-minute summary version of the video.

[1117] The above is the specific flow and operation of each processing step of the present invention.

[1118] (Application example 2)

[1119] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1120] Current video advertising distribution technology is unable to analyze viewer emotions and interests in real time and summarize and distribute videos based on that information, making it difficult to maximize viewer interest. It is also difficult to effectively provide personalized advertising based on viewing data. This reduces advertising effectiveness and creates challenges in maintaining viewer interest.

[1121] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1122] In this invention, the server includes means for acquiring video data, means for analyzing the acquired video data and identifying important portions, means for generating a video digest based on the identified important portions, means for distributing the generated digest video, means for analyzing viewer emotions, means for generating a digest advertisement based on the viewer's interests, and means for providing personalized advertisements based on viewing data. This enables the generation and distribution of effective digest advertisements based on the viewer's emotions and interests, maximizing advertising effectiveness and maintaining viewer interest.

[1123] "Video data" refers to data that contains visual video information, and is usually composed of a series of frames that have been photographed and recorded by a device such as a camera or video recorder.

[1124] The "important portion" refers to a portion of the video data that contains particularly meaningful scenes or information and is useful or interesting to the viewer.

[1125] "Abridged video" refers to a video that can be played visually and audibly in a shortened format by extracting important parts from the original video data.

[1126] "Distribution" refers to the act of sending the generated summarized video to the viewer's terminal using a communication network such as the Internet.

[1127] "Viewer emotion" refers to the emotional reaction that a viewer shows while watching a video, and refers to a mental state such as joy, surprise, interest, or dissatisfaction.

[1128] "Viewer interest" refers to a state in which a viewer shows particular interest in a particular scene or piece of information within a video.

[1129] "Personalized advertising" means advertising content that is customized based on a viewer's individual emotional state and interests.

[1130] "Viewing data" refers to detailed data that records the viewer's behavior and reactions while watching a video, including viewing time, number of clicks, number of plays, changes in emotions, and the like.

[1131] The present invention provides a system for generating a digest version of an advertisement based on a viewer's emotions and interests to provide personalized advertisements in order to improve the viewing experience of advertisement videos. The system is composed of a server, a terminal, and a user.

[1132] The server first has a means for acquiring video data. When a user uploads an advertising video to the server, the server analyzes the acquired video data and identifies important parts. This analysis includes frame analysis using OpenCV and emotion analysis using EmotionRecognizer. Once important parts are identified, a means for generating a summarized version of the video based on the identified parts is activated. Here, Moviepy is used to connect important frames and create a summarized version of the video.

[1133] EmotionRecognizer is also used as a means of analyzing viewer emotions. It collects emotional and viewing data from viewers while they are watching, and generates abridged advertisements based on their interests during viewing. It then analyzes the viewing data to provide personalized advertisements based on the viewer's interests. This process involves data analysis based on the collected viewing data. Data points such as viewing time, number of clicks, number of views, and emotional state are utilized in the analysis.

[1134] Specifically, the system operates in the following scenario: While User A is watching an advertising video using a device, the server analyzes the video in real time to identify the viewer's emotions and interests. The server extracts scenes that the viewer particularly finds interesting and creates a shortened, summarized version of the advertisement. Next, a personalized advertisement is provided to User A based on User A's viewing data. The generated summarized advertisement is then distributed via the Internet for the user to view.

[1135] Examples of prompts include:

[1136] "Your goal is to build a system that analyzes the real-time emotional state of users watching a video explaining a new product, and summarizes and displays scenes that piqued their interest. This system will analyze video data uploaded to a server and maximize the viewer's interest."

[1137] In this way, the present invention realizes summarization and personalization of advertising videos based on the viewer's emotions and interests, thereby increasing advertising effectiveness and improving the viewer experience.

[1138] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1139] Step 1:

[1140] The server acquires the advertising video data uploaded by the user. The input is the video file provided by the user, which is saved in the server's data storage. The output is the acquired video file saved on the server.

[1141] Step 2:

[1142] The server divides the acquired video data into frames and makes them available for analysis as individual image data. The input is the uploaded video file, and OpenCV is used for data processing to break the video down into frames. The output is that each frame of the video is saved as image data on the server.

[1143] Step 3:

[1144] The server analyzes the viewer's emotions for each frame. The input is the image data of each frame obtained in step 2, and performs emotion analysis using EmotionRecognizer. The output is emotional state data for each frame.

[1145] Step 4:

[1146] The server aggregates the emotional state data and identifies important segments that indicate viewer interest. The input is the emotional state data obtained in step 3, and the data is calculated by evaluating the emotional score and marking segments that exceed a certain interest score as important. The output is a list of frames that indicate important segments.

[1147] Step 5:

[1148] The server generates a digest video based on the identified key segments. The input is the frame list of key segments obtained in step 4, and Moviepy is used to concatenate these frames to create a digest video. The output is the generated digest video file.

[1149] Step 6:

[1150] The server delivers the generated summarized video to the terminal. The input is the summarized video file generated in step 5, and a link that the user can access via the Internet is generated. The output is the status that the video is available for viewing on the user's terminal.

[1151] Step 7:

[1152] The server collects and analyzes viewing data. The input is the viewing data (viewing time, clicks, plays, emotional state, etc.) generated when users watch the abridged video, and the viewing database is used for data analysis. The output is the analyzed viewing data.

[1153] Step 8:

[1154] The server predicts each user's interests based on the aggregated viewing data and provides personalized advertisements. The input is the viewing data obtained in step 7, and the generative AI model is used to predict the user's interests. The output is personalized advertising content.

[1155] In this way, a system is realized that performs a series of processes from acquiring video data to distributing it, analyzing viewing data, and providing personalized advertisements through each step between the server, terminal, and user.

[1156] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1157] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1158] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1159] [Fourth embodiment]

[1160] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1161] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1162] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1163] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1164] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1165] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1166] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1167] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1168] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1169] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1170] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1171] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1172] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1173] MODE FOR CARRYING OUT THE INVENTION

[1174] The present invention provides a system for efficiently summarizing video data from a product presentation and predicting viewer reactions. The system is composed of a server, a terminal, and a user. Each processing step is described in detail below.

[1175] System configuration

[1176] 1. The server provides an upload interface for acquiring video data. Users upload the video of the product presentation to the server through this interface.

[1177] 2. The server splits the uploaded video data into frames and processes them as individual image data, making each frame in the video ready for analysis.

[1178] 3. The server analyzes the video and audio data using generative AI and speech recognition technology. As a result of the analysis, important parts are scored and identified. These important parts are then used to generate a condensed version of the video.

[1179] 4. The server creates a video summary based on the identified key parts. It generates a shortened video by stitching together key scenes for easy user comprehension. This process also includes adding transition effects and subtitles.

[1180] 5. The server delivers the generated summary video to the device. This delivery is done via the Internet, and the user can watch or download the summary video by clicking on a link.

[1181] Creating a text summary

[1182] 1. A user can request a summary of the briefing video. If the request is accepted, the server extracts the audio data from the video.

[1183] 2. The server converts the audio data into text using speech recognition software, which is then processed with natural language processing techniques to remove noise and extract key points from the audio data.

[1184] 3. The server uses generative AI to summarize the text data and generate a text that concisely expresses the main content. This summary text can be used as a report or marketing material.

[1185] Predicting audience reactions

[1186] 1. The server collects viewing data, including viewer attributes, viewing time, number of plays, number of clicks, etc. This data is used to analyze viewer behavior patterns.

[1187] 2. The server uses generative AI to analyze the collected viewing data and learn viewer behavior patterns and trends. By comparing and analyzing past and current data, a predictive model of viewer reactions is generated.

[1188] 3. The server uses the predictive model to predict future inquiries and applications, and displays this forecast data in a user-accessible dashboard.

[1189] Specific examples

[1190] 1. The user uploads a video of a new product briefing to the server.

[1191] 2. The server analyzes the video data, identifies important parts, and generates a 5-minute video summary that is then delivered to the user.

[1192] 3. The user clicks on the download link to watch the generated summary video.

[1193] 4. When the user requests a summary, the server converts the audio data into text, summarizes the key points, and provides the text in a format that the user can download.

[1194] 5. The server collects viewing data and analyzes viewer trends. Based on this, it predicts the number of inquiries and applications and notifies users.

[1195] In this way, the present invention realizes efficient and effective video editing of product presentations and prediction of audience reactions.

[1196] The processing flow will be explained below.

[1197] Creating a summary video

[1198] Step 1:

[1199] The user uploads the video file of the briefing to the server by selecting the video file through the user interface and uploading it to the specified folder.

[1200] Step 2:

[1201] The server divides the uploaded video data into frames, dividing the video file into frames at regular intervals and generating individual image data.

[1202] Step 3:

[1203] The server uses generative AI to analyze the video data. It uses an AI algorithm to score important scenes from each frame, particularly by analyzing the speaker's facial expressions, movements, and vocal rhythm to determine their importance.

[1204] Step 4:

[1205] The server generates a summary of the video based on the scored frames, and then extracts and stitches together the frames with the highest scores to create a short summary video.

[1206] Step 5:

[1207] The server delivers the abridged video to the device, saves the completed video file, and provides a download link to the user's device, which the user can click to watch or download the abridged video.

[1208] Creating a summary of the content

[1209] Step 1:

[1210] A user requests the generation of a summary text for an uploaded video by selecting the summary generation option through a dedicated interface.

[1211] Step 2:

[1212] The server extracts the audio data from the briefing, separates it from the video, and passes it to the speech recognition engine.

[1213] Step 3:

[1214] The server uses speech recognition software to convert speech to text, converts the speech data into text data, and uses natural language processing techniques to remove noise.

[1215] Step 4:

[1216] The server uses generative AI to summarize the generated text data, extracting key keywords and phrases and summarizing them into easy-to-understand sentences.

[1217] Step 5:

[1218] The server delivers the summary text to the user's device, formats the summary results, and displays them on the user's dashboard or provides a downloadable link.

[1219] Predicting audience reactions

[1220] Step 1:

[1221] The server collects viewing data, such as the viewer's access history, viewing time, number of plays, and number of clicks, and stores this data in real time.

[1222] Step 2:

[1223] The server analyzes the collected viewing data, using statistical methods and machine learning algorithms to analyze viewer behavior patterns.

[1224] Step 3:

[1225] The server predicts viewer reactions based on viewing data and past reaction data, for example by building a model to predict the number of inquiries and applications.

[1226] Step 4:

[1227] The server notifies the user of the prediction results. The generated prediction data is displayed visually and provided to the user's dashboard. The user can then use this information to plan their next strategy.

[1228] As described above, the system of the present invention efficiently generates a summary video and text of a briefing session and predicts viewer reactions, thereby significantly improving the efficiency of users' work.

[1229] Example 1

[1230] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1231] Traditional video data from product presentations is long and consumes viewers' time, often burying important information. Furthermore, there is a lack of systematic methods for predicting viewer reactions, making it difficult to improve engagement. Furthermore, there is a need for an efficient method for quickly and accurately generating abridged video and summary text.

[1232] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1233] In this invention, the server includes means for acquiring video data, means for dividing the acquired video data into frames, means for analyzing the divided frames and identifying important portions, means for converting audio data into text and extracting important content, means for generating a summary of the video based on the identified important portions and the extracted important content, means for adding transfer effects and subtitles to the generated summary video, means for distributing the generated summary video, means for collecting and analyzing viewing data, and means for predicting viewer responses based on the collected data. This makes it possible to quickly extract important information from long videos and efficiently generate summary videos and summary text. Furthermore, predicting viewer responses can improve engagement and optimize marketing strategies.

[1234] "Video data" refers to video content such as product presentations, and is data that includes visual and audio information.

[1235] The "means for acquiring" is a function including an interface and a protocol for transmitting video data from a user to a server.

[1236] A "frame" is an individual still image that makes up a video.

[1237] The "dividing means" is a function that performs processing to cut out video data in time units and save them as individual frames.

[1238] The "analysis means" refers to a technique for extracting important information from divided frames or audio data, and performing evaluation and scoring.

[1239] "Audio data" refers to audio information contained in a video, including narration, conversation, music, and the like.

[1240] "Means for converting to text" refers to speech recognition technology for recognizing voice data as character information and converting it into text format.

[1241] "Extraction means" refers to techniques for finding important content or scenes from the analyzed text or video frames.

[1242] The "means for generating a summary version" is a function that connects the extracted important parts together to create a video that is shorter and easier to understand.

[1243] "Transfer effect" is a technique for adding visual effects when switching scenes.

[1244] "Subtitles" are the display of dialogue and explanations in a video as text.

[1245] The "distribution means" refers to a technique for providing the generated summarized video to users via a communication means such as the Internet.

[1246] "Viewing data" is information relating to the viewer's attributes, viewing time, number of plays, number of clicks, and other behavior.

[1247] "Means of collection" refers to tracking technologies and analytical tools used to automatically collect viewing data.

[1248] "Viewer response" refers to the behavioral patterns and emotional reactions of viewers when they watch a video.

[1249] "Predictive methods" are analytical techniques that use collected viewing data to predict future viewer behavior and reactions.

[1250] The present invention relates to a system for efficiently summarizing video data from a product presentation and predicting viewer reactions. The system is composed of a server, terminals, and users.

[1251] The server provides an interface for users to obtain video data. When a user uploads a video file to the server, the server divides the video data into frames. This process uses video analysis tools such as FFmpeg and OpenCV.

[1252] The server analyzes the separated frames using generative AI models and speech recognition technology. For example, it uses Google Cloud Video Intelligence API or Amazon Transcribe to convert the content and audio data of the video frames into text. Based on the analysis, important parts of the video are identified. An example of a prompt is, "What important events are contained in this video frame?"

[1253] The server then generates a condensed version of the video based on the identified key parts. During this process, video editing software such as Adobe Premiere Pro or iMovie is used to add transition effects and subtitles to the shortened video. The condensed version is then distributed to devices via the Internet, and users can watch or download the condensed version by clicking a link.

[1254] When a user requests a text summary, the server converts the audio data into text using the Google Speech-to-Text API or Amazon Transcribe. It then uses a generative AI model (e.g., GPT-4) to summarize the key content. An example prompt at this stage is "Please summarize the following text: [Extracted text]."

[1255] In addition, the server collects viewing data (viewer attributes, viewing time, number of views, number of clicks, etc.) and analyzes viewer behavior patterns using tools such as Google Analytics and Mixpanel. A generative AI model is used to analyze the data and predict viewer responses. This allows users to predict future inquiries and applications, and the results can be viewed on a dashboard.

[1256] As a concrete example, a user uploads a video of a new product briefing to a server, which analyzes the video data to generate a 5-minute summary of the video and delivers it to the user. The user clicks a link to watch the video and then requests a summary text. The server converts the audio data into text and generates a summary. Finally, the server collects viewing data and predicts the viewer's reaction.

[1257] In this way, the present invention realizes efficient and effective video editing of product presentations and prediction of audience reactions.

[1258] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1259] System program processing flow

[1260] Step 1: Upload video data

[1261] The server provides an interface for obtaining video data from the user. The user uses this interface to upload video data from their device to the server. The input in this step is a video file selected by the user, and the output is a video file stored on the server.

[1262] Specifically, the server stores the video file in cloud storage and registers the file's metadata (file size, format, resolution, etc.) in a database.

[1263] Step 2: Split the video data into frames

[1264] The server splits the uploaded video data into frames. The input of this step is the video file stored on the server, and the output is the individual frame image files.

[1265] Specifically, the server uses FFmpeg to split the video into fixed frame intervals and save each frame as an image file, generating still images that can be analyzed.

[1266] Step 3: Analyze video and audio data

[1267] The server analyzes the video and audio data using generative AI models and speech recognition technology. The input for this step is the frame image files and audio data, and the output is data identifying key parts.

[1268] Specifically, the server uses the Google Cloud Video Intelligence API to analyze the content of each frame (e.g., objects, actions, scene changes), and then uses Amazon Transcribe to convert the audio data into text. For example, a prompt sentence like "What significant events does this video frame contain?" is input to the generative AI model.

[1269] Step 4: Generate a condensed video

[1270] The server generates a digest version of the video based on the identified important parts. The input of this step is the data of the important parts, and the output is a digest version of the video file.

[1271] Specifically, the server uses Adobe Premiere Pro scripting to stitch together key frames, add transfer effects and subtitles, and generate a condensed video, which is then stored on the server.

[1272] Step 5: Publish a condensed version of your video

[1273] The server then distributes the generated digested video to the user's device via the Internet. The user then clicks on the provided link to watch or download the video. The input of this step is the digested video file, and the output is a link that the user can watch.

[1274] Specifically, the server uploads the summarized video to a cloud storage service and sends the user an email notification with a download link.

[1275] Step 6: Creating a text summary

[1276] When a user requests a text summary, the server converts the audio data into text and uses a generative AI model to create the summary. The input for this step is the audio data, and the output is the summarized text.

[1277] Specifically, the server sends the voice data to the Google Speech-to-Text API, converts it into text, and then inputs the prompt "Please summarize the following text: [Extracted text]" into the generative AI model to generate a summary text.

[1278] Step 7: Predict audience reaction

[1279] The server collects viewing data and uses a generative AI model to predict viewer reactions. The input of this step is the viewing data, and the output is the predicted results.

[1280] Specifically, the server collects viewing data (viewer attributes, viewing time, number of plays, number of clicks, etc.) using Google Analytics, inputs it into a generative AI model (for example, a custom model using TensorFlow) to learn viewer behavior patterns, and based on that, predicts the number of inquiries and applications and notifies users.

[1281] In this way, the system efficiently and effectively edits videos of product presentations and predicts audience reactions.

[1282] (Application example 1)

[1283] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1284] In the modern advertising industry, generating effective abridged advertising videos that attract viewers' attention is a key challenge. However, long advertising videos have difficulty sustaining viewers' attention, leading to lower viewer ratings. Furthermore, to maximize the effectiveness of advertising, it is necessary to predict viewer responses and select the optimal delivery timing, but this is extremely difficult to do manually.

[1285] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1286] In this invention, the server includes means for acquiring video data, means for analyzing the acquired video data and identifying important parts, means for generating a summary of the video based on the identified important parts, means for distributing the generated summary video, means for analyzing the advertising video received from the user and extracting important parts to generate a summary advertising video, and means for predicting viewer reactions based on data processing and proposing optimal distribution timing, thereby enabling the generation of effective summary advertising videos and optimal advertising distribution taking viewer reactions into consideration.

[1287] The "means for acquiring video data" is a function for users to upload multimedia data such as advertising videos and product explanation videos to the server.

[1288] "Means for analyzing video data and identifying important parts" refers to a function in which the server analyzes uploaded video data frame by frame and automatically recognizes scenes that are visually and audibly important.

[1289] The "means for generating a summary version of a video based on identified important parts" is a function that automatically generates a summary version of a video that conveys the main content in a short amount of time by linking identified important scenes.

[1290] The "means for distributing the generated summarized video" is a function for distributing the generated summarized video to users and viewers via the Internet.

[1291] "Means for analyzing advertising videos received from users, extracting important parts, and generating summarized advertising videos" refers to a function that analyzes advertising videos provided by advertisers, extracts important visual and auditory scenes, and generates effective summarized advertising videos.

[1292] "Means of predicting viewer reactions based on data processing and proposing optimal distribution timing" refers to a function that collects and analyzes viewing data, predicts future viewer reactions based on viewer behavior patterns, and proposes optimal advertising distribution timing.

[1293] MODE FOR CARRYING OUT THE INVENTION

[1294] The present invention relates to a system for efficiently summarizing video data and predicting viewer reactions. DETAILED DESCRIPTION OF THE INVENTION The present invention will be described below with reference to specific examples.

[1295] Overall system configuration

[1296] This system consists of a server that acquires and analyzes video data, a terminal that distributes the generated summarized video, and a user who uploads the video data. The main elements are as follows:

[1297] 1. How to obtain video data

[1298] Users use an interface to upload video data to the server. For example, they can upload advertising videos or product explanation videos via a web browser or smartphone application.

[1299] 2. A method for analyzing video data and identifying important parts

[1300] The server analyzes the uploaded video data. Specifically, it uses OpenCV to divide the video into frames and identify important visual and auditory scenes. It also uses speech recognition technology to extract important parts from the audio data.

[1301] 3. A method for generating a summary of the video based on the identified key parts

[1302] The server then concatenates key scenes based on the analysis results and generates a condensed video using MoviePy, which can be enhanced with visual effects to make it more visually appealing.

[1303] 4. Means of distributing the generated summarized video

[1304] The summarized video generated by the server is delivered to the terminal via the Internet, and the user can view or download the summarized video from the provided link.

[1305] 5. Means for analyzing the advertisement video received from the user, extracting important parts, and generating a summary advertisement video

[1306] The server particularly analyzes the advertisement video received from the advertiser, extracts important parts that will attract the viewer's attention, and generates an effective summary advertisement video.

[1307] 6. A method for predicting viewer reactions based on data processing and proposing optimal distribution timing

[1308] The server collects viewing data, analyzes viewer behavior patterns based on the collected data, and uses a generative AI model to predict these patterns and suggest optimal distribution timing and channels.

[1309] Hardware and Software

[1310] Hardware: General PCs, servers, smartphones

[1311] Software: Python, OpenCV, MoviePy, transformers, OpenAI API

[1312] Operational procedures and examples

[1313] 1. Upload a video

[1314] Users log in to the system from their smartphones or PCs and upload their own advertising videos. For example, advertisers may upload promotional videos for new products.

[1315] 2. Video analysis and summary generation

[1316] The server analyzes the uploaded video and identifies important scenes using frame analysis with OpenCV and speech recognition technology, then concatenates the identified important parts and generates a summarized video using MoviePy.

[1317] 3. Summary Text Generation

[1318] The server also analyzes the audio data from the video and converts it into text using speech recognition technology, then generates a summary text using a generative AI model.

[1319] 4. Predictions and Recommendations

[1320] The server collects viewing data and uses a generative AI model to predict viewer reactions, and then suggests optimal ad delivery timing based on this.

[1321] Example prompts for generative AI models

[1322] Extract key parts from promotional videos for new products and generate visually impactful summary videos. Convert the video audio into text and summarize the key points. Analyze viewing data to predict the effectiveness of the next advertising campaign.

[1323] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1324] Step 1:

[1325] Users log in to the system using a smartphone or PC and upload their own advertising video to the server. The advertising video file is provided as input. The server receives this file and stores it in a database. As output, a message indicating that the file has been saved is generated. Specifically, a file selection dialog is displayed and the selected video file is sent to the server.

[1326] Step 2:

[1327] The server uses OpenCV to split the uploaded video file into frames. The video file saved in step 1 is provided as input. Data processing involves splitting the video into frames based on timestamps and saving each frame image as a list. The output is a set of frame images in list format. Specifically, the server reads the video file and analyzes each frame sequentially.

[1328] Step 3:

[1329] The server converts the video's audio data into text using speech recognition technology (for example, the speech recognition model from the transformers package). The video file saved in step 1 is provided as input. Data calculations involve extracting the audio data and converting it into text. The output is text data generated from the audio data. Specifically, the audio portion of the video file is analyzed and transcribed using a speech recognition engine.

[1330] Step 4:

[1331] The server uses the generative AI model to summarize the text data and extract key points. The text data generated in step 3 is provided as input. The data is calculated by summarizing the text data based on natural language processing technology. The output is the summarized text data. Specifically, the text data is input into the generative AI model, and a summary is generated.

[1332] Step 5:

[1333] The server identifies important scenes based on the image data and audio analysis results for each frame. The input is the frame image data from step 2 and the audio analysis results from step 3. The data calculation involves scoring and selecting important scenes based on visual and auditory features. The output is a list of important scenes. Specifically, the analysis results for each frame are evaluated using a scoring algorithm.

[1334] Step 6:

[1335] The server concatenates key scenes to generate a condensed video. The input is the list of key scenes identified in step 5. The data is processed by concatenating these scenes in order and combining them into a single short video. The output is a condensed video file. Specifically, MoviePy is used to edit the key scenes and combine them into a single video file.

[1336] Step 7:

[1337] The server distributes the generated summarized video to users and viewers. The input is the summarized video file generated in step 6. The data calculation involves uploading the video file and generating a link. The output is the generation of a viewing link, which is notified to the user. Specifically, the operation involves uploading the video using a video hosting service and providing a viewing link.

[1338] Step 8:

[1339] The server collects viewing data and analyzes viewer behavior patterns. Viewing data such as the number of clicks on viewing links and viewing time is provided as input. Data calculation involves compiling and analyzing the viewing data. The output is analysis results showing viewer behavior patterns. Specifically, the server uses a viewing tracking system to collect data and perform statistical analysis.

[1340] Step 9:

[1341] A generative AI model is used to predict viewer reactions based on viewing data, and the optimal timing for ad delivery is proposed. The input is the viewing data analysis results from step 8. Data calculation involves analyzing behavioral patterns and applying a predictive model. The output is a proposal for the optimal timing for ad delivery. Specifically, the predictive algorithm from the generative AI model is applied to generate the proposed results.

[1342] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1343] MODE FOR CARRYING OUT THE INVENTION

[1344] The present invention is a system for efficiently summarizing video data from product presentations and predicting viewer reactions. This system consists of a server, a terminal, and a user. By combining it with an emotion engine, the system can recognize viewer emotions and identify the importance of content with high accuracy based on those emotions. Each processing step is explained in detail below.

[1345] System configuration

[1346] 1. The server provides an upload interface for acquiring video data. Users upload the video of the product presentation to the server through this interface.

[1347] 2. The server splits the uploaded video data into frames and processes them as individual image data, making each frame in the video ready for analysis.

[1348] 3. The server analyzes the video and audio data using generative AI and speech recognition technology. As a result of the analysis, important parts are scored and identified. These important parts are then used to generate a condensed version of the video.

[1349] 4. The server uses an emotion engine to analyze the viewer's emotional state. Based on the viewer's facial expression and voice data collected during viewing, the server recognizes the viewer's emotions and reflects this emotional data in the video analysis. For example, scenes that the viewer is particularly interested in are given a higher importance rating.

[1350] 5. The server generates a video summary based on the identified key parts and emotional data. It stitches together key scenes to make it easier for users to understand, and generates a shortened video that reflects the viewer's emotional state. This process also includes adding transition effects and subtitles.

[1351] 6. The server delivers the generated summary video to the device. This delivery is done via the Internet, and the user can watch or download the summary video by clicking on a link.

[1352] Creating a text summary

[1353] 1. A user can request a summary of the briefing video. If the request is accepted, the server extracts the audio data from the video.

[1354] 2. The server converts the audio data into text using speech recognition software, which is then processed with natural language processing techniques to remove noise and extract key points from the audio data.

[1355] 3. The server uses generative AI to summarize the text data and generate a text that concisely expresses the main content. This summary text can be used as a report or marketing material.

[1356] Predicting audience reactions

[1357] 1. The server collects viewing data, including viewer attributes, viewing time, number of plays, number of clicks, emotional data, etc. This data is used to analyze viewer behavior patterns.

[1358] 2. The server uses generative AI to analyze the collected viewing data and learn viewer behavior patterns and trends. By comparing and analyzing past and current data, a predictive model of viewer reactions is generated.

[1359] 3. The server uses the predictive model to predict future inquiries and applications, and displays this forecast data in a user-accessible dashboard.

[1360] Specific examples

[1361] 1. The user uploads a video of a new product briefing to the server.

[1362] 2. The server analyzes the video data and uses an emotion engine to analyze the viewer's emotional state. Scenes that the viewer particularly enjoyed are identified as important parts with particularly high scores.

[1363] 3. The server generates a five-minute video summary based on these key scenes, edits it to maximize viewer interest, and delivers the video to the user.

[1364] 4. The user clicks on the download link to watch the generated summary video.

[1365] 5. When the user requests a summary, the server converts the audio data into text, summarizes the key points, and provides the text in a format that the user can download.

[1366] 6. The server collects viewing data and analyzes the viewer's behavioral patterns and emotional state. Based on this, it predicts the number of inquiries and applications and notifies the user.

[1367] In this way, the present invention efficiently generates summary video and text of a briefing session and predicts viewer reactions, greatly improving the efficiency of users' work. Furthermore, by combining it with an emotion engine, it becomes possible to provide content that reflects the viewer's emotional state, thereby drawing out their interest with greater accuracy.

[1368] The processing flow will be explained below.

[1369] Creating a summary video

[1370] Step 1:

[1371] The user uploads the video file of the briefing to the server by selecting the video file through the user interface and uploading it to the specified folder.

[1372] Step 2:

[1373] The server divides the uploaded video data into frames, dividing the video file into frames at regular intervals and generating individual image data.

[1374] Step 3:

[1375] The server uses generative AI to analyze the video data. It uses an AI algorithm to score important scenes from each frame, particularly by analyzing the speaker's facial expressions, movements, and vocal rhythm to determine their importance.

[1376] Step 4:

[1377] The server uses an emotion engine to analyze the viewer's emotional state, acquiring facial expression and voice data and recognizing emotions in real time.

[1378] Step 5:

[1379] Based on the results of the emotion engine, the server re-scores frames that viewers showed particular interest in as high scores, allowing the emotion data to be reflected in the selection of important scenes in the video.

[1380] Step 6:

[1381] The server generates a video summary based on the scored frames. High-scoring frames are extracted and stitched together to create a short video summary.

[1382] Step 7:

[1383] The server delivers the abridged video to the device, saves the completed video file, and provides a download link to the user's device, which the user can click to watch or download the abridged video.

[1384] Creating a summary of the content

[1385] Step 1:

[1386] A user requests the generation of a summary text for an uploaded video by selecting the summary generation option through a dedicated interface.

[1387] Step 2:

[1388] The server extracts the audio data from the briefing, separates it from the video, and passes it to the speech recognition engine.

[1389] Step 3:

[1390] The server uses speech recognition software to convert speech to text, converts the speech data into text data, and uses natural language processing techniques to remove noise.

[1391] Step 4:

[1392] The server uses generative AI to summarize the generated text data, extracting key keywords and phrases and summarizing them into easy-to-understand sentences.

[1393] Step 5:

[1394] The server delivers the summary text to the user's device, formats the summary results, and displays them on the user's dashboard or provides a downloadable link.

[1395] Predicting audience reactions

[1396] Step 1:

[1397] The server collects viewing data, such as the viewer's access history, viewing time, number of plays, number of clicks, and emotional data, and stores this data in real time.

[1398] Step 2:

[1399] The server analyzes the collected viewing data, using statistical methods and machine learning algorithms to analyze viewer behavior patterns.

[1400] Step 3:

[1401] The server predicts viewer reactions based on viewing data and past reaction data, for example by building a model to predict the number of inquiries and applications.

[1402] Step 4:

[1403] The server notifies the user of the prediction results. The generated prediction data is displayed visually and provided to the user's dashboard. The user can then use this information to plan their next strategy.

[1404] As described above, the system of the present invention efficiently generates summary videos and texts of briefing sessions and predicts viewer reactions, greatly improving the efficiency of users' work. Furthermore, by combining it with an emotion engine, it becomes possible to provide content that reflects the viewer's emotional state, thereby more accurately attracting viewer interest.

[1405] Example 2

[1406] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1407] Conventional video summarization systems have difficulty accurately identifying important parts of a video, and have been unable to generate summaries that take into account the viewer's emotions. It has also been difficult to accurately predict viewer reactions. This has led to issues such as users being unable to provide content that piques the viewer's interest, making it difficult to carry out effective marketing and presentations.

[1408] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1409] In this invention, the server includes means for acquiring video data, means for dividing the acquired video data into frames, means for analyzing the divided frame data and identifying important portions, means for collecting and analyzing viewer emotion data, means for generating a summary of the video based on the identified important portions and the viewer emotion data, means for adding transfer effects and subtitles to the generated summary video, and means for delivering the generated summary video. This makes it possible to automatically extract important scenes that reflect the viewer's emotional state, generate highly accurate video summaries, and predict viewer reactions.

[1410] "Video data" refers to data in a file format that continuously records visual information and is played back on a computer or electronic device.

[1411] A "frame" is an individual still image that constitutes video data, and is recognized as a video when it is displayed continuously at regular intervals of seconds.

[1412] "Analysis" is the process of analyzing input data in detail and extracting useful information based on that analysis.

[1413] An "important portion" is a scene or element that is considered to be particularly important within the video data, and is a portion that contains information that is highly valuable to the viewer.

[1414] "Viewer emotional data" refers to data that indicates the emotional state of a viewer that can be obtained from facial expressions, voice, behavior, etc. that the viewer displays while watching a video.

[1415] A "summary video" is a video that has been reconstructed in a shortened format by extracting important parts from the original video data.

[1416] A "transition effect" is a visual effect that creates a smooth transition between different scenes or frames.

[1417] "Subtitles" are textual information displayed to supplement the visual content of a video, providing explanations and translations to viewers.

[1418] "Distribution" is the process of transmitting created content to users over the Internet or a communication network, making it accessible to them.

[1419] "Voice data" refers to data in a file format that records a voice signal, and is the subject of voice recognition.

[1420] "Viewing data" refers to data such as the viewer's behavior and attributes when viewing video content, viewing time, number of plays, and number of clicks.

[1421] A "generative AI model" is an artificial intelligence algorithm or software that learns from large amounts of data and generates appropriate outputs based on the input data.

[1422] This invention is a system for efficiently summarizing video data, such as product presentations, and predicting viewer reactions. This system is composed of a server, terminals, and users, and by combining an emotion engine, it is possible to recognize viewer emotions and identify the importance of content based on those emotions.

[1423] Server Configuration

[1424] The server has the following main functions:

[1425] 1. Acquiring video data:

[1426] The server provides an interface for users to upload video data. Specifically, files can be uploaded using a web page or an application.

[1427] 2. Video data frame division:

[1428] The server uses video processing software such as FFmpeg to split the uploaded video data into frames, which makes later analysis easier.

[1429] 3. Video and audio data analysis:

[1430] The server analyzes the video and audio data using a generative AI model and speech recognition technology. In this step, a generative AI model (e.g., OpenAI's GPT model) is used to understand the text and images, and speech recognition technology such as Google Speech-to-Text is used to convert the speech to text.

[1431] 4. Identify the important parts:

[1432] The server identifies important parts of the video based on the analysis results, for example, by using information scored by a generative AI model to extract particularly important scenes.

[1433] 5. Emotional Data Analysis:

[1434] The server collects viewers' viewing data (facial expressions and voice data) and analyzes it using an emotion engine (e.g., Amazon Rekognition or Microsoft Azure's emotion recognition API). Scenes that viewers find particularly interesting are given high marks.

[1435] 6. Generate a condensed video:

[1436] The server generates a condensed version of the video based on the identified key parts and viewer sentiment data. It uses Adobe Premiere Pro APIs and other tools to stitch together key scenes to create a shortened video. It also adds transition effects and subtitles.

[1437] 7. Video Distribution:

[1438] The server generates a URL link to deliver the digested video, and users can view the digest by clicking on this link.

[1439] Specific examples

[1440] Uploading videos

[1441] The user uploads a video of a new product briefing to the server.

[1442] Video and audio analysis

[1443] The server divides the uploaded video data into frames using Ffmpeg and analyzes the video and audio data using a generative AI model and voice recognition technology.

[1444] Key points and sentiment analysis

[1445] The server identifies important parts of the video based on the analysis results and collects and analyzes viewer emotion data.

[1446] Summary video generation and delivery

[1447] The server uses the Adobe Premiere Pro API to generate a condensed version of the video based on the identified key parts and viewer sentiment data, and delivers it to the user via a URL link.

[1448] Prompt Sentence Examples

[1449] I've uploaded a video of a new product briefing. I want you to analyze the viewer's emotional state and generate a 5-minute summary version of the video.

[1450] This invention automatically extracts important scenes that reflect the viewer's emotional state, generates highly accurate video summaries, and predicts viewer reactions, allowing users to maximize viewer interest and conduct effective marketing and presentations.

[1451] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1452] Step 1:

[1453] Uploading videos

[1454] The server provides an interface for users to upload video data, which may include platforms such as web pages or applications.

[1455] The user presses the file selection button in the browser or app, selects the briefing video file, and sends the video to the server.

[1456] Input: Product presentation video file

[1457] Output: Video file stored on the server

[1458] Step 2:

[1459] Video data frame division

[1460] The server uses video processing software such as FFmpeg to split the uploaded video data into frames.

[1461] Using FFmpeg, the video data is split into still images at 30 frames per second, and each frame is saved as individual image data.

[1462] Input: Video file

[1463] Output: Split frame image data

[1464] Step 3:

[1465] Video and audio data analysis

[1466] The server analyzes the video and audio data using generative AI models and voice recognition technology.

[1467] A generative AI model (e.g., GPT model) is used to analyze the frame images and score important parts. Google Speech-to-Text technology is used to convert the audio data into text.

[1468] Input: Frame image data and audio data

[1469] Output: Text data with scores and frame analysis results

[1470] Step 4:

[1471] Identifying the important parts

[1472] The server lists important scenes based on the analysis results and identifies particularly important parts.

[1473] The scored data is used to extract scenes that are likely to be of particular interest to viewers.

[1474] Input: Text data with scores and frame analysis results

[1475] Output: Important Scene List

[1476] Step 5:

[1477] Emotional Data Analysis

[1478] The server collects viewers' viewing data and analyzes it using an emotion engine.

[1479] The viewer's facial expression and voice data are input into an emotion engine to analyze the viewer's emotional state and evaluate their emotional response to a particular scene.

[1480] Input: Viewing data (facial expression and voice data)

[1481] Output: Emotion analysis results

[1482] Step 6:

[1483] Generate a summary video

[1484] The server generates a summary version of the video based on the identified important parts and viewer emotion data.

[1485] Use video editing software such as Adobe Premiere Pro API to stitch together key scenes to create a short video, add transfer effects and subtitles.

[1486] Input: Important scene list and sentiment analysis results

[1487] Output: Abridged video

[1488] Step 7:

[1489] Video distribution

[1490] The server generates a URL link to distribute the generated summarized video and sends it to the user.

[1491] Users click on the link they receive to watch or download the abridged video.

[1492] Input: Abridged video

[1493] Output: URL link of the summary video

[1494] Prompt Sentence Examples

[1495] I've uploaded a video of a new product briefing. I want you to analyze the viewer's emotional state and generate a 5-minute summary version of the video.

[1496] The above is the specific flow and operation of each processing step of the present invention.

[1497] (Application example 2)

[1498] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1499] Current video advertising distribution technology is unable to analyze viewer emotions and interests in real time and summarize and distribute videos based on that information, making it difficult to maximize viewer interest. It is also difficult to effectively provide personalized advertising based on viewing data. This reduces advertising effectiveness and creates challenges in maintaining viewer interest.

[1500] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1501] In this invention, the server includes means for acquiring video data, means for analyzing the acquired video data and identifying important portions, means for generating a video digest based on the identified important portions, means for distributing the generated digest video, means for analyzing viewer emotions, means for generating a digest advertisement based on the viewer's interests, and means for providing personalized advertisements based on viewing data. This enables the generation and distribution of effective digest advertisements based on the viewer's emotions and interests, maximizing advertising effectiveness and maintaining viewer interest.

[1502] "Video data" refers to data that contains visual video information, and is usually composed of a series of frames that have been photographed and recorded by a device such as a camera or video recorder.

[1503] The "important portion" refers to a portion of the video data that contains particularly meaningful scenes or information and is useful or interesting to the viewer.

[1504] "Abridged video" refers to a video that can be played visually and audibly in a shortened format by extracting important parts from the original video data.

[1505] "Distribution" refers to the act of sending the generated summarized video to the viewer's terminal using a communication network such as the Internet.

[1506] "Viewer emotion" refers to the emotional reaction that a viewer shows while watching a video, and refers to a mental state such as joy, surprise, interest, or dissatisfaction.

[1507] "Viewer interest" refers to a state in which a viewer shows particular interest in a particular scene or piece of information within a video.

[1508] "Personalized advertising" means advertising content that is customized based on a viewer's individual emotional state and interests.

[1509] "Viewing data" refers to detailed data that records the viewer's behavior and reactions while watching a video, including viewing time, number of clicks, number of plays, changes in emotions, and the like.

[1510] The present invention provides a system for generating a digest version of an advertisement based on a viewer's emotions and interests to provide personalized advertisements in order to improve the viewing experience of advertisement videos. The system is composed of a server, a terminal, and a user.

[1511] The server first has a means for acquiring video data. When a user uploads an advertising video to the server, the server analyzes the acquired video data and identifies important parts. This analysis includes frame analysis using OpenCV and emotion analysis using EmotionRecognizer. Once important parts are identified, a means for generating a summarized version of the video based on the identified parts is activated. Here, Moviepy is used to connect important frames and create a summarized version of the video.

[1512] EmotionRecognizer is also used as a means of analyzing viewer emotions. It collects emotional and viewing data from viewers while they are watching, and generates abridged advertisements based on their interests during viewing. It then analyzes the viewing data to provide personalized advertisements based on the viewer's interests. This process involves data analysis based on the collected viewing data. Data points such as viewing time, number of clicks, number of views, and emotional state are utilized in the analysis.

[1513] Specifically, the system operates in the following scenario: While User A is watching an advertising video using a device, the server analyzes the video in real time to identify the viewer's emotions and interests. The server extracts scenes that the viewer particularly finds interesting and creates a shortened, summarized version of the advertisement. Next, a personalized advertisement is provided to User A based on User A's viewing data. The generated summarized advertisement is then distributed via the Internet for the user to view.

[1514] Examples of prompts include:

[1515] "Your goal is to build a system that analyzes the real-time emotional state of users watching a video explaining a new product, and summarizes and displays scenes that piqued their interest. This system will analyze video data uploaded to a server and maximize the viewer's interest."

[1516] In this way, the present invention realizes summarization and personalization of advertising videos based on the viewer's emotions and interests, thereby increasing advertising effectiveness and improving the viewer experience.

[1517] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1518] Step 1:

[1519] The server acquires the advertising video data uploaded by the user. The input is the video file provided by the user, which is saved in the server's data storage. The output is the acquired video file saved on the server.

[1520] Step 2:

[1521] The server divides the acquired video data into frames and makes them available for analysis as individual image data. The input is the uploaded video file, and OpenCV is used for data processing to break the video down into frames. The output is that each frame of the video is saved as image data on the server.

[1522] Step 3:

[1523] The server analyzes the viewer's emotions for each frame. The input is the image data of each frame obtained in step 2, and performs emotion analysis using EmotionRecognizer. The output is emotional state data for each frame.

[1524] Step 4:

[1525] The server aggregates the emotional state data and identifies important segments that indicate viewer interest. The input is the emotional state data obtained in step 3, and the data is calculated by evaluating the emotional score and marking segments that exceed a certain interest score as important. The output is a list of frames that indicate important segments.

[1526] Step 5:

[1527] The server generates a digest video based on the identified key segments. The input is the frame list of key segments obtained in step 4, and Moviepy is used to concatenate these frames to create a digest video. The output is the generated digest video file.

[1528] Step 6:

[1529] The server delivers the generated summarized video to the terminal. The input is the summarized video file generated in step 5, and a link that the user can access via the Internet is generated. The output is the status that the video is available for viewing on the user's terminal.

[1530] Step 7:

[1531] The server collects and analyzes viewing data. The input is the viewing data (viewing time, clicks, plays, emotional state, etc.) generated when users watch the abridged video, and the viewing database is used for data analysis. The output is the analyzed viewing data.

[1532] Step 8:

[1533] The server predicts each user's interests based on the aggregated viewing data and provides personalized advertisements. The input is the viewing data obtained in step 7, and the generative AI model is used to predict the user's interests. The output is personalized advertising content.

[1534] In this way, a system is realized that performs a series of processes from acquiring video data to distributing it, analyzing viewing data, and providing personalized advertisements through each step between the server, terminal, and user.

[1535] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1536] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1537] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1538] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1539] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1540] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1541] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1542] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1543] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1544] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1545] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1546] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1547] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1548] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1549] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1550] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1551] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1552] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1553] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1554] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1555] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1556] The following is further disclosed regarding the above embodiment.

[1557] (Claim 1)

[1558] a means for acquiring video data;

[1559] A means for analyzing the acquired video data and identifying important parts;

[1560] a means for generating a summary of the video based on the identified key portions;

[1561] a means for distributing the generated abridged video;

[1562] A system including:

[1563] (Claim 2)

[1564] 10. The system of claim 1, further comprising means for analyzing the audio data and summarizing key content.

[1565] (Claim 3)

[1566] a means for collecting and analyzing viewing data;

[1567] 10. The system of claim 1, further comprising: means for predicting viewer response based on the collected data.

[1568] "Example 1"

[1569] (Claim 1)

[1570] a means for acquiring video data;

[1571] A means for dividing the acquired video data into frames;

[1572] A means for analyzing the divided frames and identifying important portions;

[1573] a means for converting the audio data into text and extracting key content;

[1574] a means for generating a summary of the video based on the identified important parts and extracted important content;

[1575] A means to add transfer effects and subtitles to the generated summary video,

[1576] a means for distributing the generated abridged video;

[1577] a means for collecting and analyzing viewing data;

[1578] A means of predicting audience reactions based on collected data;

[1579] A system including:

[1580] (Claim 2)

[1581] 10. The system of claim 1, further comprising means for analyzing the audio data and summarizing key content.

[1582] (Claim 3)

[1583] 10. The system of claim 1, further comprising: means for collecting and analyzing viewing data; and means for predicting viewer responses based on the collected data.

[1584] "Application Example 1"

[1585] (Claim 1)

[1586] a means for acquiring video data;

[1587] A means for analyzing the acquired video data and identifying important parts;

[1588] a means for generating a summary of the video based on the identified key portions;

[1589] a means for distributing the generated abridged video;

[1590] A means for analyzing the advertisement video received from the user, extracting important parts, and generating a digest advertisement video;

[1591] A method to predict viewer reactions based on data processing and suggest the optimal timing for distribution,

[1592] A system including:

[1593] (Claim 2)

[1594] 10. The system of claim 1, further comprising means for analyzing the audio data and summarizing key content.

[1595] (Claim 3)

[1596] a means for collecting and analyzing viewing data;

[1597] 10. The system of claim 1, further comprising: means for predicting viewer response based on the collected data.

[1598] "Example 2: Combining Emotion Engines"

[1599] (Claim 1)

[1600] a means for acquiring video data;

[1601] means for dividing the acquired video data into frames;

[1602] A means for analyzing the divided frame data and identifying important portions;

[1603] A means for collecting and analyzing viewer emotional data;

[1604] a means for generating a summary of the video based on the identified key portions and viewer sentiment data;

[1605] A means to add transfer effects and subtitles to the generated summary video,

[1606] a means for distributing the generated abridged video;

[1607] A system including:

[1608] (Claim 2)

[1609] 10. The system of claim 1, further comprising means for analyzing the audio data and summarizing key content.

[1610] (Claim 3)

[1611] a means for collecting and analyzing viewing data;

[1612] 10. The system of claim 1, further comprising: means for predicting viewer response based on the collected data.

[1613] "Application example 2 when combining emotion engines"

[1614] (Claim 1)

[1615] a means for acquiring video data;

[1616] A means for analyzing the acquired video data and identifying important parts;

[1617] a means for generating a summary of the video based on the identified key portions;

[1618] a means for distributing the generated abridged video;

[1619] A means of analyzing audience emotions,

[1620] means for generating abridged advertisements based on viewer interests;

[1621] A means for providing personalized advertising based on viewing data;

[1622] A system including:

[1623] (Claim 2)

[1624] 10. The system of claim 1, further comprising means for analyzing the audio data and summarizing key content.

[1625] (Claim 3)

[1626] a means for collecting and analyzing viewing data;

[1627] 10. The system of claim 1, further comprising: means for predicting viewer response based on the collected data. [Explanation of symbols]

[1628] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for acquiring video data; A means for analyzing the acquired video data and identifying important parts; a means for generating a summary of the video based on the identified key portions; a means for distributing the generated abridged video; A system including:

2. 10. The system of claim 1, further comprising means for analyzing the audio data and summarizing key content.

3. a means for collecting and analyzing viewing data; The system of claim 1 further comprising means for predicting audience response based on the collected data.

Images