system

Abstract

Provide a system. A means for collecting video data and comment data; A means of analyzing the streamer's emotions from the collected video data, A means of analyzing viewer reactions from collected comment data, A means of analyzing the content of video data and identifying highlights; A means for generating a clipped video based on the identified points of interest; A means for distributing the generated clipped video; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Video streaming is extremely popular today, with streamers and viewers consuming a wide variety of video content every day. However, watching an entire lengthy streamed video requires time and effort, making it difficult to find the highlights. Furthermore, manually cutting and editing videos is costly and requires specialized skills. For this reason, streamers and viewers are seeking an efficient, low-cost way to access videos with highlights. The present invention aims to solve these problems by providing a system that automatically extracts and edits the highlights of streamed videos. [Means for solving the problem]

[0005] The present invention provides the following means. First, it includes a means for collecting video data and comment data. Next, it includes a means for analyzing the emotions of a streamer from the collected video data and a means for analyzing viewer reactions from the collected comment data. It also includes a means for analyzing the content of the video data to identify highlights, and a means for generating cut-out videos based on the identified highlights. Finally, it includes a means for distributing the generated cut-out videos, thereby realizing low-cost and efficient video editing. Specifically, it includes a means for identifying emotions based on an analysis of the streamer's voice and facial expressions, and a means for classifying viewer reactions based on a text analysis of the comment data. This makes it possible to automatically provide cut-out videos of consistent quality without relying on the subjectivity of a video editor.

[0006] "Video data" refers to media files containing video and audio provided by a distributor.

[0007] "Comment data" is text information posted by viewers in real time or non-real time in response to a video.

[0008] "Means for analyzing emotions" refers to a technology or method that analyzes the streamer's voice and facial expressions to identify their emotional state at that time (e.g., joy, anger, surprise).

[0009] "Means for analyzing viewer reactions" refers to a technique or method for analyzing comment data and classifying viewer reactions as positive, negative, neutral, etc.

[0010] "Means for identifying highlights by analyzing the content of video data" refers to a technology or method for understanding the content of a video and automatically identifying important scenes or turning points in the story.

[0011] The "means for generating cut-out videos" refers to a technique or method for cutting out identified highlight scenes, editing them, and compiling them into a single video.

[0012] "Means of distribution" refers to the technology or method by which the generated clipped video is provided to viewers via the Internet.

[0013] A "voice emotion analysis model" is a machine learning model for analyzing voice data to identify emotional states.

[0014] A "text analysis model" is a natural language processing technology that analyzes text information such as comment data and classifies viewers' intentions and emotions.

[0015] An "automatic speech recognition (ASR) model" is a machine learning model for converting voice data into text data. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The embodiment of the present invention will be described in terms of a server, a terminal, and a user. The processing of the program of the system for automatically generating a cut-out video will be described in detail with specific examples.

[0038] Collection of video and comment data

[0039] The server calls the video distribution platform API to collect the video data of the specified broadcaster and the comment data associated with that video. Specifically, it receives the URL and broadcast ID of the broadcast video, obtains the video file through the API, and saves it in the server's storage. It also uses the API to collect comment data and stores it in a database.

[0040] Data analysis

[0041] The server performs the following analysis based on the collected data.

[0042] 1. Emotion analysis

[0043] The server detects the streamer's face from the video frame and inputs it into an emotion analysis model to identify the streamer's emotional state (e.g., joy, anger, surprise, etc.) along with a timestamp. It also analyzes the audio data in parallel to estimate emotions from the audio.

[0044] 2. Comment Analysis

[0045] The server runs the comment data through a text analysis model to categorize viewer comments into negative, positive, and neutral categories, and also extracts keywords to identify times when viewer responses are most likely to be concentrated.

[0046] 3. Content Analysis

[0047] The server transcribes the video content using an automatic speech recognition (ASR) model, and then analyzes the text data using a natural language processing (NLP) model, which identifies turning points and important scenes in the video.

[0048] Extracting highlights and generating clipped videos

[0049] Based on the results of the analysis, the server identifies scenes that are worth watching.

[0050] From the results of the emotion analysis, scenes in which the streamer became particularly emotional are extracted.

[0051] From the results of the comment analysis, the time periods when viewers responded most strongly are identified and designated as highlight scenes.

[0052] The results of content analysis identify turning points in the topic and scenes in which specific important keywords appear frequently.

[0053] Each element is combined to determine the start and end timestamps of the highlight scenes.

[0054] The server cuts out and concatenates the scenes identified as highlights to generate a single cut-out video, allowing users to easily view high-quality highlights.

[0055] Video distribution

[0056] The server distributes the generated clipped video to the user in the following manner.

[0057] Upload to a video distribution platform and generate a link to watch the video.

[0058] The generated viewing link will be provided to the user via a method such as email or push notification.

[0059] Specific examples

[0060] For example, consider a case where this system is applied to a live streaming video of a broadcaster A.

[0061] 1. Collection

[0062] The server collects the video data and comment data of distributor A.

[0063] 2. Analysis

[0064] The server uses emotion analysis to identify scenes in the video where streamer A is laughing out loud (e.g., 45:30-45:35), and also uses comment analysis to identify scenes where the number of comments suddenly increases (e.g., 1:15:00-1:15:30).

[0065] 3. Extraction and generation

[0066] These scenes are extracted as highlights and linked together as cut-out videos to create a five-minute highlight video.

[0067] 4. Distribution

[0068] The server uploads this highlight video to a video distribution platform and provides users with a viewing link.

[0069] Thus, the present invention is a system that automatically cuts out moving images, enabling efficient and low-cost video editing.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The server uses the video distribution platform API to collect video data and comment data from the specified broadcaster. First, it receives the URL or broadcast ID of the broadcast video and makes an API call to obtain the video file. The obtained video file is then saved in the server's storage. It also makes a similar API call to obtain comment data related to the video, and stores the obtained comment data in the database.

[0073] Step 2:

[0074] The server performs emotion analysis based on the collected video data. Specifically, it detects the streamer's face from the video frames and inputs it into an emotion analysis model to identify the streamer's emotion (e.g., joy, anger, surprise, etc.) at each timestamp. It also applies the audio data to an audio emotion analysis model to analyze the streamer's emotion from the audio content. The analysis results are recorded along with the timestamp.

[0075] Step 3:

[0076] The server analyzes the comment data through a text analysis model. Specifically, it inputs the comment data into a natural language processing (NLP) model and classifies it into positive, negative, and neutral categories. It also extracts keywords and identifies which timestamps are generating the most active viewer reactions. This allows the server to understand the level of enthusiasm among viewer reactions.

[0077] Step 4:

[0078] The server transcribes the video data using an automatic speech recognition (ASR) model, and then runs the transcribed text data through a natural language processing (NLP) model to identify specific topics and important scenes, thereby understanding what the streamer is saying and which parts are important.

[0079] Step 5:

[0080] The server identifies highlight scenes by combining the results of sentiment analysis, comment analysis, and content analysis. First, it extracts scenes with large emotional fluctuations from the sentiment analysis results. Next, it identifies scenes with extremely positive viewer reactions from the comment analysis results. Finally, it extracts scenes with turning points in the conversation and scenes with many specific important keywords from the content analysis results. It then combines these data to determine the start and end timestamps of each scene.

[0081] Step 6:

[0082] The server generates clipped videos based on the identified highlight scenes. It clips the video according to the timestamp of each scene and connects multiple scenes to create a single highlight video. If necessary, it adds subtitles and sound effects to improve the quality of the video.

[0083] Step 7:

[0084] The server distributes the generated clipped video so that users can view it. Specifically, it uploads the highlight video to a video distribution platform and generates a viewing link. This viewing link is provided to users using a method to notify them (e.g., email or push notification) so that they can easily access it.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] In recent years, with the spread of video streaming, many viewers are enjoying live streaming and uploaded videos. However, these videos are often long, and watching them in their entirety takes a huge amount of time. Furthermore, manually finding particularly interesting or important scenes in a video is extremely time-consuming and places a considerable burden on viewers. Furthermore, editing and cropping videos requires specialized knowledge and skills, making them difficult for average users. To solve these problems, a system that can automatically extract the highlights of a video and provide them to viewers is needed.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes means for collecting video data and comment data, means for analyzing the distributor's emotions from the collected video data, means for analyzing the viewer's reactions from the collected comment data, means for analyzing the content of the video data using a natural language processing model to identify highlights, means for generating clipped videos based on the identified highlights, and means for distributing the generated clipped videos. This makes it possible to automatically extract highlights from videos and make them easy to watch.

[0090] "Moving image data" refers to moving image information recorded in video format that can be viewed by a viewer.

[0091] "Comment data" refers to information including opinions and reactions posted by viewers in text format to videos.

[0092] "Emotion analysis" is the process of identifying a person's emotional state (e.g., joy, anger, surprise, etc.) based on video frames and audio data.

[0093] "Viewer reaction" is a concept that refers to the reaction or feedback (e.g., positive, negative, neutral) that viewers give to a video.

[0094] A "natural language processing model" is a machine learning model used to understand and process human language.

[0095] "Keyword extraction" is the process of identifying and extracting important words and phrases from text data.

[0096] A "speech recognition model" is a machine learning model for converting speech into text.

[0097] A "cut-out video" is a short video clip created by cutting out specific scenes from the original video and connecting them together.

[0098] "Video editing software" refers to software tools used to cut, join, and add effects to video clips.

[0099] An "online platform" is a web service that allows users to share and view videos over the Internet.

[0100] "Viewing Link" refers to the URL that allows a user to view a video online.

[0101] An "automatic speech recognition model" is an algorithm that analyzes an input speech signal and converts it into a corresponding string of characters.

[0102] A "turning point in the topic" refers to a moment when the content of the video changes significantly or when the topic switches to a new one.

[0103] An "important scene" refers to a part of a video that deserves particular attention or that will interest viewers.

[0104] This invention is a system that collects video data and comment data from a video distribution platform, analyzes them, automatically extracts highlights, and generates and distributes clipped videos. The following describes the implementation of this system, focusing on a server, a terminal, and a user.

[0105] Collection of video and comment data

[0106] The server calls the video distribution platform API to collect the video data of the specified broadcaster and the comment data associated with that video. Specifically, it receives the URL or broadcast ID of the broadcaster's video and obtains the video file via the API. The obtained video file is saved in storage within the server. Comment data is similarly obtained using the API and stored in a database. For example, the YouTube (registered trademark) API can be used to collect the video file and comment data corresponding to the video ID "exampleID."

[0107] Data analysis

[0108] The server performs the following analysis based on the collected video data and comment data.

[0109] 1. Emotion analysis

[0110] The streamer's face is detected from the video frame and input into an emotion analysis model to identify the streamer's emotional state (e.g., joy, anger, surprise, etc.) along with a timestamp. Audio data is also analyzed in parallel to estimate emotions from the audio.

[0111] 2. Comment Analysis

[0112] The comment data is run through a text analysis model to categorize viewer comments into negative, positive, and neutral categories, and keywords are extracted to identify times when viewer responses are most concentrated.

[0113] 3. Content Analysis

[0114] The video content is transcribed using an automatic speech recognition (ASR) model, and the text data is then analyzed using a natural language processing (NLP) model to identify turning points and key moments in the video.

[0115] Extracting highlights and generating clipped videos

[0116] The server identifies scenes that are worth watching based on the analysis results. It determines which scenes are particularly emotional based on the sentiment analysis results, which time periods had the strongest reactions from viewers based on the comment analysis results, and which scenes show turning points in the conversation or many important keywords based on the content analysis results. It then determines the start and end timestamps of the identified scenes and uses video editing software (e.g., FFmpeg) to generate a cut-out video.

[0117] Video distribution

[0118] The server uploads the generated clipped video to a video distribution platform and generates a viewing link to provide it to users. The server also provides the generated viewing link to users by means of email, push notification, etc. For example, the YouTube API can be used to upload the clipped video and notify viewers of the link.

[0119] Specific examples

[0120] For example, consider a case where this system is applied to a live streaming video of a broadcaster A.

[0121] 1. Collection

[0122] The server collects the video data and comment data of distributor A.

[0123] 2. Analysis

[0124] The server uses emotion analysis to identify scenes in the video where streamer A is laughing out loud (e.g., 45:30-45:35), and also uses comment analysis to identify scenes where the number of comments suddenly increases (e.g., 1:15:00-1:15:30).

[0125] 3. Extraction and generation

[0126] These scenes are extracted as highlights and linked together as cut-out videos to create a five-minute highlight video.

[0127] 4. Distribution

[0128] The server uploads this highlight video to a video distribution platform and provides users with a viewing link.

[0129] Prompt Sentence Examples

[0130] "Design a system to naturally crop streams from Streamer A's live stream video and meet the following requirements:

[0131] 1. Collection of video and comment data

[0132] 2. Data analysis using sentiment analysis, comment analysis, and content analysis

[0133] 3. Extract the highlights and create a single video clip

[0134] 4. Upload to a video distribution platform and issue a viewing link

[0135] Please explain with specific examples."

[0136] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0137] Step 1:

[0138] The server calls the API of the video distribution platform to collect video data and comment data. (Input) Receives the URL and distribution ID of the distributed video. (Processing) Obtains the video file via the API and saves it in storage on the server. Similarly, comment data is collected using the API and stored in a database. (Output) The video file and comment data are saved on the server. For example, the YouTube API is used to obtain the video file and comments corresponding to the video ID "exampleID" and save them in their respective destinations.

[0139] Step 2:

[0140] The server analyzes the streamer's emotions from the collected video data. (Input) Saved video file. (Processing) The streamer's face is detected from the video frames and input into an emotion analysis model to identify the emotional state (e.g., joy, anger, surprise, etc.) along with the timestamp. In addition, the audio data is analyzed to estimate emotions from the audio. (Output) Emotional state data corresponding to the timestamp of each frame. For example, it identifies the scene where the streamer is laughing out loud (e.g., timestamp 45:30).

[0141] Step 3:

[0142] The server analyzes viewer reactions from the collected comment data. (Input) Saved comment data. (Processing) The comment data is run through a text analysis model to classify it into negative, positive, and neutral categories. Keywords are also extracted to identify time periods when viewer reactions are concentrated. (Output) The classified comment data and the timestamps where viewer reactions peak. For example, the scene where the number of comments suddenly increased (e.g., timestamp 1:15:00) is identified.

[0143] Step 4:

[0144] The server analyzes the content of the video data to identify highlights. (Input) Saved video file and transcription data. (Processing) The audio from the video is transcribed using an automatic speech recognition (ASR) model, and the resulting text data is analyzed using a natural language processing (NLP) model. This allows for the identification of turning points in the topic and important scenes. (Output) Timestamps of turning points in the topic and important scenes. For example, identifying the scene at 45 minutes and 50 seconds where the topic changed significantly.

[0145] Step 5:

[0146] The server generates a clipped video based on the highlight extraction results. (Input) Sentiment analysis, comment analysis, and content analysis results. (Processing) The server determines the start and end timestamps by comprehensively determining particularly emotional scenes from the sentiment analysis results, time periods with strong viewer reactions from the comment analysis results, and scenes with turning points in the topic or frequent appearance of important keywords from the content analysis results. The clipped video is generated using video editing software (e.g., FFmpeg). (Output) The generated clipped video file. For example, the identified scenes (30:45-31:00, 45:30-45:35, 1:15:00-1:15:30) are concatenated to create a single highlight video "highlights.mp4."

[0147] Step 6:

[0148] The server distributes the generated cut-out video. (Input) Generated cut-out video file. (Processing) Upload the video to an online platform and generate a viewing link. The viewing link is then provided to the user via a method such as email or push notification. (Output) Notification of upload completion to the video distribution platform and a viewing link. For example, upload using the YouTube API and notify the user of the generated viewing link via email.

[0149] (Application example 1)

[0150] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0151] Conventional video streaming services have the drawback of requiring viewers to manually edit long video content to extract the most interesting scenes, which is time-consuming and inefficient. Furthermore, viewers often find it difficult to watch the entire video, potentially missing important or interesting scenes. Furthermore, a lack of real-time analysis and notifications makes it difficult for users to quickly access important information. To address these issues, a system was needed that could automatically analyze video and comment data and notify viewers in real time.

[0152] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0153] In this invention, the server includes means for collecting video data and comment data, means for analyzing the streamer's emotions from the collected video data, means for analyzing viewer reactions from the collected comment data, means for analyzing the content of the video data to identify highlights, means for generating clipped videos based on the identified highlights, means for distributing the generated clipped videos, and means for sending notifications to smartphones. This allows important scenes to be automatically extracted from long video content and provided as clipped videos, allowing viewers to efficiently view highlights without any hassle. Furthermore, the generated clipped videos are notified to the user's smartphone in real time, allowing viewers to quickly access important information without missing anything.

[0154] "Video Data" means digital files containing video and audio information that a User views or broadcasts on an online platform.

[0155] "Comment data" refers to collected textual reactions and feedback entered by viewers while watching video content.

[0156] "Emotion analysis means" refers to algorithms or models that detect facial expressions, vocal tone, etc. from the streamer's video frames to identify the streamer's emotional state.

[0157] The "means for analyzing viewer reactions" refers to natural language processing technology that analyzes comment data and classifies its content as positive, negative, neutral, etc.

[0158] The "means of identifying highlights" refers to technology that analyzes video data and comment data to identify scenes that are particularly noteworthy, such as scenes that elicited strong emotions from viewers and streamers, or time periods when viewers responded most strongly.

[0159] The "means for generating cut-out videos" is a technology that extracts scenes identified as highlights and connects them to generate a single highlight video.

[0160] The "means for sending notifications to smartphones" refers to a system that notifies users of the information in real time on their smartphones when generated highlight videos and highlight scenes become available.

[0161] The means of "distribution" is the technology of uploading the generated clipped video to an online platform, generating a viewing link, and providing it to users.

[0162] The following describes an embodiment of the present invention, focusing on a server, a terminal, and a user. The processing of a program in a system for automatically generating a cut-out video will be described in detail with specific examples.

[0163] Collection of video and comment data

[0164] The server calls the video distribution platform API to collect the video data of the specified broadcaster and the comment data associated with that video. Specifically, it receives the URL and broadcast ID of the broadcast video, obtains the video file through the API, and saves it in the server's storage. It also uses the API to collect comment data and stores it in a database.

[0165] Data analysis

[0166] The server performs the following analysis based on the collected data. First, it detects the streamer's face from the video frames and inputs it into an emotion analysis model to identify the streamer's emotional state along with a timestamp. It also analyzes the audio data in parallel to infer emotions from the audio. Next, it runs the comment data through a text analysis model to classify viewer comments into negative, positive, or neutral categories. It also extracts keywords and identifies time periods when viewer reactions are concentrated. Finally, it transcribes the video content using an automatic speech recognition (ASR) model, and then analyzes the text data using a natural language processing (NLP) model. This allows it to identify turning points in the video and important scenes.

[0167] Extracting highlights and generating clipped videos

[0168] The server identifies scenes that are worth watching based on the results of the analysis. From the results of sentiment analysis, it extracts scenes in which the streamer was particularly emotional. From the results of comment analysis, it identifies time periods when viewers responded most strongly and designates these as scenes worth watching. From the results of content analysis, it identifies scenes where the topic changed or specific important keywords frequently appear. By combining all elements, it determines the start and end timestamps of the scenes worth watching. The server then cuts out the scenes identified as worth watching and concatenates them to generate a single cut-out video. This allows users to easily watch high-quality highlights.

[0169] Video distribution

[0170] The server distributes the generated clipped video to users in the following ways: Uploads it to a video distribution platform and generates a video viewing link. Provides the generated viewing link to users via a method (e.g., email or push notification). Notifying users in real time via push notification has the advantage of allowing them to watch without missing any important information.

[0171] Specific examples

[0172] For example, consider the case where this system is applied to a live streaming video of streamer A. The server collects streamer A's video data and comment data. The server uses emotion analysis to identify scenes in the video where streamer A is laughing out loud, and also uses comment analysis to identify scenes where the number of comments has increased dramatically. These scenes are extracted as highlights, and then linked together as clipped videos to generate a five-minute highlight video. The server then uploads this highlight video to a video distribution platform and provides users with a viewing link via push notification.

[0173] Example prompt sentence:

[0174] Retrieve video data and comments from the provided URLs. Analyze the video frames to detect emotions such as joy, sadness, and surprise. Simultaneously, analyze the comments to classify them into positive, negative, and neutral categories. Identify key moments from the emotion and comment analyzes to create highlight clips. Finally, stitch these highlights together into a 5-minute video and notify the user via push notification.

[0175] As described above, the present invention provides a system that automatically clips videos, enabling efficient and low-cost video editing. Furthermore, by notifying users of the clipped videos in real time, the viewing experience is improved.

[0176] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0177] Step 1:

[0178] Collection of video and comment data

[0179] The server collects the video data and comment data of the specified streamer from the video streaming platform API. Specifically, it calls the API based on the stream video URL or stream ID provided by the user to retrieve the video file and save it in the server's storage. In parallel, it also collects comment data using the same API and stores it in a database.

[0180] Input: Streaming video URL or Streaming ID

[0181] Output: Video files stored in storage, comment data stored in a database

[0182] Step 2:

[0183] Video sentiment analysis

[0184] The server analyzes the streamer's emotions using the collected video data. It detects the streamer's face from the video frames and inputs it into an emotion analysis model to identify the streamer's emotional state. It also analyzes the audio data in parallel to perform voice emotion analysis.

[0185] Input: Video file saved in storage

[0186] Output: Streamer sentiment data with timestamps

[0187] Step 3:

[0188] Text analysis of comment data

[0189] The server runs the comment data through a text analysis model to classify each comment as negative, positive, or neutral, and also uses comment timestamps to identify times when viewers tend to respond more frequently.

[0190] Input: Comment data stored in the database

[0191] Output: Viewer response data (negative, positive, neutral) with timestamps

[0192] Step 4:

[0193] Video content analysis

[0194] The server transcribes the video data using an automatic speech recognition (ASR) model, and then runs the text data through a natural language processing (NLP) model to identify topic changes and important scenes within the video.

[0195] Input: Video file saved in storage

[0196] Output: Text data of important scenes in the video

[0197] Step 5:

[0198] Extraction of highlights

[0199] The server combines the results of emotion analysis, comment analysis, and content analysis to identify highlights. Based on timestamps, it extracts scenes in which the streamer was particularly emotional, times when viewers responded most strongly, and turning points in the conversation, and determines the start and end timestamps of highlights.

[0200] Input: Streamer's emotional data, viewer's reaction data, text data of important scenes

[0201] Output: List of highlights (with start and end timestamps)

[0202] Step 6:

[0203] Generate cropped video

[0204] The server extracts the identified scenes based on the list of highlight scenes, concatenates them, and generates a single cut-out video. Using a video editing library, the server combines each highlight scene to generate a highlight video.

[0205] Input: List of highlights, video files saved in storage

[0206] Output: Generated highlight video file

[0207] Step 7:

[0208] Highlight video distribution

[0209] The server uploads the generated highlight video to a video distribution platform. A link to view the video is generated and provided to users via email, push notification, or other means. In particular, push notifications are sent to smartphones, allowing users to access the highlight video in real time.

[0210] Input: Generated highlight video file

[0211] Output: Viewing link on video streaming platform, notification to user

[0212] By following the above steps, the present invention is a system that automatically cuts out moving images and provides users with efficient, high-quality highlight scenes.

[0213] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0214] The following describes an embodiment of the present invention, focusing on a server, a terminal, and a user. The processing of a system program that automatically generates a cut-out video and recognizes the user's emotions will be described in detail with specific examples.

[0215] Collection of video and comment data

[0216] The server uses the API of the video distribution platform to collect video data and comment data from the specified broadcaster. This involves receiving the URL and broadcast ID of the broadcast video and making an API call to retrieve the video file. The retrieved video file is saved in the server's storage, and the comment data is stored in a database.

[0217] Data analysis

[0218] The server performs the following analysis based on the collected data:

[0219] 1. Emotion analysis

[0220] The server detects the streamer's face from the video frames and uses an emotion analysis model to identify the streamer's emotion (e.g., joy, anger, surprise) at each timestamp. It also analyzes the audio data and estimates emotions from the audio.

[0221] 2. Comment Analysis

[0222] The server then runs the comment data through a text analysis model to categorize viewer comments into positive, negative, and neutral categories, extracting key keywords and identifying times when viewer reactions are most likely to be high.

[0223] 3. Content Analysis

[0224] The server transcribes the video data using an automatic speech recognition (ASR) model and identifies turning points and important scenes in the video using a natural language processing (NLP) model.

[0225] Analysis using emotion engines

[0226] The server integrates the results of the streamer's emotion analysis and the results of the viewer's comment analysis, and uses an emotion engine to comprehensively recognize the user's emotions. This emotion engine analyzes the emotional states of the streamer and the viewer in real time, and combines that data to more accurately identify scenes that users are interested in.

[0227] Extracting highlights and generating clipped videos

[0228] Based on the results of the analysis, the server identifies scenes that are likely to attract significant attention. Specifically, it extracts scenes with large emotional fluctuations from the emotion analysis results, scenes that have drawn active reactions from viewers from the comment analysis results, and important scenes from the content analysis results, and combines these to determine the start and end timestamps of the scenes that are likely to attract significant attention.

[0229] For example, based on the user's live streaming viewing data, the server performs the following steps:

[0230] 1. Collection: The server collects the broadcast video and comment data.

[0231] 2. Analysis: Analyze the streamer's emotions (e.g., scenes where they laugh out loud), scenes that get a lot of viewers' reactions (e.g., when the number of comments suddenly increases), and turning points in the topic.

[0232] 3. Comprehensive analysis by the emotion engine: The emotion engine integrates this data and identifies scenes that are particularly interesting as highlights.

[0233] 4. Cutting and generation: Generate a 5-minute cut video based on the highlights.

[0234] Video distribution

[0235] The server delivers the generated clipped video to the user in the following way:

[0236] The server uploads the highlight video to a video distribution platform and generates a viewing link.

[0237] The server provides the viewing link to the user using a means of notifying the user (e.g., email, push notification) so that the user can easily access it.

[0238] In this way, by combining an emotion engine, the present invention provides a system that automatically extracts high-precision highlight scenes and generates and distributes cut-out videos efficiently and at low cost.

[0239] The processing flow will be explained below.

[0240] Step 1:

[0241] The server uses the API of the video distribution platform to collect video data and comment data from the specified broadcaster. Specifically, it receives the URL or broadcast ID of the broadcast video, makes an API call to obtain the video file, and saves it in storage. At the same time, it also obtains comment data and stores it in a database.

[0242] Step 2:

[0243] The server performs emotion analysis based on the collected video data. It detects the streamer's face from each frame of the video and uses an emotion analysis model to identify the streamer's emotion at each timestamp. It also analyzes the audio data, infers emotions from the audio content, and records the analysis results along with the timestamp.

[0244] Step 3:

[0245] The server then analyzes the collected comment data using a text analysis model. Specifically, it classifies comments as positive, negative, or neutral, and extracts important keywords. This analysis identifies times when viewers are particularly enthusiastic.

[0246] Step 4:

[0247] The server transcribes the video data using an automatic speech recognition (ASR) model, and analyzes the transcribed text data using a natural language processing (NLP) model to identify turning points and important scenes in the video.

[0248] Step 5:

[0249] The server uses an emotion engine to integrate the results of previous analyses and recognize the user's emotions. The server combines the results of the streamer's emotion analysis with the results of the viewer's comments analysis to identify scenes that are of particular interest to the user. This comprehensive analysis allows the server to extract highlights from the video scenes that show large emotional fluctuations or that elicit strong reactions from viewers.

[0250] Step 6:

[0251] The server generates clipped videos based on the highlight scenes. It clips the video using the start and end timestamps of the identified highlight scenes, and connects multiple scenes as needed to create a single highlight video. It also performs editing such as adding subtitles and sound effects.

[0252] Step 7:

[0253] The server provides the generated clipped video to the user. First, it uploads the highlight video to a video distribution platform and generates a viewing link. Next, it provides the viewing link to the user by a means of notifying the user (e.g., email or push notification), allowing the user to easily access the video.

[0254] Example 2

[0255] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0256] Conventional video distribution systems lack the technology to automatically identify scenes of interest to users and generate clipped videos. In particular, they lack a means to integrate and analyze the streamer's emotions and the viewer's reactions, making it difficult to accurately extract highlight scenes. This poses a challenge in terms of the quality and experience of the videos provided to users.

[0257] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0258] In this invention, the server includes means for collecting video data and comment data, means for analyzing the emotions of the distributor from the collected video data, means for analyzing viewer reactions from the collected comment data, means for analyzing the content of the video data to identify highlights, means for integrating the results of the distributor's emotion analysis and the results of the viewer's comment analysis to identify scenes of interest to the user, means for generating clipped videos based on the identified highlights, and means for distributing the generated clipped videos. This makes it possible to comprehensively analyze the emotions of the distributor and the viewer, automatically identify high-precision highlight scenes of interest to the user, and generate and provide optimized clipped videos.

[0259] "Video Data" refers to digital files containing video and audio collected from a video distribution platform designated by the user.

[0260] "Comment data" refers to text messages left by viewers on video distribution platforms.

[0261] "Distributor" means an individual or organization that publishes videos on a video distribution platform.

[0262] "Emotion analysis" is the process of identifying emotions from a streamer's facial expressions and voice.

[0263] A "viewer" is an individual or group that watches a streamer's video on a video streaming platform and leaves comments.

[0264] "Reactions" refers to comments and reactions that viewers make to a video.

[0265] "Highlights" are scenes in the video that are judged to be of particular interest to viewers.

[0266] A "cut-out video" is a short video generated based on a highlight scene.

[0267] "Integration" is the process of combining multiple analysis results into one.

[0268] "Distribution" refers to the act of providing the generated clipped video to a user.

[0269] This invention provides a system that collects and analyzes video data and comment data to identify highlight scenes, and automatically generates and distributes cut-out videos of scenes that interest users.

[0270] Collection of video and comment data

[0271] The server collects video data and comment data using the API of the video distribution platform. For example, by using an API key provided by the video distribution platform and specifying the URL of the video or distribution ID to be acquired, the video file and comment data are downloaded to the server. The acquired video data is saved in storage within the server (e.g., Amazon S3), and the comment data is stored in a database (e.g., MySQL (registered trademark)).

[0272] Example: Use the YouTube API to retrieve videos and their comments from a specific streamer.

[0273] Data analysis

[0274] The server performs the following analysis on the collected video data and comment data:

[0275] 1. Emotion analysis

[0276] The server detects the streamer's face from video frames and identifies the streamer's emotions using an emotion analysis model (e.g., Microsoft® Azure® Face API). It also uses Google® Cloud Speech-to-Text API to estimate emotions from audio data.

[0277] Example: Identifying smile frames and detecting scenes of people laughing out loud.

[0278] 2. Comment Analysis

[0279] The server runs the comment data through a text analysis model (e.g., Google Cloud Natural Language API) to categorize it into positive, negative, and neutral categories, extract important keywords, and identify times when viewers' reactions are concentrated.

[0280] Example: Identify moments when positive comments like "That's funny!" or "That's amazing!" spike.

[0281] 3. Content Analysis

[0282] The server transcribes the video data using an automatic speech recognition (ASR) model (e.g., AWS (registered trademark) Transcribe) and identifies topical turning points and key scenes in the video using a natural language processing (NLP) model (e.g., GPT-3 (registered trademark)).

[0283] Example: Analyzing the timing of the transition from "game tips" to "questions from viewers."

[0284] Analysis using emotion engines

[0285] The server uses an emotion engine that combines the results of the individually analyzed emotion analysis and comment analysis to identify scenes that users are interested in. This emotion engine analyzes the streamer's emotions and the viewers' reactions in real time, and combines the data to identify highlight scenes with greater accuracy.

[0286] Extracting highlight scenes and generating cut-out videos

[0287] The server identifies interesting scenes based on the analysis results, determines their start and end timestamps, and then uses a video editing tool such as FFMPEG to generate a five-minute excerpt based on the identified interesting scenes.

[0288] Examples:

[0289] 1. Collection: The server collects the streamer's video data and comment data.

[0290] 2. Analysis: Analyze the streamer's emotions (e.g., laughter), audience reactions, and timing of topic changes.

[0291] 3. Emotion engine integration: Integrate emotions and viewer responses to identify interesting scenes.

[0292] 4. Extract and generate: Extract key scenes and generate a 5-minute video.

[0293] Video distribution

[0294] The server uploads the generated clipped video to a video distribution platform and generates a viewing link to provide it to users. The viewing link is then sent via email using services such as SendGrid, or via a push notification sent via a mobile app.

[0295] Example: Use the YouTube API to upload a highlight video and send users a link to watch it.

[0296] Prompt Sentence Examples

[0297] "Analyze the streamer's emotions and combine them with the analysis of viewers' comments to identify the highlights of the stream. Generate a 5-minute video clip based on the identified scenes."

[0298] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0299] Step 1:

[0300] Collecting videos and comments

[0301] The server uses the API of the video distribution platform to collect video and comment data. It receives the video URL and distribution ID provided by the user as input. Based on this input, the server makes an API call to obtain the video file and comment data. Specifically, it obtains the data using the YouTube API, saves the video file in the server's storage as output, and stores the comment data in a database.

[0302] Specific operation: The server inputs a video URL to the YouTube API, calls the "videos.list" endpoint to retrieve video data, and calls the "liveChat / messages" endpoint to retrieve comment data, then saves them to Amazon S3 and MySQL, respectively.

[0303] Step 2:

[0304] Emotion analysis

[0305] The server receives the captured video data as input, detects the streamer's face from each frame, and analyzes their emotions. This is done using emotion analysis models such as the Microsoft Azure Face API and Google Cloud Vision API. To estimate emotions from audio data, the audio is converted to text using the Google Cloud Speech-to-Text API, and the text is then passed through the emotion analysis model. The output is the emotional state for each timestamp.

[0306] How it works: The server splits the video file and sends each frame to the Azure Face API to detect emotions like smile and surprise, and sends the audio portion to the Google Cloud Speech-to-Text API for transcription, which then analyzes emotions based on the text.

[0307] Step 3:

[0308] Comment Analysis

[0309] The server receives comment data as input and runs it through a text analysis model (e.g., Google Cloud Natural Language API). It classifies comments into positive, negative, and neutral sentiment categories and extracts important keywords. The output is the identified sentiment categories, important keywords, and time periods when reactions are concentrated.

[0310] How it works: The server sends comment data to the Natural Language API for sentiment analysis and keyword extraction, identifying times when comments such as "Amazing!" or "Horrible..." spike.

[0311] Step 4:

[0312] Content Analysis

[0313] The server takes video data as input, transcribes it using an automatic speech recognition (ASR) model (e.g., AWS Transcribe), and then uses a natural language processing (NLP) model (e.g., GPT-3) to identify topical turning points and key scenes in the video. The output is a list of key scenes in the video.

[0314] How it works: The server sends the audio portion of the video to AWS Transcribe, then sends the generated text to GPT-3 to identify the point at which the topic changes, for example, from "How to play" to "Viewer Questions."

[0315] Step 5:

[0316] Integrating data and using sentiment engines

[0317] The server receives the results of sentiment analysis and comment analysis as input and uses an emotion engine to integrate them. The emotion engine analyzes the data in real time and integrates the emotional states of the streamer and viewers. The output is a list of highlight scenes.

[0318] What it does: The emotion engine identifies when a streamer's laugh or surprise coincides with a positive comment from the viewer.

[0319] Step 6:

[0320] Extracting highlight scenes and generating cut-out videos

[0321] The server uses the integrated data as input to identify the start and end timestamps of the highlight scenes, and then generates a 5-minute excerpt using a video editing tool such as FFMPEG. The output is the excerpted video file.

[0322] What it does: The server selects the highlight scenes (scenes with lots of laughs or high comment activity) for FFMPEG, which then cuts out the video and combines it into a single file.

[0323] Step 7:

[0324] Distribution of clipped videos

[0325] The server receives the generated clipped video as input, uploads it to a video distribution platform, generates a viewing link, and provides it to the user via a means of notifying them (e.g., email, push notification). The output is the distributed video link.

[0326] What it does: Upload highlight videos using the YouTube API, send emails with SendGrid, or send push notifications through a mobile app.

[0327] (Application example 2)

[0328] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0329] With conventional content distribution services, it has been difficult to efficiently provide viewers with the highlights of videos. In particular, accurately extracting highlights based on the streamer's emotions and viewer comments is time-consuming and resource-intensive. As a result, ordinary users often miss interesting scenes, resulting in an unsatisfactory viewing experience.

[0330] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting video data and comment data, means for analyzing the distributor's emotions from the collected video data, and means for analyzing the viewer's reactions from the collected comment data. This makes it possible to accurately analyze the distributor's emotions and viewer comments and automatically extract interesting scenes.

[0331] The system further includes a means for analyzing the content of the video data to identify highlights, a means for generating a clipped video based on the identified highlights, a means for distributing the generated clipped video, and a means for distributing the clipped video to a smartphone or a head-mounted display, thereby making it easier for users to view particularly noteworthy scenes as short clips, improving the viewing experience.

[0332] "Video data" refers to digital files containing video and audio distributed by a distributor.

[0333] "Comment Data" refers to the text messages and associated metadata that viewers submit during a broadcast.

[0334] "Emotion analysis" is the process of identifying and analyzing the streamer's emotions from video data.

[0335] "Means for analyzing viewer reactions" refers to technology that classifies viewer reactions based on comment data and identifies their intentions and emotions.

[0336] "Means for identifying highlights by analyzing the content of video data" refers to analytical technology for automatically extracting important scenes and turning points in themes in videos.

[0337] The "means for generating clipped videos" refers to a technology for generating short clips based on identified highlight scenes.

[0338] "Means for distributing the generated cut-out video" refers to a distribution technology for providing the generated cut-out video to users.

[0339] A "smartphone" is a mobile device that has the ability to connect to the Internet and install applications.

[0340] A "head-mounted display" is a device that displays images when worn by the user on the head.

[0341] The "emotion engine" is a technology that comprehensively analyzes the emotions and reactions of streamers and viewers, and recognizes emotions in real time.

[0342] A "natural language processing (NLP) model" is an algorithm that analyzes text data and understands its meaning and intent.

[0343] An "automatic speech recognition (ASR) model" is a speech recognition technology used to convert speech data into text.

[0344] The embodiment of the present invention will be described with reference to a server, a terminal, and a user. A detailed description is provided below.

[0345] Data collection by the server

[0346] The server uses the API of the video distribution platform to collect video data and comment data from the specified broadcaster. Specifically, it receives the URL of the video data and broadcast ID, and makes an API call to retrieve the video file. This retrieved video data is saved in the server's storage, and the comment data is stored in a database.

[0347] Data analysis and emotion recognition

[0348] The server performs the following analysis based on the collected data:

[0349] 1. Emotion analysis:

[0350] The server detects the streamer's face from the video frame and uses an emotion analysis model to identify the streamer's emotion at each timestamp. It also analyzes the audio data and estimates emotions from the audio.

[0351] 2. Comment Analysis:

[0352] The server then runs the comment data through a text analysis model to categorize viewer comments into positive, negative, and neutral categories, extracting key keywords and identifying times when viewer reactions are most likely to be high.

[0353] 3. Content Analysis:

[0354] The server transcribes the video data using an automatic speech recognition (ASR) model and identifies turning points and important scenes in the video using a natural language processing (NLP) model.

[0355] Extracting highlights and generating clipped videos

[0356] Based on this data, the server identifies the highlight scenes that users are interested in. Specifically, it extracts scenes with large emotional fluctuations from the results of emotion analysis, scenes with active viewer reactions from the results of comment analysis, and important scenes from the results of content analysis, and determines the start and end timestamps of the highlight scenes.

[0357] Generate and deliver clipped videos

[0358] The server generates short clips based on the identified highlights, and the resulting clips are delivered to a smartphone or head-mounted display application.

[0359] For example, users can install the application and use a feature called "Emotion Clips" to automatically extract the most interesting scenes based on the streamer's emotions and viewers' reactions, and watch them in short clips.

[0360] The example prompt shown

[0361] We want to create an application that allows users to analyze the streamer's emotions and viewer comments to automatically cut out the most interesting scenes and easily watch them as short clips.

[0362] In this way, the present invention provides a system that combines emotion analysis, comment data analysis, and content analysis to make it easy for users to obtain an advanced viewing experience in a short amount of time.

[0363] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0364] Step 1:

[0365] The server collects video data and comment data using the video distribution platform's API. Specifically, it receives the URL and broadcast ID of the broadcaster's broadcast video and makes an API call. The video file obtained as a result of the API call is saved in the server's storage, and the comment data is stored in a database. The input is the URL or broadcast ID of the broadcast video, and the output is the saved video file and comment data stored in the database.

[0366] Step 2:

[0367] The server analyzes the streamer's emotions by applying the collected video data to an emotion analysis model. Specifically, it detects the streamer's face from the video frames and uses the emotion analysis model to identify the emotion at each timestamp. It also analyzes audio data to complement the emotion. The input is the video data, and the output is the streamer's emotion data at each timestamp.

[0368] Step 3:

[0369] The server performs text analysis on the collected comment data. Specifically, it applies a text analysis model to the comment data, classifying it into positive, negative, and neutral categories, and extracting important keywords. Based on this analysis, it identifies time periods when viewer responses are concentrated. The input is the comment data, and the output is the classified comment data, important keywords, and information on time periods.

[0370] Step 4:

[0371] The server transcribes the video data using an automatic speech recognition (ASR) model and identifies important scenes and turning points in the content using a natural language processing (NLP) model. The input is the video data, and the output is the transcribed text and information about the identified important scenes.

[0372] Step 5:

[0373] The server integrates the results of each analysis to identify highlight scenes. Specifically, it extracts scenes with large emotional fluctuations from the emotion analysis results, scenes with active viewer reactions from the comment analysis results, and important scenes from the content analysis results, and integrates these to determine the start and end timestamps of highlight scenes. The input is the result data of each analysis, and the output is timestamp information for highlight scenes.

[0374] Step 6:

[0375] The server generates a clipped video based on the identified highlight scenes. Specifically, it extracts the relevant scenes from the original video based on the timestamps of the highlight scenes and edits them. The input is the timestamp information of the highlight scenes, and the output is the generated clipped video.

[0376] Step 7:

[0377] The server delivers the generated clips to smartphones or head-mounted displays. Specifically, it uploads short clips to a distribution platform and generates a viewing link. The generated viewing link is then sent to the device. The input is the generated clip, and the output is a notification containing the viewing link.

[0378] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0379] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0380] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0381] [Second embodiment]

[0382] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0383] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0384] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0385] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0386] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0387] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0388] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0389] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0390] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0391] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0392] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0393] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0394] The embodiment of the present invention will be described in terms of a server, a terminal, and a user. The processing of the program of the system for automatically generating a cut-out video will be described in detail with specific examples.

[0395] Collection of video and comment data

[0396] The server calls the video distribution platform API to collect the video data of the specified broadcaster and the comment data associated with that video. Specifically, it receives the URL and broadcast ID of the broadcast video, obtains the video file through the API, and saves it in the server's storage. It also uses the API to collect comment data and stores it in a database.

[0397] Data analysis

[0398] The server performs the following analysis based on the collected data.

[0399] 1. Emotion analysis

[0400] The server detects the streamer's face from the video frame and inputs it into an emotion analysis model to identify the streamer's emotional state (e.g., joy, anger, surprise, etc.) along with a timestamp. It also analyzes the audio data in parallel to estimate emotions from the audio.

[0401] 2. Comment Analysis

[0402] The server runs the comment data through a text analysis model to categorize viewer comments into negative, positive, and neutral categories, and also extracts keywords to identify times when viewer responses are most likely to be concentrated.

[0403] 3. Content Analysis

[0404] The server transcribes the video content using an automatic speech recognition (ASR) model, and then analyzes the text data using a natural language processing (NLP) model, which identifies turning points and important scenes in the video.

[0405] Extracting highlights and generating clipped videos

[0406] Based on the results of the analysis, the server identifies scenes that are worth watching.

[0407] From the results of the emotion analysis, scenes in which the streamer became particularly emotional are extracted.

[0408] From the results of the comment analysis, the time periods when viewers responded most strongly are identified and designated as highlight scenes.

[0409] The results of content analysis identify turning points in the topic and scenes in which specific important keywords appear frequently.

[0410] Each element is combined to determine the start and end timestamps of the highlight scenes.

[0411] The server cuts out and concatenates the scenes identified as highlights to generate a single cut-out video, allowing users to easily view high-quality highlights.

[0412] Video distribution

[0413] The server distributes the generated clipped video to the user in the following manner.

[0414] Upload to a video distribution platform and generate a link to watch the video.

[0415] The generated viewing link will be provided to the user via a method such as email or push notification.

[0416] Specific examples

[0417] For example, consider a case where this system is applied to a live streaming video of a broadcaster A.

[0418] 1. Collection

[0419] The server collects the video data and comment data of distributor A.

[0420] 2. Analysis

[0421] The server uses emotion analysis to identify scenes in the video where streamer A is laughing out loud (e.g., 45:30-45:35), and also uses comment analysis to identify scenes where the number of comments suddenly increases (e.g., 1:15:00-1:15:30).

[0422] 3. Extraction and generation

[0423] These scenes are extracted as highlights and linked together as cut-out videos to create a five-minute highlight video.

[0424] 4. Distribution

[0425] The server uploads this highlight video to a video distribution platform and provides users with a viewing link.

[0426] Thus, the present invention is a system that automatically cuts out moving images, enabling efficient and low-cost video editing.

[0427] The processing flow will be explained below.

[0428] Step 1:

[0429] The server uses the video distribution platform API to collect video data and comment data from the specified broadcaster. First, it receives the URL or broadcast ID of the broadcast video and makes an API call to obtain the video file. The obtained video file is then saved in the server's storage. It also makes a similar API call to obtain comment data related to the video, and stores the obtained comment data in the database.

[0430] Step 2:

[0431] The server performs emotion analysis based on the collected video data. Specifically, it detects the streamer's face from the video frames and inputs it into an emotion analysis model to identify the streamer's emotion (e.g., joy, anger, surprise, etc.) at each timestamp. It also applies the audio data to an audio emotion analysis model to analyze the streamer's emotion from the audio content. The analysis results are recorded along with the timestamp.

[0432] Step 3:

[0433] The server analyzes the comment data through a text analysis model. Specifically, it inputs the comment data into a natural language processing (NLP) model and classifies it into positive, negative, and neutral categories. It also extracts keywords and identifies which timestamps are generating the most active viewer reactions. This allows the server to understand the level of enthusiasm among viewer reactions.

[0434] Step 4:

[0435] The server transcribes the video data using an automatic speech recognition (ASR) model, and then runs the transcribed text data through a natural language processing (NLP) model to identify specific topics and important scenes, thereby understanding what the streamer is saying and which parts are important.

[0436] Step 5:

[0437] The server identifies highlight scenes by combining the results of sentiment analysis, comment analysis, and content analysis. First, it extracts scenes with large emotional fluctuations from the sentiment analysis results. Next, it identifies scenes with extremely positive viewer reactions from the comment analysis results. Finally, it extracts scenes with turning points in the conversation and scenes with many specific important keywords from the content analysis results. It then combines these data to determine the start and end timestamps of each scene.

[0438] Step 6:

[0439] The server generates clipped videos based on the identified highlight scenes. It clips the video according to the timestamp of each scene and connects multiple scenes to create a single highlight video. If necessary, it adds subtitles and sound effects to improve the quality of the video.

[0440] Step 7:

[0441] The server distributes the generated clipped video so that users can view it. Specifically, it uploads the highlight video to a video distribution platform and generates a viewing link. This viewing link is provided to users using a method to notify them (e.g., email or push notification) so that they can easily access it.

[0442] Example 1

[0443] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0444] In recent years, with the spread of video streaming, many viewers are enjoying live streaming and uploaded videos. However, these videos are often long, and watching them in their entirety takes a huge amount of time. Furthermore, manually finding particularly interesting or important scenes in a video is extremely time-consuming and places a considerable burden on viewers. Furthermore, editing and cropping videos requires specialized knowledge and skills, making them difficult for average users. To solve these problems, a system that can automatically extract the highlights of a video and provide them to viewers is needed.

[0445] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0446] In this invention, the server includes means for collecting video data and comment data, means for analyzing the distributor's emotions from the collected video data, means for analyzing the viewer's reactions from the collected comment data, means for analyzing the content of the video data using a natural language processing model to identify highlights, means for generating clipped videos based on the identified highlights, and means for distributing the generated clipped videos. This makes it possible to automatically extract highlights from videos and make them easy to watch.

[0447] "Moving image data" refers to moving image information recorded in video format that can be viewed by a viewer.

[0448] "Comment data" refers to information including opinions and reactions posted by viewers in text format to videos.

[0449] "Emotion analysis" is the process of identifying a person's emotional state (e.g., joy, anger, surprise, etc.) based on video frames and audio data.

[0450] "Viewer reaction" is a concept that refers to the reaction or feedback (e.g., positive, negative, neutral) that viewers give to a video.

[0451] A "natural language processing model" is a machine learning model used to understand and process human language.

[0452] "Keyword extraction" is the process of identifying and extracting important words and phrases from text data.

[0453] A "speech recognition model" is a machine learning model for converting speech into text.

[0454] A "cut-out video" is a short video clip created by cutting out specific scenes from the original video and connecting them together.

[0455] "Video editing software" refers to software tools used to cut, join, and add effects to video clips.

[0456] An "online platform" is a web service that allows users to share and view videos over the Internet.

[0457] "Viewing Link" refers to the URL that allows a user to view a video online.

[0458] An "automatic speech recognition model" is an algorithm that analyzes an input speech signal and converts it into a corresponding string of characters.

[0459] A "turning point in the topic" refers to a moment when the content of the video changes significantly or when the topic switches to a new one.

[0460] An "important scene" refers to a part of a video that deserves particular attention or that will interest viewers.

[0461] This invention is a system that collects video data and comment data from a video distribution platform, analyzes them, automatically extracts highlights, and generates and distributes clipped videos. The following describes the implementation of this system, focusing on a server, a terminal, and a user.

[0462] Collection of video and comment data

[0463] The server calls the video distribution platform API to collect the video data of the specified broadcaster and the comment data associated with that video. Specifically, it receives the URL or broadcast ID of the broadcaster's video and retrieves the video file via the API. The retrieved video file is saved in the server's storage. Comment data is similarly retrieved using the API and stored in a database. For example, the YouTube API can be used to collect the video file and comment data corresponding to the video ID "exampleID."

[0464] Data analysis

[0465] The server performs the following analysis based on the collected video data and comment data.

[0466] 1. Emotion analysis

[0467] The streamer's face is detected from the video frame and input into an emotion analysis model to identify the streamer's emotional state (e.g., joy, anger, surprise, etc.) along with a timestamp. Audio data is also analyzed in parallel to estimate emotions from the audio.

[0468] 2. Comment Analysis

[0469] The comment data is run through a text analysis model to categorize viewer comments into negative, positive, and neutral categories, and keywords are extracted to identify times when viewer responses are most concentrated.

[0470] 3. Content Analysis

[0471] The video content is transcribed using an automatic speech recognition (ASR) model, and the text data is then analyzed using a natural language processing (NLP) model to identify turning points and key moments in the video.

[0472] Extracting highlights and generating clipped videos

[0473] The server identifies scenes that are worth watching based on the analysis results. It determines which scenes are particularly emotional based on the sentiment analysis results, which time periods had the strongest reactions from viewers based on the comment analysis results, and which scenes show turning points in the conversation or many important keywords based on the content analysis results. It then determines the start and end timestamps of the identified scenes and uses video editing software (e.g., FFmpeg) to generate a cut-out video.

[0474] Video distribution

[0475] The server uploads the generated clipped video to a video distribution platform and generates a viewing link to provide it to users. The server also provides the generated viewing link to users by means of email, push notification, etc. For example, the YouTube API can be used to upload the clipped video and notify viewers of the link.

[0476] Specific examples

[0477] For example, consider a case where this system is applied to a live streaming video of a broadcaster A.

[0478] 1. Collection

[0479] The server collects the video data and comment data of distributor A.

[0480] 2. Analysis

[0481] The server uses emotion analysis to identify scenes in the video where streamer A is laughing out loud (e.g., 45:30-45:35), and also uses comment analysis to identify scenes where the number of comments suddenly increases (e.g., 1:15:00-1:15:30).

[0482] 3. Extraction and generation

[0483] These scenes are extracted as highlights and linked together as cut-out videos to create a five-minute highlight video.

[0484] 4. Distribution

[0485] The server uploads this highlight video to a video distribution platform and provides users with a viewing link.

[0486] Prompt Sentence Examples

[0487] "Design a system to naturally crop streams from Streamer A's live stream video and meet the following requirements:

[0488] 1. Collection of video and comment data

[0489] 2. Data analysis using sentiment analysis, comment analysis, and content analysis

[0490] 3. Extract the highlights and create a single video clip

[0491] 4. Upload to a video distribution platform and issue a viewing link

[0492] Please explain with specific examples."

[0493] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0494] Step 1:

[0495] The server calls the API of the video distribution platform to collect video data and comment data. (Input) Receives the URL and distribution ID of the distributed video. (Processing) Obtains the video file via the API and saves it in storage on the server. Similarly, comment data is collected using the API and stored in a database. (Output) The video file and comment data are saved on the server. For example, the YouTube API is used to obtain the video file and comments corresponding to the video ID "exampleID" and save them in their respective destinations.

[0496] Step 2:

[0497] The server analyzes the streamer's emotions from the collected video data. (Input) Saved video file. (Processing) The streamer's face is detected from the video frames and input into an emotion analysis model to identify the emotional state (e.g., joy, anger, surprise, etc.) along with the timestamp. In addition, the audio data is analyzed to estimate emotions from the audio. (Output) Emotional state data corresponding to the timestamp of each frame. For example, it identifies the scene where the streamer is laughing out loud (e.g., timestamp 45:30).

[0498] Step 3:

[0499] The server analyzes viewer reactions from the collected comment data. (Input) Saved comment data. (Processing) The comment data is run through a text analysis model to classify it into negative, positive, and neutral categories. Keywords are also extracted to identify time periods when viewer reactions are concentrated. (Output) The classified comment data and the timestamps where viewer reactions peak. For example, the scene where the number of comments suddenly increased (e.g., timestamp 1:15:00) is identified.

[0500] Step 4:

[0501] The server analyzes the content of the video data to identify highlights. (Input) Saved video file and transcription data. (Processing) The audio from the video is transcribed using an automatic speech recognition (ASR) model, and the resulting text data is analyzed using a natural language processing (NLP) model. This allows for the identification of turning points in the topic and important scenes. (Output) Timestamps of turning points in the topic and important scenes. For example, identifying the scene at 45 minutes and 50 seconds where the topic changed significantly.

[0502] Step 5:

[0503] The server generates a clipped video based on the highlight extraction results. (Input) Sentiment analysis, comment analysis, and content analysis results. (Processing) The server determines the start and end timestamps by comprehensively determining particularly emotional scenes from the sentiment analysis results, time periods with strong viewer reactions from the comment analysis results, and scenes with turning points in the topic or frequent appearance of important keywords from the content analysis results. The clipped video is generated using video editing software (e.g., FFmpeg). (Output) The generated clipped video file. For example, the identified scenes (30:45-31:00, 45:30-45:35, 1:15:00-1:15:30) are concatenated to create a single highlight video "highlights.mp4."

[0504] Step 6:

[0505] The server distributes the generated cut-out video. (Input) Generated cut-out video file. (Processing) Upload the video to an online platform and generate a viewing link. The viewing link is then provided to the user via a method such as email or push notification. (Output) Notification of upload completion to the video distribution platform and a viewing link. For example, upload using the YouTube API and notify the user of the generated viewing link via email.

[0506] (Application example 1)

[0507] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0508] Conventional video streaming services have the drawback of requiring viewers to manually edit long video content to extract the most interesting scenes, which is time-consuming and inefficient. Furthermore, viewers often find it difficult to watch the entire video, potentially missing important or interesting scenes. Furthermore, a lack of real-time analysis and notifications makes it difficult for users to quickly access important information. To address these issues, a system was needed that could automatically analyze video and comment data and notify viewers in real time.

[0509] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0510] In this invention, the server includes means for collecting video data and comment data, means for analyzing the streamer's emotions from the collected video data, means for analyzing viewer reactions from the collected comment data, means for analyzing the content of the video data to identify highlights, means for generating clipped videos based on the identified highlights, means for distributing the generated clipped videos, and means for sending notifications to smartphones. This allows important scenes to be automatically extracted from long video content and provided as clipped videos, allowing viewers to efficiently view highlights without any hassle. Furthermore, the generated clipped videos are notified to the user's smartphone in real time, allowing viewers to quickly access important information without missing anything.

[0511] "Video Data" means digital files containing video and audio information that a User views or broadcasts on an online platform.

[0512] "Comment data" refers to collected textual reactions and feedback entered by viewers while watching video content.

[0513] "Emotion analysis means" refers to algorithms or models that detect facial expressions, vocal tone, etc. from the streamer's video frames to identify the streamer's emotional state.

[0514] The "means for analyzing viewer reactions" refers to natural language processing technology that analyzes comment data and classifies its content as positive, negative, neutral, etc.

[0515] The "means of identifying highlights" refers to technology that analyzes video data and comment data to identify scenes that are particularly noteworthy, such as scenes that elicited strong emotions from viewers and streamers, or time periods when viewers responded most strongly.

[0516] The "means for generating cut-out videos" is a technology that extracts scenes identified as highlights and connects them to generate a single highlight video.

[0517] The "means for sending notifications to smartphones" refers to a system that notifies users of the information in real time on their smartphones when generated highlight videos and highlight scenes become available.

[0518] The means of "distribution" is the technology of uploading the generated clipped video to an online platform, generating a viewing link, and providing it to users.

[0519] The following describes an embodiment of the present invention, focusing on a server, a terminal, and a user. The processing of a program in a system for automatically generating a cut-out video will be described in detail with specific examples.

[0520] Collection of video and comment data

[0521] The server calls the video distribution platform API to collect the video data of the specified broadcaster and the comment data associated with that video. Specifically, it receives the URL and broadcast ID of the broadcast video, obtains the video file through the API, and saves it in the server's storage. It also uses the API to collect comment data and stores it in a database.

[0522] Data analysis

[0523] The server performs the following analysis based on the collected data. First, it detects the streamer's face from the video frames and inputs it into an emotion analysis model to identify the streamer's emotional state along with a timestamp. It also analyzes the audio data in parallel to infer emotions from the audio. Next, it runs the comment data through a text analysis model to classify viewer comments into negative, positive, or neutral categories. It also extracts keywords and identifies time periods when viewer reactions are concentrated. Finally, it transcribes the video content using an automatic speech recognition (ASR) model, and then analyzes the text data using a natural language processing (NLP) model. This allows it to identify turning points in the video and important scenes.

[0524] Extracting highlights and generating clipped videos

[0525] The server identifies scenes that are worth watching based on the results of the analysis. From the results of sentiment analysis, it extracts scenes in which the streamer was particularly emotional. From the results of comment analysis, it identifies time periods when viewers responded most strongly and designates these as scenes worth watching. From the results of content analysis, it identifies scenes where the topic changed or specific important keywords frequently appear. By combining all elements, it determines the start and end timestamps of the scenes worth watching. The server then cuts out the scenes identified as worth watching and concatenates them to generate a single cut-out video. This allows users to easily watch high-quality highlights.

[0526] Video distribution

[0527] The server distributes the generated clipped video to users in the following ways: Uploads it to a video distribution platform and generates a video viewing link. Provides the generated viewing link to users via a method (e.g., email or push notification). Notifying users in real time via push notification has the advantage of allowing them to watch without missing any important information.

[0528] Specific examples

[0529] For example, consider the case where this system is applied to a live streaming video of streamer A. The server collects streamer A's video data and comment data. The server uses emotion analysis to identify scenes in the video where streamer A is laughing out loud, and also uses comment analysis to identify scenes where the number of comments has increased dramatically. These scenes are extracted as highlights, and then linked together as clipped videos to generate a five-minute highlight video. The server then uploads this highlight video to a video distribution platform and provides users with a viewing link via push notification.

[0530] Example prompt sentence:

[0531] Retrieve video data and comments from the provided URLs. Analyze the video frames to detect emotions such as joy, sadness, and surprise. Simultaneously, analyze the comments to classify them into positive, negative, and neutral categories. Identify key moments from the emotion and comment analyzes to create highlight clips. Finally, stitch these highlights together into a 5-minute video and notify the user via push notification.

[0532] As described above, the present invention provides a system that automatically clips videos, enabling efficient and low-cost video editing. Furthermore, by notifying users of the clipped videos in real time, the viewing experience is improved.

[0533] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0534] Step 1:

[0535] Collection of video and comment data

[0536] The server collects the video data and comment data of the specified streamer from the video streaming platform API. Specifically, it calls the API based on the stream video URL or stream ID provided by the user to retrieve the video file and save it in the server's storage. In parallel, it also collects comment data using the same API and stores it in a database.

[0537] Input: Streaming video URL or Streaming ID

[0538] Output: Video files stored in storage, comment data stored in a database

[0539] Step 2:

[0540] Video sentiment analysis

[0541] The server analyzes the streamer's emotions using the collected video data. It detects the streamer's face from the video frames and inputs it into an emotion analysis model to identify the streamer's emotional state. It also analyzes the audio data in parallel to perform voice emotion analysis.

[0542] Input: Video file saved in storage

[0543] Output: Streamer sentiment data with timestamps

[0544] Step 3:

[0545] Text analysis of comment data

[0546] The server runs the comment data through a text analysis model to classify each comment as negative, positive, or neutral, and also uses comment timestamps to identify times when viewers tend to respond more frequently.

[0547] Input: Comment data stored in the database

[0548] Output: Viewer response data (negative, positive, neutral) with timestamps

[0549] Step 4:

[0550] Video content analysis

[0551] The server transcribes the video data using an automatic speech recognition (ASR) model, and then runs the text data through a natural language processing (NLP) model to identify topic changes and important scenes within the video.

[0552] Input: Video file saved in storage

[0553] Output: Text data of important scenes in the video

[0554] Step 5:

[0555] Extraction of highlights

[0556] The server combines the results of emotion analysis, comment analysis, and content analysis to identify highlights. Based on timestamps, it extracts scenes in which the streamer was particularly emotional, times when viewers responded most strongly, and turning points in the conversation, and determines the start and end timestamps of highlights.

[0557] Input: Streamer's emotional data, viewer's reaction data, text data of important scenes

[0558] Output: List of highlights (with start and end timestamps)

[0559] Step 6:

[0560] Generate cropped video

[0561] The server extracts the identified scenes based on the list of highlight scenes, concatenates them, and generates a single cut-out video. Using a video editing library, the server combines each highlight scene to generate a highlight video.

[0562] Input: List of highlights, video files saved in storage

[0563] Output: Generated highlight video file

[0564] Step 7:

[0565] Highlight video distribution

[0566] The server uploads the generated highlight video to a video distribution platform. A link to view the video is generated and provided to users via email, push notification, or other means. In particular, push notifications are sent to smartphones, allowing users to access the highlight video in real time.

[0567] Input: Generated highlight video file

[0568] Output: Viewing link on video streaming platform, notification to user

[0569] By following the above steps, the present invention is a system that automatically cuts out moving images and provides users with efficient, high-quality highlight scenes.

[0570] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0571] The following describes an embodiment of the present invention, focusing on a server, a terminal, and a user. The processing of a system program that automatically generates a cut-out video and recognizes the user's emotions will be described in detail with specific examples.

[0572] Collection of video and comment data

[0573] The server uses the API of the video distribution platform to collect video data and comment data from the specified broadcaster. This involves receiving the URL and broadcast ID of the broadcast video and making an API call to retrieve the video file. The retrieved video file is saved in the server's storage, and the comment data is stored in a database.

[0574] Data analysis

[0575] The server performs the following analysis based on the collected data:

[0576] 1. Emotion analysis

[0577] The server detects the streamer's face from the video frames and uses an emotion analysis model to identify the streamer's emotion (e.g., joy, anger, surprise) at each timestamp. It also analyzes the audio data and estimates emotions from the audio.

[0578] 2. Comment Analysis

[0579] The server then runs the comment data through a text analysis model to categorize viewer comments into positive, negative, and neutral categories, extracting key keywords and identifying times when viewer reactions are most likely to be high.

[0580] 3. Content Analysis

[0581] The server transcribes the video data using an automatic speech recognition (ASR) model and identifies turning points and important scenes in the video using a natural language processing (NLP) model.

[0582] Analysis using emotion engines

[0583] The server integrates the results of the streamer's emotion analysis and the results of the viewer's comment analysis, and uses an emotion engine to comprehensively recognize the user's emotions. This emotion engine analyzes the emotional states of the streamer and the viewer in real time, and combines that data to more accurately identify scenes that users are interested in.

[0584] Extracting highlights and generating clipped videos

[0585] Based on the results of the analysis, the server identifies scenes that are likely to attract significant attention. Specifically, it extracts scenes with large emotional fluctuations from the emotion analysis results, scenes that have drawn active reactions from viewers from the comment analysis results, and important scenes from the content analysis results, and combines these to determine the start and end timestamps of the scenes that are likely to attract significant attention.

[0586] For example, based on the user's live streaming viewing data, the server performs the following steps:

[0587] 1. Collection: The server collects the broadcast video and comment data.

[0588] 2. Analysis: Analyze the streamer's emotions (e.g., scenes where they laugh out loud), scenes that get a lot of viewers' reactions (e.g., when the number of comments suddenly increases), and turning points in the topic.

[0589] 3. Comprehensive analysis by the emotion engine: The emotion engine integrates this data and identifies scenes that are particularly interesting as highlights.

[0590] 4. Cutting and generation: Generate a 5-minute cut video based on the highlights.

[0591] Video distribution

[0592] The server delivers the generated clipped video to the user in the following way:

[0593] The server uploads the highlight video to a video distribution platform and generates a viewing link.

[0594] The server provides the viewing link to the user using a means of notifying the user (e.g., email, push notification) so that the user can easily access it.

[0595] In this way, by combining an emotion engine, the present invention provides a system that automatically extracts high-precision highlight scenes and generates and distributes cut-out videos efficiently and at low cost.

[0596] The processing flow will be explained below.

[0597] Step 1:

[0598] The server uses the API of the video distribution platform to collect video data and comment data from the specified broadcaster. Specifically, it receives the URL or broadcast ID of the broadcast video, makes an API call to obtain the video file, and saves it in storage. At the same time, it also obtains comment data and stores it in a database.

[0599] Step 2:

[0600] The server performs emotion analysis based on the collected video data. It detects the streamer's face from each frame of the video and uses an emotion analysis model to identify the streamer's emotion at each timestamp. It also analyzes the audio data, infers emotions from the audio content, and records the analysis results along with the timestamp.

[0601] Step 3:

[0602] The server then analyzes the collected comment data using a text analysis model. Specifically, it classifies comments as positive, negative, or neutral, and extracts important keywords. This analysis identifies times when viewers are particularly enthusiastic.

[0603] Step 4:

[0604] The server transcribes the video data using an automatic speech recognition (ASR) model, and analyzes the transcribed text data using a natural language processing (NLP) model to identify turning points and important scenes in the video.

[0605] Step 5:

[0606] The server uses an emotion engine to integrate the results of previous analyses and recognize the user's emotions. The server combines the results of the streamer's emotion analysis with the results of the viewer's comments analysis to identify scenes that are of particular interest to the user. This comprehensive analysis allows the server to extract highlights from the video scenes that show large emotional fluctuations or that elicit strong reactions from viewers.

[0607] Step 6:

[0608] The server generates clipped videos based on the highlight scenes. It clips the video using the start and end timestamps of the identified highlight scenes, and connects multiple scenes as needed to create a single highlight video. It also performs editing such as adding subtitles and sound effects.

[0609] Step 7:

[0610] The server provides the generated clipped video to the user. First, it uploads the highlight video to a video distribution platform and generates a viewing link. Next, it provides the viewing link to the user by a means of notifying the user (e.g., email or push notification), allowing the user to easily access the video.

[0611] Example 2

[0612] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0613] Conventional video distribution systems lack the technology to automatically identify scenes of interest to users and generate clipped videos. In particular, they lack a means to integrate and analyze the streamer's emotions and the viewer's reactions, making it difficult to accurately extract highlight scenes. This poses a challenge in terms of the quality and experience of the videos provided to users.

[0614] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0615] In this invention, the server includes means for collecting video data and comment data, means for analyzing the emotions of the distributor from the collected video data, means for analyzing viewer reactions from the collected comment data, means for analyzing the content of the video data to identify highlights, means for integrating the results of the distributor's emotion analysis and the results of the viewer's comment analysis to identify scenes of interest to the user, means for generating clipped videos based on the identified highlights, and means for distributing the generated clipped videos. This makes it possible to comprehensively analyze the emotions of the distributor and the viewer, automatically identify high-precision highlight scenes of interest to the user, and generate and provide optimized clipped videos.

[0616] "Video Data" refers to digital files containing video and audio collected from a video distribution platform designated by the user.

[0617] "Comment data" refers to text messages left by viewers on video distribution platforms.

[0618] "Distributor" means an individual or organization that publishes videos on a video distribution platform.

[0619] "Emotion analysis" is the process of identifying emotions from a streamer's facial expressions and voice.

[0620] A "viewer" is an individual or group that watches a streamer's video on a video streaming platform and leaves comments.

[0621] "Reactions" refers to comments and reactions that viewers make to a video.

[0622] "Highlights" are scenes in the video that are judged to be of particular interest to viewers.

[0623] A "cut-out video" is a short video generated based on a highlight scene.

[0624] "Integration" is the process of combining multiple analysis results into one.

[0625] "Distribution" refers to the act of providing the generated clipped video to a user.

[0626] This invention provides a system that collects and analyzes video data and comment data to identify highlight scenes, and automatically generates and distributes cut-out videos of scenes that interest users.

[0627] Collection of video and comment data

[0628] The server uses the API of the video distribution platform to collect video data and comment data. For example, by using the API key provided by the video distribution platform and specifying the URL of the video and distribution ID to be retrieved, the video file and comment data are downloaded to the server. The retrieved video data is saved in storage within the server (e.g., Amazon S3), and the comment data is stored in a database (e.g., MySQL).

[0629] Example: Use the YouTube API to retrieve videos and their comments from a specific streamer.

[0630] Data analysis

[0631] The server performs the following analysis on the collected video data and comment data:

[0632] 1. Emotion analysis

[0633] The server detects the streamer's face from the video frames and identifies the streamer's emotions using an emotion analysis model (e.g., Microsoft Azure Face API). It also uses Google Cloud Speech-to-Text API to estimate emotions from audio data.

[0634] Example: Identifying smile frames and detecting scenes of people laughing out loud.

[0635] 2. Comment Analysis

[0636] The server runs the comment data through a text analysis model (e.g., Google Cloud Natural Language API) to categorize it into positive, negative, and neutral categories, extract important keywords, and identify times when viewers' reactions are concentrated.

[0637] Example: Identify moments when positive comments like "That's funny!" or "That's amazing!" spike.

[0638] 3. Content Analysis

[0639] The server transcribes the video data using an automatic speech recognition (ASR) model (e.g., AWS Transcribe) and identifies turning points and key scenes in the video using a natural language processing (NLP) model (e.g., GPT-3).

[0640] Example: Analyzing the timing of the transition from "game tips" to "questions from viewers."

[0641] Analysis using emotion engines

[0642] The server uses an emotion engine that combines the results of the individually analyzed emotion analysis and comment analysis to identify scenes that users are interested in. This emotion engine analyzes the streamer's emotions and the viewers' reactions in real time, and combines the data to identify highlight scenes with greater accuracy.

[0643] Extracting highlight scenes and generating cut-out videos

[0644] The server identifies interesting scenes based on the analysis results, determines their start and end timestamps, and then uses a video editing tool such as FFMPEG to generate a five-minute excerpt based on the identified interesting scenes.

[0645] Examples:

[0646] 1. Collection: The server collects the streamer's video data and comment data.

[0647] 2. Analysis: Analyze the streamer's emotions (e.g., laughter), audience reactions, and timing of topic changes.

[0648] 3. Emotion engine integration: Integrate emotions and viewer responses to identify interesting scenes.

[0649] 4. Extract and generate: Extract key scenes and generate a 5-minute video.

[0650] Video distribution

[0651] The server uploads the generated clipped video to a video distribution platform and generates a viewing link to provide it to users. The viewing link is then sent via email using services such as SendGrid, or via a push notification sent via a mobile app.

[0652] Example: Use the YouTube API to upload a highlight video and send users a link to watch it.

[0653] Prompt Sentence Examples

[0654] "Analyze the streamer's emotions and combine them with the analysis of viewers' comments to identify the highlights of the stream. Generate a 5-minute video clip based on the identified scenes."

[0655] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0656] Step 1:

[0657] Collecting videos and comments

[0658] The server uses the API of the video distribution platform to collect video and comment data. It receives the video URL and distribution ID provided by the user as input. Based on this input, the server makes an API call to obtain the video file and comment data. Specifically, it obtains the data using the YouTube API, saves the video file in the server's storage as output, and stores the comment data in a database.

[0659] Specific operation: The server inputs a video URL to the YouTube API, calls the "videos.list" endpoint to retrieve video data, and calls the "liveChat / messages" endpoint to retrieve comment data, then saves them to Amazon S3 and MySQL, respectively.

[0660] Step 2:

[0661] Emotion analysis

[0662] The server receives the captured video data as input, detects the streamer's face from each frame, and analyzes their emotions. This is done using emotion analysis models such as the Microsoft Azure Face API and Google Cloud Vision API. To estimate emotions from audio data, the audio is converted to text using the Google Cloud Speech-to-Text API, and the text is then passed through the emotion analysis model. The output is the emotional state for each timestamp.

[0663] How it works: The server splits the video file and sends each frame to the Azure Face API to detect emotions like smile and surprise, and sends the audio portion to the Google Cloud Speech-to-Text API for transcription, which then analyzes emotions based on the text.

[0664] Step 3:

[0665] Comment Analysis

[0666] The server receives comment data as input and runs it through a text analysis model (e.g., Google Cloud Natural Language API). It classifies comments into positive, negative, and neutral sentiment categories and extracts important keywords. The output is the identified sentiment categories, important keywords, and time periods when reactions are concentrated.

[0667] How it works: The server sends comment data to the Natural Language API for sentiment analysis and keyword extraction, identifying times when comments such as "Amazing!" or "Horrible..." spike.

[0668] Step 4:

[0669] Content Analysis

[0670] The server takes video data as input, transcribes it using an automatic speech recognition (ASR) model (e.g., AWS Transcribe), and then uses a natural language processing (NLP) model (e.g., GPT-3) to identify topical turning points and key scenes in the video. The output is a list of key scenes in the video.

[0671] How it works: The server sends the audio portion of the video to AWS Transcribe, then sends the generated text to GPT-3 to identify the point at which the topic changes, for example, from "How to play" to "Viewer Questions."

[0672] Step 5:

[0673] Integrating data and using sentiment engines

[0674] The server receives the results of sentiment analysis and comment analysis as input and uses an emotion engine to integrate them. The emotion engine analyzes the data in real time and integrates the emotional states of the streamer and viewers. The output is a list of highlight scenes.

[0675] What it does: The emotion engine identifies when a streamer's laugh or surprise coincides with a positive comment from the viewer.

[0676] Step 6:

[0677] Extracting highlight scenes and generating cut-out videos

[0678] The server uses the integrated data as input to identify the start and end timestamps of the highlight scenes, and then generates a 5-minute excerpt using a video editing tool such as FFMPEG. The output is the excerpted video file.

[0679] What it does: The server selects the highlight scenes (scenes with lots of laughs or high comment activity) for FFMPEG, which then cuts out the video and combines it into a single file.

[0680] Step 7:

[0681] Distribution of clipped videos

[0682] The server receives the generated clipped video as input, uploads it to a video distribution platform, generates a viewing link, and provides it to the user via a means of notifying them (e.g., email, push notification). The output is the distributed video link.

[0683] What it does: Upload highlight videos using the YouTube API, send emails with SendGrid, or send push notifications through a mobile app.

[0684] (Application example 2)

[0685] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0686] With conventional content distribution services, it has been difficult to efficiently provide viewers with the highlights of videos. In particular, accurately extracting highlights based on the streamer's emotions and viewer comments is time-consuming and resource-intensive. As a result, ordinary users often miss interesting scenes, resulting in an unsatisfactory viewing experience.

[0687] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting video data and comment data, means for analyzing the distributor's emotions from the collected video data, and means for analyzing the viewer's reactions from the collected comment data. This makes it possible to accurately analyze the distributor's emotions and viewer comments and automatically extract interesting scenes.

[0688] The system further includes a means for analyzing the content of the video data to identify highlights, a means for generating a clipped video based on the identified highlights, a means for distributing the generated clipped video, and a means for distributing the clipped video to a smartphone or a head-mounted display, thereby making it easier for users to view particularly noteworthy scenes as short clips, improving the viewing experience.

[0689] "Video data" refers to digital files containing video and audio distributed by a distributor.

[0690] "Comment Data" refers to the text messages and associated metadata that viewers submit during a broadcast.

[0691] "Emotion analysis" is the process of identifying and analyzing the streamer's emotions from video data.

[0692] "Means for analyzing viewer reactions" refers to technology that classifies viewer reactions based on comment data and identifies their intentions and emotions.

[0693] "Means for identifying highlights by analyzing the content of video data" refers to analytical technology for automatically extracting important scenes and turning points in themes in videos.

[0694] The "means for generating clipped videos" refers to a technology for generating short clips based on identified highlight scenes.

[0695] "Means for distributing the generated cut-out video" refers to a distribution technology for providing the generated cut-out video to users.

[0696] A "smartphone" is a mobile device that has the ability to connect to the Internet and install applications.

[0697] A "head-mounted display" is a device that displays images when worn by the user on the head.

[0698] The "emotion engine" is a technology that comprehensively analyzes the emotions and reactions of streamers and viewers, and recognizes emotions in real time.

[0699] A "natural language processing (NLP) model" is an algorithm that analyzes text data and understands its meaning and intent.

[0700] An "automatic speech recognition (ASR) model" is a speech recognition technology used to convert speech data into text.

[0701] The embodiment of the present invention will be described with reference to a server, a terminal, and a user. A detailed description is provided below.

[0702] Data collection by the server

[0703] The server uses the API of the video distribution platform to collect video data and comment data from the specified broadcaster. Specifically, it receives the URL of the video data and broadcast ID, and makes an API call to retrieve the video file. This retrieved video data is saved in the server's storage, and the comment data is stored in a database.

[0704] Data analysis and emotion recognition

[0705] The server performs the following analysis based on the collected data:

[0706] 1. Emotion analysis:

[0707] The server detects the streamer's face from the video frame and uses an emotion analysis model to identify the streamer's emotion at each timestamp. It also analyzes the audio data and estimates emotions from the audio.

[0708] 2. Comment Analysis:

[0709] The server then runs the comment data through a text analysis model to categorize viewer comments into positive, negative, and neutral categories, extracting key keywords and identifying times when viewer reactions are most likely to be high.

[0710] 3. Content Analysis:

[0711] The server transcribes the video data using an automatic speech recognition (ASR) model and identifies turning points and important scenes in the video using a natural language processing (NLP) model.

[0712] Extracting highlights and generating clipped videos

[0713] Based on this data, the server identifies the highlight scenes that users are interested in. Specifically, it extracts scenes with large emotional fluctuations from the results of emotion analysis, scenes with active viewer reactions from the results of comment analysis, and important scenes from the results of content analysis, and determines the start and end timestamps of the highlight scenes.

[0714] Generate and deliver clipped videos

[0715] The server generates short clips based on the identified highlights, and the resulting clips are delivered to a smartphone or head-mounted display application.

[0716] For example, users can install the application and use a feature called "Emotion Clips" to automatically extract the most interesting scenes based on the streamer's emotions and viewers' reactions, and watch them in short clips.

[0717] The example prompt shown

[0718] We want to create an application that allows users to analyze the streamer's emotions and viewer comments to automatically cut out the most interesting scenes and easily watch them as short clips.

[0719] In this way, the present invention provides a system that combines emotion analysis, comment data analysis, and content analysis to make it easy for users to obtain an advanced viewing experience in a short amount of time.

[0720] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0721] Step 1:

[0722] The server collects video data and comment data using the video distribution platform's API. Specifically, it receives the URL and broadcast ID of the broadcaster's broadcast video and makes an API call. The video file obtained as a result of the API call is saved in the server's storage, and the comment data is stored in a database. The input is the URL or broadcast ID of the broadcast video, and the output is the saved video file and comment data stored in the database.

[0723] Step 2:

[0724] The server analyzes the streamer's emotions by applying the collected video data to an emotion analysis model. Specifically, it detects the streamer's face from the video frames and uses the emotion analysis model to identify the emotion at each timestamp. It also analyzes audio data to complement the emotion. The input is the video data, and the output is the streamer's emotion data at each timestamp.

[0725] Step 3:

[0726] The server performs text analysis on the collected comment data. Specifically, it applies a text analysis model to the comment data, classifying it into positive, negative, and neutral categories, and extracting important keywords. Based on this analysis, it identifies time periods when viewer responses are concentrated. The input is the comment data, and the output is the classified comment data, important keywords, and information on time periods.

[0727] Step 4:

[0728] The server transcribes the video data using an automatic speech recognition (ASR) model and identifies important scenes and turning points in the content using a natural language processing (NLP) model. The input is the video data, and the output is the transcribed text and information about the identified important scenes.

[0729] Step 5:

[0730] The server integrates the results of each analysis to identify highlight scenes. Specifically, it extracts scenes with large emotional fluctuations from the emotion analysis results, scenes with active viewer reactions from the comment analysis results, and important scenes from the content analysis results, and integrates these to determine the start and end timestamps of highlight scenes. The input is the result data of each analysis, and the output is timestamp information for highlight scenes.

[0731] Step 6:

[0732] The server generates a clipped video based on the identified highlight scenes. Specifically, it extracts the relevant scenes from the original video based on the timestamps of the highlight scenes and edits them. The input is the timestamp information of the highlight scenes, and the output is the generated clipped video.

[0733] Step 7:

[0734] The server delivers the generated clips to smartphones or head-mounted displays. Specifically, it uploads short clips to a distribution platform and generates a viewing link. The generated viewing link is then sent to the device. The input is the generated clip, and the output is a notification containing the viewing link.

[0735] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0736] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0737] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0738] [Third embodiment]

[0739] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0740] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0741] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0742] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0743] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0744] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0745] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0746] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0747] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0748] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0749] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0750] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0751] The embodiment of the present invention will be described in terms of a server, a terminal, and a user. The processing of the program of the system for automatically generating a cut-out video will be described in detail with specific examples.

[0752] Collection of video and comment data

[0753] The server calls the video distribution platform API to collect the video data of the specified broadcaster and the comment data associated with that video. Specifically, it receives the URL and broadcast ID of the broadcast video, obtains the video file through the API, and saves it in the server's storage. It also uses the API to collect comment data and stores it in a database.

[0754] Data analysis

[0755] The server performs the following analysis based on the collected data.

[0756] 1. Emotion analysis

[0757] The server detects the streamer's face from the video frame and inputs it into an emotion analysis model to identify the streamer's emotional state (e.g., joy, anger, surprise, etc.) along with a timestamp. It also analyzes the audio data in parallel to estimate emotions from the audio.

[0758] 2. Comment Analysis

[0759] The server runs the comment data through a text analysis model to categorize viewer comments into negative, positive, and neutral categories, and also extracts keywords to identify times when viewer responses are most likely to be concentrated.

[0760] 3. Content Analysis

[0761] The server transcribes the video content using an automatic speech recognition (ASR) model, and then analyzes the text data using a natural language processing (NLP) model, which identifies turning points and important scenes in the video.

[0762] Extracting highlights and generating clipped videos

[0763] Based on the results of the analysis, the server identifies scenes that are worth watching.

[0764] From the results of the emotion analysis, scenes in which the streamer became particularly emotional are extracted.

[0765] From the results of the comment analysis, the time periods when viewers responded most strongly are identified and designated as highlight scenes.

[0766] The results of content analysis identify turning points in the topic and scenes in which specific important keywords appear frequently.

[0767] Each element is combined to determine the start and end timestamps of the highlight scenes.

[0768] The server cuts out and concatenates the scenes identified as highlights to generate a single cut-out video, allowing users to easily view high-quality highlights.

[0769] Video distribution

[0770] The server distributes the generated clipped video to the user in the following manner.

[0771] Upload to a video distribution platform and generate a link to watch the video.

[0772] The generated viewing link will be provided to the user via a method such as email or push notification.

[0773] Specific examples

[0774] For example, consider a case where this system is applied to a live streaming video of a broadcaster A.

[0775] 1. Collection

[0776] The server collects the video data and comment data of distributor A.

[0777] 2. Analysis

[0778] The server uses emotion analysis to identify scenes in the video where streamer A is laughing out loud (e.g., 45:30-45:35), and also uses comment analysis to identify scenes where the number of comments suddenly increases (e.g., 1:15:00-1:15:30).

[0779] 3. Extraction and generation

[0780] These scenes are extracted as highlights and linked together as cut-out videos to create a five-minute highlight video.

[0781] 4. Distribution

[0782] The server uploads this highlight video to a video distribution platform and provides users with a viewing link.

[0783] Thus, the present invention is a system that automatically cuts out moving images, enabling efficient and low-cost video editing.

[0784] The processing flow will be explained below.

[0785] Step 1:

[0786] The server uses the video distribution platform API to collect video data and comment data from the specified broadcaster. First, it receives the URL or broadcast ID of the broadcast video and makes an API call to obtain the video file. The obtained video file is then saved in the server's storage. It also makes a similar API call to obtain comment data related to the video, and stores the obtained comment data in the database.

[0787] Step 2:

[0788] The server performs emotion analysis based on the collected video data. Specifically, it detects the streamer's face from the video frames and inputs it into an emotion analysis model to identify the streamer's emotion (e.g., joy, anger, surprise, etc.) at each timestamp. It also applies the audio data to an audio emotion analysis model to analyze the streamer's emotion from the audio content. The analysis results are recorded along with the timestamp.

[0789] Step 3:

[0790] The server analyzes the comment data through a text analysis model. Specifically, it inputs the comment data into a natural language processing (NLP) model and classifies it into positive, negative, and neutral categories. It also extracts keywords and identifies which timestamps are generating the most active viewer reactions. This allows the server to understand the level of enthusiasm among viewer reactions.

[0791] Step 4:

[0792] The server transcribes the video data using an automatic speech recognition (ASR) model, and then runs the transcribed text data through a natural language processing (NLP) model to identify specific topics and important scenes, thereby understanding what the streamer is saying and which parts are important.

[0793] Step 5:

[0794] The server identifies highlight scenes by combining the results of sentiment analysis, comment analysis, and content analysis. First, it extracts scenes with large emotional fluctuations from the sentiment analysis results. Next, it identifies scenes with extremely positive viewer reactions from the comment analysis results. Finally, it extracts scenes with turning points in the conversation and scenes with many specific important keywords from the content analysis results. It then combines these data to determine the start and end timestamps of each scene.

[0795] Step 6:

[0796] The server generates clipped videos based on the identified highlight scenes. It clips the video according to the timestamp of each scene and connects multiple scenes to create a single highlight video. If necessary, it adds subtitles and sound effects to improve the quality of the video.

[0797] Step 7:

[0798] The server distributes the generated clipped video so that users can view it. Specifically, it uploads the highlight video to a video distribution platform and generates a viewing link. This viewing link is provided to users using a method to notify them (e.g., email or push notification) so that they can easily access it.

[0799] Example 1

[0800] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0801] In recent years, with the spread of video streaming, many viewers are enjoying live streaming and uploaded videos. However, these videos are often long, and watching them in their entirety takes a huge amount of time. Furthermore, manually finding particularly interesting or important scenes in a video is extremely time-consuming and places a considerable burden on viewers. Furthermore, editing and cropping videos requires specialized knowledge and skills, making them difficult for average users. To solve these problems, a system that can automatically extract the highlights of a video and provide them to viewers is needed.

[0802] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0803] In this invention, the server includes means for collecting video data and comment data, means for analyzing the distributor's emotions from the collected video data, means for analyzing the viewer's reactions from the collected comment data, means for analyzing the content of the video data using a natural language processing model to identify highlights, means for generating clipped videos based on the identified highlights, and means for distributing the generated clipped videos. This makes it possible to automatically extract highlights from videos and make them easy to watch.

[0804] "Moving image data" refers to moving image information recorded in video format that can be viewed by a viewer.

[0805] "Comment data" refers to information including opinions and reactions posted by viewers in text format to videos.

[0806] "Emotion analysis" is the process of identifying a person's emotional state (e.g., joy, anger, surprise, etc.) based on video frames and audio data.

[0807] "Viewer reaction" is a concept that refers to the reaction or feedback (e.g., positive, negative, neutral) that viewers give to a video.

[0808] A "natural language processing model" is a machine learning model used to understand and process human language.

[0809] "Keyword extraction" is the process of identifying and extracting important words and phrases from text data.

[0810] A "speech recognition model" is a machine learning model for converting speech into text.

[0811] A "cut-out video" is a short video clip created by cutting out specific scenes from the original video and connecting them together.

[0812] "Video editing software" refers to software tools used to cut, join, and add effects to video clips.

[0813] An "online platform" is a web service that allows users to share and view videos over the Internet.

[0814] "Viewing Link" refers to the URL that allows a user to view a video online.

[0815] An "automatic speech recognition model" is an algorithm that analyzes an input speech signal and converts it into a corresponding string of characters.

[0816] A "turning point in the topic" refers to a moment when the content of the video changes significantly or when the topic switches to a new one.

[0817] An "important scene" refers to a part of a video that deserves particular attention or that will interest viewers.

[0818] This invention is a system that collects video data and comment data from a video distribution platform, analyzes them, automatically extracts highlights, and generates and distributes clipped videos. The following describes the implementation of this system, focusing on a server, a terminal, and a user.

[0819] Collection of video and comment data

[0820] The server calls the video distribution platform API to collect the video data of the specified broadcaster and the comment data associated with that video. Specifically, it receives the URL or broadcast ID of the broadcaster's video and retrieves the video file via the API. The retrieved video file is saved in the server's storage. Comment data is similarly retrieved using the API and stored in a database. For example, the YouTube API can be used to collect the video file and comment data corresponding to the video ID "exampleID."

[0821] Data analysis

[0822] The server performs the following analysis based on the collected video data and comment data.

[0823] 1. Emotion analysis

[0824] The streamer's face is detected from the video frame and input into an emotion analysis model to identify the streamer's emotional state (e.g., joy, anger, surprise, etc.) along with a timestamp. Audio data is also analyzed in parallel to estimate emotions from the audio.

[0825] 2. Comment Analysis

[0826] The comment data is run through a text analysis model to categorize viewer comments into negative, positive, and neutral categories, and keywords are extracted to identify times when viewer responses are most concentrated.

[0827] 3. Content Analysis

[0828] The video content is transcribed using an automatic speech recognition (ASR) model, and the text data is then analyzed using a natural language processing (NLP) model to identify turning points and key moments in the video.

[0829] Extracting highlights and generating clipped videos

[0830] The server identifies scenes that are worth watching based on the analysis results. It determines which scenes are particularly emotional based on the sentiment analysis results, which time periods had the strongest reactions from viewers based on the comment analysis results, and which scenes show turning points in the conversation or many important keywords based on the content analysis results. It then determines the start and end timestamps of the identified scenes and uses video editing software (e.g., FFmpeg) to generate a cut-out video.

[0831] Video distribution

[0832] The server uploads the generated clipped video to a video distribution platform and generates a viewing link to provide it to users. The server also provides the generated viewing link to users by means of email, push notification, etc. For example, the YouTube API can be used to upload the clipped video and notify viewers of the link.

[0833] Specific examples

[0834] For example, consider a case where this system is applied to a live streaming video of a broadcaster A.

[0835] 1. Collection

[0836] The server collects the video data and comment data of distributor A.

[0837] 2. Analysis

[0838] The server uses emotion analysis to identify scenes in the video where streamer A is laughing out loud (e.g., 45:30-45:35), and also uses comment analysis to identify scenes where the number of comments suddenly increases (e.g., 1:15:00-1:15:30).

[0839] 3. Extraction and generation

[0840] These scenes are extracted as highlights and linked together as cut-out videos to create a five-minute highlight video.

[0841] 4. Distribution

[0842] The server uploads this highlight video to a video distribution platform and provides users with a viewing link.

[0843] Prompt Sentence Examples

[0844] "Design a system to naturally crop streams from Streamer A's live stream video and meet the following requirements:

[0845] 1. Collection of video and comment data

[0846] 2. Data analysis using sentiment analysis, comment analysis, and content analysis

[0847] 3. Extract the highlights and create a single video clip

[0848] 4. Upload to a video distribution platform and issue a viewing link

[0849] Please explain with specific examples."

[0850] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0851] Step 1:

[0852] The server calls the API of the video distribution platform to collect video data and comment data. (Input) Receives the URL and distribution ID of the distributed video. (Processing) Obtains the video file via the API and saves it in storage on the server. Similarly, comment data is collected using the API and stored in a database. (Output) The video file and comment data are saved on the server. For example, the YouTube API is used to obtain the video file and comments corresponding to the video ID "exampleID" and save them in their respective destinations.

[0853] Step 2:

[0854] The server analyzes the streamer's emotions from the collected video data. (Input) Saved video file. (Processing) The streamer's face is detected from the video frames and input into an emotion analysis model to identify the emotional state (e.g., joy, anger, surprise, etc.) along with the timestamp. In addition, the audio data is analyzed to estimate emotions from the audio. (Output) Emotional state data corresponding to the timestamp of each frame. For example, it identifies the scene where the streamer is laughing out loud (e.g., timestamp 45:30).

[0855] Step 3:

[0856] The server analyzes viewer reactions from the collected comment data. (Input) Saved comment data. (Processing) The comment data is run through a text analysis model to classify it into negative, positive, and neutral categories. Keywords are also extracted to identify time periods when viewer reactions are concentrated. (Output) The classified comment data and the timestamps where viewer reactions peak. For example, the scene where the number of comments suddenly increased (e.g., timestamp 1:15:00) is identified.

[0857] Step 4:

[0858] The server analyzes the content of the video data to identify highlights. (Input) Saved video file and transcription data. (Processing) The audio from the video is transcribed using an automatic speech recognition (ASR) model, and the resulting text data is analyzed using a natural language processing (NLP) model. This allows for the identification of turning points in the topic and important scenes. (Output) Timestamps of turning points in the topic and important scenes. For example, identifying the scene at 45 minutes and 50 seconds where the topic changed significantly.

[0859] Step 5:

[0860] The server generates a clipped video based on the highlight extraction results. (Input) Sentiment analysis, comment analysis, and content analysis results. (Processing) The server determines the start and end timestamps by comprehensively determining particularly emotional scenes from the sentiment analysis results, time periods with strong viewer reactions from the comment analysis results, and scenes with turning points in the topic or frequent appearance of important keywords from the content analysis results. The clipped video is generated using video editing software (e.g., FFmpeg). (Output) The generated clipped video file. For example, the identified scenes (30:45-31:00, 45:30-45:35, 1:15:00-1:15:30) are concatenated to create a single highlight video "highlights.mp4."

[0861] Step 6:

[0862] The server distributes the generated cut-out video. (Input) Generated cut-out video file. (Processing) Upload the video to an online platform and generate a viewing link. The viewing link is then provided to the user via a method such as email or push notification. (Output) Notification of upload completion to the video distribution platform and a viewing link. For example, upload using the YouTube API and notify the user of the generated viewing link via email.

[0863] (Application example 1)

[0864] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0865] Conventional video streaming services have the drawback of requiring viewers to manually edit long video content to extract the most interesting scenes, which is time-consuming and inefficient. Furthermore, viewers often find it difficult to watch the entire video, potentially missing important or interesting scenes. Furthermore, a lack of real-time analysis and notifications makes it difficult for users to quickly access important information. To address these issues, a system was needed that could automatically analyze video and comment data and notify viewers in real time.

[0866] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0867] In this invention, the server includes means for collecting video data and comment data, means for analyzing the streamer's emotions from the collected video data, means for analyzing viewer reactions from the collected comment data, means for analyzing the content of the video data to identify highlights, means for generating clipped videos based on the identified highlights, means for distributing the generated clipped videos, and means for sending notifications to smartphones. This allows important scenes to be automatically extracted from long video content and provided as clipped videos, allowing viewers to efficiently view highlights without any hassle. Furthermore, the generated clipped videos are notified to the user's smartphone in real time, allowing viewers to quickly access important information without missing anything.

[0868] "Video Data" means digital files containing video and audio information that a User views or broadcasts on an online platform.

[0869] "Comment data" refers to collected textual reactions and feedback entered by viewers while watching video content.

[0870] "Emotion analysis means" refers to algorithms or models that detect facial expressions, vocal tone, etc. from the streamer's video frames to identify the streamer's emotional state.

[0871] The "means for analyzing viewer reactions" refers to natural language processing technology that analyzes comment data and classifies its content as positive, negative, neutral, etc.

[0872] The "means of identifying highlights" refers to technology that analyzes video data and comment data to identify scenes that are particularly noteworthy, such as scenes that elicited strong emotions from viewers and streamers, or time periods when viewers responded most strongly.

[0873] The "means for generating cut-out videos" is a technology that extracts scenes identified as highlights and connects them to generate a single highlight video.

[0874] The "means for sending notifications to smartphones" refers to a system that notifies users of the information in real time on their smartphones when generated highlight videos and highlight scenes become available.

[0875] The means of "distribution" is the technology of uploading the generated clipped video to an online platform, generating a viewing link, and providing it to users.

[0876] The following describes an embodiment of the present invention, focusing on a server, a terminal, and a user. The processing of a program in a system for automatically generating a cut-out video will be described in detail with specific examples.

[0877] Collection of video and comment data

[0878] The server calls the video distribution platform API to collect the video data of the specified broadcaster and the comment data associated with that video. Specifically, it receives the URL and broadcast ID of the broadcast video, obtains the video file through the API, and saves it in the server's storage. It also uses the API to collect comment data and stores it in a database.

[0879] Data analysis

[0880] The server performs the following analysis based on the collected data. First, it detects the streamer's face from the video frames and inputs it into an emotion analysis model to identify the streamer's emotional state along with a timestamp. It also analyzes the audio data in parallel to infer emotions from the audio. Next, it runs the comment data through a text analysis model to classify viewer comments into negative, positive, or neutral categories. It also extracts keywords and identifies time periods when viewer reactions are concentrated. Finally, it transcribes the video content using an automatic speech recognition (ASR) model, and then analyzes the text data using a natural language processing (NLP) model. This allows it to identify turning points in the video and important scenes.

[0881] Extracting highlights and generating clipped videos

[0882] The server identifies scenes that are worth watching based on the results of the analysis. From the results of sentiment analysis, it extracts scenes in which the streamer was particularly emotional. From the results of comment analysis, it identifies time periods when viewers responded most strongly and designates these as scenes worth watching. From the results of content analysis, it identifies scenes where the topic changed or specific important keywords frequently appear. By combining all elements, it determines the start and end timestamps of the scenes worth watching. The server then cuts out the scenes identified as worth watching and concatenates them to generate a single cut-out video. This allows users to easily watch high-quality highlights.

[0883] Video distribution

[0884] The server distributes the generated clipped video to users in the following ways: Uploads it to a video distribution platform and generates a video viewing link. Provides the generated viewing link to users via a method (e.g., email or push notification). Notifying users in real time via push notification has the advantage of allowing them to watch without missing any important information.

[0885] Specific examples

[0886] For example, consider the case where this system is applied to a live streaming video of streamer A. The server collects streamer A's video data and comment data. The server uses emotion analysis to identify scenes in the video where streamer A is laughing out loud, and also uses comment analysis to identify scenes where the number of comments has increased dramatically. These scenes are extracted as highlights, and then linked together as clipped videos to generate a five-minute highlight video. The server then uploads this highlight video to a video distribution platform and provides users with a viewing link via push notification.

[0887] Example prompt sentence:

[0888] Retrieve video data and comments from the provided URLs. Analyze the video frames to detect emotions such as joy, sadness, and surprise. Simultaneously, analyze the comments to classify them into positive, negative, and neutral categories. Identify key moments from the emotion and comment analyzes to create highlight clips. Finally, stitch these highlights together into a 5-minute video and notify the user via push notification.

[0889] As described above, the present invention provides a system that automatically clips videos, enabling efficient and low-cost video editing. Furthermore, by notifying users of the clipped videos in real time, the viewing experience is improved.

[0890] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0891] Step 1:

[0892] Collection of video and comment data

[0893] The server collects the video data and comment data of the specified streamer from the video streaming platform API. Specifically, it calls the API based on the stream video URL or stream ID provided by the user to retrieve the video file and save it in the server's storage. In parallel, it also collects comment data using the same API and stores it in a database.

[0894] Input: Streaming video URL or Streaming ID

[0895] Output: Video files stored in storage, comment data stored in a database

[0896] Step 2:

[0897] Video sentiment analysis

[0898] The server analyzes the streamer's emotions using the collected video data. It detects the streamer's face from the video frames and inputs it into an emotion analysis model to identify the streamer's emotional state. It also analyzes the audio data in parallel to perform voice emotion analysis.

[0899] Input: Video file saved in storage

[0900] Output: Streamer sentiment data with timestamps

[0901] Step 3:

[0902] Text analysis of comment data

[0903] The server runs the comment data through a text analysis model to classify each comment as negative, positive, or neutral, and also uses comment timestamps to identify times when viewers tend to respond more frequently.

[0904] Input: Comment data stored in the database

[0905] Output: Viewer response data (negative, positive, neutral) with timestamps

[0906] Step 4:

[0907] Video content analysis

[0908] The server transcribes the video data using an automatic speech recognition (ASR) model, and then runs the text data through a natural language processing (NLP) model to identify topic changes and important scenes within the video.

[0909] Input: Video file saved in storage

[0910] Output: Text data of important scenes in the video

[0911] Step 5:

[0912] Extraction of highlights

[0913] The server combines the results of emotion analysis, comment analysis, and content analysis to identify highlights. Based on timestamps, it extracts scenes in which the streamer was particularly emotional, times when viewers responded most strongly, and turning points in the conversation, and determines the start and end timestamps of highlights.

[0914] Input: Streamer's emotional data, viewer's reaction data, text data of important scenes

[0915] Output: List of highlights (with start and end timestamps)

[0916] Step 6:

[0917] Generate cropped video

[0918] The server extracts the identified scenes based on the list of highlight scenes, concatenates them, and generates a single cut-out video. Using a video editing library, the server combines each highlight scene to generate a highlight video.

[0919] Input: List of highlights, video files saved in storage

[0920] Output: Generated highlight video file

[0921] Step 7:

[0922] Highlight video distribution

[0923] The server uploads the generated highlight video to a video distribution platform. A link to view the video is generated and provided to users via email, push notification, or other means. In particular, push notifications are sent to smartphones, allowing users to access the highlight video in real time.

[0924] Input: Generated highlight video file

[0925] Output: Viewing link on video streaming platform, notification to user

[0926] By following the above steps, the present invention is a system that automatically cuts out moving images and provides users with efficient, high-quality highlight scenes.

[0927] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0928] The following describes an embodiment of the present invention, focusing on a server, a terminal, and a user. The processing of a system program that automatically generates a cut-out video and recognizes the user's emotions will be described in detail with specific examples.

[0929] Collection of video and comment data

[0930] The server uses the API of the video distribution platform to collect video data and comment data from the specified broadcaster. This involves receiving the URL and broadcast ID of the broadcast video and making an API call to retrieve the video file. The retrieved video file is saved in the server's storage, and the comment data is stored in a database.

[0931] Data analysis

[0932] The server performs the following analysis based on the collected data:

[0933] 1. Emotion analysis

[0934] The server detects the streamer's face from the video frames and uses an emotion analysis model to identify the streamer's emotion (e.g., joy, anger, surprise) at each timestamp. It also analyzes the audio data and estimates emotions from the audio.

[0935] 2. Comment Analysis

[0936] The server then runs the comment data through a text analysis model to categorize viewer comments into positive, negative, and neutral categories, extracting key keywords and identifying times when viewer reactions are most likely to be high.

[0937] 3. Content Analysis

[0938] The server transcribes the video data using an automatic speech recognition (ASR) model and identifies turning points and important scenes in the video using a natural language processing (NLP) model.

[0939] Analysis using emotion engines

[0940] The server integrates the results of the streamer's emotion analysis and the results of the viewer's comment analysis, and uses an emotion engine to comprehensively recognize the user's emotions. This emotion engine analyzes the emotional states of the streamer and the viewer in real time, and combines that data to more accurately identify scenes that users are interested in.

[0941] Extracting highlights and generating clipped videos

[0942] Based on the results of the analysis, the server identifies scenes that are likely to attract significant attention. Specifically, it extracts scenes with large emotional fluctuations from the emotion analysis results, scenes that have drawn active reactions from viewers from the comment analysis results, and important scenes from the content analysis results, and combines these to determine the start and end timestamps of the scenes that are likely to attract significant attention.

[0943] For example, based on the user's live streaming viewing data, the server performs the following steps:

[0944] 1. Collection: The server collects the broadcast video and comment data.

[0945] 2. Analysis: Analyze the streamer's emotions (e.g., scenes where they laugh out loud), scenes that get a lot of viewers' reactions (e.g., when the number of comments suddenly increases), and turning points in the topic.

[0946] 3. Comprehensive analysis by the emotion engine: The emotion engine integrates this data and identifies scenes that are particularly interesting as highlights.

[0947] 4. Cutting and generation: Generate a 5-minute cut video based on the highlights.

[0948] Video distribution

[0949] The server delivers the generated clipped video to the user in the following way:

[0950] The server uploads the highlight video to a video distribution platform and generates a viewing link.

[0951] The server provides the viewing link to the user using a means of notifying the user (e.g., email, push notification) so that the user can easily access it.

[0952] In this way, by combining an emotion engine, the present invention provides a system that automatically extracts high-precision highlight scenes and generates and distributes cut-out videos efficiently and at low cost.

[0953] The processing flow will be explained below.

[0954] Step 1:

[0955] The server uses the API of the video distribution platform to collect video data and comment data from the specified broadcaster. Specifically, it receives the URL or broadcast ID of the broadcast video, makes an API call to obtain the video file, and saves it in storage. At the same time, it also obtains comment data and stores it in a database.

[0956] Step 2:

[0957] The server performs emotion analysis based on the collected video data. It detects the streamer's face from each frame of the video and uses an emotion analysis model to identify the streamer's emotion at each timestamp. It also analyzes the audio data, infers emotions from the audio content, and records the analysis results along with the timestamp.

[0958] Step 3:

[0959] The server then analyzes the collected comment data using a text analysis model. Specifically, it classifies comments as positive, negative, or neutral, and extracts important keywords. This analysis identifies times when viewers are particularly enthusiastic.

[0960] Step 4:

[0961] The server transcribes the video data using an automatic speech recognition (ASR) model, and analyzes the transcribed text data using a natural language processing (NLP) model to identify turning points and important scenes in the video.

[0962] Step 5:

[0963] The server uses an emotion engine to integrate the results of previous analyses and recognize the user's emotions. The server combines the results of the streamer's emotion analysis with the results of the viewer's comments analysis to identify scenes that are of particular interest to the user. This comprehensive analysis allows the server to extract highlights from the video scenes that show large emotional fluctuations or that elicit strong reactions from viewers.

[0964] Step 6:

[0965] The server generates clipped videos based on the highlight scenes. It clips the video using the start and end timestamps of the identified highlight scenes, and connects multiple scenes as needed to create a single highlight video. It also performs editing such as adding subtitles and sound effects.

[0966] Step 7:

[0967] The server provides the generated clipped video to the user. First, it uploads the highlight video to a video distribution platform and generates a viewing link. Next, it provides the viewing link to the user by a means of notifying the user (e.g., email or push notification), allowing the user to easily access the video.

[0968] Example 2

[0969] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0970] Conventional video distribution systems lack the technology to automatically identify scenes of interest to users and generate clipped videos. In particular, they lack a means to integrate and analyze the streamer's emotions and the viewer's reactions, making it difficult to accurately extract highlight scenes. This poses a challenge in terms of the quality and experience of the videos provided to users.

[0971] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0972] In this invention, the server includes means for collecting video data and comment data, means for analyzing the emotions of the distributor from the collected video data, means for analyzing viewer reactions from the collected comment data, means for analyzing the content of the video data to identify highlights, means for integrating the results of the distributor's emotion analysis and the results of the viewer's comment analysis to identify scenes of interest to the user, means for generating clipped videos based on the identified highlights, and means for distributing the generated clipped videos. This makes it possible to comprehensively analyze the emotions of the distributor and the viewer, automatically identify high-precision highlight scenes of interest to the user, and generate and provide optimized clipped videos.

[0973] "Video Data" refers to digital files containing video and audio collected from a video distribution platform designated by the user.

[0974] "Comment data" refers to text messages left by viewers on video distribution platforms.

[0975] "Distributor" means an individual or organization that publishes videos on a video distribution platform.

[0976] "Emotion analysis" is the process of identifying emotions from a streamer's facial expressions and voice.

[0977] A "viewer" is an individual or group that watches a streamer's video on a video streaming platform and leaves comments.

[0978] "Reactions" refers to comments and reactions that viewers make to a video.

[0979] "Highlights" are scenes in the video that are judged to be of particular interest to viewers.

[0980] A "cut-out video" is a short video generated based on a highlight scene.

[0981] "Integration" is the process of combining multiple analysis results into one.

[0982] "Distribution" refers to the act of providing the generated clipped video to a user.

[0983] This invention provides a system that collects and analyzes video data and comment data to identify highlight scenes, and automatically generates and distributes cut-out videos of scenes that interest users.

[0984] Collection of video and comment data

[0985] The server uses the API of the video distribution platform to collect video data and comment data. For example, by using the API key provided by the video distribution platform and specifying the URL of the video and distribution ID to be retrieved, the video file and comment data are downloaded to the server. The retrieved video data is saved in storage within the server (e.g., Amazon S3), and the comment data is stored in a database (e.g., MySQL).

[0986] Example: Use the YouTube API to retrieve videos and their comments from a specific streamer.

[0987] Data analysis

[0988] The server performs the following analysis on the collected video data and comment data:

[0989] 1. Emotion analysis

[0990] The server detects the streamer's face from the video frames and identifies the streamer's emotions using an emotion analysis model (e.g., Microsoft Azure Face API). It also uses Google Cloud Speech-to-Text API to estimate emotions from audio data.

[0991] Example: Identifying smile frames and detecting scenes of people laughing out loud.

[0992] 2. Comment Analysis

[0993] The server runs the comment data through a text analysis model (e.g., Google Cloud Natural Language API) to categorize it into positive, negative, and neutral categories, extract important keywords, and identify times when viewers' reactions are concentrated.

[0994] Example: Identify moments when positive comments like "That's funny!" or "That's amazing!" spike.

[0995] 3. Content Analysis

[0996] The server transcribes the video data using an automatic speech recognition (ASR) model (e.g., AWS Transcribe) and identifies turning points and key scenes in the video using a natural language processing (NLP) model (e.g., GPT-3).

[0997] Example: Analyzing the timing of the transition from "game tips" to "questions from viewers."

[0998] Analysis using emotion engines

[0999] The server uses an emotion engine that combines the results of the individually analyzed emotion analysis and comment analysis to identify scenes that users are interested in. This emotion engine analyzes the streamer's emotions and the viewers' reactions in real time, and combines the data to identify highlight scenes with greater accuracy.

[1000] Extracting highlight scenes and generating cut-out videos

[1001] The server identifies interesting scenes based on the analysis results, determines their start and end timestamps, and then uses a video editing tool such as FFMPEG to generate a five-minute excerpt based on the identified interesting scenes.

[1002] Examples:

[1003] 1. Collection: The server collects the streamer's video data and comment data.

[1004] 2. Analysis: Analyze the streamer's emotions (e.g., laughter), audience reactions, and timing of topic changes.

[1005] 3. Emotion engine integration: Integrate emotions and viewer responses to identify interesting scenes.

[1006] 4. Extract and generate: Extract key scenes and generate a 5-minute video.

[1007] Video distribution

[1008] The server uploads the generated clipped video to a video distribution platform and generates a viewing link to provide it to users. The viewing link is then sent via email using services such as SendGrid, or via a push notification sent via a mobile app.

[1009] Example: Use the YouTube API to upload a highlight video and send users a link to watch it.

[1010] Prompt Sentence Examples

[1011] "Analyze the streamer's emotions and combine them with the analysis of viewers' comments to identify the highlights of the stream. Generate a 5-minute video clip based on the identified scenes."

[1012] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1013] Step 1:

[1014] Collecting videos and comments

[1015] The server uses the API of the video distribution platform to collect video and comment data. It receives the video URL and distribution ID provided by the user as input. Based on this input, the server makes an API call to obtain the video file and comment data. Specifically, it obtains the data using the YouTube API, saves the video file in the server's storage as output, and stores the comment data in a database.

[1016] Specific operation: The server inputs a video URL to the YouTube API, calls the "videos.list" endpoint to retrieve video data, and calls the "liveChat / messages" endpoint to retrieve comment data, then saves them to Amazon S3 and MySQL, respectively.

[1017] Step 2:

[1018] Emotion analysis

[1019] The server receives the captured video data as input, detects the streamer's face from each frame, and analyzes their emotions. This is done using emotion analysis models such as the Microsoft Azure Face API and Google Cloud Vision API. To estimate emotions from audio data, the audio is converted to text using the Google Cloud Speech-to-Text API, and the text is then passed through the emotion analysis model. The output is the emotional state for each timestamp.

[1020] How it works: The server splits the video file and sends each frame to the Azure Face API to detect emotions like smile and surprise, and sends the audio portion to the Google Cloud Speech-to-Text API for transcription, which then analyzes emotions based on the text.

[1021] Step 3:

[1022] Comment Analysis

[1023] The server receives comment data as input and runs it through a text analysis model (e.g., Google Cloud Natural Language API). It classifies comments into positive, negative, and neutral sentiment categories and extracts important keywords. The output is the identified sentiment categories, important keywords, and time periods when reactions are concentrated.

[1024] How it works: The server sends comment data to the Natural Language API for sentiment analysis and keyword extraction, identifying times when comments such as "Amazing!" or "Horrible..." spike.

[1025] Step 4:

[1026] Content Analysis

[1027] The server takes video data as input, transcribes it using an automatic speech recognition (ASR) model (e.g., AWS Transcribe), and then uses a natural language processing (NLP) model (e.g., GPT-3) to identify topical turning points and key scenes in the video. The output is a list of key scenes in the video.

[1028] How it works: The server sends the audio portion of the video to AWS Transcribe, then sends the generated text to GPT-3 to identify the point at which the topic changes, for example, from "How to play" to "Viewer Questions."

[1029] Step 5:

[1030] Integrating data and using sentiment engines

[1031] The server receives the results of sentiment analysis and comment analysis as input and uses an emotion engine to integrate them. The emotion engine analyzes the data in real time and integrates the emotional states of the streamer and viewers. The output is a list of highlight scenes.

[1032] What it does: The emotion engine identifies when a streamer's laugh or surprise coincides with a positive comment from the viewer.

[1033] Step 6:

[1034] Extracting highlight scenes and generating cut-out videos

[1035] The server uses the integrated data as input to identify the start and end timestamps of the highlight scenes, and then generates a 5-minute excerpt using a video editing tool such as FFMPEG. The output is the excerpted video file.

[1036] What it does: The server selects the highlight scenes (scenes with lots of laughs or high comment activity) for FFMPEG, which then cuts out the video and combines it into a single file.

[1037] Step 7:

[1038] Distribution of clipped videos

[1039] The server receives the generated clipped video as input, uploads it to a video distribution platform, generates a viewing link, and provides it to the user via a means of notifying them (e.g., email, push notification). The output is the distributed video link.

[1040] What it does: Upload highlight videos using the YouTube API, send emails with SendGrid, or send push notifications through a mobile app.

[1041] (Application example 2)

[1042] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1043] With conventional content distribution services, it has been difficult to efficiently provide viewers with the highlights of videos. In particular, accurately extracting highlights based on the streamer's emotions and viewer comments is time-consuming and resource-intensive. As a result, ordinary users often miss interesting scenes, resulting in an unsatisfactory viewing experience.

[1044] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting video data and comment data, means for analyzing the distributor's emotions from the collected video data, and means for analyzing the viewer's reactions from the collected comment data. This makes it possible to accurately analyze the distributor's emotions and viewer comments and automatically extract interesting scenes.

[1045] The system further includes a means for analyzing the content of the video data to identify highlights, a means for generating a clipped video based on the identified highlights, a means for distributing the generated clipped video, and a means for distributing the clipped video to a smartphone or a head-mounted display, thereby making it easier for users to view particularly noteworthy scenes as short clips, improving the viewing experience.

[1046] "Video data" refers to digital files containing video and audio distributed by a distributor.

[1047] "Comment Data" refers to the text messages and associated metadata that viewers submit during a broadcast.

[1048] "Emotion analysis" is the process of identifying and analyzing the streamer's emotions from video data.

[1049] "Means for analyzing viewer reactions" refers to technology that classifies viewer reactions based on comment data and identifies their intentions and emotions.

[1050] "Means for identifying highlights by analyzing the content of video data" refers to analytical technology for automatically extracting important scenes and turning points in themes in videos.

[1051] The "means for generating clipped videos" refers to a technology for generating short clips based on identified highlight scenes.

[1052] "Means for distributing the generated cut-out video" refers to a distribution technology for providing the generated cut-out video to users.

[1053] A "smartphone" is a mobile device that has the ability to connect to the Internet and install applications.

[1054] A "head-mounted display" is a device that displays images when worn by the user on the head.

[1055] The "emotion engine" is a technology that comprehensively analyzes the emotions and reactions of streamers and viewers, and recognizes emotions in real time.

[1056] A "natural language processing (NLP) model" is an algorithm that analyzes text data and understands its meaning and intent.

[1057] An "automatic speech recognition (ASR) model" is a speech recognition technology used to convert speech data into text.

[1058] The embodiment of the present invention will be described with reference to a server, a terminal, and a user. A detailed description is provided below.

[1059] Data collection by the server

[1060] The server uses the API of the video distribution platform to collect video data and comment data from the specified broadcaster. Specifically, it receives the URL of the video data and broadcast ID, and makes an API call to retrieve the video file. This retrieved video data is saved in the server's storage, and the comment data is stored in a database.

[1061] Data analysis and emotion recognition

[1062] The server performs the following analysis based on the collected data:

[1063] 1. Emotion analysis:

[1064] The server detects the streamer's face from the video frame and uses an emotion analysis model to identify the streamer's emotion at each timestamp. It also analyzes the audio data and estimates emotions from the audio.

[1065] 2. Comment Analysis:

[1066] The server then runs the comment data through a text analysis model to categorize viewer comments into positive, negative, and neutral categories, extracting key keywords and identifying times when viewer reactions are most likely to be high.

[1067] 3. Content Analysis:

[1068] The server transcribes the video data using an automatic speech recognition (ASR) model and identifies turning points and important scenes in the video using a natural language processing (NLP) model.

[1069] Extracting highlights and generating clipped videos

[1070] Based on this data, the server identifies the highlight scenes that users are interested in. Specifically, it extracts scenes with large emotional fluctuations from the results of emotion analysis, scenes with active viewer reactions from the results of comment analysis, and important scenes from the results of content analysis, and determines the start and end timestamps of the highlight scenes.

[1071] Generate and deliver clipped videos

[1072] The server generates short clips based on the identified highlights, and the resulting clips are delivered to a smartphone or head-mounted display application.

[1073] For example, users can install the application and use a feature called "Emotion Clips" to automatically extract the most interesting scenes based on the streamer's emotions and viewers' reactions, and watch them in short clips.

[1074] The example prompt shown

[1075] We want to create an application that allows users to analyze the streamer's emotions and viewer comments to automatically cut out the most interesting scenes and easily watch them as short clips.

[1076] In this way, the present invention provides a system that combines emotion analysis, comment data analysis, and content analysis to make it easy for users to obtain an advanced viewing experience in a short amount of time.

[1077] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1078] Step 1:

[1079] The server collects video data and comment data using the video distribution platform's API. Specifically, it receives the URL and broadcast ID of the broadcaster's broadcast video and makes an API call. The video file obtained as a result of the API call is saved in the server's storage, and the comment data is stored in a database. The input is the URL or broadcast ID of the broadcast video, and the output is the saved video file and comment data stored in the database.

[1080] Step 2:

[1081] The server analyzes the streamer's emotions by applying the collected video data to an emotion analysis model. Specifically, it detects the streamer's face from the video frames and uses the emotion analysis model to identify the emotion at each timestamp. It also analyzes audio data to complement the emotion. The input is the video data, and the output is the streamer's emotion data at each timestamp.

[1082] Step 3:

[1083] The server performs text analysis on the collected comment data. Specifically, it applies a text analysis model to the comment data, classifying it into positive, negative, and neutral categories, and extracting important keywords. Based on this analysis, it identifies time periods when viewer responses are concentrated. The input is the comment data, and the output is the classified comment data, important keywords, and information on time periods.

[1084] Step 4:

[1085] The server transcribes the video data using an automatic speech recognition (ASR) model and identifies important scenes and turning points in the content using a natural language processing (NLP) model. The input is the video data, and the output is the transcribed text and information about the identified important scenes.

[1086] Step 5:

[1087] The server integrates the results of each analysis to identify highlight scenes. Specifically, it extracts scenes with large emotional fluctuations from the emotion analysis results, scenes with active viewer reactions from the comment analysis results, and important scenes from the content analysis results, and integrates these to determine the start and end timestamps of highlight scenes. The input is the result data of each analysis, and the output is timestamp information for highlight scenes.

[1088] Step 6:

[1089] The server generates a clipped video based on the identified highlight scenes. Specifically, it extracts the relevant scenes from the original video based on the timestamps of the highlight scenes and edits them. The input is the timestamp information of the highlight scenes, and the output is the generated clipped video.

[1090] Step 7:

[1091] The server delivers the generated clips to smartphones or head-mounted displays. Specifically, it uploads short clips to a distribution platform and generates a viewing link. The generated viewing link is then sent to the device. The input is the generated clip, and the output is a notification containing the viewing link.

[1092] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1093] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1094] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1095] [Fourth embodiment]

[1096] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1097] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1098] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1099] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1100] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1101] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1102] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1103] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1104] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1105] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1106] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1107] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1108] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1109] The embodiment of the present invention will be described in terms of a server, a terminal, and a user. The processing of the program of the system for automatically generating a cut-out video will be described in detail with specific examples.

[1110] Collection of video and comment data

[1111] The server calls the video distribution platform API to collect the video data of the specified broadcaster and the comment data associated with that video. Specifically, it receives the URL and broadcast ID of the broadcast video, obtains the video file through the API, and saves it in the server's storage. It also uses the API to collect comment data and stores it in a database.

[1112] Data analysis

[1113] The server performs the following analysis based on the collected data.

[1114] 1. Emotion analysis

[1115] The server detects the streamer's face from the video frame and inputs it into an emotion analysis model to identify the streamer's emotional state (e.g., joy, anger, surprise, etc.) along with a timestamp. It also analyzes the audio data in parallel to estimate emotions from the audio.

[1116] 2. Comment Analysis

[1117] The server runs the comment data through a text analysis model to categorize viewer comments into negative, positive, and neutral categories, and also extracts keywords to identify times when viewer responses are most likely to be concentrated.

[1118] 3. Content Analysis

[1119] The server transcribes the video content using an automatic speech recognition (ASR) model, and then analyzes the text data using a natural language processing (NLP) model, which identifies turning points and important scenes in the video.

[1120] Extracting highlights and generating clipped videos

[1121] Based on the results of the analysis, the server identifies scenes that are worth watching.

[1122] From the results of the emotion analysis, scenes in which the streamer became particularly emotional are extracted.

[1123] From the results of the comment analysis, the time periods when viewers responded most strongly are identified and designated as highlight scenes.

[1124] The results of content analysis identify turning points in the topic and scenes in which specific important keywords appear frequently.

[1125] Each element is combined to determine the start and end timestamps of the highlight scenes.

[1126] The server cuts out and concatenates the scenes identified as highlights to generate a single cut-out video, allowing users to easily view high-quality highlights.

[1127] Video distribution

[1128] The server distributes the generated clipped video to the user in the following manner.

[1129] Upload to a video distribution platform and generate a link to watch the video.

[1130] The generated viewing link will be provided to the user via a method such as email or push notification.

[1131] Specific examples

[1132] For example, consider a case where this system is applied to a live streaming video of a broadcaster A.

[1133] 1. Collection

[1134] The server collects the video data and comment data of distributor A.

[1135] 2. Analysis

[1136] The server uses emotion analysis to identify scenes in the video where streamer A is laughing out loud (e.g., 45:30-45:35), and also uses comment analysis to identify scenes where the number of comments suddenly increases (e.g., 1:15:00-1:15:30).

[1137] 3. Extraction and generation

[1138] These scenes are extracted as highlights and linked together as cut-out videos to create a five-minute highlight video.

[1139] 4. Distribution

[1140] The server uploads this highlight video to a video distribution platform and provides users with a viewing link.

[1141] Thus, the present invention is a system that automatically cuts out moving images, enabling efficient and low-cost video editing.

[1142] The processing flow will be explained below.

[1143] Step 1:

[1144] The server uses the video distribution platform API to collect video data and comment data from the specified broadcaster. First, it receives the URL or broadcast ID of the broadcast video and makes an API call to obtain the video file. The obtained video file is then saved in the server's storage. It also makes a similar API call to obtain comment data related to the video, and stores the obtained comment data in the database.

[1145] Step 2:

[1146] The server performs emotion analysis based on the collected video data. Specifically, it detects the streamer's face from the video frames and inputs it into an emotion analysis model to identify the streamer's emotion (e.g., joy, anger, surprise, etc.) at each timestamp. It also applies the audio data to an audio emotion analysis model to analyze the streamer's emotion from the audio content. The analysis results are recorded along with the timestamp.

[1147] Step 3:

[1148] The server analyzes the comment data through a text analysis model. Specifically, it inputs the comment data into a natural language processing (NLP) model and classifies it into positive, negative, and neutral categories. It also extracts keywords and identifies which timestamps are generating the most active viewer reactions. This allows the server to understand the level of enthusiasm among viewer reactions.

[1149] Step 4:

[1150] The server transcribes the video data using an automatic speech recognition (ASR) model, and then runs the transcribed text data through a natural language processing (NLP) model to identify specific topics and important scenes, thereby understanding what the streamer is saying and which parts are important.

[1151] Step 5:

[1152] The server identifies highlight scenes by combining the results of sentiment analysis, comment analysis, and content analysis. First, it extracts scenes with large emotional fluctuations from the sentiment analysis results. Next, it identifies scenes with extremely positive viewer reactions from the comment analysis results. Finally, it extracts scenes with turning points in the conversation and scenes with many specific important keywords from the content analysis results. It then combines these data to determine the start and end timestamps of each scene.

[1153] Step 6:

[1154] The server generates clipped videos based on the identified highlight scenes. It clips the video according to the timestamp of each scene and connects multiple scenes to create a single highlight video. If necessary, it adds subtitles and sound effects to improve the quality of the video.

[1155] Step 7:

[1156] The server distributes the generated clipped video so that users can view it. Specifically, it uploads the highlight video to a video distribution platform and generates a viewing link. This viewing link is provided to users using a method to notify them (e.g., email or push notification) so that they can easily access it.

[1157] Example 1

[1158] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1159] In recent years, with the spread of video streaming, many viewers are enjoying live streaming and uploaded videos. However, these videos are often long, and watching them in their entirety takes a huge amount of time. Furthermore, manually finding particularly interesting or important scenes in a video is extremely time-consuming and places a considerable burden on viewers. Furthermore, editing and cropping videos requires specialized knowledge and skills, making them difficult for average users. To solve these problems, a system that can automatically extract the highlights of a video and provide them to viewers is needed.

[1160] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1161] In this invention, the server includes means for collecting video data and comment data, means for analyzing the distributor's emotions from the collected video data, means for analyzing the viewer's reactions from the collected comment data, means for analyzing the content of the video data using a natural language processing model to identify highlights, means for generating clipped videos based on the identified highlights, and means for distributing the generated clipped videos. This makes it possible to automatically extract highlights from videos and make them easy to watch.

[1162] "Moving image data" refers to moving image information recorded in video format that can be viewed by a viewer.

[1163] "Comment data" refers to information including opinions and reactions posted by viewers in text format to videos.

[1164] "Emotion analysis" is the process of identifying a person's emotional state (e.g., joy, anger, surprise, etc.) based on video frames and audio data.

[1165] "Viewer reaction" is a concept that refers to the reaction or feedback (e.g., positive, negative, neutral) that viewers give to a video.

[1166] A "natural language processing model" is a machine learning model used to understand and process human language.

[1167] "Keyword extraction" is the process of identifying and extracting important words and phrases from text data.

[1168] A "speech recognition model" is a machine learning model for converting speech into text.

[1169] A "cut-out video" is a short video clip created by cutting out specific scenes from the original video and connecting them together.

[1170] "Video editing software" refers to software tools used to cut, join, and add effects to video clips.

[1171] An "online platform" is a web service that allows users to share and view videos over the Internet.

[1172] "Viewing Link" refers to the URL that allows a user to view a video online.

[1173] An "automatic speech recognition model" is an algorithm that analyzes an input speech signal and converts it into a corresponding string of characters.

[1174] A "turning point in the topic" refers to a moment when the content of the video changes significantly or when the topic switches to a new one.

[1175] An "important scene" refers to a part of a video that deserves particular attention or that will interest viewers.

[1176] This invention is a system that collects video data and comment data from a video distribution platform, analyzes them, automatically extracts highlights, and generates and distributes clipped videos. The following describes the implementation of this system, focusing on a server, a terminal, and a user.

[1177] Collection of video and comment data

[1178] The server calls the video distribution platform API to collect the video data of the specified broadcaster and the comment data associated with that video. Specifically, it receives the URL or broadcast ID of the broadcaster's video and retrieves the video file via the API. The retrieved video file is saved in the server's storage. Comment data is similarly retrieved using the API and stored in a database. For example, the YouTube API can be used to collect the video file and comment data corresponding to the video ID "exampleID."

[1179] Data analysis

[1180] The server performs the following analysis based on the collected video data and comment data.

[1181] 1. Emotion analysis

[1182] The streamer's face is detected from the video frame and input into an emotion analysis model to identify the streamer's emotional state (e.g., joy, anger, surprise, etc.) along with a timestamp. Audio data is also analyzed in parallel to estimate emotions from the audio.

[1183] 2. Comment Analysis

[1184] The comment data is run through a text analysis model to categorize viewer comments into negative, positive, and neutral categories, and keywords are extracted to identify times when viewer responses are most concentrated.

[1185] 3. Content Analysis

[1186] The video content is transcribed using an automatic speech recognition (ASR) model, and the text data is then analyzed using a natural language processing (NLP) model to identify turning points and key moments in the video.

[1187] Extracting highlights and generating clipped videos

[1188] The server identifies scenes that are worth watching based on the analysis results. It determines which scenes are particularly emotional based on the sentiment analysis results, which time periods had the strongest reactions from viewers based on the comment analysis results, and which scenes show turning points in the conversation or many important keywords based on the content analysis results. It then determines the start and end timestamps of the identified scenes and uses video editing software (e.g., FFmpeg) to generate a cut-out video.

[1189] Video distribution

[1190] The server uploads the generated clipped video to a video distribution platform and generates a viewing link to provide it to users. The server also provides the generated viewing link to users by means of email, push notification, etc. For example, the YouTube API can be used to upload the clipped video and notify viewers of the link.

[1191] Specific examples

[1192] For example, consider a case where this system is applied to a live streaming video of a broadcaster A.

[1193] 1. Collection

[1194] The server collects the video data and comment data of distributor A.

[1195] 2. Analysis

[1196] The server uses emotion analysis to identify scenes in the video where streamer A is laughing out loud (e.g., 45:30-45:35), and also uses comment analysis to identify scenes where the number of comments suddenly increases (e.g., 1:15:00-1:15:30).

[1197] 3. Extraction and generation

[1198] These scenes are extracted as highlights and linked together as cut-out videos to create a five-minute highlight video.

[1199] 4. Distribution

[1200] The server uploads this highlight video to a video distribution platform and provides users with a viewing link.

[1201] Prompt Sentence Examples

[1202] "Design a system to naturally crop streams from Streamer A's live stream video and meet the following requirements:

[1203] 1. Collection of video and comment data

[1204] 2. Data analysis using sentiment analysis, comment analysis, and content analysis

[1205] 3. Extract the highlights and create a single video clip

[1206] 4. Upload to a video distribution platform and issue a viewing link

[1207] Please explain with specific examples."

[1208] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1209] Step 1:

[1210] The server calls the API of the video distribution platform to collect video data and comment data. (Input) Receives the URL and distribution ID of the distributed video. (Processing) Obtains the video file via the API and saves it in storage on the server. Similarly, comment data is collected using the API and stored in a database. (Output) The video file and comment data are saved on the server. For example, the YouTube API is used to obtain the video file and comments corresponding to the video ID "exampleID" and save them in their respective destinations.

[1211] Step 2:

[1212] The server analyzes the streamer's emotions from the collected video data. (Input) Saved video file. (Processing) The streamer's face is detected from the video frames and input into an emotion analysis model to identify the emotional state (e.g., joy, anger, surprise, etc.) along with the timestamp. In addition, the audio data is analyzed to estimate emotions from the audio. (Output) Emotional state data corresponding to the timestamp of each frame. For example, it identifies the scene where the streamer is laughing out loud (e.g., timestamp 45:30).

[1213] Step 3:

[1214] The server analyzes viewer reactions from the collected comment data. (Input) Saved comment data. (Processing) The comment data is run through a text analysis model to classify it into negative, positive, and neutral categories. Keywords are also extracted to identify time periods when viewer reactions are concentrated. (Output) The classified comment data and the timestamps where viewer reactions peak. For example, the scene where the number of comments suddenly increased (e.g., timestamp 1:15:00) is identified.

[1215] Step 4:

[1216] The server analyzes the content of the video data to identify highlights. (Input) Saved video file and transcription data. (Processing) The audio from the video is transcribed using an automatic speech recognition (ASR) model, and the resulting text data is analyzed using a natural language processing (NLP) model. This allows for the identification of turning points in the topic and important scenes. (Output) Timestamps of turning points in the topic and important scenes. For example, identifying the scene at 45 minutes and 50 seconds where the topic changed significantly.

[1217] Step 5:

[1218] The server generates a clipped video based on the highlight extraction results. (Input) Sentiment analysis, comment analysis, and content analysis results. (Processing) The server determines the start and end timestamps by comprehensively determining particularly emotional scenes from the sentiment analysis results, time periods with strong viewer reactions from the comment analysis results, and scenes with turning points in the topic or frequent appearance of important keywords from the content analysis results. The clipped video is generated using video editing software (e.g., FFmpeg). (Output) The generated clipped video file. For example, the identified scenes (30:45-31:00, 45:30-45:35, 1:15:00-1:15:30) are concatenated to create a single highlight video "highlights.mp4."

[1219] Step 6:

[1220] The server distributes the generated cut-out video. (Input) Generated cut-out video file. (Processing) Upload the video to an online platform and generate a viewing link. The viewing link is then provided to the user via a method such as email or push notification. (Output) Notification of upload completion to the video distribution platform and a viewing link. For example, upload using the YouTube API and notify the user of the generated viewing link via email.

[1221] (Application example 1)

[1222] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1223] Conventional video streaming services have the drawback of requiring viewers to manually edit long video content to extract the most interesting scenes, which is time-consuming and inefficient. Furthermore, viewers often find it difficult to watch the entire video, potentially missing important or interesting scenes. Furthermore, a lack of real-time analysis and notifications makes it difficult for users to quickly access important information. To address these issues, a system was needed that could automatically analyze video and comment data and notify viewers in real time.

[1224] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1225] In this invention, the server includes means for collecting video data and comment data, means for analyzing the streamer's emotions from the collected video data, means for analyzing viewer reactions from the collected comment data, means for analyzing the content of the video data to identify highlights, means for generating clipped videos based on the identified highlights, means for distributing the generated clipped videos, and means for sending notifications to smartphones. This allows important scenes to be automatically extracted from long video content and provided as clipped videos, allowing viewers to efficiently view highlights without any hassle. Furthermore, the generated clipped videos are notified to the user's smartphone in real time, allowing viewers to quickly access important information without missing anything.

[1226] "Video Data" means digital files containing video and audio information that a User views or broadcasts on an online platform.

[1227] "Comment data" refers to collected textual reactions and feedback entered by viewers while watching video content.

[1228] "Emotion analysis means" refers to algorithms or models that detect facial expressions, vocal tone, etc. from the streamer's video frames to identify the streamer's emotional state.

[1229] The "means for analyzing viewer reactions" refers to natural language processing technology that analyzes comment data and classifies its content as positive, negative, neutral, etc.

[1230] The "means of identifying highlights" refers to technology that analyzes video data and comment data to identify scenes that are particularly noteworthy, such as scenes that elicited strong emotions from viewers and streamers, or time periods when viewers responded most strongly.

[1231] The "means for generating cut-out videos" is a technology that extracts scenes identified as highlights and connects them to generate a single highlight video.

[1232] The "means for sending notifications to smartphones" refers to a system that notifies users of the information in real time on their smartphones when generated highlight videos and highlight scenes become available.

[1233] The means of "distribution" is the technology of uploading the generated clipped video to an online platform, generating a viewing link, and providing it to users.

[1234] The following describes an embodiment of the present invention, focusing on a server, a terminal, and a user. The processing of a program in a system for automatically generating a cut-out video will be described in detail with specific examples.

[1235] Collection of video and comment data

[1236] The server calls the video distribution platform API to collect the video data of the specified broadcaster and the comment data associated with that video. Specifically, it receives the URL and broadcast ID of the broadcast video, obtains the video file through the API, and saves it in the server's storage. It also uses the API to collect comment data and stores it in a database.

[1237] Data analysis

[1238] The server performs the following analysis based on the collected data. First, it detects the streamer's face from the video frames and inputs it into an emotion analysis model to identify the streamer's emotional state along with a timestamp. It also analyzes the audio data in parallel to infer emotions from the audio. Next, it runs the comment data through a text analysis model to classify viewer comments into negative, positive, or neutral categories. It also extracts keywords and identifies time periods when viewer reactions are concentrated. Finally, it transcribes the video content using an automatic speech recognition (ASR) model, and then analyzes the text data using a natural language processing (NLP) model. This allows it to identify turning points in the video and important scenes.

[1239] Extracting highlights and generating clipped videos

[1240] The server identifies scenes that are worth watching based on the results of the analysis. From the results of sentiment analysis, it extracts scenes in which the streamer was particularly emotional. From the results of comment analysis, it identifies time periods when viewers responded most strongly and designates these as scenes worth watching. From the results of content analysis, it identifies scenes where the topic changed or specific important keywords frequently appear. By combining all elements, it determines the start and end timestamps of the scenes worth watching. The server then cuts out the scenes identified as worth watching and concatenates them to generate a single cut-out video. This allows users to easily watch high-quality highlights.

[1241] Video distribution

[1242] The server distributes the generated clipped video to users in the following ways: Uploads it to a video distribution platform and generates a video viewing link. Provides the generated viewing link to users via a method (e.g., email or push notification). Notifying users in real time via push notification has the advantage of allowing them to watch without missing any important information.

[1243] Specific examples

[1244] For example, consider the case where this system is applied to a live streaming video of streamer A. The server collects streamer A's video data and comment data. The server uses emotion analysis to identify scenes in the video where streamer A is laughing out loud, and also uses comment analysis to identify scenes where the number of comments has increased dramatically. These scenes are extracted as highlights, and then linked together as clipped videos to generate a five-minute highlight video. The server then uploads this highlight video to a video distribution platform and provides users with a viewing link via push notification.

[1245] Example prompt sentence:

[1246] Retrieve video data and comments from the provided URLs. Analyze the video frames to detect emotions such as joy, sadness, and surprise. Simultaneously, analyze the comments to classify them into positive, negative, and neutral categories. Identify key moments from the emotion and comment analyzes to create highlight clips. Finally, stitch these highlights together into a 5-minute video and notify the user via push notification.

[1247] As described above, the present invention provides a system that automatically clips videos, enabling efficient and low-cost video editing. Furthermore, by notifying users of the clipped videos in real time, the viewing experience is improved.

[1248] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1249] Step 1:

[1250] Collection of video and comment data

[1251] The server collects the video data and comment data of the specified streamer from the video streaming platform API. Specifically, it calls the API based on the stream video URL or stream ID provided by the user to retrieve the video file and save it in the server's storage. In parallel, it also collects comment data using the same API and stores it in a database.

[1252] Input: Streaming video URL or Streaming ID

[1253] Output: Video files stored in storage, comment data stored in a database

[1254] Step 2:

[1255] Video sentiment analysis

[1256] The server analyzes the streamer's emotions using the collected video data. It detects the streamer's face from the video frames and inputs it into an emotion analysis model to identify the streamer's emotional state. It also analyzes the audio data in parallel to perform voice emotion analysis.

[1257] Input: Video file saved in storage

[1258] Output: Streamer sentiment data with timestamps

[1259] Step 3:

[1260] Text analysis of comment data

[1261] The server runs the comment data through a text analysis model to classify each comment as negative, positive, or neutral, and also uses comment timestamps to identify times when viewers tend to respond more frequently.

[1262] Input: Comment data stored in the database

[1263] Output: Viewer response data (negative, positive, neutral) with timestamps

[1264] Step 4:

[1265] Video content analysis

[1266] The server transcribes the video data using an automatic speech recognition (ASR) model, and then runs the text data through a natural language processing (NLP) model to identify topic changes and important scenes within the video.

[1267] Input: Video file saved in storage

[1268] Output: Text data of important scenes in the video

[1269] Step 5:

[1270] Extraction of highlights

[1271] The server combines the results of emotion analysis, comment analysis, and content analysis to identify highlights. Based on timestamps, it extracts scenes in which the streamer was particularly emotional, times when viewers responded most strongly, and turning points in the conversation, and determines the start and end timestamps of highlights.

[1272] Input: Streamer's emotional data, viewer's reaction data, text data of important scenes

[1273] Output: List of highlights (with start and end timestamps)

[1274] Step 6:

[1275] Generate cropped video

[1276] The server extracts the identified scenes based on the list of highlight scenes, concatenates them, and generates a single cut-out video. Using a video editing library, the server combines each highlight scene to generate a highlight video.

[1277] Input: List of highlights, video files saved in storage

[1278] Output: Generated highlight video file

[1279] Step 7:

[1280] Highlight video distribution

[1281] The server uploads the generated highlight video to a video distribution platform. A link to view the video is generated and provided to users via email, push notification, or other means. In particular, push notifications are sent to smartphones, allowing users to access the highlight video in real time.

[1282] Input: Generated highlight video file

[1283] Output: Viewing link on video streaming platform, notification to user

[1284] By following the above steps, the present invention is a system that automatically cuts out moving images and provides users with efficient, high-quality highlight scenes.

[1285] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1286] The following describes an embodiment of the present invention, focusing on a server, a terminal, and a user. The processing of a system program that automatically generates a cut-out video and recognizes the user's emotions will be described in detail with specific examples.

[1287] Collection of video and comment data

[1288] The server uses the API of the video distribution platform to collect video data and comment data from the specified broadcaster. This involves receiving the URL and broadcast ID of the broadcast video and making an API call to retrieve the video file. The retrieved video file is saved in the server's storage, and the comment data is stored in a database.

[1289] Data analysis

[1290] The server performs the following analysis based on the collected data:

[1291] 1. Emotion analysis

[1292] The server detects the streamer's face from the video frames and uses an emotion analysis model to identify the streamer's emotion (e.g., joy, anger, surprise) at each timestamp. It also analyzes the audio data and estimates emotions from the audio.

[1293] 2. Comment Analysis

[1294] The server then runs the comment data through a text analysis model to categorize viewer comments into positive, negative, and neutral categories, extracting key keywords and identifying times when viewer reactions are most likely to be high.

[1295] 3. Content Analysis

[1296] The server transcribes the video data using an automatic speech recognition (ASR) model and identifies turning points and important scenes in the video using a natural language processing (NLP) model.

[1297] Analysis using emotion engines

[1298] The server integrates the results of the streamer's emotion analysis and the results of the viewer's comment analysis, and uses an emotion engine to comprehensively recognize the user's emotions. This emotion engine analyzes the emotional states of the streamer and the viewer in real time, and combines that data to more accurately identify scenes that users are interested in.

[1299] Extracting highlights and generating clipped videos

[1300] Based on the results of the analysis, the server identifies scenes that are likely to attract significant attention. Specifically, it extracts scenes with large emotional fluctuations from the emotion analysis results, scenes that have drawn active reactions from viewers from the comment analysis results, and important scenes from the content analysis results, and combines these to determine the start and end timestamps of the scenes that are likely to attract significant attention.

[1301] For example, based on the user's live streaming viewing data, the server performs the following steps:

[1302] 1. Collection: The server collects the broadcast video and comment data.

[1303] 2. Analysis: Analyze the streamer's emotions (e.g., scenes where they laugh out loud), scenes that get a lot of viewers' reactions (e.g., when the number of comments suddenly increases), and turning points in the topic.

[1304] 3. Comprehensive analysis by the emotion engine: The emotion engine integrates this data and identifies scenes that are particularly interesting as highlights.

[1305] 4. Cutting and generation: Generate a 5-minute cut video based on the highlights.

[1306] Video distribution

[1307] The server delivers the generated clipped video to the user in the following way:

[1308] The server uploads the highlight video to a video distribution platform and generates a viewing link.

[1309] The server provides the viewing link to the user using a means of notifying the user (e.g., email, push notification) so that the user can easily access it.

[1310] In this way, by combining an emotion engine, the present invention provides a system that automatically extracts high-precision highlight scenes and generates and distributes cut-out videos efficiently and at low cost.

[1311] The processing flow will be explained below.

[1312] Step 1:

[1313] The server uses the API of the video distribution platform to collect video data and comment data from the specified broadcaster. Specifically, it receives the URL or broadcast ID of the broadcast video, makes an API call to obtain the video file, and saves it in storage. At the same time, it also obtains comment data and stores it in a database.

[1314] Step 2:

[1315] The server performs emotion analysis based on the collected video data. It detects the streamer's face from each frame of the video and uses an emotion analysis model to identify the streamer's emotion at each timestamp. It also analyzes the audio data, infers emotions from the audio content, and records the analysis results along with the timestamp.

[1316] Step 3:

[1317] The server then analyzes the collected comment data using a text analysis model. Specifically, it classifies comments as positive, negative, or neutral, and extracts important keywords. This analysis identifies times when viewers are particularly enthusiastic.

[1318] Step 4:

[1319] The server transcribes the video data using an automatic speech recognition (ASR) model, and analyzes the transcribed text data using a natural language processing (NLP) model to identify turning points and important scenes in the video.

[1320] Step 5:

[1321] The server uses an emotion engine to integrate the results of previous analyses and recognize the user's emotions. The server combines the results of the streamer's emotion analysis with the results of the viewer's comments analysis to identify scenes that are of particular interest to the user. This comprehensive analysis allows the server to extract highlights from the video scenes that show large emotional fluctuations or that elicit strong reactions from viewers.

[1322] Step 6:

[1323] The server generates clipped videos based on the highlight scenes. It clips the video using the start and end timestamps of the identified highlight scenes, and connects multiple scenes as needed to create a single highlight video. It also performs editing such as adding subtitles and sound effects.

[1324] Step 7:

[1325] The server provides the generated clipped video to the user. First, it uploads the highlight video to a video distribution platform and generates a viewing link. Next, it provides the viewing link to the user by a means of notifying the user (e.g., email or push notification), allowing the user to easily access the video.

[1326] Example 2

[1327] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1328] Conventional video distribution systems lack the technology to automatically identify scenes of interest to users and generate clipped videos. In particular, they lack a means to integrate and analyze the streamer's emotions and the viewer's reactions, making it difficult to accurately extract highlight scenes. This poses a challenge in terms of the quality and experience of the videos provided to users.

[1329] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1330] In this invention, the server includes means for collecting video data and comment data, means for analyzing the emotions of the distributor from the collected video data, means for analyzing viewer reactions from the collected comment data, means for analyzing the content of the video data to identify highlights, means for integrating the results of the distributor's emotion analysis and the results of the viewer's comment analysis to identify scenes of interest to the user, means for generating clipped videos based on the identified highlights, and means for distributing the generated clipped videos. This makes it possible to comprehensively analyze the emotions of the distributor and the viewer, automatically identify high-precision highlight scenes of interest to the user, and generate and provide optimized clipped videos.

[1331] "Video Data" refers to digital files containing video and audio collected from a video distribution platform designated by the user.

[1332] "Comment data" refers to text messages left by viewers on video distribution platforms.

[1333] "Distributor" means an individual or organization that publishes videos on a video distribution platform.

[1334] "Emotion analysis" is the process of identifying emotions from a streamer's facial expressions and voice.

[1335] A "viewer" is an individual or group that watches a streamer's video on a video streaming platform and leaves comments.

[1336] "Reactions" refers to comments and reactions that viewers make to a video.

[1337] "Highlights" are scenes in the video that are judged to be of particular interest to viewers.

[1338] A "cut-out video" is a short video generated based on a highlight scene.

[1339] "Integration" is the process of combining multiple analysis results into one.

[1340] "Distribution" refers to the act of providing the generated clipped video to a user.

[1341] This invention provides a system that collects and analyzes video data and comment data to identify highlight scenes, and automatically generates and distributes cut-out videos of scenes that interest users.

[1342] Collection of video and comment data

[1343] The server uses the API of the video distribution platform to collect video data and comment data. For example, by using the API key provided by the video distribution platform and specifying the URL of the video and distribution ID to be retrieved, the video file and comment data are downloaded to the server. The retrieved video data is saved in storage within the server (e.g., Amazon S3), and the comment data is stored in a database (e.g., MySQL).

[1344] Example: Use the YouTube API to retrieve videos and their comments from a specific streamer.

[1345] Data analysis

[1346] The server performs the following analysis on the collected video data and comment data:

[1347] 1. Emotion analysis

[1348] The server detects the streamer's face from the video frames and identifies the streamer's emotions using an emotion analysis model (e.g., Microsoft Azure Face API). It also uses Google Cloud Speech-to-Text API to estimate emotions from audio data.

[1349] Example: Identifying smile frames and detecting scenes of people laughing out loud.

[1350] 2. Comment Analysis

[1351] The server runs the comment data through a text analysis model (e.g., Google Cloud Natural Language API) to categorize it into positive, negative, and neutral categories, extract important keywords, and identify times when viewers' reactions are concentrated.

[1352] Example: Identify moments when positive comments like "That's funny!" or "That's amazing!" spike.

[1353] 3. Content Analysis

[1354] The server transcribes the video data using an automatic speech recognition (ASR) model (e.g., AWS Transcribe) and identifies turning points and key scenes in the video using a natural language processing (NLP) model (e.g., GPT-3).

[1355] Example: Analyzing the timing of the transition from "game tips" to "questions from viewers."

[1356] Analysis using emotion engines

[1357] The server uses an emotion engine that combines the results of the individually analyzed emotion analysis and comment analysis to identify scenes that users are interested in. This emotion engine analyzes the streamer's emotions and the viewers' reactions in real time, and combines the data to identify highlight scenes with greater accuracy.

[1358] Extracting highlight scenes and generating cut-out videos

[1359] The server identifies interesting scenes based on the analysis results, determines their start and end timestamps, and then uses a video editing tool such as FFMPEG to generate a five-minute excerpt based on the identified interesting scenes.

[1360] Examples:

[1361] 1. Collection: The server collects the streamer's video data and comment data.

[1362] 2. Analysis: Analyze the streamer's emotions (e.g., laughter), audience reactions, and timing of topic changes.

[1363] 3. Emotion engine integration: Integrate emotions and viewer responses to identify interesting scenes.

[1364] 4. Extract and generate: Extract key scenes and generate a 5-minute video.

[1365] Video distribution

[1366] The server uploads the generated clipped video to a video distribution platform and generates a viewing link to provide it to users. The viewing link is then sent via email using services such as SendGrid, or via a push notification sent via a mobile app.

[1367] Example: Use the YouTube API to upload a highlight video and send users a link to watch it.

[1368] Prompt Sentence Examples

[1369] "Analyze the streamer's emotions and combine them with the analysis of viewers' comments to identify the highlights of the stream. Generate a 5-minute video clip based on the identified scenes."

[1370] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1371] Step 1:

[1372] Collecting videos and comments

[1373] The server uses the API of the video distribution platform to collect video and comment data. It receives the video URL and distribution ID provided by the user as input. Based on this input, the server makes an API call to obtain the video file and comment data. Specifically, it obtains the data using the YouTube API, saves the video file in the server's storage as output, and stores the comment data in a database.

[1374] Specific operation: The server inputs a video URL to the YouTube API, calls the "videos.list" endpoint to retrieve video data, and calls the "liveChat / messages" endpoint to retrieve comment data, then saves them to Amazon S3 and MySQL, respectively.

[1375] Step 2:

[1376] Emotion analysis

[1377] The server receives the captured video data as input, detects the streamer's face from each frame, and analyzes their emotions. This is done using emotion analysis models such as the Microsoft Azure Face API and Google Cloud Vision API. To estimate emotions from audio data, the audio is converted to text using the Google Cloud Speech-to-Text API, and the text is then passed through the emotion analysis model. The output is the emotional state for each timestamp.

[1378] How it works: The server splits the video file and sends each frame to the Azure Face API to detect emotions like smile and surprise, and sends the audio portion to the Google Cloud Speech-to-Text API for transcription, which then analyzes emotions based on the text.

[1379] Step 3:

[1380] Comment Analysis

[1381] The server receives comment data as input and runs it through a text analysis model (e.g., Google Cloud Natural Language API). It classifies comments into positive, negative, and neutral sentiment categories and extracts important keywords. The output is the identified sentiment categories, important keywords, and time periods when reactions are concentrated.

[1382] How it works: The server sends comment data to the Natural Language API for sentiment analysis and keyword extraction, identifying times when comments such as "Amazing!" or "Horrible..." spike.

[1383] Step 4:

[1384] Content Analysis

[1385] The server takes video data as input, transcribes it using an automatic speech recognition (ASR) model (e.g., AWS Transcribe), and then uses a natural language processing (NLP) model (e.g., GPT-3) to identify topical turning points and key scenes in the video. The output is a list of key scenes in the video.

[1386] How it works: The server sends the audio portion of the video to AWS Transcribe, then sends the generated text to GPT-3 to identify the point at which the topic changes, for example, from "How to play" to "Viewer Questions."

[1387] Step 5:

[1388] Integrating data and using sentiment engines

[1389] The server receives the results of sentiment analysis and comment analysis as input and uses an emotion engine to integrate them. The emotion engine analyzes the data in real time and integrates the emotional states of the streamer and viewers. The output is a list of highlight scenes.

[1390] What it does: The emotion engine identifies when a streamer's laugh or surprise coincides with a positive comment from the viewer.

[1391] Step 6:

[1392] Extracting highlight scenes and generating cut-out videos

[1393] The server uses the integrated data as input to identify the start and end timestamps of the highlight scenes, and then generates a 5-minute excerpt using a video editing tool such as FFMPEG. The output is the excerpted video file.

[1394] What it does: The server selects the highlight scenes (scenes with lots of laughs or high comment activity) for FFMPEG, which then cuts out the video and combines it into a single file.

[1395] Step 7:

[1396] Distribution of clipped videos

[1397] The server receives the generated clipped video as input, uploads it to a video distribution platform, generates a viewing link, and provides it to the user via a means of notifying them (e.g., email, push notification). The output is the distributed video link.

[1398] What it does: Upload highlight videos using the YouTube API, send emails with SendGrid, or send push notifications through a mobile app.

[1399] (Application example 2)

[1400] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1401] With conventional content distribution services, it has been difficult to efficiently provide viewers with the highlights of videos. In particular, accurately extracting highlights based on the streamer's emotions and viewer comments is time-consuming and resource-intensive. As a result, ordinary users often miss interesting scenes, resulting in an unsatisfactory viewing experience.

[1402] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting video data and comment data, means for analyzing the distributor's emotions from the collected video data, and means for analyzing the viewer's reactions from the collected comment data. This makes it possible to accurately analyze the distributor's emotions and viewer comments and automatically extract interesting scenes.

[1403] The system further includes a means for analyzing the content of the video data to identify highlights, a means for generating a clipped video based on the identified highlights, a means for distributing the generated clipped video, and a means for distributing the clipped video to a smartphone or a head-mounted display, thereby making it easier for users to view particularly noteworthy scenes as short clips, improving the viewing experience.

[1404] "Video data" refers to digital files containing video and audio distributed by a distributor.

[1405] "Comment Data" refers to the text messages and associated metadata that viewers submit during a broadcast.

[1406] "Emotion analysis" is the process of identifying and analyzing the streamer's emotions from video data.

[1407] "Means for analyzing viewer reactions" refers to technology that classifies viewer reactions based on comment data and identifies their intentions and emotions.

[1408] "Means for identifying highlights by analyzing the content of video data" refers to analytical technology for automatically extracting important scenes and turning points in themes in videos.

[1409] The "means for generating clipped videos" refers to a technology for generating short clips based on identified highlight scenes.

[1410] "Means for distributing the generated cut-out video" refers to a distribution technology for providing the generated cut-out video to users.

[1411] A "smartphone" is a mobile device that has the ability to connect to the Internet and install applications.

[1412] A "head-mounted display" is a device that displays images when worn by the user on the head.

[1413] The "emotion engine" is a technology that comprehensively analyzes the emotions and reactions of streamers and viewers, and recognizes emotions in real time.

[1414] A "natural language processing (NLP) model" is an algorithm that analyzes text data and understands its meaning and intent.

[1415] An "automatic speech recognition (ASR) model" is a speech recognition technology used to convert speech data into text.

[1416] The embodiment of the present invention will be described with reference to a server, a terminal, and a user. A detailed description is provided below.

[1417] Data collection by the server

[1418] The server uses the API of the video distribution platform to collect video data and comment data from the specified broadcaster. Specifically, it receives the URL of the video data and broadcast ID, and makes an API call to retrieve the video file. This retrieved video data is saved in the server's storage, and the comment data is stored in a database.

[1419] Data analysis and emotion recognition

[1420] The server performs the following analysis based on the collected data:

[1421] 1. Emotion analysis:

[1422] The server detects the streamer's face from the video frame and uses an emotion analysis model to identify the streamer's emotion at each timestamp. It also analyzes the audio data and estimates emotions from the audio.

[1423] 2. Comment Analysis:

[1424] The server then runs the comment data through a text analysis model to categorize viewer comments into positive, negative, and neutral categories, extracting key keywords and identifying times when viewer reactions are most likely to be high.

[1425] 3. Content Analysis:

[1426] The server transcribes the video data using an automatic speech recognition (ASR) model and identifies turning points and important scenes in the video using a natural language processing (NLP) model.

[1427] Extracting highlights and generating clipped videos

[1428] Based on this data, the server identifies the highlight scenes that users are interested in. Specifically, it extracts scenes with large emotional fluctuations from the results of emotion analysis, scenes with active viewer reactions from the results of comment analysis, and important scenes from the results of content analysis, and determines the start and end timestamps of the highlight scenes.

[1429] Generate and deliver clipped videos

[1430] The server generates short clips based on the identified highlights, and the resulting clips are delivered to a smartphone or head-mounted display application.

[1431] For example, users can install the application and use a feature called "Emotion Clips" to automatically extract the most interesting scenes based on the streamer's emotions and viewers' reactions, and watch them in short clips.

[1432] The example prompt shown

[1433] We want to create an application that allows users to analyze the streamer's emotions and viewer comments to automatically cut out the most interesting scenes and easily watch them as short clips.

[1434] In this way, the present invention provides a system that combines emotion analysis, comment data analysis, and content analysis to make it easy for users to obtain an advanced viewing experience in a short amount of time.

[1435] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1436] Step 1:

[1437] The server collects video data and comment data using the video distribution platform's API. Specifically, it receives the URL and broadcast ID of the broadcaster's broadcast video and makes an API call. The video file obtained as a result of the API call is saved in the server's storage, and the comment data is stored in a database. The input is the URL or broadcast ID of the broadcast video, and the output is the saved video file and comment data stored in the database.

[1438] Step 2:

[1439] The server analyzes the streamer's emotions by applying the collected video data to an emotion analysis model. Specifically, it detects the streamer's face from the video frames and uses the emotion analysis model to identify the emotion at each timestamp. It also analyzes audio data to complement the emotion. The input is the video data, and the output is the streamer's emotion data at each timestamp.

[1440] Step 3:

[1441] The server performs text analysis on the collected comment data. Specifically, it applies a text analysis model to the comment data, classifying it into positive, negative, and neutral categories, and extracting important keywords. Based on this analysis, it identifies time periods when viewer responses are concentrated. The input is the comment data, and the output is the classified comment data, important keywords, and information on time periods.

[1442] Step 4:

[1443] The server transcribes the video data using an automatic speech recognition (ASR) model and identifies important scenes and turning points in the content using a natural language processing (NLP) model. The input is the video data, and the output is the transcribed text and information about the identified important scenes.

[1444] Step 5:

[1445] The server integrates the results of each analysis to identify highlight scenes. Specifically, it extracts scenes with large emotional fluctuations from the emotion analysis results, scenes with active viewer reactions from the comment analysis results, and important scenes from the content analysis results, and integrates these to determine the start and end timestamps of highlight scenes. The input is the result data of each analysis, and the output is timestamp information for highlight scenes.

[1446] Step 6:

[1447] The server generates a clipped video based on the identified highlight scenes. Specifically, it extracts the relevant scenes from the original video based on the timestamps of the highlight scenes and edits them. The input is the timestamp information of the highlight scenes, and the output is the generated clipped video.

[1448] Step 7:

[1449] The server delivers the generated clips to smartphones or head-mounted displays. Specifically, it uploads short clips to a distribution platform and generates a viewing link. The generated viewing link is then sent to the device. The input is the generated clip, and the output is a notification containing the viewing link.

[1450] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1451] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1452] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1453] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1454] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1455] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1456] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1457] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1458] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1459] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1460] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1461] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1462] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1463] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1464] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1465] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1466] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1467] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1468] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1469] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1470] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1471] The following is further disclosed regarding the above embodiment.

[1472] (Claim 1)

[1473] A means for collecting video data and comment data;

[1474] A means of analyzing the streamer's emotions from the collected video data,

[1475] A means of analyzing viewer reactions from collected comment data,

[1476] A means of analyzing the content of video data and identifying highlights;

[1477] A means for generating a clipped video based on the identified points of interest;

[1478] A means for distributing the generated clipped video;

[1479] A system including:

[1480] (Claim 2)

[1481] 10. The system of claim 1, further comprising means for identifying emotions based on an analysis of the broadcaster's voice and facial expressions.

[1482] (Claim 3)

[1483] 10. The system of claim 1, further comprising means for classifying viewer responses based on text analysis of the comment data.

[1484] "Example 1"

[1485] (Claim 1)

[1486] A means for collecting video data and comment data;

[1487] A means of analyzing the streamer's emotions from the collected video data,

[1488] A means of analyzing viewer reactions from collected comment data,

[1489] A means for analyzing the content of video data using a natural language processing model to identify highlights;

[1490] A means for generating a clipped video based on the identified points of interest;

[1491] A means for distributing the generated clipped video;

[1492] A system including:

[1493] (Claim 2)

[1494] 10. The system of claim 1, further comprising means for identifying emotions based on an analysis of the broadcaster's voice and facial expressions.

[1495] (Claim 3)

[1496] 10. The system of claim 1, further comprising means for classifying viewer responses based on text analysis of the comment data.

[1497] (Claim 4)

[1498] 2. The system according to claim 1, further comprising means for extracting keywords and identifying time periods when responses from viewers are concentrated.

[1499] (Claim 5)

[1500] 2. The system of claim 1, further comprising means for transcribing the audio of the collected video data using an automatic speech recognition model.

[1501] (Claim 6)

[1502] 2. The system of claim 1, further comprising means for identifying turning points and important scenes from the transcribed text data using a natural language processing model.

[1503] (Claim 7)

[1504] The system according to claim 1, further comprising means for determining the start and end of a highlight scene based on the analysis result and generating a cut-out video using video editing software.

[1505] (Claim 8)

[1506] The system of claim 1, further comprising means for uploading the generated video to an online platform and notifying a viewing link.

[1507] "Application Example 1"

[1508] (Claim 1)

[1509] A means for collecting video data and comment data;

[1510] A means of analyzing the streamer's emotions from the collected video data,

[1511] A means of analyzing viewer reactions from collected comment data,

[1512] A means of analyzing the content of video data and identifying highlights;

[1513] A means for generating a clipped video based on the identified points of interest;

[1514] A means for distributing the generated clipped video;

[1515] A means of sending notifications to your smartphone,

[1516] A system including:

[1517] (Claim 2)

[1518] 10. The system of claim 1, further comprising means for identifying emotions based on an analysis of the broadcaster's voice and facial expressions.

[1519] (Claim 3)

[1520] 10. The system of claim 1, further comprising means for classifying viewer responses based on text analysis of the comment data.

[1521] "Example 2: Combining Emotion Engines"

[1522] (Claim 1)

[1523] A means for collecting video data and comment data;

[1524] A means of analyzing the streamer's emotions from the collected video data,

[1525] A means of analyzing viewer reactions from collected comment data,

[1526] A means of analyzing the content of video data and identifying highlights;

[1527] A means for identifying scenes of interest to users by integrating the results of emotion analysis of the distributor and the results of analysis of comments from viewers;

[1528] A means for generating a clipped video based on the identified points of interest;

[1529] A means for distributing the generated clipped video;

[1530] A system including:

[1531] (Claim 2)

[1532] 10. The system of claim 1, further comprising means for identifying emotions based on an analysis of the broadcaster's voice and facial expressions.

[1533] (Claim 3)

[1534] 10. The system of claim 1, further comprising means for classifying viewer responses based on text analysis of the comment data.

[1535] "Application example 2 when combining emotion engines"

[1536] (Claim 1)

[1537] A means for collecting video data and comment data;

[1538] A means of analyzing the streamer's emotions from the collected video data,

[1539] A means of analyzing viewer reactions from collected comment data,

[1540] A means of analyzing the content of video data and identifying highlights;

[1541] A means for generating a clipped video based on the identified points of interest;

[1542] A means for distributing the generated clipped video;

[1543] A means for delivering the clipped video to a smartphone or head-mounted display;

[1544] A system including:

[1545] (Claim 2)

[1546] 10. The system of claim 1, further comprising means for identifying emotions based on an analysis of the broadcaster's voice and facial expressions.

[1547] (Claim 3)

[1548] 10. The system of claim 1, further comprising means for classifying viewer responses based on text analysis of the comment data. [Explanation of symbols]

[1549] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for collecting video data and comment data; A means of analyzing the streamer's emotions from the collected video data, A means of analyzing viewer reactions from collected comment data, A means of analyzing the content of video data and identifying highlights, A means for generating a clipped video based on the identified points of interest; A means for distributing the generated clipped video; A system including:

2. The system of claim 1 further comprising means for identifying emotions based on an analysis of the broadcaster's voice and facial expressions.

3. The system of claim 1 , further comprising means for classifying viewer responses based on text analysis of the comment data.

Images