system

Abstract

We provide the system. [Solution] A means for analyzing a user's viewing history and preferences to generate a user profile, A means for analyzing input videos and analyzing audio, text, and video to characterize scenes within the video, A means for extracting video sections that are important to the user based on the user profile and video analysis results, A means to efficiently organize the extracted video sections and stream them to the user's device, A means for collecting user feedback and improving the deep learning model to improve the accuracy of the user profile and suggestions, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern times, there are various video contents, and users can access a huge amount of information. However, on the other hand, due to time constraints, it is particularly difficult for users to quickly access parts that they are interested in or important information. Therefore, it is required to improve the efficiency of video viewing time and provide a viewing experience tailored to the user's preferences. At present, the task of users finding interesting parts in videos by themselves is burdensome, and means to solve this problem are needed.

Means for Solving the Problems

[0005] This invention provides a system that generates user profiles by analyzing a user's viewing history and preferences. It then analyzes the input video, characterizing scenes within the video based on audio, text, and video to identify parts important to the user. Furthermore, it efficiently provides necessary information by streaming the extracted important parts to the user's terminal. The system updates the user profile by utilizing user actions and feedback during viewing, improving the viewing experience for subsequent viewings. This enables users to view content that maximizes their interest within a limited time.

[0006] "User viewing history" refers to recorded data of video content that a user has watched in the past.

[0007] "User preferences" refer to the categories and genres that users are interested in, based on their past viewing behavior and feedback.

[0008] A "user profile" is a collection of information optimized for each individual user, generated based on information such as the user's viewing history and preferences.

[0009] "Video analysis" is the process of analyzing the content of a video using audio, text, and video to characterize each scene.

[0010] "Characterization" refers to representing scenes within a video using specific keywords or tags to clarify their content and attributes.

[0011] "Important parts" are sections within a video that are particularly interesting or useful to the user.

[0012] "Streaming" is a technology that continuously sends video data to the user's device via the internet, allowing for real-time playback without waiting for downloads.

[0013] "Feedback" refers to the evaluations and opinions that users give regarding the content they watch, and is used to improve the system and update user profiles.

[0014] A "deep learning model" is a type of algorithm that uses neural networks to learn patterns from large amounts of data and perform analysis and prediction. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention aims to build a system that allows users to efficiently watch videos that best suit their interests amidst a diverse range of content. This system analyzes the user's viewing history and preferences, and optimizes viewing time by extracting and providing only the most important parts from videos.

[0037] Program Processing Outline

[0038] 1. The server has the function of initializing the user's profile and accumulating viewing history and preference data based on it. This records what genres and themes the user is interested in as a profile.

[0039] 2. The user selects the video they want to watch. Whether it's a newly uploaded video or one from an existing library, the video's metadata is sent to the server via the device.

[0040] 3. The server analyzes the received video and evaluates and characterizes each scene through speech recognition and video analysis. For example, it detects frequently used technical terms in the video and locations where laughter occurs.

[0041] 4. The server compares the video analysis results with the user profile and extracts portions deemed important to the user. It then generates video clips based on the extracted timestamps.

[0042] 5. The extracted clips are efficiently organized and then streamed from the server to the device. The device plays them seamlessly, allowing the user to continuously watch only the scenes that interest them.

[0043] 6. After viewing ends, the device sends user feedback to the server. This information is used to update the user profile and improve suggestions for future viewing.

[0044] Specific example

[0045] Suppose a user wants to watch a technical lecture video. In this case, the server identifies terms and themes that the user has shown particular interest in from their past viewing history. Next, it analyzes the lecture video to detect sections where these terms frequently appear. Based on this analysis, the system extracts important sections for the user and generates them as clips for continuous playback. In this way, the user can efficiently acquire the knowledge they seek. Thus, by implementing the present invention, the video viewing experience can be customized for each individual user, and time constraints can be greatly reduced.

[0046] The following describes the processing flow.

[0047] Step 1:

[0048] Users access the system and input their interests and preferences. This information is used for initial setup and is also treated as data added to their viewing history.

[0049] Step 2:

[0050] The server uses data on the user's viewing history and preferences to create a user profile. This profile includes the user's interests, such as genres and specific topics.

[0051] Step 3:

[0052] The user selects the video they want to watch. If the user uploads a new video, the device sends the video's metadata to the server.

[0053] Step 4:

[0054] The server analyzes the received video. Speech recognition technology is used to convert the audio into text, and video analysis technology is used to evaluate the content of each frame. This analysis characterizes and tags the scenes in the video.

[0055] Step 5:

[0056] The server identifies important parts for the user by comparing the analysis results with the user profile. It obtains timestamps of these important parts and generates video clips as needed.

[0057] Step 6:

[0058] The server generates clips of key sections and streams them to the device. The device then plays these clips sequentially, allowing the user to efficiently view the information they need.

[0059] Step 7:

[0060] The device records user actions during viewing (e.g., skipping or rewinding), and this data is sent to a server to be stored as training data.

[0061] Step 8:

[0062] After viewing ends, the device will present the user with a short survey and collect their feedback. This feedback will be used to update the user profile and improve the viewing experience for future viewings.

[0063] (Example 1)

[0064] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0065] In today's society, where diverse content abounds, it is difficult for users to efficiently view video information that matches their interests and needs. In particular, there is a need to effectively extract only the important parts within a limited time and optimize the viewing experience. However, achieving this requires technology that deeply understands, appropriately evaluates, and reflects the diverse interests of each user.

[0066] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0067] In this invention, the server includes means for analyzing user interest information and generating user characteristic information; means for analyzing input video information and processing audio information, text information, and video information to characterize scenes within the video information; and means for extracting parts that are important to the user based on the user characteristic information and the video information analysis results. This enables the user to efficiently view important video parts based on their interests.

[0068] "User interest information" refers to information about the interests and preferences that users have shown through their past actions and viewing history.

[0069] "User characteristic information" refers to user-specific profile information generated based on individual user preferences, viewing history, and evaluation history.

[0070] "Visual information" refers to input digital video and multimedia content, which includes video, audio, and text information.

[0071] "Audio information" refers to data related to the content and characteristics of audio used within video information.

[0072] "Textual information" refers to all text data contained within video information, including subtitles and captions.

[0073] "Means for processing video information" refers to technical means for analyzing and evaluating video information using technologies such as speech recognition and video analysis.

[0074] "Important parts" refer to a series of video scenes that are deemed particularly interesting to the user based on user characteristics and video information.

[0075] A "learning model" is a trained algorithm or AI model that uses large amounts of data to efficiently perform a specific task.

[0076] "Short continuous information" refers to short video clips generated by connecting important parts extracted from video information.

[0077] An "information processing system" refers to a series of infrastructure components that perform information input, processing, and output through a combination of software and hardware.

[0078] This invention is an information processing system that allows users to efficiently view video information based on their interests. By having the server, terminal, and user each fulfill their respective roles, it provides a smooth viewing experience.

[0079] The server analyzes user interest information and generates user characteristic information. To achieve this, it utilizes a database management system and machine learning algorithms to process viewing history and user feedback. By incorporating speech recognition and video analysis technologies such as Google® Cloud Speech-to-Text and OpenCV, it efficiently analyzes video information and extracts features, including audio and text information, from the video. This analysis result is combined with user characteristic information to automatically extract important parts. Furthermore, the generated AI model is used to score the importance of each element to the user and generate short, sequential pieces of information.

[0080] The device receives data from the server and streams it to the user. During this process, Adaptive Bitrate Streaming technology is used to adapt to network conditions, maintaining a high-quality video experience. User feedback after viewing is also sent to the server via the device, contributing to profile updates.

[0081] If a user wants to efficiently learn from technical lecture videos, the server analyzes and extracts appropriate segments based on specific terms and themes from their past viewing history. This allows users to obtain information relevant to their learning needs in a timely manner.

[0082] A concrete example of a prompt message is, "Detect scenes where technical terms frequently appear and generate clips based on them." In this way, the system provides a customized video viewing experience for each user, maximizing its comfort and usefulness.

[0083] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0084] Step 1:

[0085] The server retrieves user information and initializes profiles. First, when a user registers, it receives the basic information they enter (age, gender, areas of interest, etc.). Based on this input information, it stores user characteristic information in the database. This profile serves as the foundation for understanding each user's preferences.

[0086] Step 2:

[0087] The user selects the video they want to watch and sends that information to the server via their device. This input includes the video ID, title, category, and tags from the metadata. This information is stored on the server and prepares it for the next analysis step.

[0088] Step 3:

[0089] The server performs video analysis and speech recognition on the selected video information. It receives video data as input and extracts audio and text information. Specifically, it performs speech transcription using Google Cloud Speech-to-Text and scene analysis of the video using OpenCV. The output is information that lists the features of all scenes in the video.

[0090] Step 4:

[0091] The server compares the scene features obtained from the analysis with the user's characteristic information and extracts the most relevant parts. The input consists of the output information from step 3 and the user's characteristic information, and scoring is performed using an AI model. Here, timestamps of scenes important to the user are output, and a list of important segments is generated.

[0092] Step 5:

[0093] The server generates short, sequential clips based on the extracted key portions. The input is the timestamp information obtained in step 4, and a video editing tool is used to generate clips. The output is a customized video clip tailored to the user's preferences.

[0094] Step 6:

[0095] The terminal seamlessly streams clips received from the server to the viewer. The input is a video clip sent from the server, and playback is performed using Adaptive Bitrate Streaming technology, adjusting the video quality according to the network conditions. The output is smooth, uninterrupted continuous playback.

[0096] Step 7:

[0097] After watching a video, users send feedback to the server via their device. This feedback is entered as information about their satisfaction with the video and what they found interesting. The server incorporates this data into the user profile and uses it to improve the accuracy of future recommendations. This output represents updated user characteristics information.

[0098] (Application Example 1)

[0099] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0100] In today's world, the sheer volume and variety of video content makes it difficult for users to quickly and efficiently find and watch content that suits their interests. Furthermore, existing systems often fail to adequately optimize videos based on user preferences and interests, leading to wasted viewing time and content bias. This invention aims to address these issues by individually customizing the user's viewing experience and enabling effective content delivery.

[0101] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0102] In this invention, the server includes means for analyzing the user's viewing history and preferences and generating a user profile; means for analyzing input videos and analyzing audio, text, and video to characterize scenes within the videos; means for extracting video sections important to the user based on the user profile and video analysis results; means for efficiently organizing the extracted video sections and streaming them to the user terminal; and means for collecting user feedback and improving the deep learning model to improve the accuracy of the user profile and suggestions. This enables users to efficiently watch videos that match their interests in a short amount of time.

[0103] A "user profile" is a collection of data that represents a user's interests and tendencies, generated based on their viewing history and preferences.

[0104] "Video analysis" is a technique that uses audio, text, and video within a video to characterize a scene.

[0105] A "key video section" is a part of a video that is considered to be valuable to the user based on the user profile and analysis results.

[0106] "Feedback" refers to the information and reactions that users provide to the system, and the service is improved based on this feedback.

[0107] A "generative AI model" is an artificial intelligence model that uses data such as feedback to predict user interests and behavior, thereby enabling efficient content suggestions.

[0108] A "prompt message" is an instruction or guidance message generated using a generative AI model based on the user's requests and conditions.

[0109] The system implementing this invention first generates a user profile based on the user's viewing history and preferences. The viewing history is used to collect data on videos the user has watched in the past and to analyze trends. The software used at this stage is a database management system, typically such as MySQL® or PostgreSQL.

[0110] Next, the server performs video analysis. Specifically, it analyzes the received video data through speech recognition, video analysis, subtitle analysis, etc., and characterizes each scene. For video analysis, OpenCV could be used, and for speech recognition, the Google Speech-to-Text API could be used.

[0111] Subsequently, the server extracts important video sections based on the user profile and video analysis results. This is an effective process for finding information and scenes that are of interest to the user, such as specific technical terms or frequent occurrences of laughter.

[0112] The extracted video sections are streamed from the server to the user's device. In this step, the user can seamlessly and efficiently watch important scenes in sequence. The device is assumed to be a typical home PC or mobile device.

[0113] Furthermore, the system collects user feedback, which the server uses to update user profiles. This feedback data is essential for improving the system's accuracy and is used as training data to more accurately predict user preferences using generative AI models.

[0114] For example, if a user watches a lot of educational content, the system can extract important sections of technical terminology and provide videos that help them learn efficiently in a short amount of time. Similarly, for users who prefer entertainment such as comedy, the system can focus on playing humorous scenes.

[0115] The following is an example of a prompt message to input into the generative AI model.

[0116] "Please describe a method for extracting important video sections based on user interests. Design an algorithm to provide the most relevant content experience based on viewing history."

[0117] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0118] Step 1:

[0119] The server generates a user profile based on the user's viewing history. First, the server retrieves the user's past viewing data from the database. This data serves as input, and the server extracts the genres and keywords of the videos watched to create a user profile that shows the user's interests and tendencies. The user profile generated as output is used in subsequent processing steps.

[0120] Step 2:

[0121] The server analyzes newly input videos. It receives the input video data and processes it using speech recognition software and video analysis tools to analyze the audio, text, and video elements. This reveals the video's constituent elements and features, and scene feature data is generated as output.

[0122] Step 3:

[0123] The server matches user profiles with video analysis results to extract video sections that are important to the user. User profiles and scene feature data are used as input, and the data is processed to extract scenes that are likely to interest the user. The resulting output is a list of scene timestamps and clips.

[0124] Step 4:

[0125] The server organizes the extracted key video sections and prepares them for streaming to the user's device. The input consists of scene timestamps and a list of clips, which are used to generate data arranged in the optimal order for user viewing. The organized video clips are then sent as output to the user's device.

[0126] Step 5:

[0127] The device displays a streamed video clip, which the user watches. User input activates the video clip, and post-viewing feedback is provided to the server. The output is user feedback data.

[0128] Step 6:

[0129] The server analyzes user feedback and updates the user profile. The input feedback data is used to improve the training dataset for the generative AI model, enhancing the accuracy of future content recommendations. This results in an updated user profile and optimized viewing recommendations based on prompt messages.

[0130] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0131] This invention provides a system that further optimizes the video viewing experience for individual users by incorporating an emotion engine that recognizes the user's emotional state in real time. In addition to creating a profile based on the user's viewing history and preferences, this system provides content that matches the user's interests through the analysis of emotional data.

[0132] Program Processing Outline

[0133] The server creates a user profile based on the user's viewing history and preferences. The profile records the user's viewing patterns and emotional information in an integrated manner.

[0134] When a user watches a video, their device uses its camera and microphone to capture facial expressions and voice tone, and sends this data to a server. During this process, the emotion engine identifies the user's emotional state.

[0135] The server uses an emotion engine to analyze the user's emotional state in real time and measure their reactions while watching. The obtained emotional data is linked to relevant video scenes and timestamps.

[0136] The server extracts key parts of a video that are predicted to be interesting to the user, taking into account not only video analysis data but also emotional information. In this process, user emotional feedback is reflected in the creation of the video clips.

[0137] The generated clips of key segments are efficiently organized and streamed to the device. Users can watch customized content based on their emotional response to previously viewed material.

[0138] After viewing ends, the device sends feedback, including the results of the sentiment analysis, to the server. This updates the user profile, and the sentiment data is used as learning material for the system.

[0139] Specific example

[0140] The user attempts to watch a comedy scene that elicited joy or excitement. In this case, the emotion engine recognizes the user's smile and laughter from data captured by the camera and microphone. The server then analyzes this emotional feedback and incorporates similar scenes from other comedy videos into the next viewing suggestion. This matches the user with videos that suit their mood, improving the viewing experience. This format offers a new approach to video viewing that integrates the user's emotions and preferences.

[0141] The following describes the processing flow.

[0142] Step 1:

[0143] The user logs into the system and updates their personal viewing history and preferences. This keeps the user profile up to date.

[0144] Step 2:

[0145] The user selects a video they want to watch, and the video's metadata is sent to the server via their device. At this point, the user's viewing history is also taken into consideration.

[0146] Step 3:

[0147] The server analyzes the video. It evaluates each frame using audio, text, and video elements to characterize the scenes within the video.

[0148] Step 4:

[0149] The device uses its built-in camera and microphone to capture the user's facial expressions and voice in real time. This data is sent to a server, where an emotion engine analyzes the user's emotional state.

[0150] Step 5:

[0151] The server evaluates emotional data and video analysis results, extracting parts deemed important to the user. Emotional feedback is used in selecting scenes during this process.

[0152] Step 6:

[0153] The server generates clips of the extracted key portions and streams them to the device. These clips are customized based on the user's emotions.

[0154] Step 7:

[0155] New emotional responses that users exhibit while watching videos are continuously transmitted to the server in real time via their device, continuously optimizing the viewing experience.

[0156] Step 8:

[0157] After viewing ends, the device sends feedback to the server regarding the user's overall satisfaction and sentiment analysis results. Based on this information, the user profile is further updated.

[0158] (Example 2)

[0159] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0160] In recent years, with the rapid increase in video content, there has been a growing demand for providing an optimal viewing experience based on each user's individual interests and emotions. However, conventional systems have been insufficient in suggesting content that reflects users' emotions in real time. This invention aims to optimize video content while taking into account the user's emotional state.

[0161] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0162] In this invention, the server includes means for analyzing the user's viewing history and preferences and generating user attribute information; means for analyzing input visual information and analyzing audio, text, and video to characterize scenes; and means for monitoring the user's emotional state in real time and acquiring facial and audio data through the terminal. This makes it possible to suggest customized content to each user based on their emotional responses.

[0163] "User viewing history" refers to a record of content that a user has viewed in the past, and includes information such as genre and viewing time.

[0164] "User attribute information" refers to information that represents individual viewing patterns and preferences, generated based on a user's viewing history and preferences.

[0165] "Visual information" refers to data such as audio, text, and video that is necessary to characterize a scene within a video.

[0166] A "generative AI model" refers to an artificial intelligence model that generates new information or suggestions based on data.

[0167] "Emotional state" refers to the psychological reactions a user exhibits while watching, such as joy, surprise, and concentration.

[0168] "Facial expression and voice data" refers to digital data obtained from the user's facial expressions and voice tone.

[0169] "Emotional feedback" is a record of the emotional responses a user exhibits while watching content, and is used as information for system suggestions and learning.

[0170] "Customized content suggestions" refers to the provision of individually optimized viewing content suggested based on the user's past viewing history and real-time sentiment analysis.

[0171] This invention is a system for analyzing a user's emotional state in real time and providing an individually optimized video viewing experience. The system primarily involves the interaction of a server, terminal, and user's devices to realize a unique viewing experience.

[0172] First, the server generates user attribute information based on the user's viewing history and preferences. This clearly profiles individual viewing patterns and preferences. This profile serves as an indicator for determining what kind of content the user likes.

[0173] Next, the device uses its camera and microphone to capture facial and audio data while the user is watching a video. This involves using software that performs facial recognition and voice analysis. This data is crucial for determining the user's emotional state.

[0174] Furthermore, the server analyzes facial and audio data transmitted from the terminal and uses an emotion engine to identify the user's emotional state in real time. This technology is used to determine which scenes in the video had a positive impact on the user.

[0175] Based on the analyzed sentiment data, the server utilizes a generative AI model to determine the next content to watch. At this stage, the generative AI model works effectively to identify the most suitable video clips for the user.

[0176] As a concrete example, if a user smiles while watching a comedy scene, the system uses that facial expression data to recommend other comedy videos that are likely to evoke a similar emotional response. An example of a prompt used in this process would be a request such as, "Suggest a video to watch next based on my past viewing reactions." In this way, users can experience content customized to their emotions.

[0177] This invention allows users to enjoy a video viewing experience that feels as if it were made specifically for them, and the system can continuously learn from user feedback to make more accurate suggestions.

[0178] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0179] Step 1:

[0180] When a user selects video content, the device uses its built-in camera and microphone to capture the user's facial expressions and voice tone. This data is used as input to determine the user's emotional state by capturing video frames and sampling audio signals. As output, it generates real-time facial expression data and voice tone data.

[0181] Step 2:

[0182] The device sends facial expression data and voice tone data acquired from the terminal to the server. The server receives this input data and performs analysis using an emotion engine. Specifically, it detects subtle changes from facial expressions and extracts emotional features from the voice. As output, digital information representing the user's real-time emotional state is generated.

[0183] Step 3:

[0184] The server links real-time emotional data to specific scenes and timestamps in the video. This process combines the input emotional information with metadata from the content being viewed. As a result, information indicating when and in which scene an emotional response occurred is output.

[0185] Step 4:

[0186] The server integrates the user's emotional state, viewing history, and preference data and analyzes it using a generative AI model. User profiles, emotional response data, and content metadata are used as input. The output generates a set of recommended video clips.

[0187] Step 5:

[0188] Recommended content generated on the server is sent to the device and streamed. Specifically, based on the input recommendation clip information, video clips suitable for the user are collected and organized. This allows the user to watch customized content.

[0189] Step 6:

[0190] Once viewing ends, the device sends feedback to the server, including the results of the sentiment analysis during viewing. All viewing data and the final sentiment feedback are used as input. This information is used to update the user profile and improve the accuracy of future content recommendations, resulting in enhanced user attribute information being output.

[0191] (Application Example 2)

[0192] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0193] Traditional content delivery systems recommend content based solely on a user's viewing history and preferences, making it impossible to suggest optimal content that reflects the user's changing emotions in real time. Therefore, there was a need to efficiently deliver content that matches the user's emotions and create a more personalized viewing experience.

[0194] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0195] In this invention, the server includes means for analyzing the user's viewing history and preferences and generating a user profile; means for analyzing an input video and characterizing scenes with audio, text, and video; means for extracting important parts based on the user profile and video analysis results; means for recognizing the user's emotional state in real time and acquiring emotional data; means for generating and suggesting appropriate content based on the emotional data; means for streaming the generated content to the user's terminal; and means for collecting user feedback and updating the profile. This enables the suggestion of content that matches the user's real-time emotions and the personalization of the viewing experience as a result.

[0196] "Viewing history" refers to the history of content a user has viewed so far, and is data used to analyze the user's preferences.

[0197] A "user profile" is information generated based on a user's viewing history and preferences, and serves as the foundational data for providing content that is best suited to each individual user.

[0198] "Emotional data" refers to information that represents a user's real-time emotional state, and is data acquired using sensor devices such as cameras and microphones.

[0199] "Content analysis" is a method of characterizing scenes within an input video based on audio, text, and video, and is a process used to extract important parts.

[0200] "Key elements" refer to video scenes that are deemed interesting or valuable to the user, extracted based on user profiles and video analysis results.

[0201] "Feedback" refers to information collected from users' impressions and reactions after viewing content, which is used as learning material for the system and helps update user profiles.

[0202] "Streaming" is a technology that delivers generated content to user devices in real time, minimizing latency and providing a high-quality viewing experience.

[0203] An "emotion engine" refers to a core algorithm or software that analyzes a user's emotional state in real time and suggests appropriate content.

[0204] This invention is a system that combines multiple technological elements to optimize the user's viewing experience according to their emotions. The system generates a profile based on the user's viewing history and preferences, and extracts important scenes from videos based on this profile. Furthermore, by utilizing an emotion engine to acquire and analyze the user's emotional data in real time, it is possible to present content that is suitable for the user.

[0205] This system utilizes smart glasses and smart devices as terminals. These devices are equipped with cameras and microphones to capture the user's facial expressions and voice tone. This data is sent to a server, where a cloud-based emotion analysis engine analyzes the emotional data in real time. For example, services such as Amazon Rekognition are used to understand the user's emotional state from the acquired video and audio data.

[0206] The server integrates the analyzed sentiment data into user profiles and uses this to train a deep learning model. The resulting model uses a generative AI model to predict content that is likely to match the user's emotions and recommends it to the user. This allows the system to provide video clips that amplify emotions such as joy and surprise that the user feels while watching.

[0207] As a concrete example, consider a scenario where a user is watching a horror movie. In this case, the system can detect the user's surprised facial expression and suggest an appropriate next video scene to alleviate their fear. For example, based on a prompt such as, "Analyze the scene where the user is smiling and select a comedy video that matches that as the next viewing suggestion," the system can select an appropriate video.

[0208] This invention will personalize the viewing experience and provide users with the most relevant content tailored to their emotions.

[0209] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0210] Step 1:

[0211] The device acquires real-time data from the user while they are viewing content. Specifically, it uses the camera and microphone of the smart glasses to record the user's facial expressions and voice tone. The input data consists of video and audio, and the output is raw data that allows for the identification of emotions.

[0212] Step 2:

[0213] The terminal sends the acquired video and audio data to the server. Here, data transfer takes place, with the input being the raw data acquired by the terminal and the output being the server receiving this data.

[0214] Step 3:

[0215] The server uses a cloud-based sentiment analysis engine to identify the user's emotional state from the received raw data. The input is the video and audio data received in step 2, and the output is the user's identified emotional state. Data processing includes video analysis and audio analysis.

[0216] Step 4:

[0217] The server updates the user profile based on the user's emotional state. The input is real-time emotional information, and the output is an updated user profile. A deep learning model is used to perform data calculations combining emotional data with existing viewing history.

[0218] Step 5:

[0219] The server uses a generative AI model to generate recommended content based on the updated user profile. The input is the updated user profile, and the output is a list of recommended content for the user. A prompt might include, "Analyze the scenes where the user is smiling and select a comedy video that matches those scenes as the next viewing suggestion."

[0220] Step 6:

[0221] The server prepares to stream the generated recommended content to the terminal. The input is a list of recommended content, and the output is data converted into a streamable format. This step involves encoding the content and preparing it for network transfer.

[0222] Step 7:

[0223] The device receives recommended content streamed from the server and displays it in a format viewable by the user. The input is the streamed content, and the output is the video that the user views. The device handles the video rendering, realizing the user's viewing experience.

[0224] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0225] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0226] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0227] [Second Embodiment]

[0228] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0229] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0230] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0231] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0232] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0233] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0234] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0235] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0236] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0237] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0238] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0239] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0240] This invention aims to build a system that allows users to efficiently watch videos that best suit their interests amidst a diverse range of content. This system analyzes the user's viewing history and preferences, and optimizes viewing time by extracting and providing only the most important parts from videos.

[0241] Program Processing Outline

[0242] 1. The server has the function of initializing the user's profile and accumulating viewing history and preference data based on it. This records what genres and themes the user is interested in as a profile.

[0243] 2. The user selects the video they want to watch. Whether it's a newly uploaded video or one from an existing library, the video's metadata is sent to the server via the device.

[0244] 3. The server analyzes the received video and evaluates and characterizes each scene through speech recognition and video analysis. For example, it detects frequently used technical terms in the video and locations where laughter occurs.

[0245] 4. The server compares the video analysis results with the user profile and extracts portions deemed important to the user. It then generates video clips based on the extracted timestamps.

[0246] 5. The extracted clips are efficiently organized and then streamed from the server to the device. The device plays them seamlessly, allowing the user to continuously watch only the scenes that interest them.

[0247] 6. After viewing ends, the device sends user feedback to the server. This information is used to update the user profile and improve suggestions for future viewing.

[0248] Specific example

[0249] Suppose a user wants to watch a technical lecture video. In this case, the server identifies terms and themes that the user has shown particular interest in from their past viewing history. Next, it analyzes the lecture video to detect sections where these terms frequently appear. Based on this analysis, the system extracts important sections for the user and generates them as clips for continuous playback. In this way, the user can efficiently acquire the knowledge they seek. Thus, by implementing the present invention, the video viewing experience can be customized for each individual user, and time constraints can be greatly reduced.

[0250] The following describes the processing flow.

[0251] Step 1:

[0252] Users access the system and input their interests and preferences. This information is used for initial setup and is also treated as data added to their viewing history.

[0253] Step 2:

[0254] The server uses data on the user's viewing history and preferences to create a user profile. This profile includes the user's interests, such as genres and specific topics.

[0255] Step 3:

[0256] The user selects the video they want to watch. If the user uploads a new video, the device sends the video's metadata to the server.

[0257] Step 4:

[0258] The server analyzes the received video. Speech recognition technology is used to convert the audio into text, and video analysis technology is used to evaluate the content of each frame. This analysis characterizes and tags the scenes in the video.

[0259] Step 5:

[0260] The server identifies important parts for the user by comparing the analysis results with the user profile. It obtains timestamps of these important parts and generates video clips as needed.

[0261] Step 6:

[0262] The server generates clips of key sections and streams them to the device. The device then plays these clips sequentially, allowing the user to efficiently view the information they need.

[0263] Step 7:

[0264] The device records user actions during viewing (e.g., skipping or rewinding), and this data is sent to a server to be stored as training data.

[0265] Step 8:

[0266] After viewing ends, the device will present the user with a short survey and collect their feedback. This feedback will be used to update the user profile and improve the viewing experience for future viewings.

[0267] (Example 1)

[0268] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0269] In today's society, where diverse content abounds, it is difficult for users to efficiently view video information that matches their interests and needs. In particular, there is a need to effectively extract only the important parts within a limited time and optimize the viewing experience. However, achieving this requires technology that deeply understands, appropriately evaluates, and reflects the diverse interests of each user.

[0270] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0271] In this invention, the server includes means for analyzing user interest information and generating user characteristic information; means for analyzing input video information and processing audio information, text information, and video information to characterize scenes within the video information; and means for extracting parts that are important to the user based on the user characteristic information and the video information analysis results. This enables the user to efficiently view important video parts based on their interests.

[0272] "User interest information" refers to information about the interests and preferences that users have shown through their past actions and viewing history.

[0273] "User characteristic information" refers to user-specific profile information generated based on individual user preferences, viewing history, and evaluation history.

[0274] "Visual information" refers to input digital video and multimedia content, which includes video, audio, and text information.

[0275] "Audio information" refers to data related to the content and characteristics of audio used within video information.

[0276] "Textual information" refers to all text data contained within video information, including subtitles and captions.

[0277] "Means for processing video information" refers to technical means for analyzing and evaluating video information using technologies such as speech recognition and video analysis.

[0278] "Important parts" refer to a series of video scenes that are deemed particularly interesting to the user based on user characteristics and video information.

[0279] A "learning model" refers to an algorithm or AI model trained using a large amount of data to efficiently execute a specific task.

[0280] "Short continuous information" refers to a short video clip generated by connecting important parts extracted from video information.

[0281] An "information processing system" refers to a series of infrastructure that performs information input, processing, and output through a combination of software and hardware.

[0282] This invention is an information processing system that enables a user to efficiently view video information based on their interests. By each of the server, terminal, and user playing their respective roles, a smooth viewing experience is provided.

[0283] The server analyzes the user's interest information and generates the user's characteristic information. To this end, it makes full use of a database management system and machine learning algorithms to process viewing history and user feedback. By introducing speech recognition and video analysis technologies such as Google Cloud Speech-to-Text and OpenCV, it efficiently analyzes video information and extracts features including audio information and character information in the video. Combining this analysis result with the user's characteristic information, it automatically extracts important parts. Furthermore, using the generated AI model, it scores the importance for the user and generates short continuous information.

[0284] The terminal receives the data provided by the server and performs streaming playback for the user. At this time, by adapting to the network situation using Adaptive Bitrate Streaming technology, it maintains a high-quality video experience. The feedback after viewing from the user is also sent to the server via the terminal, contributing to the update of the profile.

[0285] When users want to efficiently learn lecture videos related to technology, the server analyzes and extracts appropriate segments based on specific terms or themes from past viewing histories. As a result, users can timely obtain information that suits what they want to learn.

[0286] As a specific example of the prompt text, "Detect scenes where technical terms frequently appear and generate clips based on them" can be cited. In this way, this system provides a customized video viewing experience for each individual user, maximizing its comfort and usefulness.

[0287] The flow of the specific process in Example 1 will be described using FIG. 11.

[0288] Step 1:

[0289] The server acquires user information and initializes the profile. First, when a user newly registers, the server receives the input basic information (age, gender, field of interest, etc.). Based on this input information, the server accumulates the user's characteristic information in the database. This profile serves as a basis for understanding each user's preferences.

[0290] Step 2:

[0291] The user selects the video to be viewed and transmits the information to the server via the terminal. The input here includes the video ID, title, category, tags, etc. included in the metadata. This information is saved on the server and serves as a preparation stage for the next analysis step.

[0292] Step 3:

[0293] The server performs video analysis and speech recognition on the selected video information. It receives video data as input and extracts audio and text information. Specifically, it performs speech transcription using Google Cloud Speech-to-Text and scene analysis of the video using OpenCV. The output is information that lists the features of all scenes in the video.

[0294] Step 4:

[0295] The server compares the scene features obtained from the analysis with the user's characteristic information and extracts the most relevant parts. The input consists of the output information from step 3 and the user's characteristic information, and scoring is performed using an AI model. Here, timestamps of scenes important to the user are output, and a list of important segments is generated.

[0296] Step 5:

[0297] The server generates short, sequential clips based on the extracted key portions. The input is the timestamp information obtained in step 4, and a video editing tool is used to generate clips. The output is a customized video clip tailored to the user's preferences.

[0298] Step 6:

[0299] The terminal seamlessly streams clips received from the server to the viewer. The input is a video clip sent from the server, and playback is performed using Adaptive Bitrate Streaming technology, adjusting the video quality according to the network conditions. The output is smooth, uninterrupted continuous playback.

[0300] Step 7:

[0301] After watching a video, users send feedback to the server via their device. This feedback is entered as information about their satisfaction with the video and what they found interesting. The server incorporates this data into the user profile and uses it to improve the accuracy of future recommendations. This output represents updated user characteristics information.

[0302] (Application Example 1)

[0303] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0304] In today's world, the sheer volume and variety of video content makes it difficult for users to quickly and efficiently find and watch content that suits their interests. Furthermore, existing systems often fail to adequately optimize videos based on user preferences and interests, leading to wasted viewing time and content bias. This invention aims to address these issues by individually customizing the user's viewing experience and enabling effective content delivery.

[0305] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0306] In this invention, the server includes means for analyzing the user's viewing history and preferences and generating a user profile; means for analyzing input videos and analyzing audio, text, and video to characterize scenes within the videos; means for extracting video sections important to the user based on the user profile and video analysis results; means for efficiently organizing the extracted video sections and streaming them to the user terminal; and means for collecting user feedback and improving the deep learning model to improve the accuracy of the user profile and suggestions. This enables users to efficiently watch videos that match their interests in a short amount of time.

[0307] A "user profile" is a collection of data representing a user's interests and tendencies, generated based on the user's viewing history and preferences.

[0308] "Video analysis" is a technology that characterizes scenes using the audio, text, and video within a video.

[0309] An "important video section" is a part within a video that is considered valuable to the user based on the user profile and analysis results.

[0310] "Feedback" refers to the information and reactions provided by the user to the system, based on which the service is improved.

[0311] A "generative AI model" is an artificial intelligence model that uses data such as feedback to predict the user's interests and behaviors and realizes efficient content recommendations.

[0312] A "prompt text" is an instruction text or guidance text generated based on the user's desires and conditions by utilizing a generative AI model.

[0313] The system for implementing this invention first has the server generate a user profile based on the user's viewing history and preferences. The viewing history collects data on the videos the user has watched in the past and is for analyzing the trends. The software used at this stage is a database management system, generally MySQL, PostgreSQL, etc.

[0314] Next, the server performs video analysis. Specifically, it analyzes the received video data through speech recognition, video analysis, subtitle analysis, etc., to characterize each scene. It is conceivable to use OpenCV for video analysis and Google Speech-to-Text API for speech recognition.

[0315] Subsequently, the server extracts important video sections based on the user profile and video analysis results. This is an effective process for finding information and scenes that are of interest to the user, such as specific technical terms or frequent occurrences of laughter.

[0316] The extracted video sections are streamed from the server to the user's device. In this step, the user can seamlessly and efficiently watch important scenes in sequence. The device is assumed to be a typical home PC or mobile device.

[0317] Furthermore, the system collects user feedback, which the server uses to update user profiles. This feedback data is essential for improving the system's accuracy and is used as training data to more accurately predict user preferences using generative AI models.

[0318] For example, if a user watches a lot of educational content, the system can extract important sections of technical terminology and provide videos that help them learn efficiently in a short amount of time. Similarly, for users who prefer entertainment such as comedy, the system can focus on playing humorous scenes.

[0319] The following is an example of a prompt message to input into the generative AI model.

[0320] "Please describe a method for extracting important video sections based on user interests. Design an algorithm to provide the most relevant content experience based on viewing history."

[0321] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0322] Step 1:

[0323] The server generates a user profile based on the user's viewing history. First, the server retrieves the user's past viewing data from the database. This data serves as input, and the server extracts the genres and keywords of the videos watched to create a user profile that shows the user's interests and tendencies. The user profile generated as output is used in subsequent processing steps.

[0324] Step 2:

[0325] The server analyzes newly input videos. It receives the input video data and processes it using speech recognition software and video analysis tools to analyze the audio, text, and video elements. This reveals the video's constituent elements and features, and scene feature data is generated as output.

[0326] Step 3:

[0327] The server matches user profiles with video analysis results to extract video sections that are important to the user. User profiles and scene feature data are used as input, and the data is processed to extract scenes that are likely to interest the user. The resulting output is a list of scene timestamps and clips.

[0328] Step 4:

[0329] The server organizes the extracted key video sections and prepares them for streaming to the user's device. The input consists of scene timestamps and a list of clips, which are used to generate data arranged in the optimal order for user viewing. The organized video clips are then sent as output to the user's device.

[0330] Step 5:

[0331] The device displays a streamed video clip, which the user watches. User input activates the video clip, and post-viewing feedback is provided to the server. The output is user feedback data.

[0332] Step 6:

[0333] The server analyzes user feedback and updates the user profile. The input feedback data is used to improve the training dataset for the generative AI model, enhancing the accuracy of future content recommendations. This results in an updated user profile and optimized viewing recommendations based on prompt messages.

[0334] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0335] This invention provides a system that further optimizes the video viewing experience for individual users by incorporating an emotion engine that recognizes the user's emotional state in real time. In addition to creating a profile based on the user's viewing history and preferences, this system provides content that matches the user's interests through the analysis of emotional data.

[0336] Program Processing Outline

[0337] The server creates a user profile based on the user's viewing history and preferences. The profile records the user's viewing patterns and emotional information in an integrated manner.

[0338] When a user watches a video, their device uses its camera and microphone to capture facial expressions and voice tone, and sends this data to a server. During this process, the emotion engine identifies the user's emotional state.

[0339] The server uses an emotion engine to analyze the user's emotional state in real time and measure their reactions while watching. The obtained emotional data is linked to relevant video scenes and timestamps.

[0340] The server extracts key parts of a video that are predicted to be interesting to the user, taking into account not only video analysis data but also emotional information. In this process, user emotional feedback is reflected in the creation of the video clips.

[0341] The generated clips of key segments are efficiently organized and streamed to the device. Users can watch customized content based on their emotional response to previously viewed material.

[0342] After viewing ends, the device sends feedback, including the results of the sentiment analysis, to the server. This updates the user profile, and the sentiment data is used as learning material for the system.

[0343] Specific example

[0344] The user attempts to watch a comedy scene that elicited joy or excitement. In this case, the emotion engine recognizes the user's smile and laughter from data captured by the camera and microphone. The server then analyzes this emotional feedback and incorporates similar scenes from other comedy videos into the next viewing suggestion. This matches the user with videos that suit their mood, improving the viewing experience. This format offers a new approach to video viewing that integrates the user's emotions and preferences.

[0345] The following describes the processing flow.

[0346] Step 1:

[0347] The user logs into the system and updates their personal viewing history and preferences. This keeps the user profile up to date.

[0348] Step 2:

[0349] The user selects a video they want to watch, and the video's metadata is sent to the server via their device. At this point, the user's viewing history is also taken into consideration.

[0350] Step 3:

[0351] The server analyzes the video. It evaluates each frame using audio, text, and video elements to characterize the scenes within the video.

[0352] Step 4:

[0353] The device uses its built-in camera and microphone to capture the user's facial expressions and voice in real time. This data is sent to a server, where an emotion engine analyzes the user's emotional state.

[0354] Step 5:

[0355] The server evaluates emotional data and video analysis results, extracting parts deemed important to the user. Emotional feedback is used in selecting scenes during this process.

[0356] Step 6:

[0357] The server generates clips of the extracted key portions and streams them to the device. These clips are customized based on the user's emotions.

[0358] Step 7:

[0359] New emotional responses that users exhibit while watching videos are continuously transmitted to the server in real time via their device, continuously optimizing the viewing experience.

[0360] Step 8:

[0361] After viewing ends, the device sends feedback to the server regarding the user's overall satisfaction and sentiment analysis results. Based on this information, the user profile is further updated.

[0362] (Example 2)

[0363] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0364] In recent years, with the rapid increase in video content, there has been a growing demand for providing an optimal viewing experience based on each user's individual interests and emotions. However, conventional systems have been insufficient in suggesting content that reflects users' emotions in real time. This invention aims to optimize video content while taking into account the user's emotional state.

[0365] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0366] In this invention, the server includes means for analyzing the user's viewing history and preferences and generating user attribute information; means for analyzing input visual information and analyzing audio, text, and video to characterize scenes; and means for monitoring the user's emotional state in real time and acquiring facial and audio data through the terminal. This makes it possible to suggest customized content to each user based on their emotional responses.

[0367] "User viewing history" refers to a record of content that a user has viewed in the past, and includes information such as genre and viewing time.

[0368] "User attribute information" refers to information that represents individual viewing patterns and preferences, generated based on a user's viewing history and preferences.

[0369] "Visual information" refers to data such as audio, text, and video that is necessary to characterize a scene within a video.

[0370] A "generative AI model" refers to an artificial intelligence model that generates new information or suggestions based on data.

[0371] "Emotional state" refers to the psychological reactions a user exhibits while watching, such as joy, surprise, and concentration.

[0372] "Facial expression and voice data" refers to digital data obtained from the user's facial expressions and voice tone.

[0373] "Emotional feedback" is a record of the emotional responses a user exhibits while watching content, and is used as information for system suggestions and learning.

[0374] "Customized content suggestions" refers to the provision of individually optimized viewing content suggested based on the user's past viewing history and real-time sentiment analysis.

[0375] This invention is a system for analyzing a user's emotional state in real time and providing an individually optimized video viewing experience. The system primarily involves the interaction of a server, terminal, and user's devices to realize a unique viewing experience.

[0376] First, the server generates user attribute information based on the user's viewing history and preferences. This clearly profiles individual viewing patterns and preferences. This profile serves as an indicator for determining what kind of content the user likes.

[0377] Next, the device uses its camera and microphone to capture facial and audio data while the user is watching a video. This involves using software that performs facial recognition and voice analysis. This data is crucial for determining the user's emotional state.

[0378] Furthermore, the server analyzes facial and audio data transmitted from the terminal and uses an emotion engine to identify the user's emotional state in real time. This technology is used to determine which scenes in the video had a positive impact on the user.

[0379] Based on the analyzed sentiment data, the server utilizes a generative AI model to determine the next content to watch. At this stage, the generative AI model works effectively to identify the most suitable video clips for the user.

[0380] As a concrete example, if a user smiles while watching a comedy scene, the system uses that facial expression data to recommend other comedy videos that are likely to evoke a similar emotional response. An example of a prompt used in this process would be a request such as, "Suggest a video to watch next based on my past viewing reactions." In this way, users can experience content customized to their emotions.

[0381] This invention allows users to enjoy a video viewing experience that feels as if it were made specifically for them, and the system can continuously learn from user feedback to make more accurate suggestions.

[0382] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0383] Step 1:

[0384] When a user selects video content, the device uses its built-in camera and microphone to capture the user's facial expressions and voice tone. This data is used as input to determine the user's emotional state by capturing video frames and sampling audio signals. As output, it generates real-time facial expression data and voice tone data.

[0385] Step 2:

[0386] The device sends facial expression data and voice tone data acquired from the terminal to the server. The server receives this input data and performs analysis using an emotion engine. Specifically, it detects subtle changes from facial expressions and extracts emotional features from the voice. As output, digital information representing the user's real-time emotional state is generated.

[0387] Step 3:

[0388] The server links real-time emotional data to specific scenes and timestamps in the video. This process combines the input emotional information with metadata from the content being viewed. As a result, information indicating when and in which scene an emotional response occurred is output.

[0389] Step 4:

[0390] The server integrates the user's emotional state, viewing history, and preference data and analyzes it using a generative AI model. User profiles, emotional response data, and content metadata are used as input. The output generates a set of recommended video clips.

[0391] Step 5:

[0392] Recommended content generated on the server is sent to the device and streamed. Specifically, based on the input recommendation clip information, video clips suitable for the user are collected and organized. This allows the user to watch customized content.

[0393] Step 6:

[0394] Once viewing ends, the device sends feedback to the server, including the results of the sentiment analysis during viewing. All viewing data and the final sentiment feedback are used as input. This information is used to update the user profile and improve the accuracy of future content recommendations, resulting in enhanced user attribute information being output.

[0395] (Application Example 2)

[0396] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0397] Traditional content delivery systems recommend content based solely on a user's viewing history and preferences, making it impossible to suggest optimal content that reflects the user's changing emotions in real time. Therefore, there was a need to efficiently deliver content that matches the user's emotions and create a more personalized viewing experience.

[0398] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0399] In this invention, the server includes means for analyzing the user's viewing history and preferences and generating a user profile; means for analyzing an input video and characterizing scenes with audio, text, and video; means for extracting important parts based on the user profile and video analysis results; means for recognizing the user's emotional state in real time and acquiring emotional data; means for generating and suggesting appropriate content based on the emotional data; means for streaming the generated content to the user's terminal; and means for collecting user feedback and updating the profile. This enables the suggestion of content that matches the user's real-time emotions and the personalization of the viewing experience as a result.

[0400] "Viewing history" refers to the history of content a user has viewed so far, and is data used to analyze the user's preferences.

[0401] A "user profile" is information generated based on a user's viewing history and preferences, and serves as the foundational data for providing content that is best suited to each individual user.

[0402] "Emotional data" refers to information that represents a user's real-time emotional state, and is data acquired using sensor devices such as cameras and microphones.

[0403] "Content analysis" is a method of characterizing scenes within an input video based on audio, text, and video, and is a process used to extract important parts.

[0404] "Key elements" refer to video scenes that are deemed interesting or valuable to the user, extracted based on user profiles and video analysis results.

[0405] "Feedback" refers to information collected from users' impressions and reactions after viewing content, which is used as learning material for the system and helps update user profiles.

[0406] "Streaming" is a technology that delivers generated content to user devices in real time, minimizing latency and providing a high-quality viewing experience.

[0407] An "emotion engine" refers to a core algorithm or software that analyzes a user's emotional state in real time and suggests appropriate content.

[0408] This invention is a system that combines multiple technological elements to optimize the user's viewing experience according to their emotions. The system generates a profile based on the user's viewing history and preferences, and extracts important scenes from videos based on this profile. Furthermore, by utilizing an emotion engine to acquire and analyze the user's emotional data in real time, it is possible to present content that is suitable for the user.

[0409] This system utilizes smart glasses and smart devices as terminals. These devices are equipped with cameras and microphones to capture the user's facial expressions and voice tone. This data is sent to a server, where a cloud-based emotion analysis engine analyzes the emotional data in real time. For example, services such as Amazon Rekognition are used to understand the user's emotional state from the acquired video and audio data.

[0410] The server integrates the analyzed sentiment data into user profiles and uses this to train a deep learning model. The resulting model uses a generative AI model to predict content that is likely to match the user's emotions and recommends it to the user. This allows the system to provide video clips that amplify emotions such as joy and surprise that the user feels while watching.

[0411] As a concrete example, consider a scenario where a user is watching a horror movie. In this case, the system can detect the user's surprised facial expression and suggest an appropriate next video scene to alleviate their fear. For example, based on a prompt such as, "Analyze the scene where the user is smiling and select a comedy video that matches that as the next viewing suggestion," the system can select an appropriate video.

[0412] This invention will personalize the viewing experience and provide users with the most relevant content tailored to their emotions.

[0413] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0414] Step 1:

[0415] The device acquires real-time data from the user while they are viewing content. Specifically, it uses the camera and microphone of the smart glasses to record the user's facial expressions and voice tone. The input data consists of video and audio, and the output is raw data that allows for the identification of emotions.

[0416] Step 2:

[0417] The terminal sends the acquired video and audio data to the server. Here, data transfer takes place, with the input being the raw data acquired by the terminal and the output being the server receiving this data.

[0418] Step 3:

[0419] The server uses a cloud-based sentiment analysis engine to identify the user's emotional state from the received raw data. The input is the video and audio data received in step 2, and the output is the user's identified emotional state. Data processing includes video analysis and audio analysis.

[0420] Step 4:

[0421] The server updates the user profile based on the user's emotional state. The input is real-time emotional information, and the output is an updated user profile. A deep learning model is used to perform data calculations combining emotional data with existing viewing history.

[0422] Step 5:

[0423] The server uses a generative AI model to generate recommended content based on the updated user profile. The input is the updated user profile, and the output is a list of recommended content for the user. A prompt might include, "Analyze the scenes where the user is smiling and select a comedy video that matches those scenes as the next viewing suggestion."

[0424] Step 6:

[0425] The server prepares to stream the generated recommended content to the terminal. The input is a list of recommended content, and the output is data converted into a streamable format. This step involves encoding the content and preparing it for network transfer.

[0426] Step 7:

[0427] The device receives recommended content streamed from the server and displays it in a format viewable by the user. The input is the streamed content, and the output is the video that the user views. The device handles the video rendering, realizing the user's viewing experience.

[0428] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0429] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0430] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0431] [Third Embodiment]

[0432] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0433] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0434] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0435] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0436] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0437] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0438] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0439] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0440] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0441] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0442] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0443] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0444] This invention aims to build a system that allows users to efficiently watch videos that best suit their interests amidst a diverse range of content. This system analyzes the user's viewing history and preferences, and optimizes viewing time by extracting and providing only the most important parts from videos.

[0445] Program Processing Outline

[0446] 1. The server has the function of initializing the user's profile and accumulating viewing history and preference data based on it. This records what genres and themes the user is interested in as a profile.

[0447] 2. The user selects the video they want to watch. Whether it's a newly uploaded video or one from an existing library, the video's metadata is sent to the server via the device.

[0448] 3. The server analyzes the received video and evaluates and characterizes each scene through speech recognition and video analysis. For example, it detects frequently used technical terms in the video and locations where laughter occurs.

[0449] 4. The server compares the video analysis results with the user profile and extracts portions deemed important to the user. It then generates video clips based on the extracted timestamps.

[0450] 5. The extracted clips are efficiently organized and then streamed from the server to the device. The device plays them seamlessly, allowing the user to continuously watch only the scenes that interest them.

[0451] 6. After viewing ends, the device sends user feedback to the server. This information is used to update the user profile and improve suggestions for future viewing.

[0452] Specific example

[0453] Suppose a user wants to watch a technical lecture video. In this case, the server identifies terms and themes that the user has shown particular interest in from their past viewing history. Next, it analyzes the lecture video to detect sections where these terms frequently appear. Based on this analysis, the system extracts important sections for the user and generates them as clips for continuous playback. In this way, the user can efficiently acquire the knowledge they seek. Thus, by implementing the present invention, the video viewing experience can be customized for each individual user, and time constraints can be greatly reduced.

[0454] The following describes the processing flow.

[0455] Step 1:

[0456] Users access the system and input their interests and preferences. This information is used for initial setup and is also treated as data added to their viewing history.

[0457] Step 2:

[0458] The server uses data on the user's viewing history and preferences to create a user profile. This profile includes the user's interests, such as genres and specific topics.

[0459] Step 3:

[0460] The user selects the video they want to watch. If the user uploads a new video, the device sends the video's metadata to the server.

[0461] Step 4:

[0462] The server analyzes the received video. Speech recognition technology is used to convert the audio into text, and video analysis technology is used to evaluate the content of each frame. This analysis characterizes and tags the scenes in the video.

[0463] Step 5:

[0464] The server identifies important parts for the user by comparing the analysis results with the user profile. It obtains timestamps of these important parts and generates video clips as needed.

[0465] Step 6:

[0466] The server generates clips of key sections and streams them to the device. The device then plays these clips sequentially, allowing the user to efficiently view the information they need.

[0467] Step 7:

[0468] The device records user actions during viewing (e.g., skipping or rewinding), and this data is sent to a server to be stored as training data.

[0469] Step 8:

[0470] After viewing ends, the device will present the user with a short survey and collect their feedback. This feedback will be used to update the user profile and improve the viewing experience for future viewings.

[0471] (Example 1)

[0472] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0473] In today's society, where diverse content abounds, it is difficult for users to efficiently view video information that matches their interests and needs. In particular, there is a need to effectively extract only the important parts within a limited time and optimize the viewing experience. However, achieving this requires technology that deeply understands, appropriately evaluates, and reflects the diverse interests of each user.

[0474] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0475] In this invention, the server includes means for analyzing user interest information and generating user characteristic information; means for analyzing input video information and processing audio information, text information, and video information to characterize scenes within the video information; and means for extracting parts that are important to the user based on the user characteristic information and the video information analysis results. This enables the user to efficiently view important video parts based on their interests.

[0476] "User interest information" refers to information about the interests and preferences that users have shown through their past actions and viewing history.

[0477] "User characteristic information" refers to user-specific profile information generated based on individual user preferences, viewing history, and evaluation history.

[0478] "Visual information" refers to input digital video and multimedia content, which includes video, audio, and text information.

[0479] "Audio information" refers to data related to the content and characteristics of audio used within video information.

[0480] "Textual information" refers to all text data contained within video information, including subtitles and captions.

[0481] "Means for processing video information" refers to technical means for analyzing and evaluating video information using technologies such as speech recognition and video analysis.

[0482] "Important parts" refer to a series of video scenes that are deemed particularly interesting to the user based on user characteristics and video information.

[0483] A "learning model" is a trained algorithm or AI model that uses large amounts of data to efficiently perform a specific task.

[0484] "Short continuous information" refers to short video clips generated by connecting important parts extracted from video information.

[0485] An "information processing system" refers to a series of infrastructure components that perform information input, processing, and output through a combination of software and hardware.

[0486] This invention is an information processing system that allows users to efficiently view video information based on their interests. By having the server, terminal, and user each fulfill their respective roles, it provides a smooth viewing experience.

[0487] The server analyzes user interest information and generates user characteristic information. To achieve this, it utilizes a database management system and machine learning algorithms to process viewing history and user feedback. By incorporating speech recognition and video analysis technologies such as Google Cloud Speech-to-Text and OpenCV, it efficiently analyzes video information and extracts features, including audio and text information, from the video. This analysis is combined with user characteristic information to automatically extract important parts. Furthermore, the generated AI model is used to score the importance of each element to the user and generate short, sequential pieces of information.

[0488] The device receives data from the server and streams it to the user. During this process, Adaptive Bitrate Streaming technology is used to adapt to network conditions, maintaining a high-quality video experience. User feedback after viewing is also sent to the server via the device, contributing to profile updates.

[0489] If a user wants to efficiently learn from technical lecture videos, the server analyzes and extracts appropriate segments based on specific terms and themes from their past viewing history. This allows users to obtain information relevant to their learning needs in a timely manner.

[0490] A concrete example of a prompt message is, "Detect scenes where technical terms frequently appear and generate clips based on them." In this way, the system provides a customized video viewing experience for each user, maximizing its comfort and usefulness.

[0491] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0492] Step 1:

[0493] The server retrieves user information and initializes profiles. First, when a user registers, it receives the basic information they enter (age, gender, areas of interest, etc.). Based on this input information, it stores user characteristic information in the database. This profile serves as the foundation for understanding each user's preferences.

[0494] Step 2:

[0495] The user selects the video they want to watch and sends that information to the server via their device. This input includes the video ID, title, category, and tags from the metadata. This information is stored on the server and prepares it for the next analysis step.

[0496] Step 3:

[0497] The server performs video analysis and speech recognition on the selected video information. It receives video data as input and extracts audio and text information. Specifically, it performs speech transcription using Google Cloud Speech-to-Text and scene analysis of the video using OpenCV. The output is information that lists the features of all scenes in the video.

[0498] Step 4:

[0499] The server compares the scene features obtained from the analysis with the user's characteristic information and extracts the most relevant parts. The input consists of the output information from step 3 and the user's characteristic information, and scoring is performed using an AI model. Here, timestamps of scenes important to the user are output, and a list of important segments is generated.

[0500] Step 5:

[0501] The server generates short, sequential clips based on the extracted key portions. The input is the timestamp information obtained in step 4, and a video editing tool is used to generate clips. The output is a customized video clip tailored to the user's preferences.

[0502] Step 6:

[0503] The terminal seamlessly streams clips received from the server to the viewer. The input is a video clip sent from the server, and playback is performed using Adaptive Bitrate Streaming technology, adjusting the video quality according to the network conditions. The output is smooth, uninterrupted continuous playback.

[0504] Step 7:

[0505] After watching a video, users send feedback to the server via their device. This feedback is entered as information about their satisfaction with the video and what they found interesting. The server incorporates this data into the user profile and uses it to improve the accuracy of future recommendations. This output represents updated user characteristics information.

[0506] (Application Example 1)

[0507] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0508] In today's world, the sheer volume and variety of video content makes it difficult for users to quickly and efficiently find and watch content that suits their interests. Furthermore, existing systems often fail to adequately optimize videos based on user preferences and interests, leading to wasted viewing time and content bias. This invention aims to address these issues by individually customizing the user's viewing experience and enabling effective content delivery.

[0509] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0510] In this invention, the server includes means for analyzing the user's viewing history and preferences and generating a user profile; means for analyzing input videos and analyzing audio, text, and video to characterize scenes within the videos; means for extracting video sections important to the user based on the user profile and video analysis results; means for efficiently organizing the extracted video sections and streaming them to the user terminal; and means for collecting user feedback and improving the deep learning model to improve the accuracy of the user profile and suggestions. This enables users to efficiently watch videos that match their interests in a short amount of time.

[0511] A "user profile" is a collection of data that represents a user's interests and tendencies, generated based on their viewing history and preferences.

[0512] "Video analysis" is a technique that uses audio, text, and video within a video to characterize a scene.

[0513] A "key video section" is a part of a video that is considered to be valuable to the user based on the user profile and analysis results.

[0514] "Feedback" refers to the information and reactions that users provide to the system, and the service is improved based on this feedback.

[0515] A "generative AI model" is an artificial intelligence model that uses data such as feedback to predict user interests and behavior, thereby enabling efficient content suggestions.

[0516] A "prompt message" is an instruction or guidance message generated using a generative AI model based on the user's requests and conditions.

[0517] The system implementing this invention first generates a user profile based on the user's viewing history and preferences. The viewing history is used to collect data on videos the user has watched in the past and to analyze trends. The software used at this stage is a database management system, typically such as MySQL or PostgreSQL.

[0518] Next, the server performs video analysis. Specifically, it analyzes the received video data through speech recognition, video analysis, subtitle analysis, etc., and characterizes each scene. For video analysis, OpenCV could be used, and for speech recognition, the Google Speech-to-Text API could be used.

[0519] Subsequently, the server extracts important video sections based on the user profile and video analysis results. This is an effective process for finding information and scenes that are of interest to the user, such as specific technical terms or frequent occurrences of laughter.

[0520] The extracted video sections are streamed from the server to the user's device. In this step, the user can seamlessly and efficiently watch important scenes in sequence. The device is assumed to be a typical home PC or mobile device.

[0521] Furthermore, the system collects user feedback, which the server uses to update user profiles. This feedback data is essential for improving the system's accuracy and is used as training data to more accurately predict user preferences using generative AI models.

[0522] For example, if a user watches a lot of educational content, the system can extract important sections of technical terminology and provide videos that help them learn efficiently in a short amount of time. Similarly, for users who prefer entertainment such as comedy, the system can focus on playing humorous scenes.

[0523] The following is an example of a prompt message to input into the generative AI model.

[0524] "Please describe a method for extracting important video sections based on user interests. Design an algorithm to provide the most relevant content experience based on viewing history."

[0525] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0526] Step 1:

[0527] The server generates a user profile based on the user's viewing history. First, the server retrieves the user's past viewing data from the database. This data serves as input, and the server extracts the genres and keywords of the videos watched to create a user profile that shows the user's interests and tendencies. The user profile generated as output is used in subsequent processing steps.

[0528] Step 2:

[0529] The server analyzes newly input videos. It receives the input video data and processes it using speech recognition software and video analysis tools to analyze the audio, text, and video elements. This reveals the video's constituent elements and features, and scene feature data is generated as output.

[0530] Step 3:

[0531] The server matches user profiles with video analysis results to extract video sections that are important to the user. User profiles and scene feature data are used as input, and the data is processed to extract scenes that are likely to interest the user. The resulting output is a list of scene timestamps and clips.

[0532] Step 4:

[0533] The server organizes the extracted key video sections and prepares them for streaming to the user's device. The input consists of scene timestamps and a list of clips, which are used to generate data arranged in the optimal order for user viewing. The organized video clips are then sent as output to the user's device.

[0534] Step 5:

[0535] The device displays a streamed video clip, which the user watches. User input activates the video clip, and post-viewing feedback is provided to the server. The output is user feedback data.

[0536] Step 6:

[0537] The server analyzes user feedback and updates the user profile. The input feedback data is used to improve the training dataset for the generative AI model, enhancing the accuracy of future content recommendations. This results in an updated user profile and optimized viewing recommendations based on prompt messages.

[0538] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0539] This invention provides a system that further optimizes the video viewing experience for individual users by incorporating an emotion engine that recognizes the user's emotional state in real time. In addition to creating a profile based on the user's viewing history and preferences, this system provides content that matches the user's interests through the analysis of emotional data.

[0540] Program Processing Outline

[0541] The server creates a user profile based on the user's viewing history and preferences. The profile records the user's viewing patterns and emotional information in an integrated manner.

[0542] When a user watches a video, their device uses its camera and microphone to capture facial expressions and voice tone, and sends this data to a server. During this process, the emotion engine identifies the user's emotional state.

[0543] The server uses an emotion engine to analyze the user's emotional state in real time and measure their reactions while watching. The obtained emotional data is linked to relevant video scenes and timestamps.

[0544] The server extracts key parts of a video that are predicted to be interesting to the user, taking into account not only video analysis data but also emotional information. In this process, user emotional feedback is reflected in the creation of the video clips.

[0545] The generated clips of key segments are efficiently organized and streamed to the device. Users can watch customized content based on their emotional response to previously viewed material.

[0546] After viewing ends, the device sends feedback, including the results of the sentiment analysis, to the server. This updates the user profile, and the sentiment data is used as learning material for the system.

[0547] Specific example

[0548] The user attempts to watch a comedy scene that elicited joy or excitement. In this case, the emotion engine recognizes the user's smile and laughter from data captured by the camera and microphone. The server then analyzes this emotional feedback and incorporates similar scenes from other comedy videos into the next viewing suggestion. This matches the user with videos that suit their mood, improving the viewing experience. This format offers a new approach to video viewing that integrates the user's emotions and preferences.

[0549] The following describes the processing flow.

[0550] Step 1:

[0551] The user logs into the system and updates their personal viewing history and preferences. This keeps the user profile up to date.

[0552] Step 2:

[0553] The user selects a video they want to watch, and the video's metadata is sent to the server via their device. At this point, the user's viewing history is also taken into consideration.

[0554] Step 3:

[0555] The server analyzes the video. It evaluates each frame using audio, text, and video elements to characterize the scenes within the video.

[0556] Step 4:

[0557] The device uses its built-in camera and microphone to capture the user's facial expressions and voice in real time. This data is sent to a server, where an emotion engine analyzes the user's emotional state.

[0558] Step 5:

[0559] The server evaluates emotional data and video analysis results, extracting parts deemed important to the user. Emotional feedback is used in selecting scenes during this process.

[0560] Step 6:

[0561] The server generates clips of the extracted key portions and streams them to the device. These clips are customized based on the user's emotions.

[0562] Step 7:

[0563] New emotional responses that users exhibit while watching videos are continuously transmitted to the server in real time via their device, continuously optimizing the viewing experience.

[0564] Step 8:

[0565] After viewing ends, the device sends feedback to the server regarding the user's overall satisfaction and sentiment analysis results. Based on this information, the user profile is further updated.

[0566] (Example 2)

[0567] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0568] In recent years, with the rapid increase in video content, there has been a growing demand for providing an optimal viewing experience based on each user's individual interests and emotions. However, conventional systems have been insufficient in suggesting content that reflects users' emotions in real time. This invention aims to optimize video content while taking into account the user's emotional state.

[0569] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0570] In this invention, the server includes means for analyzing the user's viewing history and preferences and generating user attribute information; means for analyzing input visual information and analyzing audio, text, and video to characterize scenes; and means for monitoring the user's emotional state in real time and acquiring facial and audio data through the terminal. This makes it possible to suggest customized content to each user based on their emotional responses.

[0571] "User viewing history" refers to a record of content that a user has viewed in the past, and includes information such as genre and viewing time.

[0572] "User attribute information" refers to information that represents individual viewing patterns and preferences, generated based on a user's viewing history and preferences.

[0573] "Visual information" refers to data such as audio, text, and video that is necessary to characterize a scene within a video.

[0574] A "generative AI model" refers to an artificial intelligence model that generates new information or suggestions based on data.

[0575] "Emotional state" refers to the psychological reactions a user exhibits while watching, such as joy, surprise, and concentration.

[0576] "Facial expression and voice data" refers to digital data obtained from the user's facial expressions and voice tone.

[0577] "Emotional feedback" is a record of the emotional responses a user exhibits while watching content, and is used as information for system suggestions and learning.

[0578] "Customized content suggestions" refers to the provision of individually optimized viewing content suggested based on the user's past viewing history and real-time sentiment analysis.

[0579] This invention is a system for analyzing a user's emotional state in real time and providing an individually optimized video viewing experience. The system primarily involves the interaction of a server, terminal, and user's devices to realize a unique viewing experience.

[0580] First, the server generates user attribute information based on the user's viewing history and preferences. This clearly profiles individual viewing patterns and preferences. This profile serves as an indicator for determining what kind of content the user likes.

[0581] Next, the device uses its camera and microphone to capture facial and audio data while the user is watching a video. This involves using software that performs facial recognition and voice analysis. This data is crucial for determining the user's emotional state.

[0582] Furthermore, the server analyzes facial and audio data transmitted from the terminal and uses an emotion engine to identify the user's emotional state in real time. This technology is used to determine which scenes in the video had a positive impact on the user.

[0583] Based on the analyzed sentiment data, the server utilizes a generative AI model to determine the next content to watch. At this stage, the generative AI model works effectively to identify the most suitable video clips for the user.

[0584] As a concrete example, if a user smiles while watching a comedy scene, the system uses that facial expression data to recommend other comedy videos that are likely to evoke a similar emotional response. An example of a prompt used in this process would be a request such as, "Suggest a video to watch next based on my past viewing reactions." In this way, users can experience content customized to their emotions.

[0585] This invention allows users to enjoy a video viewing experience that feels as if it were made specifically for them, and the system can continuously learn from user feedback to make more accurate suggestions.

[0586] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0587] Step 1:

[0588] When a user selects video content, the device uses its built-in camera and microphone to capture the user's facial expressions and voice tone. This data is used as input to determine the user's emotional state by capturing video frames and sampling audio signals. As output, it generates real-time facial expression data and voice tone data.

[0589] Step 2:

[0590] The device sends facial expression data and voice tone data acquired from the terminal to the server. The server receives this input data and performs analysis using an emotion engine. Specifically, it detects subtle changes from facial expressions and extracts emotional features from the voice. As output, digital information representing the user's real-time emotional state is generated.

[0591] Step 3:

[0592] The server links real-time emotional data to specific scenes and timestamps in the video. This process combines the input emotional information with metadata from the content being viewed. As a result, information indicating when and in which scene an emotional response occurred is output.

[0593] Step 4:

[0594] The server integrates the user's emotional state, viewing history, and preference data and analyzes it using a generative AI model. User profiles, emotional response data, and content metadata are used as input. The output generates a set of recommended video clips.

[0595] Step 5:

[0596] Recommended content generated on the server is sent to the device and streamed. Specifically, based on the input recommendation clip information, video clips suitable for the user are collected and organized. This allows the user to watch customized content.

[0597] Step 6:

[0598] Once viewing ends, the device sends feedback to the server, including the results of the sentiment analysis during viewing. All viewing data and the final sentiment feedback are used as input. This information is used to update the user profile and improve the accuracy of future content recommendations, resulting in enhanced user attribute information being output.

[0599] (Application Example 2)

[0600] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0601] Traditional content delivery systems recommend content based solely on a user's viewing history and preferences, making it impossible to suggest optimal content that reflects the user's changing emotions in real time. Therefore, there was a need to efficiently deliver content that matches the user's emotions and create a more personalized viewing experience.

[0602] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0603] In this invention, the server includes means for analyzing the user's viewing history and preferences and generating a user profile; means for analyzing an input video and characterizing scenes with audio, text, and video; means for extracting important parts based on the user profile and video analysis results; means for recognizing the user's emotional state in real time and acquiring emotional data; means for generating and suggesting appropriate content based on the emotional data; means for streaming the generated content to the user's terminal; and means for collecting user feedback and updating the profile. This enables the suggestion of content that matches the user's real-time emotions and the personalization of the viewing experience as a result.

[0604] "Viewing history" refers to the history of content a user has viewed so far, and is data used to analyze the user's preferences.

[0605] A "user profile" is information generated based on a user's viewing history and preferences, and serves as the foundational data for providing content that is best suited to each individual user.

[0606] "Emotional data" refers to information that represents a user's real-time emotional state, and is data acquired using sensor devices such as cameras and microphones.

[0607] "Content analysis" is a method of characterizing scenes within an input video based on audio, text, and video, and is a process used to extract important parts.

[0608] "Key elements" refer to video scenes that are deemed interesting or valuable to the user, extracted based on user profiles and video analysis results.

[0609] "Feedback" refers to information collected from users' impressions and reactions after viewing content, which is used as learning material for the system and helps update user profiles.

[0610] "Streaming" is a technology that delivers generated content to user devices in real time, minimizing latency and providing a high-quality viewing experience.

[0611] An "emotion engine" refers to a core algorithm or software that analyzes a user's emotional state in real time and suggests appropriate content.

[0612] This invention is a system that combines multiple technological elements to optimize the user's viewing experience according to their emotions. The system generates a profile based on the user's viewing history and preferences, and extracts important scenes from videos based on this profile. Furthermore, by utilizing an emotion engine to acquire and analyze the user's emotional data in real time, it is possible to present content that is suitable for the user.

[0613] This system utilizes smart glasses and smart devices as terminals. These devices are equipped with cameras and microphones to capture the user's facial expressions and voice tone. This data is sent to a server, where a cloud-based emotion analysis engine analyzes the emotional data in real time. For example, services such as Amazon Rekognition are used to understand the user's emotional state from the acquired video and audio data.

[0614] The server integrates the analyzed sentiment data into user profiles and uses this to train a deep learning model. The resulting model uses a generative AI model to predict content that is likely to match the user's emotions and recommends it to the user. This allows the system to provide video clips that amplify emotions such as joy and surprise that the user feels while watching.

[0615] As a concrete example, consider a scenario where a user is watching a horror movie. In this case, the system can detect the user's surprised facial expression and suggest an appropriate next video scene to alleviate their fear. For example, based on a prompt such as, "Analyze the scene where the user is smiling and select a comedy video that matches that as the next viewing suggestion," the system can select an appropriate video.

[0616] This invention will personalize the viewing experience and provide users with the most relevant content tailored to their emotions.

[0617] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0618] Step 1:

[0619] The device acquires real-time data from the user while they are viewing content. Specifically, it uses the camera and microphone of the smart glasses to record the user's facial expressions and voice tone. The input data consists of video and audio, and the output is raw data that allows for the identification of emotions.

[0620] Step 2:

[0621] The terminal sends the acquired video and audio data to the server. Here, data transfer takes place, with the input being the raw data acquired by the terminal and the output being the server receiving this data.

[0622] Step 3:

[0623] The server uses a cloud-based sentiment analysis engine to identify the user's emotional state from the received raw data. The input is the video and audio data received in step 2, and the output is the user's identified emotional state. Data processing includes video analysis and audio analysis.

[0624] Step 4:

[0625] The server updates the user profile based on the user's emotional state. The input is real-time emotional information, and the output is an updated user profile. A deep learning model is used to perform data calculations combining emotional data with existing viewing history.

[0626] Step 5:

[0627] The server uses a generative AI model to generate recommended content based on the updated user profile. The input is the updated user profile, and the output is a list of recommended content for the user. A prompt might include, "Analyze the scenes where the user is smiling and select a comedy video that matches those scenes as the next viewing suggestion."

[0628] Step 6:

[0629] The server prepares to stream the generated recommended content to the terminal. The input is a list of recommended content, and the output is data converted into a streamable format. This step involves encoding the content and preparing it for network transfer.

[0630] Step 7:

[0631] The device receives recommended content streamed from the server and displays it in a format viewable by the user. The input is the streamed content, and the output is the video that the user views. The device handles the video rendering, realizing the user's viewing experience.

[0632] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0633] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0634] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0635] [Fourth Embodiment]

[0636] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0637] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0638] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0639] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0640] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0641] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0642] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0643] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0644] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0645] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0646] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0647] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0648] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0649] This invention aims to build a system that allows users to efficiently watch videos that best suit their interests amidst a diverse range of content. This system analyzes the user's viewing history and preferences, and optimizes viewing time by extracting and providing only the most important parts from videos.

[0650] Program Processing Outline

[0651] 1. The server has the function of initializing the user's profile and accumulating viewing history and preference data based on it. This records what genres and themes the user is interested in as a profile.

[0652] 2. The user selects the video they want to watch. Whether it's a newly uploaded video or one from an existing library, the video's metadata is sent to the server via the device.

[0653] 3. The server analyzes the received video and evaluates and characterizes each scene through speech recognition and video analysis. For example, it detects frequently used technical terms in the video and locations where laughter occurs.

[0654] 4. The server compares the video analysis results with the user profile and extracts portions deemed important to the user. It then generates video clips based on the extracted timestamps.

[0655] 5. The extracted clips are efficiently organized and then streamed from the server to the device. The device plays them seamlessly, allowing the user to continuously watch only the scenes that interest them.

[0656] 6. After viewing ends, the device sends user feedback to the server. This information is used to update the user profile and improve suggestions for future viewing.

[0657] Specific example

[0658] Suppose a user wants to watch a technical lecture video. In this case, the server identifies terms and themes that the user has shown particular interest in from their past viewing history. Next, it analyzes the lecture video to detect sections where these terms frequently appear. Based on this analysis, the system extracts important sections for the user and generates them as clips for continuous playback. In this way, the user can efficiently acquire the knowledge they seek. Thus, by implementing the present invention, the video viewing experience can be customized for each individual user, and time constraints can be greatly reduced.

[0659] The following describes the processing flow.

[0660] Step 1:

[0661] Users access the system and input their interests and preferences. This information is used for initial setup and is also treated as data added to their viewing history.

[0662] Step 2:

[0663] The server uses data on the user's viewing history and preferences to create a user profile. This profile includes the user's interests, such as genres and specific topics.

[0664] Step 3:

[0665] The user selects the video they want to watch. If the user uploads a new video, the device sends the video's metadata to the server.

[0666] Step 4:

[0667] The server analyzes the received video. Speech recognition technology is used to convert the audio into text, and video analysis technology is used to evaluate the content of each frame. This analysis characterizes and tags the scenes in the video.

[0668] Step 5:

[0669] The server identifies important parts for the user by comparing the analysis results with the user profile. It obtains timestamps of these important parts and generates video clips as needed.

[0670] Step 6:

[0671] The server generates clips of key sections and streams them to the device. The device then plays these clips sequentially, allowing the user to efficiently view the information they need.

[0672] Step 7:

[0673] The device records user actions during viewing (e.g., skipping or rewinding), and this data is sent to a server to be stored as training data.

[0674] Step 8:

[0675] After viewing ends, the device will present the user with a short survey and collect their feedback. This feedback will be used to update the user profile and improve the viewing experience for future viewings.

[0676] (Example 1)

[0677] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0678] In today's society, where diverse content abounds, it is difficult for users to efficiently view video information that matches their interests and needs. In particular, there is a need to effectively extract only the important parts within a limited time and optimize the viewing experience. However, achieving this requires technology that deeply understands, appropriately evaluates, and reflects the diverse interests of each user.

[0679] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0680] In this invention, the server includes means for analyzing user interest information and generating user characteristic information; means for analyzing input video information and processing audio information, text information, and video information to characterize scenes within the video information; and means for extracting parts that are important to the user based on the user characteristic information and the video information analysis results. This enables the user to efficiently view important video parts based on their interests.

[0681] "User interest information" refers to information about the interests and preferences that users have shown through their past actions and viewing history.

[0682] "User characteristic information" refers to user-specific profile information generated based on individual user preferences, viewing history, and evaluation history.

[0683] "Visual information" refers to input digital video and multimedia content, which includes video, audio, and text information.

[0684] "Audio information" refers to data related to the content and characteristics of audio used within video information.

[0685] "Textual information" refers to all text data contained within video information, including subtitles and captions.

[0686] "Means for processing video information" refers to technical means for analyzing and evaluating video information using technologies such as speech recognition and video analysis.

[0687] "Important parts" refer to a series of video scenes that are deemed particularly interesting to the user based on user characteristics and video information.

[0688] A "learning model" is a trained algorithm or AI model that uses large amounts of data to efficiently perform a specific task.

[0689] "Short continuous information" refers to short video clips generated by connecting important parts extracted from video information.

[0690] An "information processing system" refers to a series of infrastructure components that perform information input, processing, and output through a combination of software and hardware.

[0691] This invention is an information processing system that allows users to efficiently view video information based on their interests. By having the server, terminal, and user each fulfill their respective roles, it provides a smooth viewing experience.

[0692] The server analyzes user interest information and generates user characteristic information. To achieve this, it utilizes a database management system and machine learning algorithms to process viewing history and user feedback. By incorporating speech recognition and video analysis technologies such as Google Cloud Speech-to-Text and OpenCV, it efficiently analyzes video information and extracts features, including audio and text information, from the video. This analysis is combined with user characteristic information to automatically extract important parts. Furthermore, the generated AI model is used to score the importance of each element to the user and generate short, sequential pieces of information.

[0693] The device receives data from the server and streams it to the user. During this process, Adaptive Bitrate Streaming technology is used to adapt to network conditions, maintaining a high-quality video experience. User feedback after viewing is also sent to the server via the device, contributing to profile updates.

[0694] If a user wants to efficiently learn from technical lecture videos, the server analyzes and extracts appropriate segments based on specific terms and themes from their past viewing history. This allows users to obtain information relevant to their learning needs in a timely manner.

[0695] A concrete example of a prompt message is, "Detect scenes where technical terms frequently appear and generate clips based on them." In this way, the system provides a customized video viewing experience for each user, maximizing its comfort and usefulness.

[0696] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0697] Step 1:

[0698] The server retrieves user information and initializes profiles. First, when a user registers, it receives the basic information they enter (age, gender, areas of interest, etc.). Based on this input information, it stores user characteristic information in the database. This profile serves as the foundation for understanding each user's preferences.

[0699] Step 2:

[0700] The user selects the video they want to watch and sends that information to the server via their device. This input includes the video ID, title, category, and tags from the metadata. This information is stored on the server and prepares it for the next analysis step.

[0701] Step 3:

[0702] The server performs video analysis and speech recognition on the selected video information. It receives video data as input and extracts audio and text information. Specifically, it performs speech transcription using Google Cloud Speech-to-Text and scene analysis of the video using OpenCV. The output is information that lists the features of all scenes in the video.

[0703] Step 4:

[0704] The server compares the scene features obtained from the analysis with the user's characteristic information and extracts the most relevant parts. The input consists of the output information from step 3 and the user's characteristic information, and scoring is performed using an AI model. Here, timestamps of scenes important to the user are output, and a list of important segments is generated.

[0705] Step 5:

[0706] The server generates short, sequential clips based on the extracted key portions. The input is the timestamp information obtained in step 4, and a video editing tool is used to generate clips. The output is a customized video clip tailored to the user's preferences.

[0707] Step 6:

[0708] The terminal seamlessly streams clips received from the server to the viewer. The input is a video clip sent from the server, and playback is performed using Adaptive Bitrate Streaming technology, adjusting the video quality according to the network conditions. The output is smooth, uninterrupted continuous playback.

[0709] Step 7:

[0710] After watching a video, users send feedback to the server via their device. This feedback is entered as information about their satisfaction with the video and what they found interesting. The server incorporates this data into the user profile and uses it to improve the accuracy of future recommendations. This output represents updated user characteristics information.

[0711] (Application Example 1)

[0712] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0713] In today's world, the sheer volume and variety of video content makes it difficult for users to quickly and efficiently find and watch content that suits their interests. Furthermore, existing systems often fail to adequately optimize videos based on user preferences and interests, leading to wasted viewing time and content bias. This invention aims to address these issues by individually customizing the user's viewing experience and enabling effective content delivery.

[0714] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0715] In this invention, the server includes means for analyzing the user's viewing history and preferences and generating a user profile; means for analyzing input videos and analyzing audio, text, and video to characterize scenes within the videos; means for extracting video sections important to the user based on the user profile and video analysis results; means for efficiently organizing the extracted video sections and streaming them to the user terminal; and means for collecting user feedback and improving the deep learning model to improve the accuracy of the user profile and suggestions. This enables users to efficiently watch videos that match their interests in a short amount of time.

[0716] A "user profile" is a collection of data that represents a user's interests and tendencies, generated based on their viewing history and preferences.

[0717] "Video analysis" is a technique that uses audio, text, and video within a video to characterize a scene.

[0718] A "key video section" is a part of a video that is considered to be valuable to the user based on the user profile and analysis results.

[0719] "Feedback" refers to the information and reactions that users provide to the system, and the service is improved based on this feedback.

[0720] A "generative AI model" is an artificial intelligence model that uses data such as feedback to predict user interests and behavior, thereby enabling efficient content suggestions.

[0721] A "prompt message" is an instruction or guidance message generated using a generative AI model based on the user's requests and conditions.

[0722] The system implementing this invention first generates a user profile based on the user's viewing history and preferences. The viewing history is used to collect data on videos the user has watched in the past and to analyze trends. The software used at this stage is a database management system, typically such as MySQL or PostgreSQL.

[0723] Next, the server performs video analysis. Specifically, it analyzes the received video data through speech recognition, video analysis, subtitle analysis, etc., and characterizes each scene. For video analysis, OpenCV could be used, and for speech recognition, the Google Speech-to-Text API could be used.

[0724] Subsequently, the server extracts important video sections based on the user profile and video analysis results. This is an effective process for finding information and scenes that are of interest to the user, such as specific technical terms or frequent occurrences of laughter.

[0725] The extracted video sections are streamed from the server to the user's device. In this step, the user can seamlessly and efficiently watch important scenes in sequence. The device is assumed to be a typical home PC or mobile device.

[0726] Furthermore, the system collects user feedback, which the server uses to update user profiles. This feedback data is essential for improving the system's accuracy and is used as training data to more accurately predict user preferences using generative AI models.

[0727] For example, if a user watches a lot of educational content, the system can extract important sections of technical terminology and provide videos that help them learn efficiently in a short amount of time. Similarly, for users who prefer entertainment such as comedy, the system can focus on playing humorous scenes.

[0728] The following is an example of a prompt message to input into the generative AI model.

[0729] "Please describe a method for extracting important video sections based on user interests. Design an algorithm to provide the most relevant content experience based on viewing history."

[0730] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0731] Step 1:

[0732] The server generates a user profile based on the user's viewing history. First, the server retrieves the user's past viewing data from the database. This data serves as input, and the server extracts the genres and keywords of the videos watched to create a user profile that shows the user's interests and tendencies. The user profile generated as output is used in subsequent processing steps.

[0733] Step 2:

[0734] The server analyzes newly input videos. It receives the input video data and processes it using speech recognition software and video analysis tools to analyze the audio, text, and video elements. This reveals the video's constituent elements and features, and scene feature data is generated as output.

[0735] Step 3:

[0736] The server matches user profiles with video analysis results to extract video sections that are important to the user. User profiles and scene feature data are used as input, and the data is processed to extract scenes that are likely to interest the user. The resulting output is a list of scene timestamps and clips.

[0737] Step 4:

[0738] The server organizes the extracted key video sections and prepares them for streaming to the user's device. The input consists of scene timestamps and a list of clips, which are used to generate data arranged in the optimal order for user viewing. The organized video clips are then sent as output to the user's device.

[0739] Step 5:

[0740] The device displays a streamed video clip, which the user watches. User input activates the video clip, and post-viewing feedback is provided to the server. The output is user feedback data.

[0741] Step 6:

[0742] The server analyzes user feedback and updates the user profile. The input feedback data is used to improve the training dataset for the generative AI model, enhancing the accuracy of future content recommendations. This results in an updated user profile and optimized viewing recommendations based on prompt messages.

[0743] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0744] This invention provides a system that further optimizes the video viewing experience for individual users by incorporating an emotion engine that recognizes the user's emotional state in real time. In addition to creating a profile based on the user's viewing history and preferences, this system provides content that matches the user's interests through the analysis of emotional data.

[0745] Program Processing Outline

[0746] The server creates a user profile based on the user's viewing history and preferences. The profile records the user's viewing patterns and emotional information in an integrated manner.

[0747] When a user watches a video, their device uses its camera and microphone to capture facial expressions and voice tone, and sends this data to a server. During this process, the emotion engine identifies the user's emotional state.

[0748] The server uses an emotion engine to analyze the user's emotional state in real time and measure their reactions while watching. The obtained emotional data is linked to relevant video scenes and timestamps.

[0749] The server extracts key parts of a video that are predicted to be interesting to the user, taking into account not only video analysis data but also emotional information. In this process, user emotional feedback is reflected in the creation of the video clips.

[0750] The generated clips of key segments are efficiently organized and streamed to the device. Users can watch customized content based on their emotional response to previously viewed material.

[0751] After viewing ends, the device sends feedback, including the results of the sentiment analysis, to the server. This updates the user profile, and the sentiment data is used as learning material for the system.

[0752] Specific example

[0753] The user attempts to watch a comedy scene that elicited joy or excitement. In this case, the emotion engine recognizes the user's smile and laughter from data captured by the camera and microphone. The server then analyzes this emotional feedback and incorporates similar scenes from other comedy videos into the next viewing suggestion. This matches the user with videos that suit their mood, improving the viewing experience. This format offers a new approach to video viewing that integrates the user's emotions and preferences.

[0754] The following describes the processing flow.

[0755] Step 1:

[0756] The user logs into the system and updates their personal viewing history and preferences. This keeps the user profile up to date.

[0757] Step 2:

[0758] The user selects a video they want to watch, and the video's metadata is sent to the server via their device. At this point, the user's viewing history is also taken into consideration.

[0759] Step 3:

[0760] The server analyzes the video. It evaluates each frame using audio, text, and video elements to characterize the scenes within the video.

[0761] Step 4:

[0762] The device uses its built-in camera and microphone to capture the user's facial expressions and voice in real time. This data is sent to a server, where an emotion engine analyzes the user's emotional state.

[0763] Step 5:

[0764] The server evaluates emotional data and video analysis results, extracting parts deemed important to the user. Emotional feedback is used in selecting scenes during this process.

[0765] Step 6:

[0766] The server generates clips of the extracted key portions and streams them to the device. These clips are customized based on the user's emotions.

[0767] Step 7:

[0768] New emotional responses that users exhibit while watching videos are continuously transmitted to the server in real time via their device, continuously optimizing the viewing experience.

[0769] Step 8:

[0770] After viewing ends, the device sends feedback to the server regarding the user's overall satisfaction and sentiment analysis results. Based on this information, the user profile is further updated.

[0771] (Example 2)

[0772] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0773] In recent years, with the rapid increase in video content, there has been a growing demand for providing an optimal viewing experience based on each user's individual interests and emotions. However, conventional systems have been insufficient in suggesting content that reflects users' emotions in real time. This invention aims to optimize video content while taking into account the user's emotional state.

[0774] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0775] In this invention, the server includes means for analyzing the user's viewing history and preferences and generating user attribute information; means for analyzing input visual information and analyzing audio, text, and video to characterize scenes; and means for monitoring the user's emotional state in real time and acquiring facial and audio data through the terminal. This makes it possible to suggest customized content to each user based on their emotional responses.

[0776] "User viewing history" refers to a record of content that a user has viewed in the past, and includes information such as genre and viewing time.

[0777] "User attribute information" refers to information that represents individual viewing patterns and preferences, generated based on a user's viewing history and preferences.

[0778] "Visual information" refers to data such as audio, text, and video that is necessary to characterize a scene within a video.

[0779] A "generative AI model" refers to an artificial intelligence model that generates new information or suggestions based on data.

[0780] "Emotional state" refers to the psychological reactions a user exhibits while watching, such as joy, surprise, and concentration.

[0781] "Facial expression and voice data" refers to digital data obtained from the user's facial expressions and voice tone.

[0782] "Emotional feedback" is a record of the emotional responses a user exhibits while watching content, and is used as information for system suggestions and learning.

[0783] "Customized content suggestions" refers to the provision of individually optimized viewing content suggested based on the user's past viewing history and real-time sentiment analysis.

[0784] This invention is a system for analyzing a user's emotional state in real time and providing an individually optimized video viewing experience. The system primarily involves the interaction of a server, terminal, and user's devices to realize a unique viewing experience.

[0785] First, the server generates user attribute information based on the user's viewing history and preferences. This clearly profiles individual viewing patterns and preferences. This profile serves as an indicator for determining what kind of content the user likes.

[0786] Next, the device uses its camera and microphone to capture facial and audio data while the user is watching a video. This involves using software that performs facial recognition and voice analysis. This data is crucial for determining the user's emotional state.

[0787] Furthermore, the server analyzes facial and audio data transmitted from the terminal and uses an emotion engine to identify the user's emotional state in real time. This technology is used to determine which scenes in the video had a positive impact on the user.

[0788] Based on the analyzed sentiment data, the server utilizes a generative AI model to determine the next content to watch. At this stage, the generative AI model works effectively to identify the most suitable video clips for the user.

[0789] As a concrete example, if a user smiles while watching a comedy scene, the system uses that facial expression data to recommend other comedy videos that are likely to evoke a similar emotional response. An example of a prompt used in this process would be a request such as, "Suggest a video to watch next based on my past viewing reactions." In this way, users can experience content customized to their emotions.

[0790] This invention allows users to enjoy a video viewing experience that feels as if it were made specifically for them, and the system can continuously learn from user feedback to make more accurate suggestions.

[0791] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0792] Step 1:

[0793] When a user selects video content, the device uses its built-in camera and microphone to capture the user's facial expressions and voice tone. This data is used as input to determine the user's emotional state by capturing video frames and sampling audio signals. As output, it generates real-time facial expression data and voice tone data.

[0794] Step 2:

[0795] The device sends facial expression data and voice tone data acquired from the terminal to the server. The server receives this input data and performs analysis using an emotion engine. Specifically, it detects subtle changes from facial expressions and extracts emotional features from the voice. As output, digital information representing the user's real-time emotional state is generated.

[0796] Step 3:

[0797] The server links real-time emotional data to specific scenes and timestamps in the video. This process combines the input emotional information with metadata from the content being viewed. As a result, information indicating when and in which scene an emotional response occurred is output.

[0798] Step 4:

[0799] The server integrates the user's emotional state, viewing history, and preference data and analyzes it using a generative AI model. User profiles, emotional response data, and content metadata are used as input. The output generates a set of recommended video clips.

[0800] Step 5:

[0801] Recommended content generated on the server is sent to the device and streamed. Specifically, based on the input recommendation clip information, video clips suitable for the user are collected and organized. This allows the user to watch customized content.

[0802] Step 6:

[0803] Once viewing ends, the device sends feedback to the server, including the results of the sentiment analysis during viewing. All viewing data and the final sentiment feedback are used as input. This information is used to update the user profile and improve the accuracy of future content recommendations, resulting in enhanced user attribute information being output.

[0804] (Application Example 2)

[0805] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0806] Traditional content delivery systems recommend content based solely on a user's viewing history and preferences, making it impossible to suggest optimal content that reflects the user's changing emotions in real time. Therefore, there was a need to efficiently deliver content that matches the user's emotions and create a more personalized viewing experience.

[0807] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0808] In this invention, the server includes means for analyzing the user's viewing history and preferences and generating a user profile; means for analyzing an input video and characterizing scenes with audio, text, and video; means for extracting important parts based on the user profile and video analysis results; means for recognizing the user's emotional state in real time and acquiring emotional data; means for generating and suggesting appropriate content based on the emotional data; means for streaming the generated content to the user's terminal; and means for collecting user feedback and updating the profile. This enables the suggestion of content that matches the user's real-time emotions and the personalization of the viewing experience as a result.

[0809] "Viewing history" refers to the history of content a user has viewed so far, and is data used to analyze the user's preferences.

[0810] A "user profile" is information generated based on a user's viewing history and preferences, and serves as the foundational data for providing content that is best suited to each individual user.

[0811] "Emotional data" refers to information that represents a user's real-time emotional state, and is data acquired using sensor devices such as cameras and microphones.

[0812] "Content analysis" is a method of characterizing scenes within an input video based on audio, text, and video, and is a process used to extract important parts.

[0813] "Key elements" refer to video scenes that are deemed interesting or valuable to the user, extracted based on user profiles and video analysis results.

[0814] "Feedback" refers to information collected from users' impressions and reactions after viewing content, which is used as learning material for the system and helps update user profiles.

[0815] "Streaming" is a technology that delivers generated content to user devices in real time, minimizing latency and providing a high-quality viewing experience.

[0816] An "emotion engine" refers to a core algorithm or software that analyzes a user's emotional state in real time and suggests appropriate content.

[0817] This invention is a system that combines multiple technological elements to optimize the user's viewing experience according to their emotions. The system generates a profile based on the user's viewing history and preferences, and extracts important scenes from videos based on this profile. Furthermore, by utilizing an emotion engine to acquire and analyze the user's emotional data in real time, it is possible to present content that is suitable for the user.

[0818] This system utilizes smart glasses and smart devices as terminals. These devices are equipped with cameras and microphones to capture the user's facial expressions and voice tone. This data is sent to a server, where a cloud-based emotion analysis engine analyzes the emotional data in real time. For example, services such as Amazon Rekognition are used to understand the user's emotional state from the acquired video and audio data.

[0819] The server integrates the analyzed sentiment data into user profiles and uses this to train a deep learning model. The resulting model uses a generative AI model to predict content that is likely to match the user's emotions and recommends it to the user. This allows the system to provide video clips that amplify emotions such as joy and surprise that the user feels while watching.

[0820] As a concrete example, consider a scenario where a user is watching a horror movie. In this case, the system can detect the user's surprised facial expression and suggest an appropriate next video scene to alleviate their fear. For example, based on a prompt such as, "Analyze the scene where the user is smiling and select a comedy video that matches that as the next viewing suggestion," the system can select an appropriate video.

[0821] This invention will personalize the viewing experience and provide users with the most relevant content tailored to their emotions.

[0822] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0823] Step 1:

[0824] The device acquires real-time data from the user while they are viewing content. Specifically, it uses the camera and microphone of the smart glasses to record the user's facial expressions and voice tone. The input data consists of video and audio, and the output is raw data that allows for the identification of emotions.

[0825] Step 2:

[0826] The terminal sends the acquired video and audio data to the server. Here, data transfer takes place, with the input being the raw data acquired by the terminal and the output being the server receiving this data.

[0827] Step 3:

[0828] The server uses a cloud-based sentiment analysis engine to identify the user's emotional state from the received raw data. The input is the video and audio data received in step 2, and the output is the user's identified emotional state. Data processing includes video analysis and audio analysis.

[0829] Step 4:

[0830] The server updates the user profile based on the user's emotional state. The input is real-time emotional information, and the output is an updated user profile. A deep learning model is used to perform data calculations combining emotional data with existing viewing history.

[0831] Step 5:

[0832] The server uses a generative AI model to generate recommended content based on the updated user profile. The input is the updated user profile, and the output is a list of recommended content for the user. A prompt might include, "Analyze the scenes where the user is smiling and select a comedy video that matches those scenes as the next viewing suggestion."

[0833] Step 6:

[0834] The server prepares to stream the generated recommended content to the terminal. The input is a list of recommended content, and the output is data converted into a streamable format. This step involves encoding the content and preparing it for network transfer.

[0835] Step 7:

[0836] The device receives recommended content streamed from the server and displays it in a format viewable by the user. The input is the streamed content, and the output is the video that the user views. The device handles the video rendering, realizing the user's viewing experience.

[0837] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0838] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0839] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0840] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0841] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0842] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0843] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0844] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0845] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0846] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0847] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0848] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0849] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0850] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0851] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0852] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0853] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0854] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0855] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0856] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0857] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0858] The following is further disclosed regarding the embodiments described above.

[0859] (Claim 1)

[0860] A means for analyzing a user's viewing history and preferences to generate a user profile,

[0861] A means for analyzing input videos and analyzing audio, text, and video to characterize scenes within the video,

[0862] A means for extracting important parts for the user based on the user profile and video analysis results,

[0863] A means for streaming the extracted important portion to the user terminal,

[0864] A means for collecting user feedback and updating the user profile,

[0865] A system that includes this.

[0866] (Claim 2)

[0867] The system according to claim 1, further comprising means for the system to learn the user's interests based on the actions the user takes while viewing and reflect this in future viewing suggestions.

[0868] (Claim 3)

[0869] The system according to claim 1, comprising means for training a deep learning model using feedback collected from users.

[0870] "Example 1"

[0871] (Claim 1)

[0872] A means of analyzing user interest information and generating user characteristic information,

[0873] A means for analyzing input video information and processing audio information, text information, and video information in order to characterize scenes within the video information,

[0874] A means for extracting parts that are important to the user based on the user characteristic information and video information analysis results,

[0875] Means for communicating the extracted important portion to the user device,

[0876] A means for collecting user feedback and updating the user characteristics information,

[0877] A method for scoring important parts using an AI model based on the user's evaluation history and generating video information as short, continuous pieces of information,

[0878] An information processing system that includes this.

[0879] (Claim 2)

[0880] The information processing system according to claim 1, comprising means for the information processing system to learn the user's interests based on the operations performed by the user while viewing, and to reflect these interests in future viewing suggestions.

[0881] (Claim 3)

[0882] The information processing system according to claim 1, comprising means for training a learning model using opinions collected from users.

[0883] "Application Example 1"

[0884] (Claim 1)

[0885] A means for analyzing a user's viewing history and preferences to generate a user profile,

[0886] A means for analyzing input videos and analyzing audio, text, and video to characterize scenes within the video,

[0887] A means for extracting video sections that are important to the user based on the user profile and video analysis results,

[0888] A means to efficiently organize the extracted video sections and stream them to the user's device,

[0889] A means for collecting user feedback and improving the deep learning model to improve the accuracy of the user profile and suggestions,

[0890] A system that includes this.

[0891] (Claim 2)

[0892] The system according to claim 1, further comprising means for the system to learn the user's interests based on the actions the user takes while viewing and reflect this in future viewing suggestions.

[0893] (Claim 3)

[0894] The system according to claim 1, comprising means for training a generative AI model using feedback collected from users and generating prompt sentences.

[0895] "Example 2 of combining an emotion engine"

[0896] (Claim 1)

[0897] A means for analyzing a user's viewing history and preferences and generating user attribute information,

[0898] A means of analyzing input visual information and analyzing audio, documents, and images to characterize the scene,

[0899] A means for extracting parts that are important to the user based on the user attribute information and the results of visual information analysis,

[0900] A means for transmitting the extracted important portion to the user's device,

[0901] A means for monitoring the user's emotional state in real time and acquiring facial and voice data through a terminal,

[0902] A means for identifying emotions using acquired facial and voice data and analyzing the identification results,

[0903] A means for determining recommended viewing content for a user using a generative AI model based on the results of sentiment analysis,

[0904] A means for collecting emotional feedback from users and updating the user attribute information,

[0905] A system that includes this.

[0906] (Claim 2)

[0907] The system according to claim 1, further comprising means for the system to learn the user's interests based on the user's emotional state while viewing and reflect this in future viewing suggestions.

[0908] (Claim 3)

[0909] The system according to claim 1, comprising means for training a deep learning model using emotional feedback collected from users.

[0910] "Application example 2 when combining with an emotional engine"

[0911] (Claim 1)

[0912] A means for analyzing a user's viewing history and preferences to generate a user profile,

[0913] A means for analyzing input videos and analyzing audio, text, and video to characterize scenes within the video,

[0914] A means for extracting important parts for the user based on the user profile and video analysis results,

[0915] A means of recognizing a user's emotional state in real time and acquiring emotional data,

[0916] A means for generating and suggesting content appropriate to the user's emotions based on the aforementioned emotion data,

[0917] A means for streaming the generated content to the user terminal,

[0918] A means for collecting user feedback and updating the user profile,

[0919] A system that includes this.

[0920] (Claim 2)

[0921] The system according to claim 1, further comprising means for the system to learn the user's interests based on the user's actions while viewing and real-time sentiment data, and to reflect this in future viewing suggestions.

[0922] (Claim 3)

[0923] The system according to claim 1, comprising means for training a deep learning model using feedback and sentiment data collected from users. [Explanation of Symbols]

[0924] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for analyzing a user's viewing history and preferences to generate a user profile, A means for analyzing input videos and analyzing audio, text, and video to characterize scenes within the video, A means for extracting video sections that are important to the user based on the user profile and video analysis results, A means to efficiently organize the extracted video sections and stream them to the user's device, A means for collecting user feedback and improving the deep learning model to improve the accuracy of the user profile and suggestions, A system that includes this.

2. The system according to claim 1, further comprising means for the system to learn the user's interests based on the actions the user takes while viewing and reflect this in future viewing suggestions.

3. The system according to claim 1, comprising means for training a generative AI model using feedback collected from users and generating prompt sentences.

Images