System

Abstract

To provide a system that generates sub-audio based on contents of a moving image and a user's level of understanding and adds it to the moving image.SOLUTION: A system according to an embodiment includes an analysis unit, an evaluation unit, a generation unit, and an addition unit. The analysis unit analyzes contents of a moving image. The evaluation unit evaluates a user's level of understanding based on the contents analyzed by the analysis unit. The generation unit generates sub-audio based on the level of understanding evaluated by the evaluation unit. The addition unit adds the sub-audio generated by the generation unit to the moving image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional techniques do not adequately generate secondary audio to improve comprehension of video content, and there is room for improvement.

[0005] The system according to the embodiment aims to generate secondary audio based on the content of the video and the user's level of understanding, and to add the secondary audio to the video. [Means for solving the problem]

[0006] The system according to the embodiment includes an analysis unit, an evaluation unit, a generation unit, and an attachment unit. The analysis unit analyzes the content of the video. The evaluation unit evaluates the user's level of understanding based on the content analyzed by the analysis unit. The generation unit generates a secondary audio based on the level of understanding evaluated by the evaluation unit. The attachment unit attaches the secondary audio generated by the generation unit to the video. [Effects of the Invention]

[0007] The system according to the embodiment can generate secondary audio based on the content of the video and the user's level of understanding, and can add the secondary audio to the video. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE INVENTION

[0009] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (for example, a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). Display 40A displays visible information such as text and images in accordance with instructions from processor 46. Speaker 40B outputs audio in accordance with instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device owned by a user (e.g., a mobile phone, a robot, a home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) A secondary audio generation system according to an embodiment of the present invention is a system for deepening understanding of video content. This secondary audio generation system analyzes the content of a video, evaluates the user's level of comprehension, and then uses a generation AI to generate secondary audio based on the video content and the user's level of comprehension, and then adds the secondary audio to the video. This secondary audio blends in with the video, allowing the user to enjoy the video content without any sense of incongruity. Furthermore, the secondary audio itself can be registered by the user as content, and business value can be added by inserting adfusion-style advertisements. For example, the secondary audio generation system analyzes the content of a video. For example, it extracts important points from each scene of the video and organizes them to make them easier for the user to understand. In this process, the generation AI analyzes the content of the video and identifies the important points. Next, the secondary audio generation system evaluates the user's level of comprehension. For example, data such as the user's viewing history, viewing time, and playback speed are collected when the user watches a video, and the generation AI analyzes this data to evaluate the user's level of comprehension. This allows the user to identify areas where their understanding is lacking. The generation AI generates secondary audio based on the content of the video and the user's level of comprehension. For example, it generates secondary audio that explains important points of the video or supplements parts that are difficult for the user to understand. This audio commentary blends in with the video, allowing users to enjoy the video content without any sense of incongruity. Furthermore, the audio commentary itself can be registered as content by the user. For example, users can share audio commentary they have created with other users. This allows users to share information with each other and deepen their understanding of the video. Business value can also be added by inserting adfusion-like advertisements. For example, by inserting advertisements into the audio commentary, advertising revenue can be earned. This also benefits the video content provider and the audio commentary creator. This system deepens understanding of the video content and allows users to enjoy a more fulfilling viewing experience. Business value is also increased by sharing audio commentary and advertising revenue. This allows the audio commentary generation system to deepen understanding of video content and provide users with a fulfilling viewing experience.

[0029] The audio secondary generation system according to the embodiment includes an analysis unit, an evaluation unit, a generation unit, and an assignment unit. The analysis unit analyzes the content of a video. For example, the analysis unit extracts important points from each scene of the video and organizes them to make them easier for the user to understand. The analysis unit uses a generation AI to analyze the content of the video and identify important points. For example, the generation AI extracts important keyframes from each scene of the video and analyzes the main points of each scene. The evaluation unit evaluates the user's level of comprehension based on the content analyzed by the analysis unit. The evaluation unit collects data such as the user's viewing history, viewing time, and playback speed when watching a video, and the generation AI analyzes the data to evaluate the user's level of comprehension. For example, the evaluation unit identifies which parts of the video the user lacks understanding of, based on the user's viewing history when watching the video. The generation unit generates audio secondary based on the level of comprehension evaluated by the evaluation unit. The generation unit uses the generation AI to generate audio secondary based on the content of the video and the user's level of comprehension. For example, the generation unit generates audio commentary that explains important points of a video or that supplements parts that are difficult for the user to understand. The generation AI uses a text generation AI (e.g., LLM) to summarize the content of the video and generate explanatory audio. The attachment unit attaches the audio commentary generated by the generation unit to the video. For example, the attachment unit inserts the generated audio commentary at an appropriate timing in the video to allow the user to watch without feeling uncomfortable. The attachment unit uses the generation AI to adjust the balance between the audio of the video and the audio commentary to maintain a natural flow. As a result, the audio commentary generation system according to the embodiment analyzes the content of the video, generates audio commentary based on the user's level of understanding, and attaches it to the video, thereby deepening the user's understanding.

[0030] The analysis unit analyzes the content of the video. For example, it extracts important points from each video scene and organizes them for easy user understanding. Specifically, the analysis unit sequentially analyzes video frames and uses image recognition and natural language processing technologies to detect scene changes and important events. The generation AI extracts important keyframes from each video scene and analyzes the key points of each scene. For example, in sports game videos, it identifies goal scenes and important plays, and in educational videos, it extracts important concepts and steps. The generation AI also analyzes the video's audio track to detect changes in the speaker's speech and emotions. This allows the analysis unit to comprehensively analyze the video's visual and audio information and efficiently extract information important to the user. Furthermore, the analysis unit utilizes video metadata and subtitle information to perform more accurate analysis. For example, by analyzing subtitle information, it can accurately grasp the speaker's speech and identify important keywords and phrases. This allows the analysis unit to analyze the video content from multiple angles and provide valuable information to users.

[0031] The evaluation unit evaluates the user's level of comprehension based on the content analyzed by the analysis unit. The evaluation unit collects data such as the user's viewing history, viewing time, and playback speed when watching a video, and the generation AI analyzes this data to evaluate the user's level of comprehension. Specifically, the evaluation unit closely monitors the user's viewing behavior and collects data such as at what point playback was stopped, which parts were repeatedly viewed, and whether the playback speed was changed. The generation AI analyzes this data to identify which parts the user is having difficulty understanding. For example, if a user repeatedly watches a particular scene, it determines that that scene is likely to be difficult to understand. The generation AI can also evaluate the user's level of comprehension of the current video based on the user's viewing history, taking into account the content and comprehension level of videos viewed in the past. Furthermore, the evaluation unit can collect user feedback and reflect it in the evaluation of comprehension. For example, it can analyze questionnaires and comments provided by users after watching a video and use them to evaluate comprehension. This allows the evaluation unit to comprehensively analyze the user's viewing behavior and feedback and accurately evaluate comprehension.

[0032] The generation unit generates audio commentary based on the comprehension level evaluated by the evaluation unit. The generation unit uses a generation AI to generate audio commentary based on the video content and the user's comprehension level. Specifically, the generation unit generates audio commentary that explains important points of the video or supplements parts that are difficult for the user to understand. The generation AI uses a text generation AI (e.g., LLM) to summarize the content of the video and generate explanatory audio. For example, in the case of educational videos, the generation AI generates audio commentary that concisely explains important concepts and procedures to make it easier for the user to understand. In addition, in the case of entertainment videos, the generation AI generates audio commentary that explains background information about scenes and details about characters to enrich the viewing experience. The generation unit can adjust the content and tone of the audio commentary according to the user's comprehension level. For example, it can provide more detailed and thorough explanations to users with low comprehension levels, and provide concise and to-the-point explanations to users with high comprehension levels. This allows the generation unit to generate audio commentary customized to the user's needs and improve the viewing experience.

[0033] The audio adder adds the audio secondary generated by the generator to the video. For example, the audio adder inserts the generated audio secondary at the appropriate timing in the video to ensure a natural viewing experience for the user. Specifically, the audio adder adjusts the timing of audio secondary insertion for each video scene to prevent overlapping of the original audio with the audio secondary. The generation AI adjusts the balance between the video's audio and the audio secondary to maintain a natural flow. For example, it emphasizes the audio secondary in scenes where the original audio is quiet and reduces the audio secondary in scenes where the original audio is important. The audio adder also provides a function that allows users to turn the audio secondary on and off and adjust the volume according to their preferences. This allows users to customize the audio secondary to suit their viewing style. Furthermore, the audio adder can provide audio secondary in multiple languages. For example, it generates audio secondary in the user's native language, such as English, Japanese, or Spanish, and adds it to the video. This allows the audio adder to provide audio secondary that is easy to understand even for users around the world.

[0034] The collection unit can collect data on the user's viewing history, viewing time, and pause or rewind. The collection unit, for example, collects the user's viewing history. The viewing history includes the titles of videos viewed, viewing time, viewing frequency, etc. The collection unit analyzes the user's viewing behavior based on the viewing history and provides data for evaluating the user's level of comprehension. The collection unit can also collect viewing time. The viewing time includes the video playback time, viewing start time, viewing end time, etc. The collection unit provides data for evaluating the user's level of concentration and comprehension based on the viewing time. The collection unit can also collect pause and rewind data. The pause data includes the number of pauses, pause duration, pause location, etc. The rewind data includes the number of rewinds, rewind duration, rewind location, etc. The collection unit identifies parts that the user finds difficult to understand based on the pause and rewind data and provides data for evaluating the user's level of comprehension. In this way, the collection unit can more accurately evaluate the user's level of comprehension by collecting the user's viewing history, viewing time, and pause and rewind data. Some or all of the above-described processing in the collection unit may be performed using, for example, AI, or may be performed without using AI. For example, the collection unit may input data on viewing history and viewing time into the generation AI and have the generation AI analyze the data.

[0035] The analysis unit can analyze the content of the video using natural language processing technology. The analysis unit analyzes the content of the video using, for example, natural language processing technology. Natural language processing technology includes morphological analysis, grammatical analysis, semantic analysis, etc. The analysis unit uses morphological analysis to divide the text data of the video into words and identify the part of speech of each word. Uses grammatical analysis to analyze the grammatical structure of the text data and understand the structure of the sentence. Uses semantic analysis to analyze the meaning of the text data and extract important information. For example, the analysis unit performs morphological analysis on the subtitle data of the video and identifies the part of speech of each word. Next, uses grammatical analysis to analyze the grammatical structure of the subtitle data and understand the structure of the sentence. Finally, uses semantic analysis to analyze the meaning of the subtitle data and extract important information. In this way, the analysis unit can analyze the content of the video in more detail by using natural language processing technology. Some or all of the above-mentioned processing in the analysis unit may be performed using, for example, AI, or may be performed without using AI. For example, the analysis unit can input text data of a video into the generation AI and have the generation AI analyze the text data.

[0036] The system includes a registration unit that allows users to register audio commentary. The registration unit allows users to share audio commentary created by themselves with other users. For example, a user can register audio commentary created by themselves in the system, allowing other users to listen to that audio commentary. The registration unit must clarify the audio file format, registration procedure, registration conditions, etc. For example, audio file formats include MP3 format, WAV format, etc. The registration procedure includes uploading the audio file, inputting metadata, and setting the public profile. The registration conditions include audio file quality, appropriateness of the content, and copyright confirmation. This allows the registration unit to share audio commentary created by users with other users. This allows users to share information with each other and deepen their understanding of videos.

[0037] The system includes an advertising unit that inserts advertisements. The advertising unit can insert advertisements into the secondary audio. For example, the advertising unit can insert video advertisements, banner advertisements, text advertisements, etc. into the secondary audio. The advertising unit needs to clarify the content of the advertisements and how they are inserted. For example, video advertisements include product introduction videos and service promotion videos. Banner advertisements include product banner images and service banner images. Text advertisements include product descriptions and service descriptions. The advertising unit needs to clarify how the advertisements are inserted. For example, the advertising unit sets the timing and position at which advertisements are inserted into the secondary audio. The advertising unit adjusts the timing of advertisement insertion according to the video scene, allowing the user to watch without feeling uncomfortable. In this way, the advertising unit can earn advertising revenue by inserting advertisements into the secondary audio. This also allows the video content provider and the secondary audio creator to benefit.

[0038] The analysis unit can apply a different analysis algorithm to each video scene and perform analysis according to the characteristics of the scene. The analysis unit, for example, applies a different analysis algorithm to each video scene. Analysis algorithms applied to each scene include action scenes, dialogue scenes, and landscape scenes. In action scenes, the generation AI focuses on analyzing fast-moving parts. In dialogue scenes, the generation AI uses voice recognition to analyze the conversation in detail. In landscape scenes, the generation AI analyzes the video features and extracts important information. For example, in action scenes, the generation AI analyzes fast-moving parts and identifies important action scenes. In dialogue scenes, the generation AI analyzes the conversation using voice recognition and extracts important lines. In landscape scenes, the generation AI analyzes the video features and identifies important landscape scenes. This allows the analysis unit to perform analysis according to the characteristics of the scene, thereby improving the accuracy of the analysis. Some or all of the above-mentioned processing in the analysis unit may be performed, for example, using AI or without AI. For example, the analysis unit can input data for each scene into the generation AI and have the generation AI perform analysis according to the scene characteristics.

[0039] The analysis unit can analyze the audio and video of a video separately and evaluate the relevance between them. For example, the analysis unit analyzes the audio and video of a video separately. Audio analysis includes feature extraction of audio data, audio recognition, and emotion analysis of the audio. Video analysis includes feature extraction of video data, video recognition, and emotion analysis of the video. The analysis unit analyzes the audio data and extracts important keywords. The analysis unit analyzes the video data and identifies important scenes. Next, the analysis unit evaluates the relevance between the audio and video. For example, the analysis unit evaluates the synchronization between the audio and video, the degree of agreement between the content of the audio and the content of the video, etc. This allows the analysis unit to evaluate the relevance between the audio and video and integrate and analyze important points. Some or all of the above-mentioned processing in the analysis unit may be performed using, for example, AI, or may be performed without using AI. For example, the analysis unit can input the audio data and video data into a generation AI and have the generation AI evaluate the relevance between the audio and video.

[0040] When analyzing a video, the analysis unit can improve the accuracy of the analysis by referring to the user's viewing history. The analysis unit, for example, refers to the user's viewing history. The viewing history includes the titles of videos viewed, viewing time, viewing frequency, etc. The analysis unit analyzes the user's viewing behavior based on the viewing history and improves the accuracy of the analysis. For example, the generation AI analyzes the user's past viewing history and refers to the analysis results of similar videos. The generation AI evaluates the user's understanding of a specific genre from the viewing history and reflects this in the analysis. The generation AI prioritizes analysis of points of interest based on the user's viewing history. In this way, the analysis unit improves the accuracy of the analysis by referring to the user's viewing history. Some or all of the above-mentioned processing in the analysis unit may be performed using AI, for example, or may be performed without using AI. For example, the analysis unit may input viewing history data into the generation AI and have the generation AI improve the accuracy of the analysis.

[0041] When analyzing a video, the analysis unit can perform the analysis by referring to related external data. The analysis unit, for example, refers to related external data. The external data includes subtitle data, related literature, news articles, social media posts, etc. The analysis unit analyzes the subtitle data to complement the content of the video. The analysis unit refers to related literature to reflect background information about the video in the analysis. The analysis unit refers to news articles to obtain the latest information related to the content of the video. The analysis unit refers to social media posts to analyze user reactions to the video. For example, the generation AI analyzes the subtitle data to complement the content of the video. The generation AI refers to related literature to reflect background information about the video in the analysis. The generation AI refers to news articles to obtain the latest information related to the content of the video. The generation AI refers to social media posts to analyze user reactions to the video. In this way, the analysis unit can analyze the content of the video in more detail by referring to related external data. Some or all of the above-mentioned processing in the analysis unit may be performed using AI, for example, or may be performed without using AI. For example, the analysis unit may input external data into the generation AI and have the generation AI analyze the data.

[0042] The evaluation unit can analyze the user's viewing history and viewing time in detail to improve the accuracy of the evaluation of the level of comprehension. The evaluation unit, for example, analyzes the user's viewing history in detail. The viewing history includes the titles of videos viewed, viewing time, viewing frequency, etc. The evaluation unit analyzes the user's viewing behavior based on the viewing history to improve the accuracy of the evaluation of the level of comprehension. For example, the generation AI analyzes the user's viewing history and evaluates the level of comprehension for a specific genre. The generation AI evaluates the user's concentration level based on the viewing time. The generation AI integrates the viewing history and viewing time to improve the accuracy of the evaluation of the level of comprehension. In this way, the evaluation unit improves the accuracy of the evaluation of the level of comprehension by analyzing the viewing history and viewing time in detail. Some or all of the above-mentioned processes in the evaluation unit may be performed using AI, for example, or may be performed without using AI. For example, the evaluation unit may input viewing history data into the generation AI and have the generation AI analyze the data.

[0043] The evaluation unit can reflect the user's operations while watching in the evaluation in real time. The evaluation unit, for example, collects the user's operations while watching in real time and reflects them in the evaluation. Operations while watching include pausing, rewinding, fast-forwarding, etc. The evaluation unit evaluates the user's level of comprehension based on the frequency of pausing. Based on the number of rewinds, it identifies parts that are difficult for the user to understand. Based on the frequency of fast-forwarding, it identifies parts that the user is not interested in. For example, the generation AI collects the frequency of pausing and evaluates the user's level of comprehension. The generation AI collects the number of rewinds and identifies parts that are difficult for the user to understand. The generation AI collects the frequency of fast-forwarding and identifies parts that the user is not interested in. In this way, the evaluation unit reflects the operations while watching in the evaluation in real time, thereby improving the accuracy of the evaluation of the level of comprehension. Some or all of the above-mentioned processes in the evaluation unit may be performed using AI, for example, or may be performed without using AI. For example, the evaluation unit can input operation data during watching into the generation AI and have the generation AI analyze the data.

[0044] When evaluating the level of comprehension, the evaluation unit can improve the accuracy of the evaluation by referring to the user's learning history. The evaluation unit, for example, refers to the user's learning history. The learning history includes topics studied, study time, learning outcomes, etc. The evaluation unit analyzes the user's learning behavior based on the learning history and improves the accuracy of the evaluation of the level of comprehension. For example, the generation AI analyzes the user's learning history and evaluates the level of understanding in a specific field. The generation AI evaluates the user's level of comprehension in detail based on the learning history. The generation AI integrates the learning history and viewing history to improve the accuracy of the evaluation. In this way, the evaluation unit improves the accuracy of the evaluation of the level of comprehension by referring to the learning history. Some or all of the above-mentioned processing in the evaluation unit may be performed using AI, for example, or may be performed without using AI. For example, the evaluation unit may input learning history data into the generation AI and have the generation AI analyze the data.

[0045] When evaluating the level of comprehension, the evaluation unit can adjust the evaluation method by taking into account the user's device information. The evaluation unit, for example, takes into account the user's device information. The device information includes the device type, OS version, screen size, etc. The evaluation unit analyzes the user's viewing behavior based on the device information and adjusts the evaluation method. For example, the generation AI selects the optimal evaluation method based on the user's device information. The generation AI adjusts the display method of the evaluation results by taking into account the device's screen size. The generation AI improves the accuracy of the evaluation based on the device's performance. In this way, the evaluation unit improves the accuracy of the evaluation method by taking the device information into account. Some or all of the above-mentioned processing in the evaluation unit may be performed using AI, for example, or may be performed without using AI. For example, the evaluation unit can input device information into the generation AI and have the generation AI analyze the data.

[0046] The generation unit generates audio subtitles that emphasize important points of the video, thereby helping the user understand. For example, the generation unit generates audio subtitles that emphasize important points of the video. Important points include key frames, important lines, major events, etc. The generation unit uses a generation AI to analyze important scenes in the video and generate audio subtitles corresponding to those scenes. The generation AI extracts key points of the video and generates audio subtitles that emphasize the key points. The generation AI generates audio subtitles that include supplemental information to help understand based on the content of the video. For example, the generation AI analyzes important scenes in the video and generates audio subtitles corresponding to those scenes. The generation AI extracts key points of the video and generates audio subtitles that emphasize the key points. The generation AI generates audio subtitles that include supplemental information to help understand based on the content of the video. In this way, the generation unit generates audio subtitles that emphasize important points, thereby deepening the user's understanding. Some or all of the above-mentioned processing in the generation unit may be performed using, for example, AI, or may be performed without using AI. For example, the generation unit may input data on important points of the video to the generation AI and cause the generation AI to generate audio subtitles.

[0047] The generation unit can generate different audio commentary styles depending on the content of the video. The generation unit generates different audio commentary styles depending on the content of the video, for example. Audio commentary styles include commentary style, narrative style, and dialogue style. The generation unit uses a generation AI to analyze the content of the video and generate audio commentary style. The generation AI generates audio commentary that provides supplemental information based on the content of the video. The generation AI generates audio commentary in question format depending on the content of the video. For example, the generation AI analyzes the content of the video and generates audio commentary style. The generation AI generates audio commentary that provides supplemental information based on the content of the video. The generation AI generates audio commentary in question format depending on the content of the video. In this way, the generation unit generates different audio commentary styles depending on the content of the video, thereby deepening the user's understanding. Some or all of the above-mentioned processing in the generation unit may be performed using AI, for example, or may be performed without using AI. For example, the generation unit may input video content data to the generation AI and cause the generation AI to generate an audio commentary style.

[0048] When generating the secondary audio, the generation unit can improve the accuracy of generation by referring to the user's viewing history. The generation unit, for example, refers to the user's viewing history. The viewing history includes the titles of videos viewed, viewing time, viewing frequency, etc. The generation unit analyzes the user's viewing behavior based on the viewing history and improves the accuracy of the secondary audio generation. For example, the generation AI analyzes the user's viewing history and references the secondary audio of similar videos. The generation AI prioritizes reflecting points of interest to the user in the secondary audio based on the viewing history. The generation AI integrates the viewing history and the comprehension level to improve the accuracy of the secondary audio. In this way, the generation unit improves the accuracy of the secondary audio generation by referring to the user's viewing history. Some or all of the above-mentioned processes in the generation unit may be performed using AI, for example, or may be performed without using AI. For example, the generation unit may input viewing history data to the generation AI and have the generation AI generate the secondary audio.

[0049] The generation unit can generate the secondary audio by referring to related external data. The generation unit references, for example, related external data. External data includes subtitle data, related literature, news articles, social media posts, etc. The generation unit analyzes the subtitle data and reflects it in the secondary audio. The generation unit references related literature to generate secondary audio including supplemental information. The generation unit references news articles to reflect the latest information related to the content of the video in the secondary audio. The generation unit references social media posts to reflect user reactions to the video in the secondary audio. For example, a generation AI analyzes the subtitle data and reflects it in the secondary audio. The generation AI references related literature to generate secondary audio including supplemental information. The generation AI references news articles to reflect the latest information related to the content of the video in the secondary audio. The generation AI references social media posts to reflect user reactions to the video in the secondary audio. In this way, by the generation unit referring to the related external data, the accuracy of generating the secondary audio is improved. Some or all of the above-mentioned processes in the generation unit may be performed, for example, using AI or without using AI. For example, the generation unit can input external data into the generation AI and cause the generation AI to generate secondary audio.

[0050] The adding unit can add a different audio subtitle to each scene in the video, providing audio subtitles that correspond to the characteristics of the scene. For example, the adding unit adds a different audio subtitle to each scene in the video. Audio subtitles applied to each scene include action scenes, dialogue scenes, and landscape scenes. The adding unit uses a generation AI to add audio subtitles that correspond to fast-moving parts in action scenes. In dialogue scenes, audio subtitles that supplement the dialogue content are added. In landscape scenes, audio subtitles that explain the characteristics of the video are added. For example, the generation AI analyzes action scenes and adds audio subtitles that correspond to fast-moving parts. The generation AI analyzes dialogue scenes and adds audio subtitles that supplement the dialogue content. The generation AI analyzes landscape scenes and adds audio subtitles that explain the characteristics of the video. In this way, the adding unit provides audio subtitles that correspond to the characteristics of the scene, thereby deepening the user's understanding. Some or all of the above-mentioned processing in the adding unit may be performed, for example, using AI or without AI. For example, the adding unit can input data for each scene into the generation AI and cause the generation AI to add secondary audio.

[0051] The attachment unit can adjust the balance between the audio and secondary audio of a video to allow the user to watch without feeling uncomfortable. The attachment unit, for example, adjusts the balance between the audio and secondary audio of a video. The balance between the audio and secondary audio includes volume adjustment, audio priority, audio mixing method, etc. The attachment unit uses a generation AI to analyze the audio level of the video and automatically adjust the volume of the secondary audio. The generation AI adjusts the timing of the audio and secondary audio to maintain a natural flow. The generation AI provides an optimal audio balance taking into account the user's device settings. For example, the generation AI analyzes the audio level of a video and automatically adjusts the volume of the secondary audio. The generation AI adjusts the timing of the audio and secondary audio to maintain a natural flow. The generation AI provides an optimal audio balance taking into account the user's device settings. As a result, the attachment unit adjusts the balance between the audio and secondary audio to allow the user to watch without feeling uncomfortable. Some or all of the above-mentioned processing in the attachment unit may be performed using, for example, AI, or may be performed without using AI. For example, the attachment unit may input audio data to the generation AI and cause the generation AI to adjust the audio balance.

[0052] When adding secondary audio, the adding unit can improve the accuracy of the addition by referring to the user's viewing history. The adding unit may, for example, refer to the user's viewing history. The viewing history may include the titles of videos viewed, viewing time, viewing frequency, etc. The adding unit may analyze the user's viewing behavior based on the viewing history and improve the accuracy of the secondary audio addition. For example, the generation AI may analyze the user's viewing history and refer to secondary audio from similar videos. The generation AI may prioritize points of interest to the user based on the viewing history and reflect them in the secondary audio. The generation AI may integrate the viewing history and the comprehension level to improve the accuracy of the secondary audio. In this way, the adding unit may improve the accuracy of the secondary audio addition by referring to the user's viewing history. Some or all of the above-mentioned processing in the adding unit may be performed using AI, for example, or may be performed without using AI. For example, the adding unit may input viewing history data to the generation AI and have the generation AI add secondary audio.

[0053] The adding unit can add the secondary audio by referring to related external data. The adding unit, for example, refers to related external data. The external data includes subtitle data, related literature, news articles, social media posts, etc. The adding unit analyzes the subtitle data and reflects it in the secondary audio. The adding unit refers to related literature and adds secondary audio including supplemental information. The adding unit refers to news articles and reflects the latest information related to the content of the video in the secondary audio. The adding unit refers to social media posts and reflects user reactions to the video in the secondary audio. For example, a generation AI analyzes the subtitle data and reflects it in the secondary audio. The generation AI refers to related literature and adds secondary audio including supplemental information. The generation AI refers to news articles and reflects the latest information related to the content of the video in the secondary audio. The generation AI refers to social media posts and reflects user reactions to the video in the secondary audio. In this way, the adding unit refers to related external data, thereby improving the accuracy of adding secondary audio. Some or all of the above-mentioned processing in the adding unit may be performed, for example, using AI or without using AI. For example, the adding unit can input external data to the generation AI and cause the generation AI to add secondary audio.

[0054] The collection unit can collect detailed information about operations performed by a user while viewing content for analysis. The collection unit, for example, collects detailed information about operations performed by a user while viewing content. Operations performed while viewing content include pausing, rewinding, fast-forwarding, and the like. The collection unit collects the frequency of pauses to evaluate the user's level of understanding. The number of rewinds is collected to identify parts that are difficult for the user to understand. The frequency of fast-forwarding is collected to identify parts that the user is not interested in. For example, a generation AI collects the frequency of pauses to evaluate the user's level of understanding. The generation AI collects the number of rewinds to identify parts that are difficult for the user to understand. The generation AI collects the frequency of fast-forwarding to identify parts that the user is not interested in. In this way, the collection unit collects detailed information about operations performed while viewing content, thereby improving the accuracy of analysis. Some or all of the above-described processing in the collection unit may be performed using AI, for example, or may be performed without using AI. For example, the collection unit can input data about operations performed while viewing content into a generation AI and have the generation AI analyze the data.

[0055] The collection unit collects information about the user's viewing environment to improve the accuracy of analysis. The collection unit, for example, collects information about the user's viewing environment. The viewing environment includes the type of device, network environment, viewing location, etc. The collection unit collects device information and selects the optimal analysis method. The collection unit collects network status and adjusts the data collection speed. The collection unit collects viewing locations and selects an analysis method according to the environment. For example, a generation AI collects information about the user's device and selects the optimal analysis method. The generation AI collects network status and adjusts the data collection speed. The generation AI collects viewing locations and selects an analysis method according to the environment. In this way, the collection unit collects information about the viewing environment to improve the accuracy of analysis. Some or all of the above-mentioned processing in the collection unit may be performed using AI, for example, or may be performed without using AI. For example, the collection unit may input viewing environment data into a generation AI and have the generation AI analyze the data.

[0056] When collecting data, the collection unit can prioritize collecting highly relevant data by taking into account the user's geographical location information. The collection unit, for example, considers the user's geographical location information. The geographical location information includes GPS data, IP address, location information services, etc. The collection unit collects relevant data based on the geographical location information. For example, the generation AI collects relevant data based on the user's geographical location information. The generation AI selects an optimal data collection method by taking into account the geographical location information. The generation AI improves the accuracy of data collection based on the geographical location information. In this way, the collection unit can prioritize collecting highly relevant data by taking into account the geographical location information. Some or all of the above-mentioned processing in the collection unit may be performed using AI, for example, or may be performed without using AI. For example, the collection unit can input geographical location information data to the generation AI and cause the generation AI to collect data.

[0057] When collecting data, the collection unit can analyze the user's social media activity and collect related data. The collection unit, for example, analyzes the user's social media activity. The social media activity includes the content of posts, the number of likes, the number of followers, etc. The collection unit collects related data based on the social media activity. For example, the generation AI analyzes the user's social media activity and collects related data. The generation AI identifies the user's interests based on the content of social media posts. The generation AI improves the accuracy of the collected data based on the social media data. As a result, the collection unit analyzes the social media activity and improves the accuracy of the collection of related data. Some or all of the above-mentioned processing in the collection unit may be performed using, for example, AI, or may be performed without using AI. For example, the collection unit may input social media data into the generation AI and have the generation AI collect the data.

[0058] The registration unit can evaluate the quality of the secondary audio registered by the user and provide appropriate feedback. The registration unit, for example, evaluates the quality of the secondary audio registered by the user. The quality of the secondary audio includes audio clarity, accuracy of the content, and the user's evaluation. The registration unit uses a generation AI to evaluate the sound quality of the secondary audio and provide feedback on areas for improvement. The generation AI evaluates the content of the secondary audio and provides supplementary information. The generation AI evaluates the overall quality of the secondary audio and provides feedback to the user. For example, the generation AI evaluates the sound quality of the secondary audio and provides feedback on areas for improvement. The generation AI evaluates the content of the secondary audio and provides supplementary information. The generation AI evaluates the overall quality of the secondary audio and provides feedback to the user. In this way, the registration unit evaluates the quality of the secondary audio and provides appropriate feedback, thereby improving the quality of the secondary audio for the user. Some or all of the above-mentioned processing in the registration unit may be performed using, for example, AI, or may be performed without using AI. For example, the registration unit can input secondary audio data to the generation AI and cause the generation AI to evaluate the quality and provide feedback.

[0059] The registration unit automatically categorizes the audio commentary registered by the user, making it easier for other users to search. For example, the registration unit automatically categorizes the audio commentary registered by the user. Audio commentary categories include education, entertainment, and news. The registration unit uses a generation AI to analyze the content of the audio commentary and classify it into an appropriate category. The generation AI extracts keywords from the audio commentary and tags them for easy search. The generation AI automatically updates the category of the audio commentary to reflect the latest information. For example, the generation AI analyzes the content of the audio commentary and classifies it into an appropriate category. The generation AI extracts keywords from the audio commentary and tags them for easy search. The generation AI automatically updates the category of the audio commentary to reflect the latest information. In this way, the registration unit automatically categorizes the audio commentary, making it easier for other users to search. Some or all of the above-described processing in the registration unit may be performed using, for example, AI, or may be performed without using AI. For example, the registration unit may input audio commentary data to the generation AI and have the generation AI classify the category.

[0060] The registration unit can improve the accuracy of registration by referring to the user's viewing history when registering the secondary audio. The registration unit, for example, refers to the user's viewing history. The viewing history includes the titles of videos viewed, viewing time, viewing frequency, etc. The registration unit analyzes the user's viewing behavior based on the viewing history and improves the accuracy of the secondary audio registration. For example, the generation AI analyzes the user's viewing history and references the secondary audio of similar videos. The generation AI prioritizes reflecting points of interest to the user in the secondary audio based on the viewing history. The generation AI integrates the viewing history and the comprehension level to improve the accuracy of the secondary audio. In this way, the registration unit improves the accuracy of the secondary audio registration by referring to the user's viewing history. Some or all of the above-mentioned processes in the registration unit may be performed using AI, for example, or may be performed without using AI. For example, the registration unit may input viewing history data to the generation AI and have the generation AI register the secondary audio.

[0061] The registration unit can refer to related external data when registering the secondary audio. The registration unit refers to, for example, related external data. The external data includes subtitle data, related literature, news articles, social media posts, etc. The registration unit analyzes the subtitle data and reflects it in the secondary audio. The registration unit refers to related literature and registers secondary audio including supplemental information. The registration unit refers to news articles and reflects the latest information related to the content of the video in the secondary audio. The registration unit refers to social media posts and reflects user reactions to the video in the secondary audio. For example, a generation AI analyzes the subtitle data and reflects it in the secondary audio. The generation AI refers to related literature and registers secondary audio including supplemental information. The generation AI refers to news articles and reflects the latest information related to the content of the video in the secondary audio. The generation AI refers to social media posts and reflects user reactions to the video in the secondary audio. In this way, by the registration unit referring to the related external data, the accuracy of registering the secondary audio is improved. Some or all of the above-mentioned processing in the registration unit may be performed, for example, using AI or without using AI. For example, the registration unit can input external data into the generation AI and cause the generation AI to register the secondary audio.

[0062] The advertising unit can insert different advertisements for each scene of the video and provide advertisements according to the characteristics of the scene. The advertising unit, for example, inserts different advertisements for each scene of the video. Advertisements applied to each scene include action scenes, dialogue scenes, landscape scenes, etc. The advertising unit uses a generation AI to insert advertisements corresponding to fast-moving parts in action scenes. In dialogue scenes, it inserts advertisements related to the content of the conversation. In landscape scenes, it inserts advertisements related to the characteristics of the video. For example, the generation AI analyzes an action scene and inserts advertisements corresponding to fast-moving parts. The generation AI analyzes a dialogue scene and inserts advertisements related to the content of the conversation. The generation AI analyzes a landscape scene and inserts advertisements related to the characteristics of the video. In this way, the advertising unit provides advertisements according to the characteristics of the scene, thereby inserting advertisements that attract the user's interest. Some or all of the above-mentioned processing in the advertising unit may be performed using, for example, AI, or may be performed without using AI. For example, the advertising unit may input data for each scene into the generation AI and cause the generation AI to insert advertisements.

[0063] The advertising unit can automatically select advertisements related to the content of the video and insert advertisements that interest the user. The advertising unit automatically selects advertisements related to the content of the video, for example. Advertisements include product introduction videos, service promotion videos, banner advertisements, and text advertisements. The advertising unit uses a generation AI to analyze the content of the video and select advertisements for related products and services. The generation AI inserts advertisements of interest to the user based on the user's viewing history. The generation AI inserts advertisements at the optimal timing depending on the scene in the video. For example, the generation AI analyzes the content of the video and selects advertisements for related products and services. The generation AI inserts advertisements of interest to the user based on the user's viewing history. The generation AI inserts advertisements at the optimal timing depending on the scene in the video. In this way, the advertising unit provides advertisements that interest the user by inserting advertisements related to the content of the video. Some or all of the above-described processing in the advertising unit may be performed using, for example, AI, or may be performed without using AI. For example, the advertising unit may input video content data into the generation AI and cause the generation AI to select and insert advertisements.

[0064] When inserting an advertisement, the advertising unit can improve the accuracy of the advertisement by referring to the user's viewing history. The advertising unit, for example, refers to the user's viewing history. The viewing history includes the titles of videos viewed, viewing time, viewing frequency, etc. The advertising unit analyzes the user's viewing behavior based on the viewing history and improves the accuracy of the advertisement. For example, the generation AI analyzes the user's viewing history and selects relevant advertisements. The generation AI preferentially inserts advertisements that are of interest to the user based on the viewing history. The generation AI integrates the viewing history and comprehension level to improve the accuracy of the advertisement. In this way, the advertising unit improves the accuracy of the advertisement by referring to the user's viewing history. Some or all of the above-mentioned processing in the advertising unit may be performed using AI, for example, or may be performed without using AI. For example, the advertising unit can input viewing history data into the generation AI and have the generation AI select and insert advertisements.

[0065] When inserting an advertisement, the advertising unit can select an advertisement by referring to related external data. The advertising unit, for example, refers to related external data. The external data includes the user's purchase history, interest data, news articles, social media posts, etc. The advertising unit analyzes the purchase history and selects a relevant advertisement. The advertising unit inserts an advertisement that is of interest to the user based on the interest data. The advertising unit refers to news articles and reflects the latest information related to the content of the video in the advertisement. The advertising unit refers to social media posts and reflects the user's reaction to the video in the advertisement. For example, the generation AI analyzes the user's purchase history and selects a relevant advertisement. The generation AI inserts an advertisement that is of interest to the user based on the interest data. The generation AI refers to news articles and reflects the latest information related to the content of the video in the advertisement. The generation AI refers to social media posts and reflects the user's reaction to the video in the advertisement. In this way, the advertising unit improves the accuracy of the advertisement by referring to the related external data. Some or all of the above-mentioned processing in the advertising unit may be performed, for example, using AI or without using AI. For example, the advertising department can input external data into the generation AI and have the generation AI select and insert advertisements.

[0066] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0067] When analyzing the content of a video, the analysis unit can apply an analysis algorithm that is appropriate for the video genre. For example, in the case of an educational video, the generation AI will perform an analysis that emphasizes educational elements and enhances learning effectiveness. In the case of an entertainment video, the generation AI will perform an analysis that emphasizes visual elements and storytelling. In the case of a news video, the generation AI will perform an analysis that emphasizes fact-checking and extracting important information. This allows the analysis unit to generate more appropriate secondary audio by performing an analysis that is appropriate for the video genre.

[0068] The collection unit can collect biometric data such as the user's heart rate and electrodermal response while watching, in addition to the user's viewing history and viewing time. For example, the collection unit can collect heart rate fluctuations while the user is watching a video to evaluate the user's level of excitement and concentration. The collection unit can collect electrodermal response to evaluate the user's emotional response. This allows the collection unit to more accurately evaluate the user's level of comprehension based on the user's biometric data.

[0069] When analyzing the content of a video, the analysis unit can perform an integrated analysis of the video's visual and audio elements. For example, it can analyze the synchronization of the video's video and audio to identify important scenes and lines, analyze the color and composition of the video to extract visually important points, and analyze the tone and rhythm of the audio to identify emotional elements. This allows the analysis unit to generate richer audio descriptions by taking both visual and audio into account.

[0070] When a user registers a secondary audio track, the registration unit can automatically convert it into text using voice recognition technology and register it as text data. For example, when a user registers a secondary audio track by voice, the voice recognition technology converts the content into text and saves it as text data. This makes it easier for other users to search the text data and share the content of the secondary audio track more easily.

[0071] The advertising unit can generate personalized advertisements based on the user's viewing history and viewing time. For example, it generates advertisements for related products based on the genre and content of videos the user has previously viewed. It inserts advertisements at appropriate times depending on the user's viewing time period. This allows the advertising unit to provide advertisements that match the user's interests and concerns, thereby increasing advertising effectiveness.

[0072] The processing flow of the first embodiment will be briefly explained below.

[0073] Step 1: The analysis unit analyzes the content of the video. For example, the analysis unit extracts important points from each scene in the video and organizes them so that they are easy for users to understand. The analysis unit uses the generation AI to analyze the content of the video and identify important points. For example, the generation AI extracts important keyframes from each scene in the video and analyzes the main points of that scene. Step 2: The evaluation unit evaluates the user's level of understanding based on the content analyzed by the analysis unit. For example, the evaluation unit collects data such as the user's viewing history, viewing time, and playback speed when watching a video, and the generation AI analyzes this data to evaluate the user's level of understanding. For example, the evaluation unit identifies which parts of the video the user is lacking in understanding based on the viewing history when watching the video. Step 3: The generation unit generates audio commentary based on the comprehension level assessed by the evaluation unit. The generation unit uses a generation AI to generate audio commentary based on the content of the video and the user's level of comprehension. For example, the generation unit generates audio commentary that explains important points of the video or supplements parts that are difficult for the user to understand. The generation AI uses a text generation AI (e.g., LLM) to summarize the content of the video and generate explanatory audio. Step 4: The audio adder adds the audio commentary generated by the generator to the video. For example, the audio adder inserts the generated audio commentary at the appropriate timing in the video to ensure a natural viewing experience for the user. The audio adder uses the generation AI to adjust the balance between the video audio and the audio commentary to maintain a natural flow.

[0074] (Example 2) A secondary audio generation system according to an embodiment of the present invention is a system for deepening understanding of video content. This secondary audio generation system analyzes the content of a video, evaluates the user's level of comprehension, and then uses a generation AI to generate secondary audio based on the video content and the user's level of comprehension, and then adds the secondary audio to the video. This secondary audio blends in with the video, allowing the user to enjoy the video content without any sense of incongruity. Furthermore, the secondary audio itself can be registered by the user as content, and business value can be added by inserting adfusion-style advertisements. For example, the secondary audio generation system analyzes the content of a video. For example, it extracts important points from each scene of the video and organizes them to make them easier for the user to understand. In this process, the generation AI analyzes the content of the video and identifies the important points. Next, the secondary audio generation system evaluates the user's level of comprehension. For example, data such as the user's viewing history, viewing time, and playback speed are collected when the user watches a video, and the generation AI analyzes this data to evaluate the user's level of comprehension. This allows the user to identify areas where their understanding is lacking. The generation AI generates secondary audio based on the content of the video and the user's level of comprehension. For example, it generates secondary audio that explains important points of the video or supplements parts that are difficult for the user to understand. This audio commentary blends in with the video, allowing users to enjoy the video content without any sense of incongruity. Furthermore, the audio commentary itself can be registered as content by the user. For example, users can share audio commentary they have created with other users. This allows users to share information with each other and deepen their understanding of the video. Business value can also be added by inserting adfusion-like advertisements. For example, by inserting advertisements into the audio commentary, advertising revenue can be earned. This also benefits the video content provider and the audio commentary creator. This system deepens understanding of the video content and allows users to enjoy a more fulfilling viewing experience. Business value is also increased by sharing audio commentary and advertising revenue. This allows the audio commentary generation system to deepen understanding of video content and provide users with a fulfilling viewing experience.

[0075] The audio secondary generation system according to the embodiment includes an analysis unit, an evaluation unit, a generation unit, and an assignment unit. The analysis unit analyzes the content of a video. For example, the analysis unit extracts important points from each scene of the video and organizes them to make them easier for the user to understand. The analysis unit uses a generation AI to analyze the content of the video and identify important points. For example, the generation AI extracts important keyframes from each scene of the video and analyzes the main points of each scene. The evaluation unit evaluates the user's level of comprehension based on the content analyzed by the analysis unit. The evaluation unit collects data such as the user's viewing history, viewing time, and playback speed when watching a video, and the generation AI analyzes the data to evaluate the user's level of comprehension. For example, the evaluation unit identifies which parts of the video the user lacks understanding of, based on the user's viewing history when watching the video. The generation unit generates audio secondary based on the level of comprehension evaluated by the evaluation unit. The generation unit uses the generation AI to generate audio secondary based on the content of the video and the user's level of comprehension. For example, the generation unit generates audio commentary that explains important points of a video or that supplements parts that are difficult for the user to understand. The generation AI uses a text generation AI (e.g., LLM) to summarize the content of the video and generate explanatory audio. The attachment unit attaches the audio commentary generated by the generation unit to the video. For example, the attachment unit inserts the generated audio commentary at an appropriate timing in the video to allow the user to watch without feeling uncomfortable. The attachment unit uses the generation AI to adjust the balance between the audio of the video and the audio commentary to maintain a natural flow. As a result, the audio commentary generation system according to the embodiment analyzes the content of the video, generates audio commentary based on the user's level of understanding, and attaches it to the video, thereby deepening the user's understanding.

[0076] The analysis unit analyzes the content of the video. For example, it extracts important points from each video scene and organizes them for easy user understanding. Specifically, the analysis unit sequentially analyzes video frames and uses image recognition and natural language processing technologies to detect scene changes and important events. The generation AI extracts important keyframes from each video scene and analyzes the key points of each scene. For example, in sports game videos, it identifies goal scenes and important plays, and in educational videos, it extracts important concepts and steps. The generation AI also analyzes the video's audio track to detect changes in the speaker's speech and emotions. This allows the analysis unit to comprehensively analyze the video's visual and audio information and efficiently extract information important to the user. Furthermore, the analysis unit utilizes video metadata and subtitle information to perform more accurate analysis. For example, by analyzing subtitle information, it can accurately grasp the speaker's speech and identify important keywords and phrases. This allows the analysis unit to analyze the video content from multiple angles and provide valuable information to users.

[0077] The evaluation unit evaluates the user's level of comprehension based on the content analyzed by the analysis unit. The evaluation unit collects data such as the user's viewing history, viewing time, and playback speed when watching a video, and the generation AI analyzes this data to evaluate the user's level of comprehension. Specifically, the evaluation unit closely monitors the user's viewing behavior and collects data such as at what point playback was stopped, which parts were repeatedly viewed, and whether the playback speed was changed. The generation AI analyzes this data to identify which parts the user is having difficulty understanding. For example, if a user repeatedly watches a particular scene, it determines that that scene is likely to be difficult to understand. The generation AI can also evaluate the user's level of comprehension of the current video based on the user's viewing history, taking into account the content and comprehension level of videos viewed in the past. Furthermore, the evaluation unit can collect user feedback and reflect it in the evaluation of comprehension. For example, it can analyze questionnaires and comments provided by users after watching a video and use them to evaluate comprehension. This allows the evaluation unit to comprehensively analyze the user's viewing behavior and feedback and accurately evaluate comprehension.

[0078] The generation unit generates audio commentary based on the comprehension level evaluated by the evaluation unit. The generation unit uses a generation AI to generate audio commentary based on the video content and the user's comprehension level. Specifically, the generation unit generates audio commentary that explains important points of the video or supplements parts that are difficult for the user to understand. The generation AI uses a text generation AI (e.g., LLM) to summarize the content of the video and generate explanatory audio. For example, in the case of educational videos, the generation AI generates audio commentary that concisely explains important concepts and procedures to make it easier for the user to understand. In addition, in the case of entertainment videos, the generation AI generates audio commentary that explains background information about scenes and details about characters to enrich the viewing experience. The generation unit can adjust the content and tone of the audio commentary according to the user's comprehension level. For example, it can provide more detailed and thorough explanations to users with low comprehension levels, and provide concise and to-the-point explanations to users with high comprehension levels. This allows the generation unit to generate audio commentary customized to the user's needs and improve the viewing experience.

[0079] The audio adder adds the audio secondary generated by the generator to the video. For example, the audio adder inserts the generated audio secondary at the appropriate timing in the video to ensure a natural viewing experience for the user. Specifically, the audio adder adjusts the timing of audio secondary insertion for each video scene to prevent overlapping of the original audio with the audio secondary. The generation AI adjusts the balance between the video's audio and the audio secondary to maintain a natural flow. For example, it emphasizes the audio secondary in scenes where the original audio is quiet and reduces the audio secondary in scenes where the original audio is important. The audio adder also provides a function that allows users to turn the audio secondary on and off and adjust the volume according to their preferences. This allows users to customize the audio secondary to suit their viewing style. Furthermore, the audio adder can provide audio secondary in multiple languages. For example, it generates audio secondary in the user's native language, such as English, Japanese, or Spanish, and adds it to the video. This allows the audio adder to provide audio secondary that is easy to understand even for users around the world.

[0080] The collection unit can collect data on the user's viewing history, viewing time, and pause or rewind. The collection unit, for example, collects the user's viewing history. The viewing history includes the titles of videos viewed, viewing time, viewing frequency, etc. The collection unit analyzes the user's viewing behavior based on the viewing history and provides data for evaluating the user's level of comprehension. The collection unit can also collect viewing time. The viewing time includes the video playback time, viewing start time, viewing end time, etc. The collection unit provides data for evaluating the user's level of concentration and comprehension based on the viewing time. The collection unit can also collect pause and rewind data. The pause data includes the number of pauses, pause duration, pause location, etc. The rewind data includes the number of rewinds, rewind duration, rewind location, etc. The collection unit identifies parts that the user finds difficult to understand based on the pause and rewind data and provides data for evaluating the user's level of comprehension. In this way, the collection unit can more accurately evaluate the user's level of comprehension by collecting the user's viewing history, viewing time, and pause and rewind data. Some or all of the above-described processing in the collection unit may be performed using, for example, AI, or may be performed without using AI. For example, the collection unit may input data on viewing history and viewing time into the generation AI and have the generation AI analyze the data.

[0081] The analysis unit can analyze the content of the video using natural language processing technology. The analysis unit analyzes the content of the video using, for example, natural language processing technology. Natural language processing technology includes morphological analysis, grammatical analysis, semantic analysis, etc. The analysis unit uses morphological analysis to divide the text data of the video into words and identify the part of speech of each word. Uses grammatical analysis to analyze the grammatical structure of the text data and understand the structure of the sentence. Uses semantic analysis to analyze the meaning of the text data and extract important information. For example, the analysis unit performs morphological analysis on the subtitle data of the video and identifies the part of speech of each word. Next, uses grammatical analysis to analyze the grammatical structure of the subtitle data and understand the structure of the sentence. Finally, uses semantic analysis to analyze the meaning of the subtitle data and extract important information. In this way, the analysis unit can analyze the content of the video in more detail by using natural language processing technology. Some or all of the above-mentioned processing in the analysis unit may be performed using, for example, AI, or may be performed without using AI. For example, the analysis unit can input text data of a video into the generation AI and have the generation AI analyze the text data.

[0082] The system includes a registration unit that allows users to register audio commentary. The registration unit allows users to share audio commentary created by themselves with other users. For example, a user can register audio commentary created by themselves in the system, allowing other users to listen to that audio commentary. The registration unit must clarify the audio file format, registration procedure, registration conditions, etc. For example, audio file formats include MP3 format, WAV format, etc. The registration procedure includes uploading the audio file, inputting metadata, and setting the public profile. The registration conditions include audio file quality, appropriateness of the content, and copyright confirmation. This allows the registration unit to share audio commentary created by users with other users. This allows users to share information with each other and deepen their understanding of videos.

[0083] The system includes an advertising unit that inserts advertisements. The advertising unit can insert advertisements into the secondary audio. For example, the advertising unit can insert video advertisements, banner advertisements, text advertisements, etc. into the secondary audio. The advertising unit needs to clarify the content of the advertisements and how they are inserted. For example, video advertisements include product introduction videos and service promotion videos. Banner advertisements include product banner images and service banner images. Text advertisements include product descriptions and service descriptions. The advertising unit needs to clarify how the advertisements are inserted. For example, the advertising unit sets the timing and position at which advertisements are inserted into the secondary audio. The advertising unit adjusts the timing of advertisement insertion according to the video scene, allowing the user to watch without feeling uncomfortable. In this way, the advertising unit can earn advertising revenue by inserting advertisements into the secondary audio. This also allows the video content provider and the secondary audio creator to benefit.

[0084] The analysis unit can estimate the user's emotions and adjust the video analysis method based on the estimated user emotions. The analysis unit, for example, estimates the user's emotions. Specific methods for estimating the user's emotions include facial expression recognition, voice analysis, and survey results. The analysis unit uses facial expression recognition to estimate emotions from the user's facial expressions. Voice analysis can estimate emotions from the user's tone and speed of voice. Survey results can be used to estimate the user's self-reported emotions. The analysis unit adjusts the video analysis method based on the estimated user emotions. For example, if the user is excited, the generation AI can quickly analyze the video and emphasize important points. If the user is relaxed, the generation AI can perform a detailed analysis and extract even the finer points. If the user is tired, the generation AI can perform a concise analysis and extract only the main points. This allows the analysis unit to adjust the video analysis method according to the user's emotions, enabling more appropriate analysis. Emotion estimation is achieved, for example, using an emotion estimation function using an emotion engine or generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the analysis unit may be performed using AI, or may be performed without using AI. For example, the analysis unit may input user emotion data into the generation AI and cause the generation AI to adjust the analysis method based on the emotion.

[0085] The analysis unit can apply a different analysis algorithm to each video scene and perform analysis according to the characteristics of the scene. The analysis unit, for example, applies a different analysis algorithm to each video scene. Analysis algorithms applied to each scene include action scenes, dialogue scenes, and landscape scenes. In action scenes, the generation AI focuses on analyzing fast-moving parts. In dialogue scenes, the generation AI uses voice recognition to analyze the conversation in detail. In landscape scenes, the generation AI analyzes the video features and extracts important information. For example, in action scenes, the generation AI analyzes fast-moving parts and identifies important action scenes. In dialogue scenes, the generation AI analyzes the conversation using voice recognition and extracts important lines. In landscape scenes, the generation AI analyzes the video features and identifies important landscape scenes. This allows the analysis unit to perform analysis according to the characteristics of the scene, thereby improving the accuracy of the analysis. Some or all of the above-mentioned processing in the analysis unit may be performed, for example, using AI or without AI. For example, the analysis unit can input data for each scene into the generation AI and have the generation AI perform analysis according to the scene characteristics.

[0086] The analysis unit can analyze the audio and video of a video separately and evaluate the relevance between them. For example, the analysis unit analyzes the audio and video of a video separately. Audio analysis includes feature extraction of audio data, audio recognition, and emotion analysis of the audio. Video analysis includes feature extraction of video data, video recognition, and emotion analysis of the video. The analysis unit analyzes the audio data and extracts important keywords. The analysis unit analyzes the video data and identifies important scenes. Next, the analysis unit evaluates the relevance between the audio and video. For example, the analysis unit evaluates the synchronization between the audio and video, the degree of agreement between the content of the audio and the content of the video, etc. This allows the analysis unit to evaluate the relevance between the audio and video and integrate and analyze important points. Some or all of the above-mentioned processing in the analysis unit may be performed using, for example, AI, or may be performed without using AI. For example, the analysis unit can input the audio data and video data into a generation AI and have the generation AI evaluate the relevance between the audio and video.

[0087] The analysis unit can estimate the user's emotions and adjust the display method of the analysis results based on the estimated user emotions. The analysis unit, for example, estimates the user's emotions. Specific methods for estimating the user's emotions include facial expression recognition, voice analysis, and survey results. The analysis unit uses facial expression recognition to estimate the emotion from the user's facial expressions. Voice analysis to estimate the emotion from the tone and speed of the user's voice. Survey results are used to estimate the user's self-reported emotions. The analysis unit adjusts the display method of the analysis results based on the estimated user emotions. For example, if the user is excited, the generation AI provides a visually stimulating display method. If the user is relaxed, the generation AI provides a calming display method. If the user is tired, the generation AI provides a simple, highly visible display method. This allows the analysis unit to adjust the display method of the analysis results according to the user's emotions, enabling more appropriate display. Emotion estimation is achieved, for example, using an emotion estimation function using an emotion engine or generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the analysis unit may be performed using AI, or may be performed without using AI. For example, the analysis unit may input user emotion data into the generation AI and cause the generation AI to adjust the display method based on the emotion.

[0088] When analyzing a video, the analysis unit can improve the accuracy of the analysis by referring to the user's viewing history. The analysis unit, for example, refers to the user's viewing history. The viewing history includes the titles of videos viewed, viewing time, viewing frequency, etc. The analysis unit analyzes the user's viewing behavior based on the viewing history and improves the accuracy of the analysis. For example, the generation AI analyzes the user's past viewing history and refers to the analysis results of similar videos. The generation AI evaluates the user's understanding of a specific genre from the viewing history and reflects this in the analysis. The generation AI prioritizes analysis of points of interest based on the user's viewing history. In this way, the analysis unit improves the accuracy of the analysis by referring to the user's viewing history. Some or all of the above-mentioned processing in the analysis unit may be performed using AI, for example, or may be performed without using AI. For example, the analysis unit may input viewing history data into the generation AI and have the generation AI improve the accuracy of the analysis.

[0089] When analyzing a video, the analysis unit can perform the analysis by referring to related external data. The analysis unit, for example, refers to related external data. The external data includes subtitle data, related literature, news articles, social media posts, etc. The analysis unit analyzes the subtitle data to complement the content of the video. The analysis unit refers to related literature to reflect background information about the video in the analysis. The analysis unit refers to news articles to obtain the latest information related to the content of the video. The analysis unit refers to social media posts to analyze user reactions to the video. For example, the generation AI analyzes the subtitle data to complement the content of the video. The generation AI refers to related literature to reflect background information about the video in the analysis. The generation AI refers to news articles to obtain the latest information related to the content of the video. The generation AI refers to social media posts to analyze user reactions to the video. In this way, the analysis unit can analyze the content of the video in more detail by referring to related external data. Some or all of the above-mentioned processing in the analysis unit may be performed using AI, for example, or may be performed without using AI. For example, the analysis unit may input external data into the generation AI and have the generation AI analyze the data.

[0090] The evaluation unit can estimate the user's emotions and adjust the evaluation criteria for comprehension based on the estimated user emotions. The evaluation unit, for example, estimates the user's emotions. Specific methods for estimating the user's emotions include facial expression recognition, voice analysis, and survey results. The evaluation unit uses facial expression recognition to estimate the user's emotions from their facial expressions. Voice analysis to estimate the user's emotions from the tone and speed of their voice. Survey results are used to estimate the user's self-reported emotions. The evaluation unit adjusts the evaluation criteria for comprehension based on the estimated user emotions. For example, if the user is excited, the generation AI tightens the evaluation criteria for comprehension. If the user is relaxed, the generation AI loosens the evaluation criteria for comprehension. If the user is tired, the generation AI simplifies the evaluation criteria for comprehension. This allows the evaluation unit to adjust the evaluation criteria for comprehension according to the user's emotions, enabling a more appropriate evaluation. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the evaluation unit may be performed using AI, or may be performed without using AI. For example, the evaluation unit may input user emotion data into the generation AI and cause the generation AI to adjust the evaluation criteria based on the emotion.

[0091] The evaluation unit can analyze the user's viewing history and viewing time in detail to improve the accuracy of the evaluation of the level of comprehension. The evaluation unit, for example, analyzes the user's viewing history in detail. The viewing history includes the titles of videos viewed, viewing time, viewing frequency, etc. The evaluation unit analyzes the user's viewing behavior based on the viewing history to improve the accuracy of the evaluation of the level of comprehension. For example, the generation AI analyzes the user's viewing history and evaluates the level of comprehension for a specific genre. The generation AI evaluates the user's concentration level based on the viewing time. The generation AI integrates the viewing history and viewing time to improve the accuracy of the evaluation of the level of comprehension. In this way, the evaluation unit improves the accuracy of the evaluation of the level of comprehension by analyzing the viewing history and viewing time in detail. Some or all of the above-mentioned processes in the evaluation unit may be performed using AI, for example, or may be performed without using AI. For example, the evaluation unit may input viewing history data into the generation AI and have the generation AI analyze the data.

[0092] The evaluation unit can reflect the user's operations while watching in the evaluation in real time. The evaluation unit, for example, collects the user's operations while watching in real time and reflects them in the evaluation. Operations while watching include pausing, rewinding, fast-forwarding, etc. The evaluation unit evaluates the user's level of comprehension based on the frequency of pausing. Based on the number of rewinds, it identifies parts that are difficult for the user to understand. Based on the frequency of fast-forwarding, it identifies parts that the user is not interested in. For example, the generation AI collects the frequency of pausing and evaluates the user's level of comprehension. The generation AI collects the number of rewinds and identifies parts that are difficult for the user to understand. The generation AI collects the frequency of fast-forwarding and identifies parts that the user is not interested in. In this way, the evaluation unit reflects the operations while watching in the evaluation in real time, thereby improving the accuracy of the evaluation of the level of comprehension. Some or all of the above-mentioned processes in the evaluation unit may be performed using AI, for example, or may be performed without using AI. For example, the evaluation unit can input operation data during watching into the generation AI and have the generation AI analyze the data.

[0093] The evaluation unit can estimate the user's emotions and adjust the order in which the comprehension assessment results are displayed based on the estimated user emotions. The evaluation unit, for example, estimates the user's emotions. Specific methods for estimating the user's emotions include facial expression recognition, voice analysis, and survey results. The evaluation unit uses facial expression recognition to estimate the user's emotions from the user's facial expressions. Voice analysis to estimate the user's emotions from the tone and speed of the user's voice. Survey results are used to estimate the user's self-reported emotions. The evaluation unit adjusts the order in which the comprehension assessment results are displayed based on the estimated user emotions. For example, if the user is excited, the generation AI displays important assessment results first. If the user is relaxed, the generation AI displays detailed assessment results sequentially. If the user is tired, the generation AI displays concise assessment results first. This allows the evaluation unit to adjust the display order of the assessment results according to the user's emotions, enabling more appropriate display. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the evaluation unit may be performed using AI, or may be performed without using AI. For example, the evaluation unit may input user emotion data into the generation AI and cause the generation AI to adjust the display order based on the emotion.

[0094] When evaluating the level of comprehension, the evaluation unit can improve the accuracy of the evaluation by referring to the user's learning history. The evaluation unit, for example, refers to the user's learning history. The learning history includes topics studied, study time, learning outcomes, etc. The evaluation unit analyzes the user's learning behavior based on the learning history and improves the accuracy of the evaluation of the level of comprehension. For example, the generation AI analyzes the user's learning history and evaluates the level of understanding in a specific field. The generation AI evaluates the user's level of comprehension in detail based on the learning history. The generation AI integrates the learning history and viewing history to improve the accuracy of the evaluation. In this way, the evaluation unit improves the accuracy of the evaluation of the level of comprehension by referring to the learning history. Some or all of the above-mentioned processing in the evaluation unit may be performed using AI, for example, or may be performed without using AI. For example, the evaluation unit may input learning history data into the generation AI and have the generation AI analyze the data.

[0095] When evaluating the level of comprehension, the evaluation unit can adjust the evaluation method by taking into account the user's device information. The evaluation unit, for example, takes into account the user's device information. The device information includes the device type, OS version, screen size, etc. The evaluation unit analyzes the user's viewing behavior based on the device information and adjusts the evaluation method. For example, the generation AI selects the optimal evaluation method based on the user's device information. The generation AI adjusts the display method of the evaluation results by taking into account the device's screen size. The generation AI improves the accuracy of the evaluation based on the device's performance. In this way, the evaluation unit improves the accuracy of the evaluation method by taking the device information into account. Some or all of the above-mentioned processing in the evaluation unit may be performed using AI, for example, or may be performed without using AI. For example, the evaluation unit can input device information into the generation AI and have the generation AI analyze the data.

[0096] The generation unit can estimate the user's emotions and adjust the audio subtitle generation method based on the estimated user's emotions. The generation unit, for example, estimates the user's emotions. Specific methods for estimating the user's emotions include facial expression recognition, voice analysis, and survey results. The generation unit uses facial expression recognition to estimate the emotion from the user's facial expression. Voice analysis to estimate the emotion from the tone and speed of the user's voice. Survey results are used to estimate the user's self-reported emotions. The generation unit adjusts the audio subtitle generation method based on the estimated user's emotions. For example, if the user is excited, the generation AI generates visually stimulating audio subtitles. If the user is relaxed, the generation AI generates calm audio subtitles. If the user is tired, the generation AI generates concise and to-the-point audio subtitles. This allows the generation unit to adjust the audio subtitle generation method according to the user's emotions, thereby generating more appropriate audio subtitles. Emotion estimation is achieved, for example, using an emotion estimation function using an emotion engine or generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the generation unit may be performed using AI, or may be performed without using AI. For example, the generation unit may input user emotion data into the generation AI and cause the generation AI to adjust the generation method based on the emotion.

[0097] The generation unit generates audio subtitles that emphasize important points of the video, thereby helping the user understand. For example, the generation unit generates audio subtitles that emphasize important points of the video. Important points include key frames, important lines, major events, etc. The generation unit uses a generation AI to analyze important scenes in the video and generate audio subtitles corresponding to those scenes. The generation AI extracts key points of the video and generates audio subtitles that emphasize the key points. The generation AI generates audio subtitles that include supplemental information to help understand based on the content of the video. For example, the generation AI analyzes important scenes in the video and generates audio subtitles corresponding to those scenes. The generation AI extracts key points of the video and generates audio subtitles that emphasize the key points. The generation AI generates audio subtitles that include supplemental information to help understand based on the content of the video. In this way, the generation unit generates audio subtitles that emphasize important points, thereby deepening the user's understanding. Some or all of the above-mentioned processing in the generation unit may be performed using, for example, AI, or may be performed without using AI. For example, the generation unit may input data on important points of the video to the generation AI and cause the generation AI to generate audio subtitles.

[0098] The generation unit can generate different audio commentary styles depending on the content of the video. The generation unit generates different audio commentary styles depending on the content of the video, for example. Audio commentary styles include commentary style, narrative style, and dialogue style. The generation unit uses a generation AI to analyze the content of the video and generate audio commentary style. The generation AI generates audio commentary that provides supplemental information based on the content of the video. The generation AI generates audio commentary in question format depending on the content of the video. For example, the generation AI analyzes the content of the video and generates audio commentary style. The generation AI generates audio commentary that provides supplemental information based on the content of the video. The generation AI generates audio commentary in question format depending on the content of the video. In this way, the generation unit generates different audio commentary styles depending on the content of the video, thereby deepening the user's understanding. Some or all of the above-mentioned processing in the generation unit may be performed using AI, for example, or may be performed without using AI. For example, the generation unit may input video content data to the generation AI and cause the generation AI to generate an audio commentary style.

[0099] The generation unit can estimate the user's emotion and adjust the length of the audio commentary based on the estimated user emotion. The generation unit, for example, estimates the user's emotion. Specific methods for estimating the user's emotion include facial expression recognition, voice analysis, and survey results. The generation unit uses facial expression recognition to estimate the emotion from the user's facial expression. Voice analysis to estimate the emotion from the tone and speed of the user's voice. Survey results are used to estimate the user's self-reported emotion. The generation unit adjusts the length of the audio commentary based on the estimated user emotion. For example, if the user is excited, the generation AI generates a short, to-the-point audio commentary. If the user is relaxed, the generation AI generates a longer audio commentary with detailed explanations. If the user is tired, the generation AI generates a concise, highly legible audio commentary. The generation unit adjusts the length of the audio commentary according to the user's emotion, thereby generating a more appropriate audio commentary. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the generation unit may be performed using AI, for example, or may be performed without using AI. For example, the generation unit may input user emotion data to the generation AI and cause the generation AI to adjust the length of the secondary audio based on the emotion.

[0100] When generating the secondary audio, the generation unit can improve the accuracy of generation by referring to the user's viewing history. The generation unit, for example, refers to the user's viewing history. The viewing history includes the titles of videos viewed, viewing time, viewing frequency, etc. The generation unit analyzes the user's viewing behavior based on the viewing history and improves the accuracy of the secondary audio generation. For example, the generation AI analyzes the user's viewing history and references the secondary audio of similar videos. The generation AI prioritizes reflecting points of interest to the user in the secondary audio based on the viewing history. The generation AI integrates the viewing history and the comprehension level to improve the accuracy of the secondary audio. In this way, the generation unit improves the accuracy of the secondary audio generation by referring to the user's viewing history. Some or all of the above-mentioned processes in the generation unit may be performed using AI, for example, or may be performed without using AI. For example, the generation unit may input viewing history data to the generation AI and have the generation AI generate the secondary audio.

[0101] The generation unit can generate the secondary audio by referring to related external data. The generation unit references, for example, related external data. External data includes subtitle data, related literature, news articles, social media posts, etc. The generation unit analyzes the subtitle data and reflects it in the secondary audio. The generation unit references related literature to generate secondary audio including supplemental information. The generation unit references news articles to reflect the latest information related to the content of the video in the secondary audio. The generation unit references social media posts to reflect user reactions to the video in the secondary audio. For example, a generation AI analyzes the subtitle data and reflects it in the secondary audio. The generation AI references related literature to generate secondary audio including supplemental information. The generation AI references news articles to reflect the latest information related to the content of the video in the secondary audio. The generation AI references social media posts to reflect user reactions to the video in the secondary audio. In this way, by the generation unit referring to the related external data, the accuracy of generating the secondary audio is improved. Some or all of the above-mentioned processes in the generation unit may be performed, for example, using AI or without using AI. For example, the generation unit can input external data into the generation AI and cause the generation AI to generate secondary audio.

[0102] The adding unit can estimate the user's emotion and adjust the audio commentary adding method based on the estimated user's emotion. The adding unit, for example, estimates the user's emotion. Specific methods for estimating the user's emotion include facial expression recognition, voice analysis, and survey results. The adding unit uses facial expression recognition to estimate the emotion from the user's facial expression. It uses voice analysis to estimate the emotion from the tone and speed of the user's voice. It uses survey results to estimate the user's self-reported emotion. The adding unit adjusts the audio commentary adding method based on the estimated user's emotion. For example, if the user is excited, the generation AI adds a visually stimulating audio commentary. If the user is relaxed, the generation AI adds a calming audio commentary. If the user is tired, the generation AI adds a concise and to-the-point audio commentary. Thus, the adding unit adjusts the audio commentary adding method according to the user's emotion, thereby adding a more appropriate audio commentary. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the attachment unit may be performed using AI, or may be performed without using AI. For example, the attachment unit may input user emotion data to the generation AI and cause the generation AI to adjust the attachment method based on the emotion.

[0103] The adding unit can add a different audio subtitle to each scene in the video, providing audio subtitles that correspond to the characteristics of the scene. For example, the adding unit adds a different audio subtitle to each scene in the video. Audio subtitles applied to each scene include action scenes, dialogue scenes, and landscape scenes. The adding unit uses a generation AI to add audio subtitles that correspond to fast-moving parts in action scenes. In dialogue scenes, audio subtitles that supplement the dialogue content are added. In landscape scenes, audio subtitles that explain the characteristics of the video are added. For example, the generation AI analyzes action scenes and adds audio subtitles that correspond to fast-moving parts. The generation AI analyzes dialogue scenes and adds audio subtitles that supplement the dialogue content. The generation AI analyzes landscape scenes and adds audio subtitles that explain the characteristics of the video. In this way, the adding unit provides audio subtitles that correspond to the characteristics of the scene, thereby deepening the user's understanding. Some or all of the above-mentioned processing in the adding unit may be performed, for example, using AI or without AI. For example, the adding unit can input data for each scene into the generation AI and cause the generation AI to add secondary audio.

[0104] The attachment unit can adjust the balance between the audio and secondary audio of a video to allow the user to watch without feeling uncomfortable. The attachment unit, for example, adjusts the balance between the audio and secondary audio of a video. The balance between the audio and secondary audio includes volume adjustment, audio priority, audio mixing method, etc. The attachment unit uses a generation AI to analyze the audio level of the video and automatically adjust the volume of the secondary audio. The generation AI adjusts the timing of the audio and secondary audio to maintain a natural flow. The generation AI provides an optimal audio balance taking into account the user's device settings. For example, the generation AI analyzes the audio level of a video and automatically adjusts the volume of the secondary audio. The generation AI adjusts the timing of the audio and secondary audio to maintain a natural flow. The generation AI provides an optimal audio balance taking into account the user's device settings. As a result, the attachment unit adjusts the balance between the audio and secondary audio to allow the user to watch without feeling uncomfortable. Some or all of the above-mentioned processing in the attachment unit may be performed using, for example, AI, or may be performed without using AI. For example, the attachment unit may input audio data to the generation AI and cause the generation AI to adjust the audio balance.

[0105] The attachment unit can estimate the user's emotion and adjust the display method of the secondary audio based on the estimated user's emotion. The attachment unit, for example, estimates the user's emotion. Specific methods for estimating the user's emotion include facial expression recognition, voice analysis, and survey results. The attachment unit uses facial expression recognition to estimate the emotion from the user's facial expression. It uses voice analysis to estimate the emotion from the tone and speed of the user's voice. It uses survey results to estimate the user's self-reported emotion. The attachment unit adjusts the display method of the secondary audio based on the estimated user's emotion. For example, if the user is excited, the generation AI provides a visually stimulating display method. If the user is relaxed, the generation AI provides a calming display method. If the user is tired, the generation AI provides a simple, highly visible display method. This allows the attachment unit to adjust the display method of the secondary audio according to the user's emotion, enabling more appropriate display. Emotion estimation is achieved, for example, using an emotion estimation function using an emotion engine or generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the annotation unit may be performed using AI, or may be performed without using AI. For example, the annotation unit may input user emotion data to the generation AI and cause the generation AI to adjust the display method based on the emotion.

[0106] When adding secondary audio, the adding unit can improve the accuracy of the addition by referring to the user's viewing history. The adding unit may, for example, refer to the user's viewing history. The viewing history may include the titles of videos viewed, viewing time, viewing frequency, etc. The adding unit may analyze the user's viewing behavior based on the viewing history and improve the accuracy of the secondary audio addition. For example, the generation AI may analyze the user's viewing history and refer to secondary audio from similar videos. The generation AI may prioritize points of interest to the user based on the viewing history and reflect them in the secondary audio. The generation AI may integrate the viewing history and the comprehension level to improve the accuracy of the secondary audio. In this way, the adding unit may improve the accuracy of the secondary audio addition by referring to the user's viewing history. Some or all of the above-mentioned processing in the adding unit may be performed using AI, for example, or may be performed without using AI. For example, the adding unit may input viewing history data to the generation AI and have the generation AI add secondary audio.

[0107] The adding unit can add the secondary audio by referring to related external data. The adding unit, for example, refers to related external data. The external data includes subtitle data, related literature, news articles, social media posts, etc. The adding unit analyzes the subtitle data and reflects it in the secondary audio. The adding unit refers to related literature and adds secondary audio including supplemental information. The adding unit refers to news articles and reflects the latest information related to the content of the video in the secondary audio. The adding unit refers to social media posts and reflects user reactions to the video in the secondary audio. For example, a generation AI analyzes the subtitle data and reflects it in the secondary audio. The generation AI refers to related literature and adds secondary audio including supplemental information. The generation AI refers to news articles and reflects the latest information related to the content of the video in the secondary audio. The generation AI refers to social media posts and reflects user reactions to the video in the secondary audio. In this way, the adding unit refers to related external data, thereby improving the accuracy of adding secondary audio. Some or all of the above-mentioned processing in the adding unit may be performed, for example, using AI or without using AI. For example, the adding unit can input external data to the generation AI and cause the generation AI to add secondary audio.

[0108] The collection unit can estimate the user's emotions and adjust the data collection method based on the estimated user's emotions. The collection unit, for example, estimates the user's emotions. Specific methods for estimating the user's emotions include facial expression recognition, voice analysis, and survey results. The collection unit uses facial expression recognition to estimate the user's emotions from the user's facial expressions. Voice analysis to estimate the user's emotions from the tone and speed of the user's voice. Survey results are used to estimate the user's self-reported emotions. The collection unit adjusts the data collection method based on the estimated user's emotions. For example, if the user is excited, the generation AI collects data quickly. If the user is relaxed, the generation AI collects detailed data. If the user is tired, the generation AI collects concise data. This allows the collection unit to adjust the data collection method according to the user's emotions, enabling more appropriate data collection. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the collection unit may be performed using, for example, AI, or may be performed without using AI. For example, the collection unit may input user emotion data to the generation AI and cause the generation AI to adjust the collection method based on the emotion.

[0109] The collection unit can collect detailed information about operations performed by a user while viewing content for analysis. The collection unit, for example, collects detailed information about operations performed by a user while viewing content. Operations performed while viewing content include pausing, rewinding, fast-forwarding, and the like. The collection unit collects the frequency of pauses to evaluate the user's level of understanding. The number of rewinds is collected to identify parts that are difficult for the user to understand. The frequency of fast-forwarding is collected to identify parts that the user is not interested in. For example, a generation AI collects the frequency of pauses to evaluate the user's level of understanding. The generation AI collects the number of rewinds to identify parts that are difficult for the user to understand. The generation AI collects the frequency of fast-forwarding to identify parts that the user is not interested in. In this way, the collection unit collects detailed information about operations performed while viewing content, thereby improving the accuracy of analysis. Some or all of the above-described processing in the collection unit may be performed using AI, for example, or may be performed without using AI. For example, the collection unit can input data about operations performed while viewing content into a generation AI and have the generation AI analyze the data.

[0110] The collection unit collects information about the user's viewing environment to improve the accuracy of analysis. The collection unit, for example, collects information about the user's viewing environment. The viewing environment includes the type of device, network environment, viewing location, etc. The collection unit collects device information and selects the optimal analysis method. The collection unit collects network status and adjusts the data collection speed. The collection unit collects viewing locations and selects an analysis method according to the environment. For example, a generation AI collects information about the user's device and selects the optimal analysis method. The generation AI collects network status and adjusts the data collection speed. The generation AI collects viewing locations and selects an analysis method according to the environment. In this way, the collection unit collects information about the viewing environment to improve the accuracy of analysis. Some or all of the above-mentioned processing in the collection unit may be performed using AI, for example, or may be performed without using AI. For example, the collection unit may input viewing environment data into a generation AI and have the generation AI analyze the data.

[0111] The collection unit can estimate the user's emotions and prioritize collected data based on the estimated user emotions. The collection unit, for example, estimates the user's emotions. Specific methods for estimating the user's emotions include facial expression recognition, voice analysis, and survey results. The collection unit uses facial expression recognition to estimate the user's emotions from the user's facial expressions. Voice analysis to estimate the user's emotions from the tone and speed of the user's voice. Survey results are used to estimate the user's self-reported emotions. The collection unit prioritizes the collected data based on the estimated user emotions. For example, if the user is excited, the generation AI prioritizes collecting important data. If the user is relaxed, the generation AI prioritizes collecting detailed data. If the user is tired, the generation AI prioritizes collecting concise data. This allows the collection unit to prioritize the collected data according to the user's emotions, enabling more appropriate data collection. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the collection unit may be performed using, for example, AI, or may be performed without using AI. For example, the collection unit may input user emotion data to the generation AI and cause the generation AI to determine the priority of the collected data based on the emotion.

[0112] When collecting data, the collection unit can prioritize collecting highly relevant data by taking into account the user's geographical location information. The collection unit, for example, considers the user's geographical location information. The geographical location information includes GPS data, IP address, location information services, etc. The collection unit collects relevant data based on the geographical location information. For example, the generation AI collects relevant data based on the user's geographical location information. The generation AI selects an optimal data collection method by taking into account the geographical location information. The generation AI improves the accuracy of data collection based on the geographical location information. In this way, the collection unit can prioritize collecting highly relevant data by taking into account the geographical location information. Some or all of the above-mentioned processing in the collection unit may be performed using AI, for example, or may be performed without using AI. For example, the collection unit can input geographical location information data to the generation AI and cause the generation AI to collect data.

[0113] When collecting data, the collection unit can analyze the user's social media activity and collect related data. The collection unit, for example, analyzes the user's social media activity. The social media activity includes the content of posts, the number of likes, the number of followers, etc. The collection unit collects related data based on the social media activity. For example, the generation AI analyzes the user's social media activity and collects related data. The generation AI identifies the user's interests based on the content of social media posts. The generation AI improves the accuracy of the collected data based on the social media data. As a result, the collection unit analyzes the social media activity and improves the accuracy of the collection of related data. Some or all of the above-mentioned processing in the collection unit may be performed using, for example, AI, or may be performed without using AI. For example, the collection unit may input social media data into the generation AI and have the generation AI collect the data.

[0114] The registration unit can estimate the user's emotion and adjust the registration method of the audio subtitles based on the estimated user's emotion. The registration unit, for example, estimates the user's emotion. Specific methods for estimating the user's emotion include facial expression recognition, voice analysis, and survey results. The registration unit uses facial expression recognition to estimate the emotion from the user's facial expression. Voice analysis to estimate the emotion from the tone and speed of the user's voice. Survey results are used to estimate the user's self-reported emotion. The registration unit adjusts the registration method of the audio subtitles based on the estimated user's emotion. For example, if the user is excited, the generation AI provides a quick method for registering the audio subtitles. If the user is relaxed, the generation AI provides a detailed registration method. If the user is tired, the generation AI provides a simple registration method. This allows the registration unit to adjust the registration method of the audio subtitles according to the user's emotion, enabling more appropriate registration. Emotion estimation is achieved, for example, using an emotion estimation function using an emotion engine or generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the registration unit may be performed using AI, or may be performed without using AI. For example, the registration unit may input user emotion data into the generation AI and cause the generation AI to adjust the registration method based on the emotion.

[0115] The registration unit can evaluate the quality of the secondary audio registered by the user and provide appropriate feedback. The registration unit, for example, evaluates the quality of the secondary audio registered by the user. The quality of the secondary audio includes audio clarity, accuracy of the content, and the user's evaluation. The registration unit uses a generation AI to evaluate the sound quality of the secondary audio and provide feedback on areas for improvement. The generation AI evaluates the content of the secondary audio and provides supplementary information. The generation AI evaluates the overall quality of the secondary audio and provides feedback to the user. For example, the generation AI evaluates the sound quality of the secondary audio and provides feedback on areas for improvement. The generation AI evaluates the content of the secondary audio and provides supplementary information. The generation AI evaluates the overall quality of the secondary audio and provides feedback to the user. In this way, the registration unit evaluates the quality of the secondary audio and provides appropriate feedback, thereby improving the quality of the secondary audio for the user. Some or all of the above-mentioned processing in the registration unit may be performed using, for example, AI, or may be performed without using AI. For example, the registration unit can input secondary audio data to the generation AI and cause the generation AI to evaluate the quality and provide feedback.

[0116] The registration unit automatically categorizes the audio commentary registered by the user, making it easier for other users to search. For example, the registration unit automatically categorizes the audio commentary registered by the user. Audio commentary categories include education, entertainment, and news. The registration unit uses a generation AI to analyze the content of the audio commentary and classify it into an appropriate category. The generation AI extracts keywords from the audio commentary and tags them for easy search. The generation AI automatically updates the category of the audio commentary to reflect the latest information. For example, the generation AI analyzes the content of the audio commentary and classifies it into an appropriate category. The generation AI extracts keywords from the audio commentary and tags them for easy search. The generation AI automatically updates the category of the audio commentary to reflect the latest information. In this way, the registration unit automatically categorizes the audio commentary, making it easier for other users to search. Some or all of the above-described processing in the registration unit may be performed using, for example, AI, or may be performed without using AI. For example, the registration unit may input audio commentary data to the generation AI and have the generation AI classify the category.

[0117] The registration unit can estimate the user's emotion and adjust the display method of the secondary audio based on the estimated user's emotion. The registration unit, for example, estimates the user's emotion. Specific methods for estimating the user's emotion include facial expression recognition, voice analysis, and questionnaire results. The registration unit uses facial expression recognition to estimate the emotion from the user's facial expression. Voice analysis to estimate the emotion from the tone and speed of the user's voice. Questionnaire results are used to estimate the user's self-reported emotion. The registration unit adjusts the display method of the secondary audio based on the estimated user's emotion. For example, if the user is excited, the generation AI provides a visually stimulating display method. If the user is relaxed, the generation AI provides a calming display method. If the user is tired, the generation AI provides a simple, highly visible display method. This allows the registration unit to adjust the display method of the secondary audio according to the user's emotion, enabling more appropriate display. Emotion estimation is achieved, for example, using an emotion estimation function using an emotion engine or generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the registration unit may be performed using AI, or may be performed without using AI. For example, the registration unit may input user emotion data into the generation AI and cause the generation AI to adjust the display method based on the emotion.

[0118] The registration unit can improve the accuracy of registration by referring to the user's viewing history when registering the secondary audio. The registration unit, for example, refers to the user's viewing history. The viewing history includes the titles of videos viewed, viewing time, viewing frequency, etc. The registration unit analyzes the user's viewing behavior based on the viewing history and improves the accuracy of the secondary audio registration. For example, the generation AI analyzes the user's viewing history and references the secondary audio of similar videos. The generation AI prioritizes reflecting points of interest to the user in the secondary audio based on the viewing history. The generation AI integrates the viewing history and the comprehension level to improve the accuracy of the secondary audio. In this way, the registration unit improves the accuracy of the secondary audio registration by referring to the user's viewing history. Some or all of the above-mentioned processes in the registration unit may be performed using AI, for example, or may be performed without using AI. For example, the registration unit may input viewing history data to the generation AI and have the generation AI register the secondary audio.

[0119] The registration unit can refer to related external data when registering the secondary audio. The registration unit refers to, for example, related external data. The external data includes subtitle data, related literature, news articles, social media posts, etc. The registration unit analyzes the subtitle data and reflects it in the secondary audio. The registration unit refers to related literature and registers secondary audio including supplemental information. The registration unit refers to news articles and reflects the latest information related to the content of the video in the secondary audio. The registration unit refers to social media posts and reflects user reactions to the video in the secondary audio. For example, a generation AI analyzes the subtitle data and reflects it in the secondary audio. The generation AI refers to related literature and registers secondary audio including supplemental information. The generation AI refers to news articles and reflects the latest information related to the content of the video in the secondary audio. The generation AI refers to social media posts and reflects user reactions to the video in the secondary audio. In this way, by the registration unit referring to the related external data, the accuracy of registering the secondary audio is improved. Some or all of the above-mentioned processing in the registration unit may be performed, for example, using AI or without using AI. For example, the registration unit can input external data into the generation AI and cause the generation AI to register the secondary audio.

[0120] The advertising unit can estimate a user's emotions and adjust an advertisement insertion method based on the estimated user's emotions. The advertising unit, for example, estimates the user's emotions. Specific methods for estimating a user's emotions include facial expression recognition, voice analysis, and survey results. The advertising unit uses facial expression recognition to estimate the user's emotions from their facial expressions. Voice analysis to estimate the user's emotions from the tone and speed of their voice. Survey results are used to estimate the user's self-reported emotions. The advertising unit adjusts the advertisement insertion method based on the estimated user's emotions. For example, if the user is excited, the generation AI inserts a visually stimulating advertisement. If the user is relaxed, the generation AI inserts a calming advertisement. If the user is tired, the generation AI inserts a concise and to-the-point advertisement. Thus, the advertising unit adjusts the advertisement insertion method according to the user's emotions, thereby inserting more appropriate advertisements. Emotion estimation is achieved, for example, using an emotion estimation function using an emotion engine or a generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the advertising unit may be performed using AI, or may be performed without using AI. For example, the advertising unit may input user emotion data into the generation AI and cause the generation AI to adjust the advertisement insertion method based on the emotion.

[0121] The advertising unit can insert different advertisements for each scene of the video and provide advertisements according to the characteristics of the scene. The advertising unit, for example, inserts different advertisements for each scene of the video. Advertisements applied to each scene include action scenes, dialogue scenes, landscape scenes, etc. The advertising unit uses a generation AI to insert advertisements corresponding to fast-moving parts in action scenes. In dialogue scenes, it inserts advertisements related to the content of the conversation. In landscape scenes, it inserts advertisements related to the characteristics of the video. For example, the generation AI analyzes an action scene and inserts advertisements corresponding to fast-moving parts. The generation AI analyzes a dialogue scene and inserts advertisements related to the content of the conversation. The generation AI analyzes a landscape scene and inserts advertisements related to the characteristics of the video. In this way, the advertising unit provides advertisements according to the characteristics of the scene, thereby inserting advertisements that attract the user's interest. Some or all of the above-mentioned processing in the advertising unit may be performed using, for example, AI, or may be performed without using AI. For example, the advertising unit may input data for each scene into the generation AI and cause the generation AI to insert advertisements.

[0122] The advertising unit can automatically select advertisements related to the content of the video and insert advertisements that interest the user. The advertising unit automatically selects advertisements related to the content of the video, for example. Advertisements include product introduction videos, service promotion videos, banner advertisements, and text advertisements. The advertising unit uses a generation AI to analyze the content of the video and select advertisements for related products and services. The generation AI inserts advertisements of interest to the user based on the user's viewing history. The generation AI inserts advertisements at the optimal timing depending on the scene in the video. For example, the generation AI analyzes the content of the video and selects advertisements for related products and services. The generation AI inserts advertisements of interest to the user based on the user's viewing history. The generation AI inserts advertisements at the optimal timing depending on the scene in the video. In this way, the advertising unit provides advertisements that interest the user by inserting advertisements related to the content of the video. Some or all of the above-described processing in the advertising unit may be performed using, for example, AI, or may be performed without using AI. For example, the advertising unit may input video content data into the generation AI and cause the generation AI to select and insert advertisements.

[0123] The advertising unit can estimate a user's emotions and adjust the advertisement display method based on the estimated user's emotions. The advertising unit, for example, estimates the user's emotions. Specific methods for estimating the user's emotions include facial expression recognition, voice analysis, and survey results. The advertising unit uses facial expression recognition to estimate the user's emotions from the user's facial expressions. Voice analysis to estimate the user's emotions from the tone and speed of the user's voice. Survey results are used to estimate the user's self-reported emotions. The advertising unit adjusts the advertisement display method based on the estimated user's emotions. For example, if the user is excited, the generation AI displays a visually stimulating advertisement. If the user is relaxed, the generation AI displays an advertisement with a calm tone. If the user is tired, the generation AI displays a concise and to-the-point advertisement. Thus, the advertising unit adjusts the advertisement display method according to the user's emotions, thereby displaying more appropriate advertisements. Emotion estimation is realized using an emotion estimation function, for example, using an emotion engine or generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the advertising unit may be performed using, for example, AI, or may be performed without using AI. For example, the advertising unit may input user emotion data into the generation AI and cause the generation AI to adjust the advertisement display method based on the emotion.

[0124] When inserting an advertisement, the advertising unit can improve the accuracy of the advertisement by referring to the user's viewing history. The advertising unit, for example, refers to the user's viewing history. The viewing history includes the titles of videos viewed, viewing time, viewing frequency, etc. The advertising unit analyzes the user's viewing behavior based on the viewing history and improves the accuracy of the advertisement. For example, the generation AI analyzes the user's viewing history and selects relevant advertisements. The generation AI preferentially inserts advertisements that are of interest to the user based on the viewing history. The generation AI integrates the viewing history and comprehension level to improve the accuracy of the advertisement. In this way, the advertising unit improves the accuracy of the advertisement by referring to the user's viewing history. Some or all of the above-mentioned processing in the advertising unit may be performed using AI, for example, or may be performed without using AI. For example, the advertising unit can input viewing history data into the generation AI and have the generation AI select and insert advertisements.

[0125] When inserting an advertisement, the advertising unit can select an advertisement by referring to related external data. The advertising unit, for example, refers to related external data. The external data includes the user's purchase history, interest data, news articles, social media posts, etc. The advertising unit analyzes the purchase history and selects a relevant advertisement. The advertising unit inserts an advertisement that is of interest to the user based on the interest data. The advertising unit refers to news articles and reflects the latest information related to the content of the video in the advertisement. The advertising unit refers to social media posts and reflects the user's reaction to the video in the advertisement. For example, the generation AI analyzes the user's purchase history and selects a relevant advertisement. The generation AI inserts an advertisement that is of interest to the user based on the interest data. The generation AI refers to news articles and reflects the latest information related to the content of the video in the advertisement. The generation AI refers to social media posts and reflects the user's reaction to the video in the advertisement. In this way, the advertising unit improves the accuracy of the advertisement by referring to the related external data. Some or all of the above-mentioned processing in the advertising unit may be performed, for example, using AI or without using AI. For example, the advertising department can input external data into the generation AI and have the generation AI select and insert advertisements.

[0126] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0127] When analyzing the content of a video, the analysis unit can apply an analysis algorithm that is appropriate for the video genre. For example, in the case of an educational video, the generation AI will perform an analysis that emphasizes educational elements and enhances learning effectiveness. In the case of an entertainment video, the generation AI will perform an analysis that emphasizes visual elements and storytelling. In the case of a news video, the generation AI will perform an analysis that emphasizes fact-checking and extracting important information. This allows the analysis unit to generate more appropriate secondary audio by performing an analysis that is appropriate for the video genre.

[0128] The collection unit can collect biometric data such as the user's heart rate and electrodermal response while watching, in addition to the user's viewing history and viewing time. For example, the collection unit can collect heart rate fluctuations while the user is watching a video to evaluate the user's level of excitement and concentration. The collection unit can collect electrodermal response to evaluate the user's emotional response. This allows the collection unit to more accurately evaluate the user's level of comprehension based on the user's biometric data.

[0129] When analyzing the content of a video, the analysis unit can perform an integrated analysis of the video's visual and audio elements. For example, it can analyze the synchronization of the video's video and audio to identify important scenes and lines, analyze the color and composition of the video to extract visually important points, and analyze the tone and rhythm of the audio to identify emotional elements. This allows the analysis unit to generate richer audio descriptions by taking both visual and audio into account.

[0130] When a user registers a secondary audio track, the registration unit can automatically convert it into text using voice recognition technology and register it as text data. For example, when a user registers a secondary audio track by voice, the voice recognition technology converts the content into text and saves it as text data. This makes it easier for other users to search the text data and share the content of the secondary audio track more easily.

[0131] The advertising unit can generate personalized advertisements based on the user's viewing history and viewing time. For example, it generates advertisements for related products based on the genre and content of videos the user has previously viewed. It inserts advertisements at appropriate times depending on the user's viewing time period. This allows the advertising unit to provide advertisements that match the user's interests and concerns, thereby increasing advertising effectiveness.

[0132] The analysis unit can estimate the user's emotions and adjust the video analysis method based on the estimated user emotions. For example, if the user is excited, the generation AI will quickly analyze the video and emphasize the important points. If the user is relaxed, the generation AI will perform a detailed analysis and extract even the finer points. If the user is tired, the generation AI will perform a concise analysis and extract only the main points. This allows the analysis unit to adjust the video analysis method according to the user's emotions, enabling more appropriate analysis.

[0133] The evaluation unit can estimate the user's emotions and adjust the evaluation criteria for comprehension based on the estimated user emotions. For example, if the user is excited, the generation AI will tighten the evaluation criteria for comprehension. If the user is relaxed, the generation AI will loosen the evaluation criteria for comprehension. If the user is tired, the generation AI will simplify the evaluation criteria for comprehension. This allows the evaluation unit to adjust the evaluation criteria for comprehension according to the user's emotions, enabling more appropriate evaluation.

[0134] The generation unit can estimate the user's emotions and adjust the method for generating secondary audio based on the estimated user emotions. For example, if the user is excited, the generation AI generates visually stimulating secondary audio. If the user is relaxed, the generation AI generates secondary audio with a calm tone. If the user is tired, the generation AI generates secondary audio that is concise and to the point. In this way, the generation unit adjusts the method for generating secondary audio according to the user's emotions, thereby generating more appropriate secondary audio.

[0135] The audio adder can estimate the user's emotions and adjust the audio add method based on the estimated user emotions. For example, if the user is excited, the generation AI adds a visually stimulating audio add. If the user is relaxed, the generation AI adds a subtitle with a calm tone. If the user is tired, the generation AI adds a subtitle that is concise and to the point. In this way, the audio adder can add a more appropriate audio add.

[0136] The advertising unit can estimate the user's emotions and adjust the ad insertion method based on the estimated user emotions. For example, if the user is excited, the generation AI will insert a visually stimulating ad. If the user is relaxed, the generation AI will insert an ad with a calm tone. If the user is tired, the generation AI will insert a concise and to-the-point ad. In this way, the advertising unit can adjust the ad insertion method according to the user's emotions, thereby inserting more appropriate ads.

[0137] The processing flow of the second embodiment will be briefly explained below.

[0138] Step 1: The analysis unit analyzes the content of the video. For example, the analysis unit extracts important points from each scene in the video and organizes them so that they are easy for users to understand. The analysis unit uses the generation AI to analyze the content of the video and identify important points. For example, the generation AI extracts important keyframes from each scene in the video and analyzes the main points of that scene. Step 2: The evaluation unit evaluates the user's level of understanding based on the content analyzed by the analysis unit. For example, the evaluation unit collects data such as the user's viewing history, viewing time, and playback speed when watching a video, and the generation AI analyzes this data to evaluate the user's level of understanding. For example, the evaluation unit identifies which parts of the video the user is lacking in understanding based on the viewing history when watching the video. Step 3: The generation unit generates audio commentary based on the comprehension level assessed by the evaluation unit. The generation unit uses a generation AI to generate audio commentary based on the content of the video and the user's level of comprehension. For example, the generation unit generates audio commentary that explains important points of the video or supplements parts that are difficult for the user to understand. The generation AI uses a text generation AI (e.g., LLM) to summarize the content of the video and generate explanatory audio. Step 4: The audio adder adds the audio commentary generated by the generator to the video. For example, the audio adder inserts the generated audio commentary at the appropriate timing in the video to ensure a natural viewing experience for the user. The audio adder uses the generation AI to adjust the balance between the video audio and the audio commentary to maintain a natural flow.

[0139] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0140] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt including an instruction, as well as inference data such as voice data indicating speech, text data indicating text, and image data indicating an image (e.g., still image data or video data). The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in one or more data formats including voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation models 58 include AIs other than the generation AI. The AIs other than the generation AI are, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but are not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or in whole by AI, but are not limited to these examples.In addition, processing performed by AI including the generation AI may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI including the generation AI.

[0141] Furthermore, the processing by the data processing system 10 described above is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0142] Each of the multiple elements including the above-mentioned analysis unit, evaluation unit, generation unit, attachment unit, registration unit, and advertising unit is realized, for example, by at least one of the smart device 14 and the data processing device 12. For example, the analysis unit is realized by the control unit 46A of the smart device 14 and analyzes the content of the video. The evaluation unit is realized by the specific processing unit 290 of the data processing device 12 and evaluates the user's level of understanding. The generation unit is realized by the specific processing unit 290 of the data processing device 12 and generates a secondary audio. The attachment unit is realized by the control unit 46A of the smart device 14 and attaches the generated secondary audio to the video. The registration unit is realized by the control unit 46A of the smart device 14 and registers a secondary audio created by the user. The advertising unit is realized by the specific processing unit 290 of the data processing device 12 and inserts an advertisement into the secondary audio. The correspondence between each unit and the device or control unit is not limited to the above example and various modifications are possible.

[0143] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0144] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0145] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0146] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0147] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0148] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0149] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0150] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0151] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0152] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0153] In the smart glasses 214, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0154] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0155] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0156] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt including an instruction and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image (e.g., still image data or video data). The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in one or more data formats, such as voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation models 58 include AIs other than the generation AI. Examples of AIs other than the generation AI include, but are not limited to, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), and naive Bayes. These AIs can perform various types of processing, but are not limited to these examples. The AI ​​may also be an AI agent. When the processing of each of the above-described parts is performed by an AI, the processing may be performed in part or entirely by the AI, but is not limited to these examples. Processing performed by an AI, including the generation AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by an AI, including the generation AI.

[0157] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, etc., and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0158] Each of the multiple elements, including the above-described analysis unit, evaluation unit, generation unit, attachment unit, registration unit, and advertising unit, is realized, for example, by at least one of the smart glasses 214 and the data processing device 12. For example, the analysis unit is realized by the control unit 46A of the smart glasses 214 and analyzes the content of the video. The evaluation unit is realized by the specific processing unit 290 of the data processing device 12 and evaluates the user's level of understanding. The generation unit is realized by the specific processing unit 290 of the data processing device 12 and generates a secondary audio. The attachment unit is realized by the control unit 46A of the smart glasses 214 and attaches the generated secondary audio to the video. The registration unit is realized by the control unit 46A of the smart glasses 214 and registers secondary audio created by the user. The advertising unit is realized by the specific processing unit 290 of the data processing device 12 and inserts advertisements into the secondary audio. The correspondence between each unit and the device or control unit is not limited to the above example, and various modifications are possible.

[0159] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0160] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0161] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0162] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0163] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0164] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0165] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0166] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0167] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0168] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0169] In the headset type terminal 314, the identification process is performed by the processor 46. A identification program 60 is stored in the storage 50. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as a control unit 46A in accordance with the identification program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the identification processing unit 290 using these models.

[0170] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0171] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0172] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt including an instruction and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image (e.g., still image data or video data). The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in one or more data formats, such as voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation models 58 include AIs other than the generation AI. Examples of AIs other than the generation AI include, but are not limited to, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), and naive Bayes. These AIs can perform various types of processing, but are not limited to these examples. The AI ​​may also be an AI agent. When the processing of each of the above-described parts is performed by an AI, the processing may be performed in part or entirely by the AI, but is not limited to these examples. Processing performed by an AI, including the generation AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by an AI, including the generation AI.

[0173] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, etc., and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0174] Each of the multiple elements including the above-mentioned analysis unit, evaluation unit, generation unit, attachment unit, registration unit, and advertising unit is realized, for example, by at least one of the headset type terminal 314 and the data processing device 12. For example, the analysis unit is realized by the control unit 46A of the headset type terminal 314 and analyzes the content of the video. The evaluation unit is realized by the specific processing unit 290 of the data processing device 12 and evaluates the user's level of understanding. The generation unit is realized by the specific processing unit 290 of the data processing device 12 and generates a secondary audio. The attachment unit is realized by the control unit 46A of the headset type terminal 314 and attaches the generated secondary audio to the video. The registration unit is realized by the control unit 46A of the headset type terminal 314 and registers secondary audio created by the user. The advertising unit is realized by the specific processing unit 290 of the data processing device 12 and inserts advertisements into the secondary audio. The correspondence between each unit and the device or control unit is not limited to the above example and various modifications are possible.

[0175] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0176] 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0177] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0178] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0179] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0180] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0181] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0182] The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0183] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0184] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0185] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0186] In the robot 414, the processor 46 performs the identification process. The storage 50 stores the identification program 60. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as the control unit 46A in accordance with the identification program 60 executed on the RAM 48. The robot 414 also has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform the same process as the identification processing unit 290 using these models.

[0187] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0188] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0189] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt including an instruction and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image (e.g., still image data or video data). The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in one or more data formats, such as voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation models 58 include AIs other than the generation AI. Examples of AIs other than the generation AI include, but are not limited to, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), and naive Bayes. These AIs can perform various types of processing, but are not limited to these examples. The AI ​​may also be an AI agent. When the processing of each of the above-described parts is performed by an AI, the processing may be performed in part or entirely by the AI, but is not limited to these examples. Processing performed by an AI, including the generation AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by an AI, including the generation AI.

[0190] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0191] Each of the multiple elements including the above-mentioned analysis unit, evaluation unit, generation unit, attachment unit, registration unit, and advertising unit is realized, for example, by at least one of the robot 414 and the data processing device 12. For example, the analysis unit is realized by the control unit 46A of the robot 414 and analyzes the content of the video. The evaluation unit is realized by the specific processing unit 290 of the data processing device 12 and evaluates the user's level of understanding. The generation unit is realized by the specific processing unit 290 of the data processing device 12 and generates a secondary audio. The attachment unit is realized by the control unit 46A of the robot 414 and attaches the generated secondary audio to the video. The registration unit is realized by the control unit 46A of the robot 414 and registers a secondary audio created by the user. The advertising unit is realized by the specific processing unit 290 of the data processing device 12 and inserts advertisements into the secondary audio. The correspondence between each unit and the device or control unit is not limited to the above example and various modifications are possible.

[0192] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0193] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion encompasses both emotions and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0194] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0195] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0196] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. Emotions can also be created for robots, cars, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and brain physiological signal analysis systems for emotions, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the area called "reaction," where sensation is dominant. The right half of the emotion map lists emotions belonging to the area called "situation," where situational awareness is dominant.

[0197] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0198] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[0199] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0200] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0201] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0202] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0203] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. A processor also includes a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0204] The hardware resource that executes the specific process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0205] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0206] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0207] In the above example, the first to fourth embodiments have been described separately, but some or all of these embodiments may be combined. The smart device 14, smart glasses 214, headset terminal 314, and robot 414 are merely examples, and they may be combined, or other devices may be used. In the above example, the first and second embodiments have been described separately, but they may be combined.

[0208] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0209] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0210] (Appendix 1) an analysis unit that analyzes the content of the video; an evaluation unit that evaluates the user's level of understanding based on the content analyzed by the analysis unit; a generation unit that generates a secondary audio based on the intelligibility evaluated by the evaluation unit; an adding unit that adds the secondary audio generated by the generating unit to the video. A system characterized by: (Appendix 2) A collection unit is provided for collecting data on the user's viewing history, viewing time, pause, or rewind. 2. The system of claim 1. (Appendix 3) Equipped with an analysis unit that analyzes the content of videos using natural language processing technology 2. The system of claim 1. (Appendix 4) A registration unit is provided for users to register secondary audio. 2. The system of claim 1. (Appendix 5) Equipped with an advertising section for inserting advertisements 2. The system of claim 1. (Appendix 6) The analysis unit Estimate the user's emotions and adjust the video analysis method according to the estimated user emotions. 2. The system of claim 1. (Appendix 7) The analysis unit A different analysis algorithm is applied to each video scene, and analysis is performed according to the characteristics of the scene. 2. The system of claim 1. (Appendix 8) The analysis unit Analyze the audio and video of a video separately and evaluate the correlation between them 2. The system of claim 1. (Appendix 9) The analysis unit Estimate the user's emotions and adjust the display method of the analysis results according to the estimated user emotions. 2. The system of claim 1. (Appendix 10) The analysis unit When analyzing videos, refer to the user's viewing history to improve the accuracy of the analysis. 2. The system of claim 1. (Appendix 11) The analysis unit When analyzing a video, refer to related external data to perform the analysis. 2. The system of claim 1. (Appendix 12) The evaluation unit Inferring user emotions and adjusting the evaluation criteria for comprehension based on the estimated user emotions 2. The system of claim 1. (Appendix 13) The evaluation unit Improve the accuracy of comprehension assessment by analyzing users' viewing history and viewing time in detail 2. The system of claim 1. (Appendix 14) The evaluation unit User actions while watching are reflected in the evaluation in real time 2. The system of claim 1. (Appendix 15) The evaluation unit Inferring user emotions and adjusting the order in which comprehension assessment results are displayed based on the estimated user emotions 2. The system of claim 1. (Appendix 16) The evaluation unit When assessing comprehension, refer to the user's learning history to improve the accuracy of the assessment. 2. The system of claim 1. (Appendix 17) The evaluation unit When assessing comprehension, adjust the assessment method to take into account the user's device information 2. The system of claim 1. (Appendix 18) The generation unit The method estimates the user's emotion and adjusts the generation method of the secondary audio based on the estimated user's emotion. 2. The system of claim 1. (Appendix 19) The generation unit Generate audio commentary to highlight important points in the video and help users understand 2. The system of claim 1. (Appendix 20) The generation unit Generate different audio descriptions styles depending on the video content 2. The system of claim 1. (Appendix 21) The generation unit Estimating a user's emotion and adjusting the length of the secondary audio based on the estimated user's emotion 2. The system of claim 1. (Appendix 22) The generation unit When generating secondary audio, the accuracy of the generation is improved by referring to the user's viewing history. 2. The system of claim 1. (Appendix 23) The generation unit When generating audio subtitles, reference related external data 2. The system of claim 1. (Appendix 24) The granting unit The method estimates the user's emotion and adjusts the method of adding secondary audio based on the estimated user's emotion. 2. The system of claim 1. (Appendix 25) The granting unit Adds different audio subtitles to each video scene, providing audio subtitles that match the characteristics of each scene 2. The system of claim 1. (Appendix 26) The granting unit Adjust the balance between the audio and secondary audio of the video to ensure a smooth viewing experience for users 2. The system of claim 1. (Appendix 27) The granting unit Estimating a user's emotion and adjusting the display method of the secondary audio based on the estimated user's emotion 2. The system of claim 1. (Appendix 28) The granting unit When adding secondary audio, the accuracy of adding it is improved by referring to the user's viewing history. 2. The system of claim 1. (Appendix 29) The granting unit When adding audio subtitles, related external data is referenced. 2. The system of claim 1. (Appendix 30) The collecting unit Inferring user sentiment and adjusting data collection methods based on the inferred user sentiment 3. The system of claim 2. (Appendix 31) The collecting unit Collect detailed information about user actions while watching and use it for analysis 3. The system of claim 2. (Appendix 32) The collecting unit Collecting information about users' viewing environments to improve analysis accuracy 3. The system of claim 2. (Appendix 33) The collecting unit Estimate user emotions and prioritize collected data based on the estimated user emotions. 3. The system of claim 2. (Appendix 34) The collecting unit When collecting data, prioritize collecting the most relevant data by taking into account the user's geographic location. 3. The system of claim 2. (Appendix 35) The collecting unit At the time of data collection, analyze your social media activity and collect relevant data 3. The system of claim 2. (Appendix 36) The registration unit The method estimates the user's emotion and adjusts the registration method of the secondary audio based on the estimated user's emotion. 5. The system of claim 4. (Appendix 37) The registration unit Evaluate the quality of the audio description registered by the user and provide appropriate feedback 5. The system of claim 4. (Appendix 38) The registration unit Automatically categorize audio descriptions registered by users to make them easier for other users to search. 5. The system of claim 4. (Appendix 39) The registration unit Estimating a user's emotion and adjusting the display method of the secondary audio based on the estimated user's emotion 5. The system of claim 4. (Appendix 40) The registration unit When registering secondary audio, the accuracy of registration is improved by referring to the user's viewing history. 5. The system of claim 4. (Appendix 41) The registration unit When registering audio subtitles, refer to related external data 5. The system of claim 4. (Appendix 42) The advertising department Estimate user emotions and adjust ad insertion methods based on the estimated user emotions 6. The system of claim 5. (Appendix 43) The advertising department Insert different ads into each video scene and provide ads that match the characteristics of the scene. 6. The system of claim 5. (Appendix 44) The advertising department Automatically select ads relevant to the video content and insert ads that will interest users 6. The system of claim 5. (Appendix 45) The advertising department Inferring user emotions and adjusting how ads are displayed based on the estimated user emotions 6. The system of claim 5. (Appendix 46) The advertising department When inserting ads, refer to the user's viewing history to improve the accuracy of the ads. 6. The system of claim 5. (Appendix 47) The advertising department When inserting ads, referencing relevant external data to select ads 6. The system of claim 5. [Explanation of symbols]

[0211] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. an analysis unit that analyzes the content of the video; an evaluation unit that evaluates a user's level of understanding based on the content analyzed by the analysis unit; a generation unit that generates a secondary audio based on the intelligibility evaluated by the evaluation unit; an adding unit that adds the secondary audio generated by the generating unit to the video. A system characterized by:

2. A collection unit is provided for collecting data on a user's viewing history, viewing time, pause, or rewind.

2. The system of claim 1 .

3. The analysis unit is Analyzing video content using natural language processing technology 2. The system of claim 1 .

4. A registration unit is provided for the user to register secondary audio.

2. The system of claim 1 .

5. Equipped with an advertising section for inserting advertisements 2. The system of claim 1 .

6. The analysis unit is Estimate the user's emotions and adjust the video analysis method according to the estimated user emotions.

2. The system of claim 1 .

7. The analysis unit is A different analysis algorithm is applied to each scene in the video, and analysis is performed according to the characteristics of the scene.

2. The system of claim 1 .

8. The analysis unit is Analyze the audio and video of a video separately and evaluate the relevance of the two 2. The system of claim 1 .

9. The analysis unit is Estimate the user's emotions and adjust the way the analysis results are displayed according to the estimated user emotions 2. The system of claim 1 .

Images