System

Abstract

A system is provided.SOLUTION: The data processing system 10 includes means for receiving a digital file for presentation input from a user, means for analyzing the received digital file and converting the contents of each slide into text data, means for analyzing a narration prompt input from the user and generating a speech script for each slide, means for generating synthesized speech based on the generated speech script, means for generating a subtitle based on the generated synthesized speech, means for integrating the respective slides, the generated synthesized speech, and the generated subtitle to generate a presentation moving image, and means for providing the generated presentation moving image to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In traditional presentations, creating slides and generating the associated narration and subtitles is often done manually, resulting in time-consuming and labor-intensive tasks. Furthermore, when using the same slides for repeated presentations in different settings, the same content must be explained each time, which is inefficient. Furthermore, adjusting the audio quality of the narration and the timing of subtitles is also time-consuming. This makes it difficult to work efficiently and provide high-quality presentations. [Means for solving the problem]

[0005] The present invention provides a system that receives a digital presentation file input by a user, analyzes it, and converts the contents of each slide into text data. It also includes a system that analyzes narration prompts input by the user and generates a talk script for each slide. It also includes a system that generates synthetic speech based on the generated talk script and generates subtitles based on the synthetic speech. It also includes a system that integrates the slides, synthetic speech, and subtitles to generate a presentation video and provides it to the user. It also includes a system that integrates a user-selected avatar into the video and a system that automatically adjusts the intonation and timing of the narration, thereby providing a system that enables efficient presentation creation and high-quality video delivery.

[0006] A "computer device" is a type of electronic device that can process, store, and communicate data and includes a central processing unit (CPU), memory, input, and output devices.

[0007] A "digital file" is a collection of information stored and transmitted electronically, such as electronic documents, images, or audio data used on a computer system.

[0008] "Analysis" is the process of examining and breaking down input data or information in detail to understand its structure and content.

[0009] "Text data" is data that expresses information expressed in characters, symbols, etc. in a digital format that can be processed by a computer.

[0010] A "narration prompt" is a sentence of explanation or instruction entered by the user, which is the basic information for generating synthetic speech.

[0011] A "talk script" is a document that refers to the contents of explanatory text and narration generated in correspondence with each slide of a presentation.

[0012] "Synthetic voice" is an artificial voice generated by a computer program that mimics human speech.

[0013] "Subtitles" are textual information that is displayed alongside video or presentations, supplementing narration and explanations in text.

[0014] An "avatar" is a character or graphic that represents a user's alter ego or representative, and is inserted into a presentation, with movements and expressions synchronized with the narration.

[0015] A "presentation video" is a video file used for presentation purposes, which is a series of videos generated by integrating slides, narration, subtitles, and avatars.

[0016] "Providing" refers to the act of making the generated data or services available to users. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] MODE FOR CARRYING OUT THE INVENTION

[0039] The present invention provides a system for efficiently generating presentation videos and reducing the user's workload by automating a series of processes: analyzing digital files for presentations, generating synthesized speech and subtitles, and finally providing the presentation videos.

[0040] Basic system configuration

[0041] The system mainly consists of the following components:

[0042] 1. User Interface

[0043] An interface through which users upload digital files for a presentation and enter narration prompts, typically implemented as a web-based application or a dedicated desktop application.

[0044] 2. Server Module

[0045] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[0046] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[0047] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[0048] Synthetic speech generation module: Generates synthetic speech based on the talk script.

[0049] Subtitle generation module: Generates subtitles based on synthesized speech.

[0050] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[0051] Program processing

[0052] The processing of the program will be explained in natural language below.

[0053] 1. User request input

[0054] Users upload digital presentation files (e.g., PDF, PowerPoint) to the system using their terminals, and also enter narration prompts through the user interface.

[0055] 2. Analyzing the file

[0056] The server receives the uploaded digital files, identifies the file format, and then a file analysis module converts the content of each slide into text data, sorting out page numbers and header information.

[0057] 3. Parsing the prompt

[0058] The server uses a prompt parsing module to parse the input narration prompts, which then generates a talking script that corresponds to the content of each slide.

[0059] 4. Generating Synthetic Speech

[0060] The server utilizes a synthetic speech generation module to convert the talk script into synthetic speech, and in the process evaluates the quality of the generated speech and adjusts the tone and intonation if necessary.

[0061] 5. Subtitle Generation

[0062] The server uses a subtitle generation module to generate subtitles based on the generated synthesized speech, setting the time code of the subtitles to match the timing of the narration and adjusting the display timing.

[0063] 6. Avatar Integration

[0064] If the user chooses to use an avatar, the server integrates the avatar into the video, synchronizing the avatar's movements and facial expressions with the synthesized voice to create a natural presentation.

[0065] 7. Creating and providing presentation videos

[0066] The server uses the video integration module to generate a presentation video that integrates slides, synthesized speech, subtitles, and avatars, and the generated video is provided to the user, who can download it or watch it online using their device.

[0067] Specific examples

[0068] For example, when converting in-house training materials into a video presentation, the following steps are performed. First, the user uploads the PDF file of the training materials and enters a narration prompt such as, "Please explain the features of the new product on this slide." The server analyzes the file and generates a talk script based on the prompt. Next, the server converts the talk script into synthesized speech and generates subtitles based on that speech. If the user wishes, an avatar can be integrated and made to move in sync with the synthesized speech. Finally, a video presentation incorporating all elements is generated and provided to the user.

[0069] In this way, the present invention can significantly reduce the user's efforts and improve time efficiency, and also makes it possible to easily generate high-quality presentation videos.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The user uploads digital files for presentations to the system through a terminal, and inputs narration prompts through the user interface.

[0073] Step 2:

[0074] The server receives the uploaded digital file, determines the file format, and, in the case of a PDF file, extracts the text and images for each page using a library such as Apache PDFBox.

[0075] Step 3:

[0076] The server's file analysis module organizes the text data by slide, extracting page numbers and header information, then uses natural language processing (NLP) techniques to semantically analyze the extracted text and identify key elements.

[0077] Step 4:

[0078] The server's prompt analysis module analyzes the narration prompts from the user and generates a talk script, which includes technology to automatically generate appropriate explanatory text for each slide based on the prompts.

[0079] Step 5:

[0080] The server's synthetic speech generation module passes the talk script to a synthetic speech engine to generate the narration. The speech is generated using a service such as Google Cloud Text-to-Speech, and post-processing is performed to adjust the tone and intonation.

[0081] Step 6:

[0082] The subtitle generation module of the server generates subtitles based on the content of the narration, specifically, by analyzing the timing of the narration using speech recognition technology and converting the corresponding text into a subtitle format.

[0083] Step 7:

[0084] When a user requests the use of an avatar, the server adds the avatar to the system, setting the avatar's lip sync (mouth movements) and facial animation based on the timing of the synthesized voice.

[0085] Step 8:

[0086] The server's video synthesis module synthesizes each slide, narration, subtitles, and avatar to generate a presentation video, using video editing software (e.g., FFmpeg) to output the video with optimal image and sound quality.

[0087] Step 9:

[0088] The server saves the generated presentation video and makes it accessible to users by uploading the video to cloud storage and providing users with a download link.

[0089] Step 10:

[0090] Users can download the presentation video or watch it online using the provided link on their devices. Users can check the video quality and make adjustments if necessary.

[0091] This is the specific processing flow of the program.

[0092] Example 1

[0093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0094] Creating traditional presentation videos is time-consuming and laborious, and requires significant effort to adjust the quality of the narration and synchronize subtitles. It is also difficult to adjust the intonation and timing of the narration, and using avatars for natural presentations is also time-consuming. There is a need to solve these issues, reduce the user's workload, and efficiently generate high-quality presentation videos.

[0095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0096] In this invention, the server includes means for receiving a digital presentation file input by a user, means for analyzing the received digital file and converting the content of each slide into text data, means for analyzing narration prompts input by the user and generating a talk script for each slide, means for generating synthetic speech based on the generated talk script, means for generating subtitles based on the generated synthetic speech, means for evaluating the quality of the generated synthetic speech and adjusting the tone and intonation as necessary, means for setting a time code for the generated subtitles and adjusting their display timing, means for integrating an avatar selected by the user into a presentation video and synchronizing it with the synthetic speech, means for integrating each slide, the generated synthetic speech, the generated subtitles, and the avatar to generate a presentation video, and means for providing the generated presentation video to the user. This enables automatic generation of high-quality presentation videos while reducing the user's workload and improving time efficiency.

[0097] A "digital file for presentation" is a file in an electronic format, such as a PDF or PowerPoint, that a user uses to give a presentation.

[0098] "Analysis" is the process of breaking down and analyzing the content of received digital files and narration prompts to extract the necessary information.

[0099] "Text data" refers to data that expresses the content of each slide and narration prompts as text information.

[0100] A "narration prompt" is an instruction entered by the user, and is the original text that is generated as a talk script for each slide.

[0101] A "talk script" is a sentence of speech content corresponding to each slide, generated based on a narration prompt.

[0102] "Synthetic voice" refers to the generation of a talk script as voice data, and is a voice that imitates a human voice using synthetic voice technology.

[0103] "Subtitles" are texts that are generated based on synthesized speech and displayed on the screen, providing a visual complement to the narration content.

[0104] "Time code" is time information that indicates the timing of displaying and playing subtitles and narration.

[0105] An "avatar" is a computer-generated character used to give a presentation in place of a human.

[0106] "Video synthesis" is the process of combining individual slides, synthesized speech, subtitles, and avatars into a single video presentation.

[0107] "Providing" means distributing the generated presentation video to users in a form that allows them to view it online or download it.

[0108] MODE FOR CARRYING OUT THE INVENTION

[0109] This invention provides a system for efficiently generating presentation videos and reducing the user's workload. This system automates a series of processes: analyzing digital files for presentations, generating and integrating synthesized speech, subtitles, and avatars, and finally providing the presentation video.

[0110] Basic system configuration

[0111] The system mainly consists of the following components:

[0112] 1. User Interface

[0113] An interface for users to upload digital files for presentations and enter narration prompts.

[0114] It is typically implemented as a web-based application or a dedicated desktop application.

[0115] 2. Server Module

[0116] It is the main module that receives, analyzes and processes digital files.

[0117] Contains the following submodules:

[0118] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[0119] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[0120] Synthetic speech generation module: Generates synthetic speech based on the talk script.

[0121] Subtitle generation module: Generates subtitles based on synthesized speech.

[0122] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[0123] Program processing and data processing

[0124] Below, we will explain the details of this system's processing and the hardware and software used at each step.

[0125] 1. User request input

[0126] Users upload digital files (e.g., PDF, PowerPoint) for presentations to their devices through a dedicated web interface or desktop application, and simultaneously input prompts for narration.

[0127] Specifically, users upload files by dragging and dropping them into a form in their browser or using a file selection dialog, and enter narration prompts into text fields.

[0128] 2. Receiving and analyzing the file

[0129] The server receives the uploaded digital file and determines the file format using an appropriate library (e.g., Apache PDFBox, Apache POI).

[0130] Depending on the file format, the content of each slide is converted to text data. For example, if it is a PDF file, Apache PDFBox is used, and if it is a PowerPoint file, Apache POI is used.

[0131] 3. Parsing the prompt

[0132] The server uses a natural language processing library (e.g., spaCy, NLTK) to analyze the input narration prompts and generates a talk script for each slide based on the analysis results.

[0133] Specifically, the system analyzes the grammar and key keywords of the narration prompts and uses a script generation algorithm to generate a semantically consistent talk script.

[0134] 4. Generating Synthetic Speech

[0135] The server uses a synthetic speech engine (e.g., Google Text-to-Speech API, Amazon Polly) to convert the generated talk script into synthetic speech. It adjusts the tone and intonation as necessary to ensure quality.

[0136] 5. Subtitle Generation

[0137] The server generates subtitles based on the synthesized speech, extracts timestamps using speech waveform analysis, and generates subtitle data (e.g., SRT format).

[0138] Specifically, timestamps are extracted from the audio data, and the display timing is adjusted by generating and setting the subtitle time code.

[0139] 6. Avatar Integration

[0140] If the user wishes to use an avatar, the server uses an avatar generation API (e.g., Adobe Character Animator, VYOND) to generate avatar movements in accordance with the synthesized voice.

[0141] The avatar's movements and facial expressions are synchronized with the narration to create a natural presentation.

[0142] 7. Creating and providing presentation videos

[0143] The server uses a video editing tool (e.g., FFmpeg) to generate a presentation video that integrates slides, synthesized speech, subtitles, and avatars.

[0144] The final video file is uploaded to cloud storage, allowing users to download it or watch it online via their devices.

[0145] Specific examples

[0146] For example, when converting materials for in-house training into a presentation video, the following process is carried out.

[0147] 1. User request input: The user uploads the PDF file of the training material through the web interface and inputs the narration prompt, "Please explain the features of the new product on this slide."

[0148] 2. Receiving and parsing the file: The server receives the PDF file, parses the content, and converts it into text data.

[0149] 3. Parsing prompts: The server parses the narration prompts and generates a talk script.

[0150] 4. Generate synthetic voice: The server generates synthetic voice based on the talk script.

[0151] 5. Subtitle generation: The server generates subtitles from the synthesized speech and adjusts the display timing.

[0152] 6. Avatar integration: If the user wishes, the server will generate an avatar and integrate it into the video.

[0153] 7. Generating and providing presentation video: The server generates a presentation video that integrates all elements and provides it to the user.

[0154] In this way, the present invention can significantly reduce the user's workload and improve time efficiency, and also makes it possible to easily generate high-quality presentation videos.

[0155] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0156] Step 1: User Request Input

[0157] Input: The user inputs the digital file for the presentation (e.g., PDF, PowerPoint) and the narration prompt.

[0158] What it does: A user opens a dedicated web interface or desktop application, clicks the file upload button to upload a digital file, and simultaneously enters a narration prompt (e.g., "This slide should describe the features of our new product") into a text field.

[0159] Output: The digital files and narration prompts are sent to the server.

[0160] Step 2: Receiving and analyzing the file

[0161] Input: The server receives the digital file sent by the user.

[0162] What happens: The server first uses the appropriate library to determine the file format of the received file (e.g., Apache PDFBox for PDF, Apache POI for PowerPoint).

[0163] Output: The content of each slide is extracted as text data.

[0164] Step 3: Parse the prompt

[0165] Input: The server parses the narration prompt received from the user.

[0166] Specific operation: The server uses a natural language processing library (e.g., spaCy, NLTK) to parse the prompt and extract grammar and important keywords.

[0167] Output: A talk script corresponding to each slide is generated.

[0168] Step 4: Generate synthetic speech

[0169] Input: The server receives the generated talk script.

[0170] How it works: The server calls a synthetic speech engine (e.g., Google Text-to-Speech API, Amazon Polly) to convert the talk script into voice data. At this time, the voice quality and intonation are automatically set.

[0171] Output: Synthetic speech data is generated.

[0172] Step 5: Generate subtitles

[0173] Input: The server receives the generated synthesized voice data.

[0174] Specific operation: The server uses an audio waveform analysis tool to extract the timestamp of the audio data and generate subtitle data (e.g., SRT format).

[0175] Output: Subtitle files are generated.

[0176] Step 6: Integrating the Avatar

[0177] Input: Receives the user-selected avatar, synthesized voice data, and subtitle data.

[0178] Specific operation: The server uses an avatar generation API (e.g., Adobe Character Animator, VYOND) to generate avatar movements in sync with the synthesized voice, adjusting the timing and movement, and integrating the avatar data with slides, audio, and subtitles.

[0179] Output: Presentation data with avatars integrated is generated.

[0180] Step 7: Generate and serve presentation videos

[0181] Input: The server receives the consolidated presentation data.

[0182] What happens: The server uses a video editing tool (e.g. FFmpeg) to combine all elements (slides, synthesized speech, subtitles, avatars) and export them as a presentation video.

[0183] Output: The generated presentation video will be uploaded to cloud storage and a download link or streaming URL will be provided to users.

[0184] (Application example 1)

[0185] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0186] Conventional presentation video generation systems impose a heavy workload on educators when creating educational content, and have issues with the quality and adaptability of the videos created. In particular, for educational presentations, the coordination of slide content with narration and subtitles is crucial, and adjusting these elements satisfactorily can be time-consuming. Furthermore, systems lack the flexibility to adapt to different educational environments and learner needs. The present invention aims to solve these issues by providing a system that enables educators to easily and automatically generate high-quality educational presentation videos.

[0187] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0188] In this invention, the server includes means for receiving a digital presentation file input by a user, means for analyzing the received digital file and converting the content of each slide into text data, means for analyzing narration prompts input by the user and generating a talk script for each slide, means for generating synthetic speech based on the generated talk script, means for generating subtitles based on the generated synthetic speech, means for integrating the slides, the generated synthetic speech, and the generated subtitles to generate a presentation video, means for providing the generated presentation video to the user, means for generating a presentation video optimized as educational content based on content selected by the user, and means for providing an interface that is easy for educators to use and for automatically generating and adjusting content based on educator input. This enables educators to easily create high-quality educational presentation videos.

[0189] "User" refers to an individual or organization that uses this system to generate a presentation video.

[0190] A "digital presentation file" is an electronic file, such as a PDF or PowerPoint file, that contains the slides of a presentation.

[0191] "Text data" refers to the written information extracted from the slides in the digital file.

[0192] "Narration prompt" refers to text that the user enters for each slide and that serves as instructions for generating a talk script.

[0193] A "talk script" is a presentation text that is automatically generated based on narration prompts.

[0194] "Synthetic voice" is artificial voice data that is automatically generated based on a talk script.

[0195] "Subtitles" refers to text information that is displayed in response to synthesized speech.

[0196] A "presentation video" is a dynamic video file that combines slides, synthesized audio, and subtitles.

[0197] "Educational Content" refers to educational materials and videos created by educators for students and participants.

[0198] "Interface" refers to the user interface through which a user accesses the system and uploads files or enters prompts.

[0199] This invention provides a system that allows educators to easily create high-quality educational presentation videos. The system provides a series of processes that automatically generate educational content, including synthesized speech and subtitles, by simply uploading a digital file for the presentation and entering narration prompts.

[0200] Basic system configuration

[0201] The system mainly consists of the following components:

[0202] 1. User Interface

[0203] An interface through which users upload digital files for a presentation and enter narration prompts. This interface is typically implemented as a web-based application or a dedicated desktop application.

[0204] 2. Server Module

[0205] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[0206] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[0207] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[0208] Synthetic speech generation module: Generates synthetic speech based on the talk script. Software used includes Google Text-to-Speech (gTTS).

[0209] Subtitle generation module: Generates subtitles based on synthesized speech.

[0210] Video Integration Module: Generates a video presentation by integrating slides, synthesized speech, and subtitles, and optionally integrating avatars.

[0211] 3. Educational Module

[0212] Based on user input, it generates presentation videos that are optimized for educational content and automatically adjusts to meet the educator's requirements.

[0213] Program processing

[0214] The processing of the program will be explained in natural language below.

[0215] 1. User request input

[0216] Users upload digital files for presentations (e.g., PDF, PowerPoint) to the system via a web or desktop application, and also enter narration prompts through the user interface.

[0217] 2. Analyzing the file

[0218] The server receives the uploaded digital files, and a file analysis module converts the content of each slide into text data, making the presentation organized and easier to handle within the system.

[0219] 3. Parsing the prompt

[0220] The server uses a prompt analysis module to analyze the narration prompts input by the user and generate a talk script corresponding to each slide.

[0221] 4. Generating Synthetic Speech and Subtitles

[0222] The server generates speech using a synthetic speech generation module based on the generated talk script, and creates corresponding subtitles using a subtitle generation module.

[0223] 5. Video Integration

[0224] The server uses a video integration module to integrate the slides, synthesized speech, and subtitles to generate a video presentation. If the user wishes, an avatar can also be used, which is also integrated into the video.

[0225] Specific examples

[0226] For example, if an educator is creating an educational presentation video on "Fundamental Concepts of Mathematics," they might do the following:

[0227] 1. Educators upload PowerPoint files of math teaching materials to the app.

[0228] 2. For each slide, include a prompt such as "Explain the difference between integers and natural numbers" or "On the next slide, provide an example of an integer."

[0229] 3. The server automatically generates narration and creates a presentation video.

[0230] Example prompt sentence:

[0231] "On this slide, explain the difference between integers and natural numbers."

[0232] "On the next slide, give an example of an integer."

[0233] "On the last slide, please summarize what you learned today."

[0234] The present invention allows educators to easily create high-quality educational presentation videos, thereby improving the quality of educational content.

[0235] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0236] Step 1:

[0237] A user uploads a digital file for a presentation to the system using a terminal, and simultaneously inputs a narration prompt corresponding to each slide through the interface, resulting in the digital file and the narration prompt as input data.

[0238] Input: Digital files for presentation, narration prompts

[0239] Output: Received digital files, narration prompts

[0240] Step 2:

[0241] The server analyzes the received digital files using a file analysis module, converting the content of each slide into text data and organizing it, thereby obtaining the text data for each slide.

[0242] Input: Received digital files

[0243] Output: Text data for each slide

[0244] Step 3:

[0245] The server uses a prompt analysis module to analyze the narration prompts entered by the user and generate a talk script corresponding to each slide. This analysis generates a talk script suitable for narration from the prompt text.

[0246] Input: Narration prompts, text data for each slide

[0247] Output: Talk script for each slide

[0248] Step 4:

[0249] The server uses a synthetic speech generation module to convert the generated talk script into synthetic speech, during which the quality of the synthetic speech is checked and adjusted if necessary.

[0250] Input: Talk script for each slide

[0251] Output: Synthesized voice data

[0252] Step 5:

[0253] The server generates subtitles using a subtitle generation module based on the synthesized speech, and the timing of the subtitles is set based on the time code of the synthesized speech.

[0254] Input: Synthetic voice data

[0255] Output: Subtitle data with timecode

[0256] Step 6:

[0257] The server uses a video integration module to integrate each slide, the synthesized speech, and the generated subtitles to generate a presentation video. If a user selects an avatar, the avatar is also integrated into the video in synchronization with the synthesized speech.

[0258] Input: Slide image data, synthesized voice data, subtitle data

[0259] Output: Presentation video data

[0260] Step 7:

[0261] The server provides the generated presentation video to the user, who can then download the video or watch it online using their device.

[0262] Input: Presentation video data

[0263] Output: Presentation video provided to the user

[0264] Step 8:

[0265] The server optimizes the generated presentation video as educational content, creating a video in a format that is easy for educators to use.

[0266] Input: Presentation video data

[0267] Output: Presentation video optimized for education

[0268] Step 9:

[0269] Users can use the video presentations optimized for education to achieve their educational goals by providing them to learners.

[0270] Input: Presentation video optimized for education

[0271] Output: Educational presentation use

[0272] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0273] MODE FOR CARRYING OUT THE INVENTION

[0274] This invention provides a system for efficiently generating presentation videos and providing high-quality videos that take user emotions into consideration. This system automates the entire process of analyzing digital files for presentations, generating synthesized speech and subtitles, and finally providing the presentation video, and further combines it with an emotion engine that recognizes user emotions.

[0275] Basic system configuration

[0276] The system mainly consists of the following components:

[0277] 1. User Interface

[0278] An interface through which users upload digital files for a presentation and enter narration prompts, typically implemented as a web-based application or a dedicated desktop application.

[0279] 2. Server Module

[0280] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[0281] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[0282] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[0283] Emotion Engine: Recognizes the user's emotions when entering narration prompts and takes them into account during analysis.

[0284] Synthetic speech generation module: Generates synthetic speech based on the talk script. Adjusts the tone and pace of the synthetic speech based on the user's emotions.

[0285] Subtitle generation module: Generates subtitles based on synthesized speech. Adjusts the expression of the subtitles based on the user's emotions.

[0286] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[0287] Program processing

[0288] The processing of the program will be explained in natural language below.

[0289] 1. User request input

[0290] Users upload digital presentation files (e.g., PDF, PowerPoint) to the system using their terminals, and also enter narration prompts through the user interface.

[0291] 2. Analyzing the file

[0292] The server receives the uploaded digital files and identifies the file format. A file analysis module then converts the content of each slide into text data, sorting out page numbers and header information. The extracted text data is then semantically analyzed using natural language processing (NLP) techniques to identify important elements.

[0293] 3. Prompt analysis and emotion recognition

[0294] The server's prompt analysis module analyzes the narration prompts entered by the user. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice and influences the content of the prompt. For example, if the user expresses "joy" as their emotion, a talk script with positive expressions will be generated.

[0295] 4. Generating Synthetic Speech and Subtitles

[0296] The server's synthetic speech generation module converts the talk script into synthetic speech based on the results of the emotion engine. The tone and pace of the synthetic speech are adjusted based on the user's emotions. The subtitle generation module then generates subtitles based on the narration content, reflecting the user's emotions.

[0297] 5. Avatar Integration

[0298] If a user wishes to use an avatar, the server adds the avatar to the system and sets the avatar's lip sync and facial animation to match the synthesized voice.

[0299] 6. Creating and providing presentation videos

[0300] The server uses a video integration module to integrate slides, synthesized speech, subtitles, and avatars to generate a presentation video, which is then provided to users for download or online viewing on their devices.

[0301] Specific examples

[0302] For example, consider the creation of a marketing presentation. A user uploads marketing materials in PDF format and enters a narration prompt such as, "This slide should highlight the benefits of the product." As the user enters the narration prompt, an emotion engine recognizes the user's emotions, such as "excitement" or "anticipation." The server analyzes the file and generates a positive and emphatic talk script based on the emotion engine's results. A synthetic voice is then generated from the talk script, resulting in an expressive narration. Subtitles also reflect the user's emotions and use impactful expressions. If an avatar is added, the avatar will similarly behave to express excitement or anticipation. Finally, a video presentation integrating all elements is generated and served to the user.

[0303] In this way, the present invention can provide more effective and empathetic presentation videos that take into account the user's emotions, improving the quality and emotional appeal of presentations while also improving the efficiency of presentations.

[0304] The processing flow will be explained below.

[0305] Step 1:

[0306] The user uploads digital files for presentations to the system through a terminal, and inputs narration prompts through the user interface.

[0307] Step 2:

[0308] The server receives the uploaded digital file, determines the file format, and, in the case of PDF files, extracts the text and images page by page using a library such as Apache PDFBox.

[0309] Step 3:

[0310] The server's file analysis module organizes the text data by slide, extracting page numbers and header information, then uses natural language processing (NLP) techniques to semantically analyze the extracted text and identify key elements.

[0311] Step 4:

[0312] The server's prompt analysis module analyzes the narration prompts entered by the user. The emotion engine recognizes the user's emotions and reflects them in the prompt content. For example, if the user expresses "joy," a positive talk script is generated.

[0313] Step 5:

[0314] The server's synthetic speech generation module passes the talk script to a synthetic speech engine to generate the narration. The tone and pace of the speech are adjusted based on the user's emotions. Services such as Google Cloud Text-to-Speech may be used to generate the synthetic speech.

[0315] Step 6:

[0316] The server's subtitle generation module generates subtitles based on the content of the narration. It uses speech recognition technology to analyze the timing of the narration and produces text that reflects the user's emotions. For example, a cheerful expression that reflects the emotion of "joy" is used.

[0317] Step 7:

[0318] When a user requests to use an avatar, the server adds the avatar to the system. At this time, the server synchronizes the avatar's lip sync (mouth movements) and facial animation with the synthesized voice. The avatar also behaves in a way that expresses the user's emotions.

[0319] Step 8:

[0320] The server's video integration module integrates each slide, synthesized speech, subtitles, and avatars to generate a presentation video, which is then output with optimal image and sound quality using video editing software such as FFmpeg.

[0321] Step 9:

[0322] The server saves the generated presentation video and makes it accessible to users by uploading the video to cloud storage and providing users with a download link.

[0323] Step 10:

[0324] Users can download the presentation video or watch it online using the provided link on their devices. Users can check the video quality and make adjustments if necessary.

[0325] This is the specific processing flow of the program.

[0326] Example 2

[0327] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0328] The process of creating presentation videos manually is time-consuming and labor-intensive, making it inefficient. Furthermore, conventional systems struggle to generate videos that take user emotions into account, making it difficult to create high-quality presentation videos with emotional appeal. Furthermore, manually adjusting the intonation and timing of narration is tedious, making the creation of an integrated presentation video complex.

[0329] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a digital file for presentation input by a user, means for analyzing the received digital file and converting the content of each slide into text data, means for analyzing narration prompts input by the user and generating a talk script for each slide, means for recognizing the user's emotions and analyzing and generating narration prompts based on the emotions, means for generating synthetic speech based on the generated talk script, means for generating subtitles based on the generated synthetic speech, means for integrating the slides, the generated synthetic speech, and the generated subtitles to generate a presentation video, and means for providing the generated presentation video to the user. This enables automatic generation of high-quality presentation videos that take user emotions into consideration.

[0330] A "digital file for presentation" is a document for presentation that is saved in a format such as PDF or PowerPoint.

[0331] A "Presentation Video" is a video presentation generated by integrating digital file slides, narration audio, subtitles, and avatars.

[0332] A "narration prompt" is text that a user enters for each slide in a presentation to indicate what should be explained or what points should be discussed.

[0333] A "talk script" is a text of speech content corresponding to each slide of a presentation, which is generated based on file analysis and prompt analysis.

[0334] "Synthetic voice" is a mechanically generated voiced narration based on a talk script.

[0335] "Subtitles" are texts that are visually displayed within a presentation video and are used to display the contents of synthesized speech in text format.

[0336] An "avatar" is an image or animation of a character used in a presentation video that moves in sync with the synthesized voice.

[0337] An "emotion engine" is software that analyzes a user's facial expressions and voice to recognize their emotions and reflects the results in the presentation generation process.

[0338] MODE FOR CARRYING OUT THE INVENTION

[0339] System Overview

[0340] This invention provides a system for efficiently generating presentation videos and providing high-quality videos that take user emotions into consideration. This system automates the entire process of analyzing digital files for presentations, generating synthesized speech and subtitles, and finally providing the presentation video, and further combines it with an emotion engine that recognizes user emotions.

[0341] System Configuration

[0342] The system consists of the following main components:

[0343] 1. User Interface

[0344] An interface for users to upload digital files for presentations and enter narration prompts, implemented as a web-based application or a dedicated desktop application.

[0345] 2. Server Module

[0346] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[0347] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[0348] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[0349] Emotion Engine: Recognizes the user's emotions when entering narration prompts and takes them into account during analysis. For example, if the user indicates "joy" as their emotion, a talk script with positive expressions will be generated.

[0350] Synthetic speech generation module: Generates synthetic speech based on the talk script. Adjusts the tone and pace of the synthetic speech based on the user's emotions.

[0351] Subtitle generation module: Generates subtitles based on synthesized speech. Adjusts the expression of the subtitles based on the user's emotions.

[0352] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[0353] Operation explanation

[0354] 1. User request input

[0355] The user uploads digital files (e.g., PDF, PowerPoint) for the presentation through a user interface on the device, and also enters narration prompts through the user interface.

[0356] 2. Analyzing the file

[0357] The server receives the uploaded digital files and uses a file analysis module to determine the file format, which then converts the content of each slide into text data, which is then analyzed using natural language processing (NLP) techniques.

[0358] 3. Prompt analysis and emotion recognition

[0359] The server's prompt analysis module analyzes the narration prompts entered by the user. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice and reflects them in the prompt analysis. For example, if the user expresses "joy," a talk script with positive expressions is generated.

[0360] 4. Generating Synthetic Speech and Subtitles

[0361] The server's synthetic speech generation module converts the talk script into synthetic speech based on the results of the emotion engine. The tone and pace are adjusted based on the user's emotions. The subtitle generation module then generates subtitles based on the narration content, reflecting the user's emotions.

[0362] 5. Avatar Integration

[0363] If a user wishes to use an avatar, the server adds the avatar to the system and sets the avatar's lip sync and facial animation to match the synthesized voice.

[0364] 6. Creating and providing presentation videos

[0365] The server uses a video integration module to integrate slides, synthesized speech, subtitles, and avatars to generate a presentation video, which is then provided to users for download or online viewing on their devices.

[0366] Specific examples

[0367] For example, consider a user creating a market analysis presentation. The user uploads the market analysis document in PDF format and enters a narration prompt such as, "In this slide, please emphasize the advantages of the new product." When the user enters the narration prompt, the emotion engine recognizes the user's "expectations" and generates a talk script based on them. The server analyzes the file and generates content that emphasizes important points based on the results of the emotion engine. Synthetic speech is then generated from the talk script, and subtitles also reflect the user's emotions. If an avatar is added, the avatar will also behave in a similar way, reflecting the user's emotions. Finally, a video presentation integrating all elements is generated and provided to the user.

[0368] Prompt Sentence Examples

[0369] 1. "Highlight the key points of your marketing strategy."

[0370] 2. "This slide should explain your efforts to improve customer satisfaction."

[0371] 3. "Please briefly summarize the technical features of your product."

[0372] In this way, the system of the present invention can take into account the user's emotions and provide more effective and empathetic presentation videos, improving the quality and emotional appeal of presentations while also improving the efficiency of presentations.

[0373] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0374] Step 1: User Request Input

[0375] ---

[0376] Users use terminals to access the system's user interface, upload digital files (e.g., PDF, PowerPoint) for presentations, and input narration prompts for the presentation content.

[0377] Input: Digital files for presentation, narration prompts

[0378] Output: Uploaded digital files, typed narration prompts

[0379] Specific behavior: The user interface provides buttons for uploading files and text boxes for entering narration prompts. Once the user enters the file and prompt, they are sent to the server.

[0380] Step 2: Receiving and analyzing the file

[0381] ---

[0382] The server receives the uploaded digital files and uses a file analysis module to convert the content of each slide into text data, identifying the file format and organizing page numbers and header information.

[0383] Input: Uploaded digital files

[0384] Output: Parsed text data

[0385] What it does: The server's file analysis module reads PDF or PowerPoint files, extracts the text within each slide, and uses natural language processing (NLP) techniques to identify important elements.

[0386] Step 3: Parsing the prompt and recognizing emotions

[0387] ---

[0388] The server's prompt analysis module analyzes the narration prompts entered by the user and generates a talk script. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice, and reflects those emotions in the prompt analysis.

[0389] Input: Narration prompts, user emotion information

[0390] Output: Talk script reflecting emotions

[0391] Specific operation: The emotion engine analyzes the user's facial expression and voice data acquired through the user interface, identifies the user's emotional state based on this data, and uses this information to generate a talk script appropriate for the narration prompt.

[0392] Step 4: Generate synthetic speech and subtitles

[0393] ---

[0394] The server uses a synthetic speech generation module to generate synthetic speech based on the talk script, adjusting the tone and pace of the speech based on the user's emotions. The subtitle generation module converts the content of the synthetic speech into text data and generates subtitles.

[0395] Input: Talk script, emotion information

[0396] Output: Synthesized speech, generated subtitles

[0397] Specific operation: The synthetic speech generation module receives the talk script as input data and dynamically adjusts the tone and pace of the speech based on the user's emotional information. The generated speech data is then converted into text and passed to the subtitle generation module, which finally creates subtitles for the presentation.

[0398] Step 5: Integrating the Avatar

[0399] ---

[0400] If a user wishes to use an avatar, the server adds the avatar to the system and sets the avatar's lip sync and facial animation to match the synthesized voice.

[0401] Input: Synthetic voice, user avatar preference

[0402] Output: Avatar with lip sync and animation added

[0403] Specific behavior: The server loads the avatar selected by the user from available avatar resources and sets the behavior based on the synthesized voice. Lip sync and facial animation are dynamically reflected based on emotional information, resulting in more natural behavior.

[0404] Step 6: Generate and serve presentation videos

[0405] ---

[0406] The server uses a video synthesis module to synthesize each slide, the generated synthetic speech, subtitles, and avatars into a single presentation video, which is then provided to users for download or online viewing.

[0407] Input: Slide data, synthesized speech, generated subtitles, avatar data

[0408] Output: Integrated presentation video

[0409] How it works: The video integration module combines the content of each slide, the generated audio, subtitles, and avatars, arranges them on a timeline, and generates a video file. The final presentation video is saved in a specified location for user access. For online viewing, a URL is provided to the user.

[0410] (Application example 2)

[0411] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0412] Conventional presentation video generation systems require users to input slide content individually, create narration, and generate synthesized speech, which is a complex process. Furthermore, because the system automatically generates the presentation without considering the user's emotions, it lacks appeal to viewers. Therefore, there is a need for a system that can efficiently generate presentation videos that respond to the user's emotions.

[0413] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving a digital file for presentation input by a user; means for analyzing the received digital file and converting the content of each slide into text data; means for analyzing narration prompts input by the user and generating a talk script for each slide; means for generating synthetic speech based on the generated talk script; means for generating subtitles based on the generated synthetic speech; means for recognizing a user's emotion and adjusting the tone and pace of the narration based on the recognized emotion; means for integrating each slide, the generated synthetic speech, the generated subtitles, and the recognized emotion information to generate a presentation video; and means for providing the generated presentation video to the user. This makes it possible to efficiently generate a highly appealing presentation video that is tailored to the user's emotion.

[0414] "Digital files for presentations" refer to slides and materials that users use for presentation purposes and are provided in digital formats such as PDF and PowerPoint.

[0415] "Narration prompt" refers to the content or wording of the narration entered by the user, including explanations and commentary for each slide.

[0416] A "talk script" is a spoken script generated by analyzing narration prompts, and is a narration instruction manual provided for each slide.

[0417] "Synthetic voice" is an artificial voice generated based on a talk script, and is reproduced to imitate human speech.

[0418] "Subtitles" are textual information that displays the contents of the synthesized speech, and serve to visually supplement the contents of the narration.

[0419] "Emotion information" is data of emotions recognized by analyzing the user's facial expressions, voice, etc., and indicates an emotional state such as joy, sadness, excitement, etc.

[0420] "Tone and pace adjustment" refers to changing the speaking style and speed of the synthesized voice based on emotional information, as a means of providing a more emotionally rich presentation.

[0421] "Video synthesis" is the process of combining slides, synthesized audio, and subtitles into a single video presentation that viewers can watch as a continuous story.

[0422] An "avatar" is a digital character that can be selected by the user to appear in the presentation video, and has lip-sync and facial animation.

[0423] This system analyzes a digital presentation file input by a user, generates a talk script based on narration prompts entered by the user, creates synthesized speech and subtitles, and provides a presentation video that integrates them. It is particularly notable for recognizing the user's emotions and adjusting the tone and pace of the narration based on those emotions.

[0424] Basic system configuration

[0425] 1. User Interface

[0426] Users use a device to upload digital files for presentations and enter narration prompts. The user interface may be implemented as a web-based application or a dedicated desktop application, typically a web application running in a browser.

[0427] 2. Server Module

[0428] The server includes the following sub-modules to analyze the received digital files and extract the content of each slide as text data:

[0429] File Analysis Module: Identifies the digital file format (PDF, PowerPoint, etc.) of the uploaded presentation and extracts the text data for each slide.

[0430] Prompt Analysis Module: Analyzes the narration prompts entered by the user and generates a talk script for each slide.

[0431] Emotion Recognition Engine: Recognizes emotions from the user's facial expressions and voice data. This is achieved by the user using the device's camera and microphone.

[0432] Synthetic voice generation module: Creates a synthetic voice based on the generated talk script, and adjusts the tone and pace according to the user's emotions.

[0433] Subtitle generation module: Generates subtitles in synchronization with the synthesized speech. The subtitles also include expressions based on the user's emotions.

[0434] Video Integration Module: Generates a presentation video by integrating each slide, synthesized speech, subtitles, and emotional information.

[0435] 3. Use of Avatars

[0436] A user-selected avatar can be integrated into the presentation video, lip-syncing to the generated synthetic speech and changing facial expressions based on emotions, enhancing the visual and emotional appeal.

[0437] Program processing explanation

[0438] The server uses the following hardware and software:

[0439] Hardware: Camera, microphone, user device (PC, smartphone, etc.)

[0440] Software: OpenCV, Natural Language Processing Library (NLP), Emotion Recognition Engine, Synthetic Speech Generation Library

[0441] Specific examples

[0442] For example, when creating a marketing presentation, a user uploads marketing materials in PDF format and enters a narration prompt such as, "This slide should highlight the benefits of the product." As the user enters the narration prompt, an emotion recognition engine recognizes the user's "joy." The server analyzes the file and generates a talk script that emphasizes a positive tone based on the emotion recognition results. Synthetic speech is then generated from this talk script, creating an expressive narration. Subtitles also reflect the user's emotions and use impactful expressions. If an avatar is added, the avatar will also behave in a similar way, expressing joy. Finally, a video presentation integrating all elements is generated and provided to the user.

[0443] Prompt Sentence Examples

[0444] "Analyze the text of marketing presentation slides, generate a narration that adjusts the tone and pace based on user emotion, add subtitles, and create a video of the presentation. The narration prompt is "Highlight the product benefits on this slide," and the user emotion is "Delight."

[0445] In this way, the present invention takes into account the user's emotions and can provide more effective and appealing presentation videos, improving the efficiency and quality of presentation creation and eliciting empathy from viewers.

[0446] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0447] Step 1:

[0448] User request input

[0449] Users use their terminals to upload digital files (e.g., PDF, PowerPoint) for presentations to the server and input narration prompts through a user interface.

[0450] Input: Digital files for presentation and narration prompts

[0451] Output: Received digital files and prompt data

[0452] Specific operation: The terminal displays a web form for uploading a file, from which the user selects a file and enters text for the narration prompt.

[0453] Step 2:

[0454] Analyzing files

[0455] The server's file analysis module analyzes the received digital files, extracting the content of each slide as text data and organizing page numbers and header information.

[0456] Input: Received digital presentation file

[0457] Output: Text data for each slide

[0458] Specific operation: The server determines the file format and extracts the text data using an appropriate library (e.g., PyMuPDF).

[0459] Step 3:

[0460] Parsing prompts

[0461] The server's prompt analysis module analyzes the narration prompts entered by the user and generates a talk script, while the emotion recognition engine recognizes the user's emotions and influences the content of the prompts.

[0462] Input: User-supplied narration prompts and user emotion data

[0463] Output: A talk script that reflects the user's emotions

[0464] Specific operation: The server uses an emotion recognition engine (e.g., OpenCV, dlib) to analyze the user's emotions and a natural language processing library (e.g., spaCy) to analyze the narration prompts.

[0465] Step 4:

[0466] Generate synthetic speech

[0467] The server's synthetic voice generation module generates synthetic voice based on the talk script and adjusts the tone and pace according to the user's emotions.

[0468] Input: A talk script that reflects the user's emotions

[0469] Output: Emotionally adjusted synthetic speech

[0470] Specific operation: The server generates an audio file using a synthetic speech generation library (e.g., Google Text-to-Speech API) and processes the audio (adjusts tone and pace) based on emotions.

[0471] Step 5:

[0472] Subtitle generation

[0473] A subtitle generation module of the server generates subtitles based on the synthesized speech, and includes expressions according to the user's emotions.

[0474] Input: Synthesized voice and talk script

[0475] Output: Emotionally sensitive subtitle data

[0476] Specific operation: The server uses a speech recognition library (e.g., Google Speech-to-Text API) to generate text from the speech and creates subtitle data in a style that corresponds to the user's emotions.

[0477] Step 6:

[0478] Presentation video generation

[0479] The video integration module of the server integrates each slide, synthesized speech, subtitles, and emotional information to generate a presentation video.

[0480] Input: Each slide, synthesized speech, subtitle data, emotional information

[0481] Output: Finished presentation video

[0482] Specific operation: The server uses a video editing library (e.g., FFmpeg) to integrate each element and generate a video file.

[0483] Step 7:

[0484] Providing generated presentation videos

[0485] The generated presentation video is provided to the user, who can download the video or watch it online using their device.

[0486] Input: Finished presentation video

[0487] Output: User-accessible video link of the presentation

[0488] Specific operation: The server stores the video file in storage and provides a download link or streaming URL to the user's device.

[0489] This processing flow enables users to efficiently generate emotionally-reflective presentation videos and deliver visually and emotionally appealing presentations.

[0490] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0491] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0492] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0493] [Second embodiment]

[0494] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0495] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0496] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0497] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0498] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0499] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0500] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0501] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0502] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0503] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0504] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0505] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0506] MODE FOR CARRYING OUT THE INVENTION

[0507] The present invention provides a system for efficiently generating presentation videos and reducing the user's workload by automating a series of processes: analyzing digital files for presentations, generating synthesized speech and subtitles, and finally providing the presentation videos.

[0508] Basic system configuration

[0509] The system mainly consists of the following components:

[0510] 1. User Interface

[0511] An interface through which users upload digital files for a presentation and enter narration prompts, typically implemented as a web-based application or a dedicated desktop application.

[0512] 2. Server Module

[0513] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[0514] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[0515] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[0516] Synthetic speech generation module: Generates synthetic speech based on the talk script.

[0517] Subtitle generation module: Generates subtitles based on synthesized speech.

[0518] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[0519] Program processing

[0520] The processing of the program will be explained in natural language below.

[0521] 1. User request input

[0522] Users upload digital presentation files (e.g., PDF, PowerPoint) to the system using their terminals, and also enter narration prompts through the user interface.

[0523] 2. Analyzing the file

[0524] The server receives the uploaded digital files, identifies the file format, and then a file analysis module converts the content of each slide into text data, sorting out page numbers and header information.

[0525] 3. Parsing the prompt

[0526] The server uses a prompt parsing module to parse the input narration prompts, which then generates a talking script that corresponds to the content of each slide.

[0527] 4. Generating Synthetic Speech

[0528] The server utilizes a synthetic speech generation module to convert the talk script into synthetic speech, and in the process evaluates the quality of the generated speech and adjusts the tone and intonation if necessary.

[0529] 5. Subtitle Generation

[0530] The server uses a subtitle generation module to generate subtitles based on the generated synthesized speech, setting the time code of the subtitles to match the timing of the narration and adjusting the display timing.

[0531] 6. Avatar Integration

[0532] If a user chooses to use an avatar, the server integrates the avatar into the video, synchronizing the avatar's movements and facial expressions with the synthesized voice to create a natural presentation.

[0533] 7. Creating and providing presentation videos

[0534] The server uses the video integration module to generate a presentation video that integrates slides, synthesized speech, subtitles, and avatars, and the generated video is provided to the user, who can download it or watch it online using their device.

[0535] Specific examples

[0536] For example, when converting in-house training materials into a video presentation, the following steps are performed. First, the user uploads the PDF file of the training materials and enters a narration prompt, such as "Please explain the features of the new product on this slide." The server analyzes the file and generates a talk script based on the prompt. Next, the server converts the talk script into synthesized speech and generates subtitles based on that speech. If the user wishes, an avatar can be integrated and made to move in sync with the synthesized speech. Finally, a video presentation incorporating all elements is generated and provided to the user.

[0537] In this way, the present invention can significantly reduce the user's efforts and improve time efficiency, and also makes it possible to easily generate high-quality presentation videos.

[0538] The processing flow will be explained below.

[0539] Step 1:

[0540] The user uploads digital files for presentations to the system through a terminal, and inputs narration prompts through the user interface.

[0541] Step 2:

[0542] The server receives the uploaded digital file, determines the file format, and, in the case of PDF files, extracts the text and images page by page using a library such as Apache PDFBox.

[0543] Step 3:

[0544] The server's file analysis module organizes the text data by slide, extracting page numbers and header information, then uses natural language processing (NLP) techniques to semantically analyze the extracted text and identify key elements.

[0545] Step 4:

[0546] The server's prompt analysis module analyzes the narration prompts from the user and generates a talk script, which includes technology to automatically generate appropriate explanatory text for each slide based on the prompts.

[0547] Step 5:

[0548] The server's synthetic speech generation module passes the talk script to a synthetic speech engine to generate the narration, using services such as Google Cloud Text-to-Speech, followed by post-processing to adjust the tone and intonation.

[0549] Step 6:

[0550] The subtitle generation module of the server generates subtitles based on the content of the narration, specifically, by analyzing the timing of the narration using speech recognition technology and converting the corresponding text into a subtitle format.

[0551] Step 7:

[0552] When a user requests the use of an avatar, the server adds the avatar to the system, setting the avatar's lip sync (mouth movements) and facial animation based on the timing of the synthesized voice.

[0553] Step 8:

[0554] The server's video synthesis module synthesizes each slide, narration, subtitles, and avatar to generate a presentation video, using video editing software (e.g., FFmpeg) to output the video with optimal image and sound quality.

[0555] Step 9:

[0556] The server saves the generated presentation video and makes it accessible to users by uploading the video to cloud storage and providing users with a download link.

[0557] Step 10:

[0558] Users can download the presentation video or watch it online using the provided link on their devices. Users can check the video quality and make adjustments if necessary.

[0559] This is the specific processing flow of the program.

[0560] Example 1

[0561] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0562] Creating traditional presentation videos is time-consuming and laborious, and requires significant effort to adjust the quality of the narration and synchronize subtitles. It is also difficult to adjust the intonation and timing of the narration, and using avatars for natural presentations is also time-consuming. There is a need to solve these issues, reduce the user's workload, and efficiently generate high-quality presentation videos.

[0563] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0564] In this invention, the server includes means for receiving a digital presentation file input by a user, means for analyzing the received digital file and converting the content of each slide into text data, means for analyzing narration prompts input by the user and generating a talk script for each slide, means for generating synthetic speech based on the generated talk script, means for generating subtitles based on the generated synthetic speech, means for evaluating the quality of the generated synthetic speech and adjusting the tone and intonation as necessary, means for setting a time code for the generated subtitles and adjusting their display timing, means for integrating an avatar selected by the user into a presentation video and synchronizing it with the synthetic speech, means for integrating each slide, the generated synthetic speech, the generated subtitles, and the avatar to generate a presentation video, and means for providing the generated presentation video to the user. This enables automatic generation of high-quality presentation videos while reducing the user's workload and improving time efficiency.

[0565] A "digital file for presentation" is a file in an electronic format, such as a PDF or PowerPoint, that a user uses to give a presentation.

[0566] "Analysis" is the process of breaking down and analyzing the content of received digital files and narration prompts to extract the necessary information.

[0567] "Text data" refers to data that expresses the content of each slide and narration prompts as text information.

[0568] A "narration prompt" is an instruction entered by the user, and is the original text that is generated as a talk script for each slide.

[0569] A "talk script" is a sentence of speech content corresponding to each slide, generated based on a narration prompt.

[0570] "Synthetic voice" refers to the generation of a talk script as voice data, and is a voice that imitates a human voice using synthetic voice technology.

[0571] "Subtitles" are texts that are generated based on synthesized speech and displayed on the screen, providing a visual complement to the narration content.

[0572] "Time code" is time information that indicates the timing of displaying and playing subtitles and narration.

[0573] An "avatar" is a computer-generated character used to give a presentation in place of a human.

[0574] "Video synthesis" is the process of combining individual slides, synthesized speech, subtitles, and avatars into a single video presentation.

[0575] "Providing" means distributing the generated presentation video to users in a form that allows them to view it online or download it.

[0576] MODE FOR CARRYING OUT THE INVENTION

[0577] This invention provides a system for efficiently generating presentation videos and reducing the user's workload. This system automates a series of processes: analyzing digital files for presentations, generating and integrating synthesized speech, subtitles, and avatars, and finally providing the presentation video.

[0578] Basic system configuration

[0579] The system mainly consists of the following components:

[0580] 1. User Interface

[0581] An interface for users to upload digital files for presentations and enter narration prompts.

[0582] It is typically implemented as a web-based application or a dedicated desktop application.

[0583] 2. Server Module

[0584] It is the main module that receives, analyzes and processes digital files.

[0585] Contains the following submodules:

[0586] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[0587] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[0588] Synthetic speech generation module: Generates synthetic speech based on the talk script.

[0589] Subtitle generation module: Generates subtitles based on synthesized speech.

[0590] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[0591] Program processing and data processing

[0592] Below, we will explain the details of this system's processing and the hardware and software used at each step.

[0593] 1. User request input

[0594] Users upload digital files (e.g., PDF, PowerPoint) for presentations to their devices through a dedicated web interface or desktop application, and simultaneously input prompts for narration.

[0595] Specifically, users upload files by dragging and dropping them into a form in their browser or using a file selection dialog, and enter narration prompts into text fields.

[0596] 2. Receiving and analyzing the file

[0597] The server receives the uploaded digital file and determines the file format using an appropriate library (e.g., Apache PDFBox, Apache POI).

[0598] Depending on the file format, the content of each slide is converted to text data. For example, if it is a PDF file, Apache PDFBox is used, and if it is a PowerPoint file, Apache POI is used.

[0599] 3. Parsing the prompt

[0600] The server uses a natural language processing library (e.g., spaCy, NLTK) to analyze the input narration prompts and generates a talk script for each slide based on the analysis results.

[0601] Specifically, the system analyzes the grammar and key keywords of the narration prompts and uses a script generation algorithm to generate a semantically consistent talk script.

[0602] 4. Generating Synthetic Speech

[0603] The server uses a synthetic speech engine (e.g., Google Text-to-Speech API, Amazon Polly) to convert the generated talk script into synthetic speech. It adjusts the tone and intonation as necessary to ensure quality.

[0604] 5. Subtitle Generation

[0605] The server generates subtitles based on the synthesized speech, extracts timestamps using speech waveform analysis, and generates subtitle data (e.g., SRT format).

[0606] Specifically, timestamps are extracted from the audio data, and the display timing is adjusted by generating and setting the subtitle time code.

[0607] 6. Avatar Integration

[0608] If the user wishes to use an avatar, the server uses an avatar generation API (e.g., Adobe Character Animator, VYOND) to generate avatar movements in accordance with the synthesized voice.

[0609] The avatar's movements and facial expressions are synchronized with the narration to create a natural presentation.

[0610] 7. Creating and providing presentation videos

[0611] The server uses a video editing tool (e.g., FFmpeg) to generate a presentation video that integrates slides, synthesized speech, subtitles, and avatars.

[0612] The final video file is uploaded to cloud storage, allowing users to download it or watch it online via their devices.

[0613] Specific examples

[0614] For example, when converting materials for in-house training into a presentation video, the following process is carried out.

[0615] 1. User request input: The user uploads a PDF file of training materials through a web interface and inputs a narration prompt: "This slide should explain the features of the new product."

[0616] 2. Receiving and parsing the file: The server receives the PDF file, parses the content, and converts it into text data.

[0617] 3. Parsing prompts: The server parses the narration prompts and generates a talk script.

[0618] 4. Generate synthetic voice: The server generates synthetic voice based on the talk script.

[0619] 5. Subtitle generation: The server generates subtitles from the synthesized speech and adjusts the display timing.

[0620] 6. Avatar integration: If the user wishes, the server will generate an avatar and integrate it into the video.

[0621] 7. Generating and providing presentation video: The server generates a presentation video that integrates all elements and provides it to the user.

[0622] In this way, the present invention can significantly reduce the user's workload and improve time efficiency, and also makes it possible to easily generate high-quality presentation videos.

[0623] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0624] Step 1: User Request Input

[0625] Input: The user inputs the digital file for the presentation (e.g., PDF, PowerPoint) and the narration prompt.

[0626] What it does: A user opens a dedicated web interface or desktop application, clicks the file upload button to upload a digital file, and simultaneously enters a narration prompt (e.g., "This slide should describe the features of our new product") into a text field.

[0627] Output: The digital files and narration prompts are sent to the server.

[0628] Step 2: Receiving and analyzing the file

[0629] Input: The server receives the digital file sent by the user.

[0630] What happens: The server first uses the appropriate library to determine the file format of the received file (e.g., Apache PDFBox for PDF, Apache POI for PowerPoint).

[0631] Output: The content of each slide is extracted as text data.

[0632] Step 3: Parse the prompt

[0633] Input: The server parses the narration prompt received from the user.

[0634] Specific operation: The server uses a natural language processing library (e.g., spaCy, NLTK) to parse the prompt and extract grammar and important keywords.

[0635] Output: A talk script corresponding to each slide is generated.

[0636] Step 4: Generate synthetic speech

[0637] Input: The server receives the generated talk script.

[0638] How it works: The server calls a synthetic speech engine (e.g., Google Text-to-Speech API, Amazon Polly) to convert the talk script into voice data. At this time, the voice quality and intonation are automatically set.

[0639] Output: Synthetic speech data is generated.

[0640] Step 5: Generate subtitles

[0641] Input: The server receives the generated synthesized voice data.

[0642] Specific operation: The server uses an audio waveform analysis tool to extract the timestamp of the audio data and generate subtitle data (e.g., SRT format).

[0643] Output: Subtitle files are generated.

[0644] Step 6: Integrating the Avatar

[0645] Input: Receives the user-selected avatar, synthesized voice data, and subtitle data.

[0646] Specific operation: The server uses an avatar generation API (e.g., Adobe Character Animator, VYOND) to generate avatar movements in sync with the synthesized voice, adjusting the timing and movement, and integrating the avatar data with slides, audio, and subtitles.

[0647] Output: Presentation data with avatars integrated is generated.

[0648] Step 7: Generate and serve presentation videos

[0649] Input: The server receives the consolidated presentation data.

[0650] What happens: The server uses a video editing tool (e.g. FFmpeg) to combine all elements (slides, synthesized speech, subtitles, avatars) and export them as a presentation video.

[0651] Output: The generated presentation video will be uploaded to cloud storage and a download link or streaming URL will be provided to users.

[0652] (Application example 1)

[0653] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0654] Conventional presentation video generation systems impose a heavy workload on educators when creating educational content, and have issues with the quality and adaptability of the videos created. In particular, for educational presentations, the coordination of slide content with narration and subtitles is crucial, and adjusting these elements satisfactorily can be time-consuming. Furthermore, systems lack the flexibility to adapt to different educational environments and learner needs. The present invention aims to solve these issues by providing a system that enables educators to easily and automatically generate high-quality educational presentation videos.

[0655] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0656] In this invention, the server includes means for receiving a digital presentation file input by a user, means for analyzing the received digital file and converting the content of each slide into text data, means for analyzing narration prompts input by the user and generating a talk script for each slide, means for generating synthetic speech based on the generated talk script, means for generating subtitles based on the generated synthetic speech, means for integrating the slides, the generated synthetic speech, and the generated subtitles to generate a presentation video, means for providing the generated presentation video to the user, means for generating a presentation video optimized as educational content based on content selected by the user, and means for providing an interface that is easy for educators to use and for automatically generating and adjusting content based on educator input. This enables educators to easily create high-quality educational presentation videos.

[0657] "User" refers to an individual or organization that uses this system to generate a presentation video.

[0658] A "digital presentation file" is an electronic file, such as a PDF or PowerPoint file, that contains the slides of a presentation.

[0659] "Text data" refers to the written information extracted from the slides in the digital file.

[0660] "Narration prompt" refers to text that the user enters for each slide and that serves as instructions for generating a talk script.

[0661] A "talk script" is a presentation text that is automatically generated based on narration prompts.

[0662] "Synthetic voice" is artificial voice data that is automatically generated based on a talk script.

[0663] "Subtitles" refers to text information that is displayed in response to synthesized speech.

[0664] A "presentation video" is a dynamic video file that combines slides, synthesized audio, and subtitles.

[0665] "Educational Content" refers to educational materials and videos created by educators for students and participants.

[0666] "Interface" refers to the user interface through which a user accesses the system and uploads files or enters prompts.

[0667] This invention provides a system that allows educators to easily create high-quality educational presentation videos. The system provides a series of processes that automatically generate educational content, including synthesized speech and subtitles, by simply uploading a digital file for the presentation and entering narration prompts.

[0668] Basic system configuration

[0669] The system mainly consists of the following components:

[0670] 1. User Interface

[0671] An interface through which users upload digital files for a presentation and enter narration prompts. This interface is typically implemented as a web-based application or a dedicated desktop application.

[0672] 2. Server Module

[0673] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[0674] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[0675] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[0676] Synthetic speech generation module: Generates synthetic speech based on the talk script. Software used includes Google Text-to-Speech (gTTS).

[0677] Subtitle generation module: Generates subtitles based on synthesized speech.

[0678] Video Integration Module: Generates a video presentation by integrating slides, synthesized speech, and subtitles, and optionally integrating avatars.

[0679] 3. Educational Module

[0680] Based on user input, it generates presentation videos that are optimized for educational content and automatically adjusts to meet the educator's requirements.

[0681] Program processing

[0682] The processing of the program will be explained in natural language below.

[0683] 1. User request input

[0684] Users upload digital files for presentations (e.g., PDF, PowerPoint) to the system via a web or desktop application, and also enter narration prompts through the user interface.

[0685] 2. Analyzing the file

[0686] The server receives the uploaded digital files, and a file analysis module converts the content of each slide into text data, making the presentation organized and easier to handle within the system.

[0687] 3. Parsing the prompt

[0688] The server uses a prompt analysis module to analyze the narration prompts input by the user and generate a talk script corresponding to each slide.

[0689] 4. Generating Synthetic Speech and Subtitles

[0690] The server generates speech using a synthetic speech generation module based on the generated talk script, and creates corresponding subtitles using a subtitle generation module.

[0691] 5. Video Integration

[0692] The server uses a video integration module to integrate the slides, synthesized speech, and subtitles to generate a video presentation. If the user wishes, an avatar can also be used, which is also integrated into the video.

[0693] Specific examples

[0694] For example, if an educator is creating an educational presentation video on "Fundamental Concepts of Mathematics," they might do the following:

[0695] 1. Educators upload PowerPoint files of math teaching materials to the app.

[0696] 2. For each slide, include a prompt such as "Explain the difference between integers and natural numbers" or "On the next slide, provide an example of an integer."

[0697] 3. The server automatically generates narration and creates a presentation video.

[0698] Example prompt sentence:

[0699] "On this slide, explain the difference between integers and natural numbers."

[0700] "On the next slide, give an example of an integer."

[0701] "On the last slide, please summarize what you learned today."

[0702] The present invention allows educators to easily create high-quality educational presentation videos, thereby improving the quality of educational content.

[0703] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0704] Step 1:

[0705] A user uploads a digital file for a presentation to the system using a terminal, and simultaneously inputs a narration prompt corresponding to each slide through the interface, resulting in the digital file and the narration prompt as input data.

[0706] Input: Digital files for presentation, narration prompts

[0707] Output: Received digital files, narration prompts

[0708] Step 2:

[0709] The server analyzes the received digital files using a file analysis module, converting the content of each slide into text data and organizing it, thereby obtaining the text data for each slide.

[0710] Input: Received digital files

[0711] Output: Text data for each slide

[0712] Step 3:

[0713] The server uses a prompt analysis module to analyze the narration prompts entered by the user and generate a talk script corresponding to each slide. This analysis generates a talk script suitable for narration from the prompt text.

[0714] Input: Narration prompts, text data for each slide

[0715] Output: Talk script for each slide

[0716] Step 4:

[0717] The server uses a synthetic speech generation module to convert the generated talk script into synthetic speech, during which the quality of the synthetic speech is checked and adjusted if necessary.

[0718] Input: Talk script for each slide

[0719] Output: Synthesized voice data

[0720] Step 5:

[0721] The server generates subtitles using a subtitle generation module based on the synthesized speech, and the timing of the subtitles is set based on the time code of the synthesized speech.

[0722] Input: Synthetic voice data

[0723] Output: Subtitle data with timecode

[0724] Step 6:

[0725] The server uses a video integration module to integrate each slide, the synthesized speech, and the generated subtitles to generate a presentation video. If a user selects an avatar, the avatar is also integrated into the video in synchronization with the synthesized speech.

[0726] Input: Slide image data, synthesized voice data, subtitle data

[0727] Output: Presentation video data

[0728] Step 7:

[0729] The server provides the generated presentation video to the user, who can then download the video or watch it online using their device.

[0730] Input: Presentation video data

[0731] Output: Presentation video provided to the user

[0732] Step 8:

[0733] The server optimizes the generated presentation video as educational content, creating a video in a format that is easy for educators to use.

[0734] Input: Presentation video data

[0735] Output: Presentation video optimized for education

[0736] Step 9:

[0737] Users can use the video presentations optimized for education to achieve their educational goals by providing them to learners.

[0738] Input: Presentation video optimized for education

[0739] Output: Educational presentation use

[0740] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0741] MODE FOR CARRYING OUT THE INVENTION

[0742] This invention provides a system for efficiently generating presentation videos and providing high-quality videos that take user emotions into consideration. This system automates the entire process of analyzing digital files for presentations, generating synthesized speech and subtitles, and finally providing the presentation video, and further combines it with an emotion engine that recognizes user emotions.

[0743] Basic system configuration

[0744] The system mainly consists of the following components:

[0745] 1. User Interface

[0746] An interface through which users upload digital files for a presentation and enter narration prompts, typically implemented as a web-based application or a dedicated desktop application.

[0747] 2. Server Module

[0748] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[0749] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[0750] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[0751] Emotion Engine: Recognizes the user's emotions when entering narration prompts and takes them into account during analysis.

[0752] Synthetic speech generation module: Generates synthetic speech based on the talk script. Adjusts the tone and pace of the synthetic speech based on the user's emotions.

[0753] Subtitle generation module: Generates subtitles based on synthesized speech. Adjusts the expression of the subtitles based on the user's emotions.

[0754] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[0755] Program processing

[0756] The processing of the program will be explained in natural language below.

[0757] 1. User request input

[0758] Users upload digital presentation files (e.g., PDF, PowerPoint) to the system using their terminals, and also enter narration prompts through the user interface.

[0759] 2. Analyzing the file

[0760] The server receives the uploaded digital files and identifies the file format. A file analysis module then converts the content of each slide into text data, sorting out page numbers and header information. The extracted text data is then semantically analyzed using natural language processing (NLP) techniques to identify important elements.

[0761] 3. Prompt analysis and emotion recognition

[0762] The server's prompt analysis module analyzes the narration prompts entered by the user. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice and influences the content of the prompt. For example, if the user expresses "joy" as their emotion, a talk script with positive expressions will be generated.

[0763] 4. Generating Synthetic Speech and Subtitles

[0764] The server's synthetic speech generation module converts the talk script into synthetic speech based on the results of the emotion engine. The tone and pace of the synthetic speech are adjusted based on the user's emotions. The subtitle generation module then generates subtitles based on the narration content, reflecting the user's emotions.

[0765] 5. Avatar Integration

[0766] If a user wishes to use an avatar, the server adds the avatar to the system and sets the avatar's lip sync and facial animation to match the synthesized voice.

[0767] 6. Creating and providing presentation videos

[0768] The server uses a video integration module to integrate slides, synthesized speech, subtitles, and avatars to generate a presentation video, which is then provided to users for download or online viewing on their devices.

[0769] Specific examples

[0770] For example, consider the creation of a marketing presentation. A user uploads marketing materials in PDF format and enters a narration prompt such as, "This slide should highlight the benefits of the product." As the user enters the narration prompt, an emotion engine recognizes the user's emotions, such as "excitement" or "anticipation." The server analyzes the file and generates a positive and emphatic talk script based on the emotion engine's results. A synthetic voice is then generated from the talk script, resulting in an expressive narration. Subtitles also reflect the user's emotions and use impactful expressions. If an avatar is added, the avatar will similarly behave to express excitement or anticipation. Finally, a video presentation integrating all elements is generated and served to the user.

[0771] In this way, the present invention can provide more effective and empathetic presentation videos that take into account the user's emotions, improving the quality and emotional appeal of presentations while also improving the efficiency of presentations.

[0772] The processing flow will be explained below.

[0773] Step 1:

[0774] The user uploads digital files for presentations to the system through a terminal, and inputs narration prompts through the user interface.

[0775] Step 2:

[0776] The server receives the uploaded digital file, determines the file format, and, in the case of PDF files, extracts the text and images page by page using a library such as Apache PDFBox.

[0777] Step 3:

[0778] The server's file analysis module organizes the text data by slide, extracting page numbers and header information, then uses natural language processing (NLP) techniques to semantically analyze the extracted text and identify key elements.

[0779] Step 4:

[0780] The server's prompt analysis module analyzes the narration prompts entered by the user. The emotion engine recognizes the user's emotions and reflects them in the prompt content. For example, if the user expresses "joy," a positive talk script is generated.

[0781] Step 5:

[0782] The server's synthetic speech generation module passes the talk script to a synthetic speech engine to generate the narration. The tone and pace of the speech are adjusted based on the user's emotions. Services such as Google Cloud Text-to-Speech may be used to generate the synthetic speech.

[0783] Step 6:

[0784] The server's subtitle generation module generates subtitles based on the content of the narration. It uses speech recognition technology to analyze the timing of the narration and produces text that reflects the user's emotions. For example, a cheerful expression that reflects the emotion of "joy" is used.

[0785] Step 7:

[0786] When a user requests to use an avatar, the server adds the avatar to the system. At this time, the server synchronizes the avatar's lip sync (mouth movements) and facial animation with the synthesized voice. The avatar also behaves in a way that expresses the user's emotions.

[0787] Step 8:

[0788] The server's video integration module integrates each slide, synthesized speech, subtitles, and avatars to generate a presentation video, which is then output with optimal image and sound quality using video editing software such as FFmpeg.

[0789] Step 9:

[0790] The server saves the generated presentation video and makes it accessible to users by uploading the video to cloud storage and providing users with a download link.

[0791] Step 10:

[0792] Users can download the presentation video or watch it online using the provided link on their devices. Users can check the video quality and make adjustments if necessary.

[0793] This is the specific processing flow of the program.

[0794] Example 2

[0795] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0796] The process of creating presentation videos manually is time-consuming and labor-intensive, making it inefficient. Furthermore, conventional systems struggle to generate videos that take user emotions into account, making it difficult to create high-quality presentation videos with emotional appeal. Furthermore, manually adjusting the intonation and timing of narration is tedious, making the creation of an integrated presentation video complex.

[0797] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a digital file for presentation input by a user, means for analyzing the received digital file and converting the content of each slide into text data, means for analyzing narration prompts input by the user and generating a talk script for each slide, means for recognizing the user's emotions and analyzing and generating narration prompts based on the emotions, means for generating synthetic speech based on the generated talk script, means for generating subtitles based on the generated synthetic speech, means for integrating the slides, the generated synthetic speech, and the generated subtitles to generate a presentation video, and means for providing the generated presentation video to the user. This enables automatic generation of high-quality presentation videos that take user emotions into consideration.

[0798] A "digital file for presentation" is a document for presentation that is saved in a format such as PDF or PowerPoint.

[0799] A "Presentation Video" is a video presentation generated by integrating digital file slides, narration audio, subtitles, and avatars.

[0800] A "narration prompt" is text that a user enters for each slide in a presentation to indicate what should be explained or what points should be discussed.

[0801] A "talk script" is a text of speech content corresponding to each slide of a presentation, which is generated based on file analysis and prompt analysis.

[0802] "Synthetic voice" is a mechanically generated voiced narration based on a talk script.

[0803] "Subtitles" are texts that are visually displayed within a presentation video and are used to display the contents of synthesized speech in text format.

[0804] An "avatar" is an image or animation of a character used in a presentation video that moves in sync with the synthesized voice.

[0805] An "emotion engine" is software that analyzes a user's facial expressions and voice to recognize their emotions and reflects the results in the presentation generation process.

[0806] MODE FOR CARRYING OUT THE INVENTION

[0807] System Overview

[0808] This invention provides a system for efficiently generating presentation videos and providing high-quality videos that take user emotions into consideration. This system automates the entire process of analyzing digital files for presentations, generating synthesized speech and subtitles, and finally providing the presentation video, and further combines it with an emotion engine that recognizes user emotions.

[0809] System Configuration

[0810] The system consists of the following main components:

[0811] 1. User Interface

[0812] An interface for users to upload digital files for presentations and enter narration prompts, implemented as a web-based application or a dedicated desktop application.

[0813] 2. Server Module

[0814] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[0815] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[0816] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[0817] Emotion Engine: Recognizes the user's emotions when entering narration prompts and takes them into account during analysis. For example, if the user indicates "joy" as their emotion, a talk script with positive expressions will be generated.

[0818] Synthetic speech generation module: Generates synthetic speech based on the talk script. Adjusts the tone and pace of the synthetic speech based on the user's emotions.

[0819] Subtitle generation module: Generates subtitles based on synthesized speech. Adjusts the expression of the subtitles based on the user's emotions.

[0820] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[0821] Operation explanation

[0822] 1. User request input

[0823] The user uploads digital files (e.g., PDF, PowerPoint) for the presentation through a user interface on the device, and also enters narration prompts through the user interface.

[0824] 2. Analyzing the file

[0825] The server receives the uploaded digital files and uses a file analysis module to determine the file format, which then converts the content of each slide into text data, which is then analyzed using natural language processing (NLP) techniques.

[0826] 3. Prompt analysis and emotion recognition

[0827] The server's prompt analysis module analyzes the narration prompts entered by the user. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice and reflects them in the prompt analysis. For example, if the user expresses "joy," a talk script with positive expressions is generated.

[0828] 4. Generating Synthetic Speech and Subtitles

[0829] The server's synthetic speech generation module converts the talk script into synthetic speech based on the results of the emotion engine. The tone and pace are adjusted based on the user's emotions. The subtitle generation module then generates subtitles based on the narration content, reflecting the user's emotions.

[0830] 5. Avatar Integration

[0831] If a user wishes to use an avatar, the server adds the avatar to the system and sets the avatar's lip sync and facial animation to match the synthesized voice.

[0832] 6. Creating and providing presentation videos

[0833] The server uses a video integration module to integrate slides, synthesized speech, subtitles, and avatars to generate a presentation video, which is then provided to users for download or online viewing on their devices.

[0834] Specific examples

[0835] For example, consider a user creating a market analysis presentation. The user uploads the market analysis document in PDF format and enters a narration prompt such as, "In this slide, please emphasize the advantages of the new product." When the user enters the narration prompt, the emotion engine recognizes the user's "expectations" and generates a talk script based on them. The server analyzes the file and generates content that emphasizes important points based on the results of the emotion engine. Synthetic speech is then generated from the talk script, and subtitles also reflect the user's emotions. If an avatar is added, the avatar will also behave in a similar way, reflecting the user's emotions. Finally, a video presentation integrating all elements is generated and provided to the user.

[0836] Prompt Sentence Examples

[0837] 1. "Highlight the key points of your marketing strategy."

[0838] 2. "This slide should explain your efforts to improve customer satisfaction."

[0839] 3. "Please briefly summarize the technical features of your product."

[0840] In this way, the system of the present invention can take into account the user's emotions and provide more effective and empathetic presentation videos, improving the quality and emotional appeal of presentations while also improving the efficiency of presentations.

[0841] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0842] Step 1: User Request Input

[0843] ---

[0844] Users use terminals to access the system's user interface, upload digital files (e.g., PDF, PowerPoint) for presentations, and input narration prompts for the presentation content.

[0845] Input: Digital files for presentation, narration prompts

[0846] Output: Uploaded digital files, typed narration prompts

[0847] Specific behavior: The user interface provides buttons for uploading files and text boxes for entering narration prompts. Once the user enters the file and prompt, they are sent to the server.

[0848] Step 2: Receiving and analyzing the file

[0849] ---

[0850] The server receives the uploaded digital files and uses a file analysis module to convert the content of each slide into text data, identifying the file format and organizing page numbers and header information.

[0851] Input: Uploaded digital files

[0852] Output: Parsed text data

[0853] What it does: The server's file analysis module reads PDF or PowerPoint files, extracts the text within each slide, and uses natural language processing (NLP) techniques to identify important elements.

[0854] Step 3: Parsing the prompt and recognizing emotions

[0855] ---

[0856] The server's prompt analysis module analyzes the narration prompts entered by the user and generates a talk script. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice, and reflects those emotions in the prompt analysis.

[0857] Input: Narration prompts, user emotion information

[0858] Output: Talk script reflecting emotions

[0859] Specific operation: The emotion engine analyzes the user's facial expression and voice data acquired through the user interface, identifies the user's emotional state based on this data, and uses this information to generate a talk script appropriate for the narration prompt.

[0860] Step 4: Generate synthetic speech and subtitles

[0861] ---

[0862] The server uses a synthetic speech generation module to generate synthetic speech based on the talk script, adjusting the tone and pace of the speech based on the user's emotions. The subtitle generation module converts the content of the synthetic speech into text data and generates subtitles.

[0863] Input: Talk script, emotion information

[0864] Output: Synthesized speech, generated subtitles

[0865] Specific operation: The synthetic speech generation module receives the talk script as input data and dynamically adjusts the tone and pace of the speech based on the user's emotional information. The generated speech data is then converted into text and passed to the subtitle generation module, which finally creates subtitles for the presentation.

[0866] Step 5: Integrating the Avatar

[0867] ---

[0868] If a user wishes to use an avatar, the server adds the avatar to the system and sets the avatar's lip sync and facial animation to match the synthesized voice.

[0869] Input: Synthetic voice, user avatar preference

[0870] Output: Avatar with lip sync and animation added

[0871] Specific behavior: The server loads the avatar selected by the user from available avatar resources and sets the behavior based on the synthesized voice. Lip sync and facial animation are dynamically reflected based on emotional information, resulting in more natural behavior.

[0872] Step 6: Generate and serve presentation videos

[0873] ---

[0874] The server uses a video synthesis module to synthesize each slide, the generated synthetic speech, subtitles, and avatars into a single presentation video, which is then provided to users for download or online viewing.

[0875] Input: Slide data, synthesized speech, generated subtitles, avatar data

[0876] Output: Integrated presentation video

[0877] How it works: The video integration module combines the content of each slide, the generated audio, subtitles, and avatars, arranges them on a timeline, and generates a video file. The final presentation video is saved in a specified location for user access. For online viewing, a URL is provided to the user.

[0878] (Application example 2)

[0879] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0880] Conventional presentation video generation systems require users to input slide content individually, create narration, and generate synthesized speech, which is a complex process. Furthermore, because the system automatically generates the presentation without considering the user's emotions, it lacks appeal to viewers. Therefore, there is a need for a system that can efficiently generate presentation videos that respond to the user's emotions.

[0881] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving a digital file for presentation input by a user; means for analyzing the received digital file and converting the content of each slide into text data; means for analyzing narration prompts input by the user and generating a talk script for each slide; means for generating synthetic speech based on the generated talk script; means for generating subtitles based on the generated synthetic speech; means for recognizing a user's emotion and adjusting the tone and pace of the narration based on the recognized emotion; means for integrating each slide, the generated synthetic speech, the generated subtitles, and the recognized emotion information to generate a presentation video; and means for providing the generated presentation video to the user. This makes it possible to efficiently generate a highly appealing presentation video that is tailored to the user's emotion.

[0882] "Digital files for presentations" refer to slides and materials that users use for presentation purposes and are provided in digital formats such as PDF and PowerPoint.

[0883] "Narration prompt" refers to the content or wording of the narration entered by the user, including explanations and commentary for each slide.

[0884] A "talk script" is a spoken script generated by analyzing narration prompts, and is a narration instruction manual provided for each slide.

[0885] "Synthetic voice" is an artificial voice generated based on a talk script, and is reproduced to imitate human speech.

[0886] "Subtitles" are textual information that displays the contents of the synthesized speech, and serve to visually supplement the contents of the narration.

[0887] "Emotion information" is data of emotions recognized by analyzing the user's facial expressions, voice, etc., and indicates an emotional state such as joy, sadness, excitement, etc.

[0888] "Tone and pace adjustment" refers to changing the speaking style and speed of the synthesized voice based on emotional information, as a means of providing a more emotionally rich presentation.

[0889] "Video synthesis" is the process of combining slides, synthesized audio, and subtitles into a single video presentation that viewers can watch as a continuous story.

[0890] An "avatar" is a digital character that can be selected by the user to appear in the presentation video, and has lip-sync and facial animation.

[0891] This system analyzes a digital presentation file input by a user, generates a talk script based on narration prompts entered by the user, creates synthesized speech and subtitles, and provides a presentation video that integrates them. It is particularly notable for recognizing the user's emotions and adjusting the tone and pace of the narration based on those emotions.

[0892] Basic system configuration

[0893] 1. User Interface

[0894] Users use a device to upload digital files for presentations and enter narration prompts. The user interface may be implemented as a web-based application or a dedicated desktop application, typically a web application running in a browser.

[0895] 2. Server Module

[0896] The server includes the following sub-modules to analyze the received digital files and extract the content of each slide as text data:

[0897] File Analysis Module: Identifies the digital file format (PDF, PowerPoint, etc.) of the uploaded presentation and extracts the text data for each slide.

[0898] Prompt Analysis Module: Analyzes the narration prompts entered by the user and generates a talk script for each slide.

[0899] Emotion Recognition Engine: Recognizes emotions from the user's facial expressions and voice data. This is achieved by the user using the device's camera and microphone.

[0900] Synthetic voice generation module: Creates a synthetic voice based on the generated talk script, and adjusts the tone and pace according to the user's emotions.

[0901] Subtitle generation module: Generates subtitles in synchronization with the synthesized speech. The subtitles also include expressions based on the user's emotions.

[0902] Video Integration Module: Generates a presentation video by integrating each slide, synthesized speech, subtitles, and emotional information.

[0903] 3. Use of Avatars

[0904] A user-selected avatar can be integrated into the presentation video, lip-syncing to the generated synthetic speech and changing facial expressions based on emotions, enhancing the visual and emotional appeal.

[0905] Program processing explanation

[0906] The server uses the following hardware and software:

[0907] Hardware: Camera, microphone, user device (PC, smartphone, etc.)

[0908] Software: OpenCV, Natural Language Processing Library (NLP), Emotion Recognition Engine, Synthetic Speech Generation Library

[0909] Specific examples

[0910] For example, when creating a marketing presentation, a user uploads marketing materials in PDF format and enters a narration prompt such as, "This slide should highlight the benefits of the product." As the user enters the narration prompt, an emotion recognition engine recognizes the user's "joy." The server analyzes the file and generates a talk script that emphasizes a positive tone based on the emotion recognition results. Synthetic speech is then generated from this talk script, creating an expressive narration. Subtitles also reflect the user's emotions and use impactful expressions. If an avatar is added, the avatar will also behave in a similar way, expressing joy. Finally, a video presentation integrating all elements is generated and provided to the user.

[0911] Prompt Sentence Examples

[0912] "Analyze the text of marketing presentation slides, generate a narration that adjusts the tone and pace based on user emotion, add subtitles, and create a video of the presentation. The narration prompt is "Highlight the product benefits on this slide," and the user emotion is "Delight."

[0913] In this way, the present invention takes into account the user's emotions and can provide more effective and appealing presentation videos, improving the efficiency and quality of presentation creation and eliciting empathy from viewers.

[0914] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0915] Step 1:

[0916] User request input

[0917] Users use their terminals to upload digital files (e.g., PDF, PowerPoint) for presentations to the server and input narration prompts through a user interface.

[0918] Input: Digital files for presentation and narration prompts

[0919] Output: Received digital files and prompt data

[0920] Specific operation: The terminal displays a web form for uploading a file, from which the user selects a file and enters text for the narration prompt.

[0921] Step 2:

[0922] Analyzing files

[0923] The server's file analysis module analyzes the received digital files, extracting the content of each slide as text data and organizing page numbers and header information.

[0924] Input: Received digital presentation file

[0925] Output: Text data for each slide

[0926] Specific operation: The server determines the file format and extracts the text data using an appropriate library (e.g., PyMuPDF).

[0927] Step 3:

[0928] Parsing prompts

[0929] The server's prompt analysis module analyzes the narration prompts entered by the user and generates a talk script, while the emotion recognition engine recognizes the user's emotions and influences the content of the prompts.

[0930] Input: User-supplied narration prompts and user emotion data

[0931] Output: A talk script that reflects the user's emotions

[0932] Specific operation: The server uses an emotion recognition engine (e.g., OpenCV, dlib) to analyze the user's emotions and a natural language processing library (e.g., spaCy) to analyze the narration prompts.

[0933] Step 4:

[0934] Generate synthetic speech

[0935] The server's synthetic voice generation module generates synthetic voice based on the talk script and adjusts the tone and pace according to the user's emotions.

[0936] Input: A talk script that reflects the user's emotions

[0937] Output: Emotionally adjusted synthetic speech

[0938] Specific operation: The server generates an audio file using a synthetic speech generation library (e.g., Google Text-to-Speech API) and processes the audio (adjusts tone and pace) based on emotions.

[0939] Step 5:

[0940] Subtitle generation

[0941] A subtitle generation module of the server generates subtitles based on the synthesized speech, and includes expressions according to the user's emotions.

[0942] Input: Synthesized voice and talk script

[0943] Output: Emotionally sensitive subtitle data

[0944] Specific operation: The server uses a speech recognition library (e.g., Google Speech-to-Text API) to generate text from the speech and creates subtitle data in a style that corresponds to the user's emotions.

[0945] Step 6:

[0946] Presentation video generation

[0947] The video integration module of the server integrates each slide, synthesized speech, subtitles, and emotional information to generate a presentation video.

[0948] Input: Each slide, synthesized speech, subtitle data, emotional information

[0949] Output: Finished presentation video

[0950] Specific operation: The server uses a video editing library (e.g., FFmpeg) to integrate each element and generate a video file.

[0951] Step 7:

[0952] Providing generated presentation videos

[0953] The generated presentation video is provided to the user, who can download the video or watch it online using their device.

[0954] Input: Finished presentation video

[0955] Output: User-accessible video link of the presentation

[0956] Specific operation: The server stores the video file in storage and provides a download link or streaming URL to the user's device.

[0957] This processing flow enables users to efficiently generate emotionally-reflective presentation videos and deliver visually and emotionally appealing presentations.

[0958] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0959] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0960] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0961] [Third embodiment]

[0962] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0963] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0964] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0965] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0966] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0967] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0968] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0969] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0970] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0971] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0972] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0973] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0974] MODE FOR CARRYING OUT THE INVENTION

[0975] The present invention provides a system for efficiently generating presentation videos and reducing the user's workload by automating a series of processes: analyzing digital files for presentations, generating synthesized speech and subtitles, and finally providing the presentation videos.

[0976] Basic system configuration

[0977] The system mainly consists of the following components:

[0978] 1. User Interface

[0979] An interface through which users upload digital files for a presentation and enter narration prompts, typically implemented as a web-based application or a dedicated desktop application.

[0980] 2. Server Module

[0981] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[0982] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[0983] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[0984] Synthetic speech generation module: Generates synthetic speech based on the talk script.

[0985] Subtitle generation module: Generates subtitles based on synthesized speech.

[0986] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[0987] Program processing

[0988] The processing of the program will be explained in natural language below.

[0989] 1. User request input

[0990] Users upload digital presentation files (e.g., PDF, PowerPoint) to the system using their terminals, and also enter narration prompts through the user interface.

[0991] 2. Analyzing the file

[0992] The server receives the uploaded digital files, identifies the file format, and then a file analysis module converts the content of each slide into text data, sorting out page numbers and header information.

[0993] 3. Parsing the prompt

[0994] The server uses a prompt parsing module to parse the input narration prompts, which then generates a talking script that corresponds to the content of each slide.

[0995] 4. Generating Synthetic Speech

[0996] The server utilizes a synthetic speech generation module to convert the talk script into synthetic speech, and in the process evaluates the quality of the generated speech and adjusts the tone and intonation if necessary.

[0997] 5. Subtitle Generation

[0998] The server uses a subtitle generation module to generate subtitles based on the generated synthesized speech, setting the time code of the subtitles to match the timing of the narration and adjusting the display timing.

[0999] 6. Avatar Integration

[1000] If a user chooses to use an avatar, the server integrates the avatar into the video, synchronizing the avatar's movements and facial expressions with the synthesized voice to create a natural presentation.

[1001] 7. Creating and providing presentation videos

[1002] The server uses the video integration module to generate a presentation video that integrates slides, synthesized speech, subtitles, and avatars, and the generated video is provided to the user, who can download it or watch it online using their device.

[1003] Specific examples

[1004] For example, when converting in-house training materials into a video presentation, the following steps are performed. First, the user uploads the PDF file of the training materials and enters a narration prompt, such as "Please explain the features of the new product on this slide." The server analyzes the file and generates a talk script based on the prompt. Next, the server converts the talk script into synthesized speech and generates subtitles based on that speech. If the user wishes, an avatar can be integrated and made to move in sync with the synthesized speech. Finally, a video presentation incorporating all elements is generated and provided to the user.

[1005] In this way, the present invention can significantly reduce the user's efforts and improve time efficiency, and also makes it possible to easily generate high-quality presentation videos.

[1006] The processing flow will be explained below.

[1007] Step 1:

[1008] The user uploads digital files for presentations to the system through a terminal, and inputs narration prompts through the user interface.

[1009] Step 2:

[1010] The server receives the uploaded digital file, determines the file format, and, in the case of PDF files, extracts the text and images page by page using a library such as Apache PDFBox.

[1011] Step 3:

[1012] The server's file analysis module organizes the text data by slide, extracting page numbers and header information, then uses natural language processing (NLP) techniques to semantically analyze the extracted text and identify key elements.

[1013] Step 4:

[1014] The server's prompt analysis module analyzes the narration prompts from the user and generates a talk script, which includes technology to automatically generate appropriate explanatory text for each slide based on the prompts.

[1015] Step 5:

[1016] The server's synthetic speech generation module passes the talk script to a synthetic speech engine to generate the narration, using services such as Google Cloud Text-to-Speech, followed by post-processing to adjust the tone and intonation.

[1017] Step 6:

[1018] The subtitle generation module of the server generates subtitles based on the content of the narration, specifically, by analyzing the timing of the narration using speech recognition technology and converting the corresponding text into a subtitle format.

[1019] Step 7:

[1020] When a user requests the use of an avatar, the server adds the avatar to the system, setting the avatar's lip sync (mouth movements) and facial animation based on the timing of the synthesized voice.

[1021] Step 8:

[1022] The server's video synthesis module synthesizes each slide, narration, subtitles, and avatar to generate a presentation video, using video editing software (e.g., FFmpeg) to output the video with optimal image and sound quality.

[1023] Step 9:

[1024] The server saves the generated presentation video and makes it accessible to users by uploading the video to cloud storage and providing users with a download link.

[1025] Step 10:

[1026] Users can download the presentation video or watch it online using the provided link on their devices. Users can check the video quality and make adjustments if necessary.

[1027] This is the specific processing flow of the program.

[1028] Example 1

[1029] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1030] Creating traditional presentation videos is time-consuming and laborious, and requires significant effort to adjust the quality of the narration and synchronize subtitles. It is also difficult to adjust the intonation and timing of the narration, and using avatars for natural presentations is also time-consuming. There is a need to solve these issues, reduce the user's workload, and efficiently generate high-quality presentation videos.

[1031] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1032] In this invention, the server includes means for receiving a digital presentation file input by a user, means for analyzing the received digital file and converting the content of each slide into text data, means for analyzing narration prompts input by the user and generating a talk script for each slide, means for generating synthetic speech based on the generated talk script, means for generating subtitles based on the generated synthetic speech, means for evaluating the quality of the generated synthetic speech and adjusting the tone and intonation as necessary, means for setting a time code for the generated subtitles and adjusting their display timing, means for integrating an avatar selected by the user into a presentation video and synchronizing it with the synthetic speech, means for integrating each slide, the generated synthetic speech, the generated subtitles, and the avatar to generate a presentation video, and means for providing the generated presentation video to the user. This enables automatic generation of high-quality presentation videos while reducing the user's workload and improving time efficiency.

[1033] A "digital file for presentation" is a file in an electronic format, such as a PDF or PowerPoint, that a user uses to give a presentation.

[1034] "Analysis" is the process of breaking down and analyzing the content of received digital files and narration prompts to extract the necessary information.

[1035] "Text data" refers to data that expresses the content of each slide and narration prompts as text information.

[1036] A "narration prompt" is an instruction entered by the user, and is the original text that is generated as a talk script for each slide.

[1037] A "talk script" is a sentence of speech content corresponding to each slide, generated based on a narration prompt.

[1038] "Synthetic voice" refers to the generation of a talk script as voice data, and is a voice that imitates a human voice using synthetic voice technology.

[1039] "Subtitles" are texts that are generated based on synthesized speech and displayed on the screen, providing a visual complement to the narration content.

[1040] "Time code" is time information that indicates the timing of displaying and playing subtitles and narration.

[1041] An "avatar" is a computer-generated character used to give a presentation in place of a human.

[1042] "Video synthesis" is the process of combining individual slides, synthesized speech, subtitles, and avatars into a single video presentation.

[1043] "Providing" means distributing the generated presentation video to users in a form that allows them to view it online or download it.

[1044] MODE FOR CARRYING OUT THE INVENTION

[1045] This invention provides a system for efficiently generating presentation videos and reducing the user's workload. This system automates a series of processes: analyzing digital files for presentations, generating and integrating synthesized speech, subtitles, and avatars, and finally providing the presentation video.

[1046] Basic system configuration

[1047] The system mainly consists of the following components:

[1048] 1. User Interface

[1049] An interface for users to upload digital files for presentations and enter narration prompts.

[1050] It is typically implemented as a web-based application or a dedicated desktop application.

[1051] 2. Server Module

[1052] It is the main module that receives, analyzes and processes digital files.

[1053] Contains the following submodules:

[1054] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[1055] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[1056] Synthetic speech generation module: Generates synthetic speech based on the talk script.

[1057] Subtitle generation module: Generates subtitles based on synthesized speech.

[1058] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[1059] Program processing and data processing

[1060] Below, we will explain the details of this system's processing and the hardware and software used at each step.

[1061] 1. User request input

[1062] Users upload digital files (e.g., PDF, PowerPoint) for presentations to their devices through a dedicated web interface or desktop application, and simultaneously input prompts for narration.

[1063] Specifically, users upload files by dragging and dropping them into a form in their browser or using a file selection dialog, and enter narration prompts into text fields.

[1064] 2. Receiving and analyzing the file

[1065] The server receives the uploaded digital file and determines the file format using an appropriate library (e.g., Apache PDFBox, Apache POI).

[1066] Depending on the file format, the content of each slide is converted to text data. For example, if it is a PDF file, Apache PDFBox is used, and if it is a PowerPoint file, Apache POI is used.

[1067] 3. Parsing the prompt

[1068] The server uses a natural language processing library (e.g., spaCy, NLTK) to analyze the input narration prompts and generates a talk script for each slide based on the analysis results.

[1069] Specifically, the system analyzes the grammar and key keywords of the narration prompts and uses a script generation algorithm to generate a semantically consistent talk script.

[1070] 4. Generating Synthetic Speech

[1071] The server uses a synthetic speech engine (e.g., Google Text-to-Speech API, Amazon Polly) to convert the generated talk script into synthetic speech. It adjusts the tone and intonation as necessary to ensure quality.

[1072] 5. Subtitle Generation

[1073] The server generates subtitles based on the synthesized speech, extracts timestamps using speech waveform analysis, and generates subtitle data (e.g., SRT format).

[1074] Specifically, timestamps are extracted from the audio data, and the display timing is adjusted by generating and setting the subtitle time code.

[1075] 6. Avatar Integration

[1076] If the user wishes to use an avatar, the server uses an avatar generation API (e.g., Adobe Character Animator, VYOND) to generate avatar movements in accordance with the synthesized voice.

[1077] The avatar's movements and facial expressions are synchronized with the narration to create a natural presentation.

[1078] 7. Creating and providing presentation videos

[1079] The server uses a video editing tool (e.g., FFmpeg) to generate a presentation video that integrates slides, synthesized speech, subtitles, and avatars.

[1080] The final video file is uploaded to cloud storage, allowing users to download it or watch it online via their devices.

[1081] Specific examples

[1082] For example, when converting materials for in-house training into a presentation video, the following process is carried out.

[1083] 1. User request input: The user uploads a PDF file of training materials through a web interface and inputs a narration prompt: "This slide should explain the features of the new product."

[1084] 2. Receiving and parsing the file: The server receives the PDF file, parses the content, and converts it into text data.

[1085] 3. Parsing prompts: The server parses the narration prompts and generates a talk script.

[1086] 4. Generate synthetic voice: The server generates synthetic voice based on the talk script.

[1087] 5. Subtitle generation: The server generates subtitles from the synthesized speech and adjusts the display timing.

[1088] 6. Avatar integration: If the user wishes, the server will generate an avatar and integrate it into the video.

[1089] 7. Generating and providing presentation video: The server generates a presentation video that integrates all elements and provides it to the user.

[1090] In this way, the present invention can significantly reduce the user's workload and improve time efficiency, and also makes it possible to easily generate high-quality presentation videos.

[1091] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1092] Step 1: User Request Input

[1093] Input: The user inputs the digital file for the presentation (e.g., PDF, PowerPoint) and the narration prompt.

[1094] What it does: A user opens a dedicated web interface or desktop application, clicks the file upload button to upload a digital file, and simultaneously enters a narration prompt (e.g., "This slide should describe the features of our new product") into a text field.

[1095] Output: The digital files and narration prompts are sent to the server.

[1096] Step 2: Receiving and analyzing the file

[1097] Input: The server receives the digital file sent by the user.

[1098] What happens: The server first uses the appropriate library to determine the file format of the received file (e.g., Apache PDFBox for PDF, Apache POI for PowerPoint).

[1099] Output: The content of each slide is extracted as text data.

[1100] Step 3: Parse the prompt

[1101] Input: The server parses the narration prompt received from the user.

[1102] Specific operation: The server uses a natural language processing library (e.g., spaCy, NLTK) to parse the prompt and extract grammar and important keywords.

[1103] Output: A talk script corresponding to each slide is generated.

[1104] Step 4: Generate synthetic speech

[1105] Input: The server receives the generated talk script.

[1106] How it works: The server calls a synthetic speech engine (e.g., Google Text-to-Speech API, Amazon Polly) to convert the talk script into voice data. At this time, the voice quality and intonation are automatically set.

[1107] Output: Synthetic speech data is generated.

[1108] Step 5: Generate subtitles

[1109] Input: The server receives the generated synthesized voice data.

[1110] Specific operation: The server uses an audio waveform analysis tool to extract the timestamp of the audio data and generate subtitle data (e.g., SRT format).

[1111] Output: Subtitle files are generated.

[1112] Step 6: Integrating the Avatar

[1113] Input: Receives the user-selected avatar, synthesized voice data, and subtitle data.

[1114] Specific operation: The server uses an avatar generation API (e.g., Adobe Character Animator, VYOND) to generate avatar movements in sync with the synthesized voice, adjusting the timing and movement, and integrating the avatar data with slides, audio, and subtitles.

[1115] Output: Presentation data with avatars integrated is generated.

[1116] Step 7: Generate and serve presentation videos

[1117] Input: The server receives the consolidated presentation data.

[1118] What happens: The server uses a video editing tool (e.g. FFmpeg) to combine all elements (slides, synthesized speech, subtitles, avatars) and export them as a presentation video.

[1119] Output: The generated presentation video will be uploaded to cloud storage and a download link or streaming URL will be provided to users.

[1120] (Application example 1)

[1121] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1122] Conventional presentation video generation systems impose a heavy workload on educators when creating educational content, and have issues with the quality and adaptability of the videos created. In particular, for educational presentations, the coordination of slide content with narration and subtitles is crucial, and adjusting these elements satisfactorily can be time-consuming. Furthermore, systems lack the flexibility to adapt to different educational environments and learner needs. The present invention aims to solve these issues by providing a system that enables educators to easily and automatically generate high-quality educational presentation videos.

[1123] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1124] In this invention, the server includes means for receiving a digital presentation file input by a user, means for analyzing the received digital file and converting the content of each slide into text data, means for analyzing narration prompts input by the user and generating a talk script for each slide, means for generating synthetic speech based on the generated talk script, means for generating subtitles based on the generated synthetic speech, means for integrating the slides, the generated synthetic speech, and the generated subtitles to generate a presentation video, means for providing the generated presentation video to the user, means for generating a presentation video optimized as educational content based on content selected by the user, and means for providing an interface that is easy for educators to use and for automatically generating and adjusting content based on educator input. This enables educators to easily create high-quality educational presentation videos.

[1125] "User" refers to an individual or organization that uses this system to generate a presentation video.

[1126] A "digital presentation file" is an electronic file, such as a PDF or PowerPoint file, that contains the slides of a presentation.

[1127] "Text data" refers to the written information extracted from the slides in the digital file.

[1128] "Narration prompt" refers to text that the user enters for each slide and that serves as instructions for generating a talk script.

[1129] A "talk script" is a presentation text that is automatically generated based on narration prompts.

[1130] "Synthetic voice" is artificial voice data that is automatically generated based on a talk script.

[1131] "Subtitles" refers to text information that is displayed in response to synthesized speech.

[1132] A "presentation video" is a dynamic video file that combines slides, synthesized audio, and subtitles.

[1133] "Educational Content" refers to educational materials and videos created by educators for students and participants.

[1134] "Interface" refers to the user interface through which a user accesses the system and uploads files or enters prompts.

[1135] This invention provides a system that allows educators to easily create high-quality educational presentation videos. The system provides a series of processes that automatically generate educational content, including synthesized speech and subtitles, by simply uploading a digital file for the presentation and entering narration prompts.

[1136] Basic system configuration

[1137] The system mainly consists of the following components:

[1138] 1. User Interface

[1139] An interface through which users upload digital files for a presentation and enter narration prompts. This interface is typically implemented as a web-based application or a dedicated desktop application.

[1140] 2. Server Module

[1141] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[1142] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[1143] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[1144] Synthetic speech generation module: Generates synthetic speech based on the talk script. Software used includes Google Text-to-Speech (gTTS).

[1145] Subtitle generation module: Generates subtitles based on synthesized speech.

[1146] Video Integration Module: Generates a video presentation by integrating slides, synthesized speech, and subtitles, and optionally integrating avatars.

[1147] 3. Educational Module

[1148] Based on user input, it generates presentation videos that are optimized for educational content and automatically adjusts to meet the educator's requirements.

[1149] Program processing

[1150] The processing of the program will be explained in natural language below.

[1151] 1. User request input

[1152] Users upload digital files for presentations (e.g., PDF, PowerPoint) to the system via a web or desktop application, and also enter narration prompts through the user interface.

[1153] 2. Analyzing the file

[1154] The server receives the uploaded digital files, and a file analysis module converts the content of each slide into text data, making the presentation organized and easier to handle within the system.

[1155] 3. Parsing the prompt

[1156] The server uses a prompt analysis module to analyze the narration prompts input by the user and generate a talk script corresponding to each slide.

[1157] 4. Synthetic speech and subtitle generation

[1158] The server generates speech using a synthetic speech generation module based on the generated talk script, and creates corresponding subtitles using a subtitle generation module.

[1159] 5. Video Integration

[1160] The server uses a video integration module to integrate the slides, synthesized speech, and subtitles to generate a video presentation. If the user wishes, an avatar can also be used, which is also integrated into the video.

[1161] Specific examples

[1162] For example, if an educator is creating an educational presentation video on "Fundamental Concepts of Mathematics," they might do the following:

[1163] 1. Educators upload PowerPoint files of math teaching materials to the app.

[1164] 2. For each slide, include a prompt such as "Explain the difference between integers and natural numbers" or "On the next slide, provide an example of an integer."

[1165] 3. The server automatically generates narration and creates a presentation video.

[1166] Example prompt sentence:

[1167] "On this slide, explain the difference between integers and natural numbers."

[1168] "On the next slide, give an example of an integer."

[1169] "On the last slide, please summarize what you learned today."

[1170] The present invention allows educators to easily create high-quality educational presentation videos, thereby improving the quality of educational content.

[1171] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1172] Step 1:

[1173] A user uploads a digital file for a presentation to the system using a terminal, and simultaneously inputs a narration prompt corresponding to each slide through the interface, resulting in the digital file and the narration prompt as input data.

[1174] Input: Digital files for presentation, narration prompts

[1175] Output: Received digital files, narration prompts

[1176] Step 2:

[1177] The server analyzes the received digital files using a file analysis module, converting the content of each slide into text data and organizing it, thereby obtaining the text data for each slide.

[1178] Input: Received digital files

[1179] Output: Text data for each slide

[1180] Step 3:

[1181] The server uses a prompt analysis module to analyze the narration prompts entered by the user and generate a talk script corresponding to each slide. This analysis generates a talk script suitable for narration from the prompt text.

[1182] Input: Narration prompts, text data for each slide

[1183] Output: Talk script for each slide

[1184] Step 4:

[1185] The server uses a synthetic speech generation module to convert the generated talk script into synthetic speech, during which the quality of the synthetic speech is checked and adjusted if necessary.

[1186] Input: Talk script for each slide

[1187] Output: Synthesized voice data

[1188] Step 5:

[1189] The server generates subtitles using a subtitle generation module based on the synthesized speech, and the timing of the subtitles is set based on the time code of the synthesized speech.

[1190] Input: Synthetic voice data

[1191] Output: Subtitle data with timecode

[1192] Step 6:

[1193] The server uses a video integration module to integrate each slide, the synthesized speech, and the generated subtitles to generate a presentation video. If a user selects an avatar, the avatar is also integrated into the video in synchronization with the synthesized speech.

[1194] Input: Slide image data, synthesized voice data, subtitle data

[1195] Output: Presentation video data

[1196] Step 7:

[1197] The server provides the generated presentation video to the user, who can then download the video or watch it online using their device.

[1198] Input: Presentation video data

[1199] Output: Presentation video provided to the user

[1200] Step 8:

[1201] The server optimizes the generated presentation video as educational content, creating a video in a format that is easy for educators to use.

[1202] Input: Presentation video data

[1203] Output: Presentation video optimized for education

[1204] Step 9:

[1205] Users can use the video presentations optimized for education to achieve their educational goals by providing them to learners.

[1206] Input: Presentation video optimized for education

[1207] Output: Educational presentation use

[1208] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1209] MODE FOR CARRYING OUT THE INVENTION

[1210] This invention provides a system for efficiently generating presentation videos and providing high-quality videos that take user emotions into consideration. This system automates the entire process of analyzing digital files for presentations, generating synthesized speech and subtitles, and finally providing the presentation video, and further combines it with an emotion engine that recognizes user emotions.

[1211] Basic system configuration

[1212] The system mainly consists of the following components:

[1213] 1. User Interface

[1214] An interface through which users upload digital files for a presentation and enter narration prompts, typically implemented as a web-based application or a dedicated desktop application.

[1215] 2. Server Module

[1216] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[1217] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[1218] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[1219] Emotion Engine: Recognizes the user's emotions when entering narration prompts and takes them into account during analysis.

[1220] Synthetic speech generation module: Generates synthetic speech based on the talk script. Adjusts the tone and pace of the synthetic speech based on the user's emotions.

[1221] Subtitle generation module: Generates subtitles based on synthesized speech. Adjusts the expression of the subtitles based on the user's emotions.

[1222] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[1223] Program processing

[1224] The processing of the program will be explained in natural language below.

[1225] 1. User request input

[1226] Users upload digital presentation files (e.g., PDF, PowerPoint) to the system using their terminals, and also enter narration prompts through the user interface.

[1227] 2. Analyzing the file

[1228] The server receives the uploaded digital files and identifies the file format. A file analysis module then converts the content of each slide into text data, sorting out page numbers and header information. The extracted text data is then semantically analyzed using natural language processing (NLP) techniques to identify important elements.

[1229] 3. Prompt analysis and emotion recognition

[1230] The server's prompt analysis module analyzes the narration prompts entered by the user. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice and influences the content of the prompt. For example, if the user expresses "joy" as their emotion, a talk script with positive expressions will be generated.

[1231] 4. Synthetic speech and subtitle generation

[1232] The server's synthetic speech generation module converts the talk script into synthetic speech based on the results of the emotion engine. The tone and pace of the synthetic speech are adjusted based on the user's emotions. The subtitle generation module then generates subtitles based on the narration content, reflecting the user's emotions.

[1233] 5. Avatar Integration

[1234] If a user wishes to use an avatar, the server adds the avatar to the system and sets the avatar's lip sync and facial animation to match the synthesized voice.

[1235] 6. Creating and providing presentation videos

[1236] The server uses a video integration module to integrate slides, synthesized speech, subtitles, and avatars to generate a presentation video, which is then provided to users for download or online viewing on their devices.

[1237] Specific examples

[1238] For example, consider the creation of a marketing presentation. A user uploads marketing materials in PDF format and enters a narration prompt such as, "This slide should highlight the benefits of the product." As the user enters the narration prompt, an emotion engine recognizes the user's emotions, such as "excitement" or "anticipation." The server analyzes the file and generates a positive and emphatic talk script based on the emotion engine's results. A synthetic voice is then generated from the talk script, resulting in an expressive narration. Subtitles also reflect the user's emotions and use impactful expressions. If an avatar is added, the avatar will similarly behave to express excitement or anticipation. Finally, a video presentation integrating all elements is generated and served to the user.

[1239] In this way, the present invention can provide more effective and empathetic presentation videos that take into account the user's emotions, improving the quality and emotional appeal of presentations while also improving the efficiency of presentations.

[1240] The processing flow will be explained below.

[1241] Step 1:

[1242] The user uploads digital files for presentations to the system through a terminal, and inputs narration prompts through the user interface.

[1243] Step 2:

[1244] The server receives the uploaded digital file, determines the file format, and, in the case of PDF files, extracts the text and images page by page using a library such as Apache PDFBox.

[1245] Step 3:

[1246] The server's file analysis module organizes the text data by slide, extracting page numbers and header information, then uses natural language processing (NLP) techniques to semantically analyze the extracted text and identify key elements.

[1247] Step 4:

[1248] The server's prompt analysis module analyzes the narration prompts entered by the user. The emotion engine recognizes the user's emotions and reflects them in the prompt content. For example, if the user expresses "joy," a positive talk script is generated.

[1249] Step 5:

[1250] The server's synthetic speech generation module passes the talk script to a synthetic speech engine to generate the narration. The tone and pace of the speech are adjusted based on the user's emotions. Services such as Google Cloud Text-to-Speech may be used to generate the synthetic speech.

[1251] Step 6:

[1252] The server's subtitle generation module generates subtitles based on the content of the narration. It uses speech recognition technology to analyze the timing of the narration and produces text that reflects the user's emotions. For example, a cheerful expression that reflects the emotion of "joy" is used.

[1253] Step 7:

[1254] When a user requests to use an avatar, the server adds the avatar to the system. At this time, the server synchronizes the avatar's lip sync (mouth movements) and facial animation with the synthesized voice. The avatar also behaves in a way that expresses the user's emotions.

[1255] Step 8:

[1256] The server's video integration module integrates each slide, synthesized speech, subtitles, and avatars to generate a presentation video, which is then output with optimal image and sound quality using video editing software such as FFmpeg.

[1257] Step 9:

[1258] The server saves the generated presentation video and makes it accessible to users by uploading the video to cloud storage and providing users with a download link.

[1259] Step 10:

[1260] Users can download the presentation video or watch it online using the provided link on their devices. Users can check the video quality and make adjustments if necessary.

[1261] This is the specific processing flow of the program.

[1262] Example 2

[1263] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1264] The process of creating presentation videos manually is time-consuming and labor-intensive, making it inefficient. Furthermore, conventional systems struggle to generate videos that take user emotions into account, making it difficult to create high-quality presentation videos with emotional appeal. Furthermore, manually adjusting the intonation and timing of narration is tedious, making the creation of an integrated presentation video complex.

[1265] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a digital file for presentation input by a user, means for analyzing the received digital file and converting the content of each slide into text data, means for analyzing narration prompts input by the user and generating a talk script for each slide, means for recognizing the user's emotions and analyzing and generating narration prompts based on the emotions, means for generating synthetic speech based on the generated talk script, means for generating subtitles based on the generated synthetic speech, means for integrating the slides, the generated synthetic speech, and the generated subtitles to generate a presentation video, and means for providing the generated presentation video to the user. This enables automatic generation of high-quality presentation videos that take user emotions into consideration.

[1266] A "digital file for presentation" is a document for presentations saved in a format such as PDF or PowerPoint.

[1267] A "Presentation Video" is a video presentation generated by integrating digital file slides, narration audio, subtitles, and avatars.

[1268] A "narration prompt" is text that a user enters for each slide in a presentation to indicate what should be explained or what points should be discussed.

[1269] A "talk script" is a text of speech content corresponding to each slide of a presentation, which is generated based on file analysis and prompt analysis.

[1270] "Synthetic voice" is a mechanically generated voiced narration based on a talk script.

[1271] "Subtitles" are texts that are visually displayed within a presentation video and are used to display the contents of synthesized speech in text format.

[1272] An "avatar" is an image or animation of a character used in a presentation video that moves in sync with the synthesized voice.

[1273] An "emotion engine" is software that analyzes a user's facial expressions and voice to recognize their emotions and reflects the results in the presentation generation process.

[1274] MODE FOR CARRYING OUT THE INVENTION

[1275] System Overview

[1276] This invention provides a system for efficiently generating presentation videos and providing high-quality videos that take user emotions into consideration. This system automates the entire process of analyzing digital files for presentations, generating synthesized speech and subtitles, and finally providing the presentation video, and further combines it with an emotion engine that recognizes user emotions.

[1277] System Configuration

[1278] The system consists of the following main components:

[1279] 1. User Interface

[1280] An interface for users to upload digital files for presentations and enter narration prompts, implemented as a web-based application or a dedicated desktop application.

[1281] 2. Server Module

[1282] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[1283] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[1284] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[1285] Emotion Engine: Recognizes the user's emotions when entering narration prompts and analyzes them accordingly. For example, if the user expresses "joy" as their emotion, a talk script with positive expressions will be generated.

[1286] Synthetic speech generation module: Generates synthetic speech based on the talk script. Adjusts the tone and pace of the synthetic speech based on the user's emotions.

[1287] Subtitle generation module: Generates subtitles based on synthesized speech. Adjusts the expression of the subtitles based on the user's emotions.

[1288] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[1289] Operation explanation

[1290] 1. User request input

[1291] The user uploads digital files (e.g., PDF, PowerPoint) for the presentation through a user interface on the device, and also enters narration prompts through the user interface.

[1292] 2. Analyzing the file

[1293] The server receives the uploaded digital files and uses a file analysis module to determine the file format, which then converts the content of each slide into text data, which is then analyzed using natural language processing (NLP) techniques.

[1294] 3. Prompt analysis and emotion recognition

[1295] The server's prompt analysis module analyzes the narration prompts entered by the user. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice and reflects them in the prompt analysis. For example, if the user expresses "joy," a talk script with positive expressions is generated.

[1296] 4. Synthetic speech and subtitle generation

[1297] The server's synthetic speech generation module converts the talk script into synthetic speech based on the results of the emotion engine. The tone and pace are adjusted based on the user's emotions. The subtitle generation module then generates subtitles based on the narration content, reflecting the user's emotions.

[1298] 5. Avatar Integration

[1299] If a user wishes to use an avatar, the server adds the avatar to the system and sets the avatar's lip sync and facial animation to match the synthesized voice.

[1300] 6. Creating and providing presentation videos

[1301] The server uses a video integration module to integrate slides, synthesized speech, subtitles, and avatars to generate a presentation video, which is then provided to users for download or online viewing on their devices.

[1302] Specific examples

[1303] For example, consider a user creating a market analysis presentation. The user uploads the market analysis document in PDF format and enters a narration prompt such as, "In this slide, please emphasize the advantages of the new product." When the user enters the narration prompt, the emotion engine recognizes the user's "expectations" and generates a talk script based on them. The server analyzes the file and generates content that emphasizes important points based on the results of the emotion engine. Synthetic speech is then generated from the talk script, and subtitles also reflect the user's emotions. If an avatar is added, the avatar will also behave in a similar way, reflecting the emotion of expectation. Finally, a video presentation integrating all elements is generated and provided to the user.

[1304] Prompt Sentence Examples

[1305] 1. "Highlight the key points of your marketing strategy."

[1306] 2. "This slide should explain your efforts to improve customer satisfaction."

[1307] 3. "Please briefly summarize the technical features of your product."

[1308] In this way, the system of the present invention can take into account the user's emotions and provide more effective and empathetic presentation videos, improving the quality and emotional appeal of presentations while also improving the efficiency of presentations.

[1309] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1310] Step 1: User Request Input

[1311] ---

[1312] Users use terminals to access the system's user interface, upload digital files (e.g., PDF, PowerPoint) for presentations, and input narration prompts for the presentation content.

[1313] Input: Digital files for presentation, narration prompts

[1314] Output: Uploaded digital files, typed narration prompts

[1315] Specific behavior: The user interface provides buttons for uploading files and text boxes for entering narration prompts. Once the user enters the file and prompt, they are sent to the server.

[1316] Step 2: Receiving and analyzing the file

[1317] ---

[1318] The server receives the uploaded digital files and uses a file analysis module to convert the content of each slide into text data, identifying the file format and organizing page numbers and header information.

[1319] Input: Uploaded digital files

[1320] Output: Parsed text data

[1321] What it does: The server's file analysis module reads PDF or PowerPoint files, extracts the text within each slide, and uses natural language processing (NLP) techniques to identify important elements.

[1322] Step 3: Parsing the prompt and recognizing emotions

[1323] ---

[1324] The server's prompt analysis module analyzes the narration prompts entered by the user and generates a talk script. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice, and reflects those emotions in the prompt analysis.

[1325] Input: Narration prompts, user emotion information

[1326] Output: Talk script reflecting emotions

[1327] Specific operation: The emotion engine analyzes the user's facial expression and voice data acquired through the user interface, identifies the user's emotional state based on this data, and uses this information to generate a talk script appropriate for the narration prompt.

[1328] Step 4: Generate synthetic speech and subtitles

[1329] ---

[1330] The server uses a synthetic speech generation module to generate synthetic speech based on the talk script, adjusting the tone and pace of the speech based on the user's emotions. The subtitle generation module converts the content of the synthetic speech into text data and generates subtitles.

[1331] Input: Talk script, emotion information

[1332] Output: Synthesized speech, generated subtitles

[1333] Specific operation: The synthetic speech generation module receives the talk script as input data and dynamically adjusts the tone and pace of the speech based on the user's emotional information. The generated speech data is then converted into text and passed to the subtitle generation module, which finally creates subtitles for the presentation.

[1334] Step 5: Integrating the Avatar

[1335] ---

[1336] If a user wishes to use an avatar, the server adds the avatar to the system and sets the avatar's lip sync and facial animation to match the synthesized voice.

[1337] Input: Synthetic voice, user avatar preference

[1338] Output: Avatar with lip sync and animation added

[1339] Specific behavior: The server loads the avatar selected by the user from available avatar resources and sets the behavior based on the synthesized voice. Lip sync and facial animation are dynamically reflected based on emotional information, resulting in more natural behavior.

[1340] Step 6: Generate and serve presentation videos

[1341] ---

[1342] The server uses a video synthesis module to synthesize each slide, the generated synthetic speech, subtitles, and avatars into a single presentation video, which is then provided to users for download or online viewing.

[1343] Input: Slide data, synthesized speech, generated subtitles, avatar data

[1344] Output: Integrated presentation video

[1345] How it works: The video integration module combines the content of each slide, the generated audio, subtitles, and avatars, arranges them on a timeline, and generates a video file. The final presentation video is saved in a specified location for user access. For online viewing, a URL is provided to the user.

[1346] (Application example 2)

[1347] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1348] Conventional presentation video generation systems require users to input slide content individually, create narration, and generate synthesized speech, which is a complex process. Furthermore, because the system automatically generates the presentation without considering the user's emotions, it lacks appeal to viewers. Therefore, there is a need for a system that can efficiently generate presentation videos that respond to the user's emotions.

[1349] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving a digital file for presentation input by a user; means for analyzing the received digital file and converting the content of each slide into text data; means for analyzing narration prompts input by the user and generating a talk script for each slide; means for generating synthetic speech based on the generated talk script; means for generating subtitles based on the generated synthetic speech; means for recognizing a user's emotion and adjusting the tone and pace of the narration based on the recognized emotion; means for integrating each slide, the generated synthetic speech, the generated subtitles, and the recognized emotion information to generate a presentation video; and means for providing the generated presentation video to the user. This makes it possible to efficiently generate a highly appealing presentation video that is tailored to the user's emotion.

[1350] "Digital files for presentations" refer to slides and materials that users use for presentation purposes and are provided in digital formats such as PDF and PowerPoint.

[1351] "Narration prompt" refers to the content or wording of the narration entered by the user, including explanations and commentary for each slide.

[1352] A "talk script" is a spoken script generated by analyzing narration prompts, and is a narration instruction manual provided for each slide.

[1353] "Synthetic voice" is an artificial voice generated based on a talk script, and is reproduced to imitate human speech.

[1354] "Subtitles" are textual information that displays the contents of the synthesized speech, and serve to visually supplement the contents of the narration.

[1355] "Emotion information" is data of emotions recognized by analyzing the user's facial expressions, voice, etc., and indicates an emotional state such as joy, sadness, excitement, etc.

[1356] "Tone and pace adjustment" refers to changing the speaking style and speed of the synthesized voice based on emotional information, as a means of providing a more emotionally rich presentation.

[1357] "Video synthesis" is the process of combining slides, synthesized audio, and subtitles into a single video presentation that viewers can watch as a continuous story.

[1358] An "avatar" is a digital character that can be selected by the user to appear in the presentation video, and has lip-sync and facial animation.

[1359] This system analyzes a digital presentation file input by a user, generates a talk script based on narration prompts entered by the user, creates synthesized speech and subtitles, and provides a presentation video that integrates them. It is particularly notable for recognizing the user's emotions and adjusting the tone and pace of the narration based on those emotions.

[1360] Basic system configuration

[1361] 1. User Interface

[1362] Users use a device to upload digital files for presentations and enter narration prompts. The user interface may be implemented as a web-based application or a dedicated desktop application, typically a web application running in a browser.

[1363] 2. Server Module

[1364] The server includes the following sub-modules to analyze the received digital files and extract the content of each slide as text data:

[1365] File Analysis Module: Identifies the digital file format (PDF, PowerPoint, etc.) of the uploaded presentation and extracts the text data for each slide.

[1366] Prompt Analysis Module: Analyzes the narration prompts entered by the user and generates a talk script for each slide.

[1367] Emotion Recognition Engine: Recognizes emotions from the user's facial expressions and voice data. This is achieved by the user using the device's camera and microphone.

[1368] Synthetic voice generation module: Creates a synthetic voice based on the generated talk script, and adjusts the tone and pace according to the user's emotions.

[1369] Subtitle generation module: Generates subtitles in synchronization with the synthesized speech. The subtitles also include expressions based on the user's emotions.

[1370] Video Integration Module: Generates a presentation video by integrating each slide, synthesized speech, subtitles, and emotional information.

[1371] 3. Use of Avatars

[1372] A user-selected avatar can be integrated into the presentation video, lip-syncing to the generated synthetic speech and changing facial expressions based on emotions, enhancing the visual and emotional appeal.

[1373] Program processing explanation

[1374] The server uses the following hardware and software:

[1375] Hardware: Camera, microphone, user device (PC, smartphone, etc.)

[1376] Software: OpenCV, Natural Language Processing Library (NLP), Emotion Recognition Engine, Synthetic Speech Generation Library

[1377] Specific examples

[1378] For example, when creating a marketing presentation, a user uploads marketing materials in PDF format and enters a narration prompt such as, "This slide should highlight the benefits of the product." As the user enters the narration prompt, an emotion recognition engine recognizes the user's "joy." The server analyzes the file and generates a talk script that emphasizes a positive tone based on the emotion recognition results. Synthetic speech is then generated from this talk script, creating an expressive narration. Subtitles also reflect the user's emotions and use impactful expressions. If an avatar is added, the avatar will also behave in a similar way, expressing joy. Finally, a video presentation integrating all elements is generated and provided to the user.

[1379] Prompt Sentence Examples

[1380] "Analyze the text of marketing presentation slides, generate a narration that adjusts the tone and pace based on user emotion, add subtitles, and create a video of the presentation. The narration prompt is "Highlight the product benefits on this slide," and the user emotion is "Delight."

[1381] In this way, the present invention takes into account the user's emotions and can provide more effective and appealing presentation videos, improving the efficiency and quality of presentation creation and eliciting empathy from viewers.

[1382] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1383] Step 1:

[1384] User request input

[1385] Users use their terminals to upload digital files (e.g., PDF, PowerPoint) for presentations to the server and input narration prompts through a user interface.

[1386] Input: Digital files for presentation and narration prompts

[1387] Output: Received digital files and prompt data

[1388] Specific operation: The terminal displays a web form for uploading a file, from which the user selects a file and enters text for the narration prompt.

[1389] Step 2:

[1390] Analyzing files

[1391] The server's file analysis module analyzes the received digital files, extracting the content of each slide as text data and organizing page numbers and header information.

[1392] Input: Received digital presentation file

[1393] Output: Text data for each slide

[1394] Specific operation: The server determines the file format and extracts the text data using an appropriate library (e.g., PyMuPDF).

[1395] Step 3:

[1396] Parsing prompts

[1397] The server's prompt analysis module analyzes the narration prompts entered by the user and generates a talk script, while the emotion recognition engine recognizes the user's emotions and influences the content of the prompts.

[1398] Input: User-supplied narration prompts and user emotion data

[1399] Output: A talk script that reflects the user's emotions

[1400] Specific operation: The server uses an emotion recognition engine (e.g., OpenCV, dlib) to analyze the user's emotions and a natural language processing library (e.g., spaCy) to analyze the narration prompts.

[1401] Step 4:

[1402] Generate synthetic speech

[1403] The server's synthetic voice generation module generates synthetic voice based on the talk script and adjusts the tone and pace according to the user's emotions.

[1404] Input: A talk script that reflects the user's emotions

[1405] Output: Emotionally adjusted synthetic speech

[1406] Specific operation: The server generates an audio file using a synthetic speech generation library (e.g., Google Text-to-Speech API) and processes the audio (adjusts tone and pace) based on emotions.

[1407] Step 5:

[1408] Subtitle generation

[1409] A subtitle generation module of the server generates subtitles based on the synthesized speech, and includes expressions according to the user's emotions.

[1410] Input: Synthesized voice and talk script

[1411] Output: Emotionally sensitive subtitle data

[1412] Specific operation: The server uses a speech recognition library (e.g., Google Speech-to-Text API) to generate text from the speech and creates subtitle data in a style that corresponds to the user's emotions.

[1413] Step 6:

[1414] Presentation video generation

[1415] The video integration module of the server integrates each slide, synthesized speech, subtitles, and emotional information to generate a presentation video.

[1416] Input: Each slide, synthesized speech, subtitle data, emotional information

[1417] Output: Finished presentation video

[1418] Specific operation: The server uses a video editing library (e.g., FFmpeg) to integrate each element and generate a video file.

[1419] Step 7:

[1420] Providing generated presentation videos

[1421] The generated presentation video is provided to the user, who can download the video or watch it online using their device.

[1422] Input: Finished presentation video

[1423] Output: User-accessible video link of the presentation

[1424] Specific operation: The server stores the video file in storage and provides a download link or streaming URL to the user's device.

[1425] This processing flow enables users to efficiently generate emotionally-reflective presentation videos and deliver visually and emotionally appealing presentations.

[1426] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1427] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1428] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1429] [Fourth embodiment]

[1430] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1431] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1432] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1433] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1434] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1435] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1436] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1437] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1438] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1439] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1440] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1441] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1442] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1443] MODE FOR CARRYING OUT THE INVENTION

[1444] The present invention provides a system for efficiently generating presentation videos and reducing the user's workload by automating a series of processes: analyzing digital files for presentations, generating synthesized speech and subtitles, and finally providing the presentation videos.

[1445] Basic system configuration

[1446] The system mainly consists of the following components:

[1447] 1. User Interface

[1448] An interface through which users upload digital files for a presentation and enter narration prompts, typically implemented as a web-based application or a dedicated desktop application.

[1449] 2. Server Module

[1450] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[1451] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[1452] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[1453] Synthetic speech generation module: Generates synthetic speech based on the talk script.

[1454] Subtitle generation module: Generates subtitles based on synthesized speech.

[1455] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[1456] Program processing

[1457] The processing of the program will be explained in natural language below.

[1458] 1. User request input

[1459] Users upload digital presentation files (e.g., PDF, PowerPoint) to the system using their terminals, and also enter narration prompts through the user interface.

[1460] 2. Analyzing the file

[1461] The server receives the uploaded digital files, identifies the file format, and then a file analysis module converts the content of each slide into text data, sorting out page numbers and header information.

[1462] 3. Parsing the prompt

[1463] The server uses a prompt parsing module to parse the input narration prompts, which then generates a talking script that corresponds to the content of each slide.

[1464] 4. Generating Synthetic Speech

[1465] The server utilizes a synthetic speech generation module to convert the talk script into synthetic speech, and in the process evaluates the quality of the generated speech and adjusts the tone and intonation if necessary.

[1466] 5. Subtitle Generation

[1467] The server uses a subtitle generation module to generate subtitles based on the generated synthesized speech, setting the time code of the subtitles to match the timing of the narration and adjusting the display timing.

[1468] 6. Avatar Integration

[1469] If a user chooses to use an avatar, the server integrates the avatar into the video, synchronizing the avatar's movements and facial expressions with the synthesized voice to create a natural presentation.

[1470] 7. Creating and providing presentation videos

[1471] The server uses the video integration module to generate a presentation video that integrates slides, synthesized speech, subtitles, and avatars, and the generated video is provided to the user, who can download it or watch it online using their device.

[1472] Specific examples

[1473] For example, when converting in-house training materials into a video presentation, the following steps are performed. First, the user uploads the PDF file of the training materials and enters a narration prompt, such as "Please explain the features of the new product on this slide." The server analyzes the file and generates a talk script based on the prompt. Next, the server converts the talk script into synthesized speech and generates subtitles based on that speech. If the user wishes, an avatar can be integrated and made to move in sync with the synthesized speech. Finally, a video presentation incorporating all elements is generated and provided to the user.

[1474] In this way, the present invention can significantly reduce the user's efforts and improve time efficiency, and also makes it possible to easily generate high-quality presentation videos.

[1475] The processing flow will be explained below.

[1476] Step 1:

[1477] The user uploads digital files for presentations to the system through a terminal, and inputs narration prompts through the user interface.

[1478] Step 2:

[1479] The server receives the uploaded digital file, determines the file format, and, in the case of PDF files, extracts the text and images page by page using a library such as Apache PDFBox.

[1480] Step 3:

[1481] The server's file analysis module organizes the text data by slide, extracting page numbers and header information, then uses natural language processing (NLP) techniques to semantically analyze the extracted text and identify key elements.

[1482] Step 4:

[1483] The server's prompt analysis module analyzes the narration prompts from the user and generates a talk script, which includes technology to automatically generate appropriate explanatory text for each slide based on the prompts.

[1484] Step 5:

[1485] The server's synthetic speech generation module passes the talk script to a synthetic speech engine to generate the narration, using services such as Google Cloud Text-to-Speech, followed by post-processing to adjust the tone and intonation.

[1486] Step 6:

[1487] The subtitle generation module of the server generates subtitles based on the content of the narration, specifically, by analyzing the timing of the narration using speech recognition technology and converting the corresponding text into a subtitle format.

[1488] Step 7:

[1489] When a user requests the use of an avatar, the server adds the avatar to the system, setting the avatar's lip sync (mouth movements) and facial animation based on the timing of the synthesized voice.

[1490] Step 8:

[1491] The server's video synthesis module synthesizes each slide, narration, subtitles, and avatar to generate a presentation video, using video editing software (e.g., FFmpeg) to output the video with optimal image and sound quality.

[1492] Step 9:

[1493] The server saves the generated presentation video and makes it accessible to users by uploading the video to cloud storage and providing users with a download link.

[1494] Step 10:

[1495] Users can download the presentation video or watch it online using the provided link on their devices. Users can check the video quality and make adjustments if necessary.

[1496] This is the specific processing flow of the program.

[1497] Example 1

[1498] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1499] Creating traditional presentation videos is time-consuming and laborious, and requires significant effort to adjust the quality of the narration and synchronize subtitles. It is also difficult to adjust the intonation and timing of the narration, and using avatars for natural presentations is also time-consuming. There is a need to solve these issues, reduce the user's workload, and efficiently generate high-quality presentation videos.

[1500] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1501] In this invention, the server includes means for receiving a digital presentation file input by a user, means for analyzing the received digital file and converting the content of each slide into text data, means for analyzing narration prompts input by the user and generating a talk script for each slide, means for generating synthetic speech based on the generated talk script, means for generating subtitles based on the generated synthetic speech, means for evaluating the quality of the generated synthetic speech and adjusting the tone and intonation as necessary, means for setting a time code for the generated subtitles and adjusting their display timing, means for integrating an avatar selected by the user into a presentation video and synchronizing it with the synthetic speech, means for integrating each slide, the generated synthetic speech, the generated subtitles, and the avatar to generate a presentation video, and means for providing the generated presentation video to the user. This enables automatic generation of high-quality presentation videos while reducing the user's workload and improving time efficiency.

[1502] A "digital file for presentation" is a file in an electronic format, such as a PDF or PowerPoint, that a user uses to give a presentation.

[1503] "Analysis" is the process of breaking down and analyzing the content of received digital files and narration prompts to extract the necessary information.

[1504] "Text data" refers to data that expresses the content of each slide and narration prompts as text information.

[1505] A "narration prompt" is an instruction entered by the user, and is the original text that is generated as a talk script for each slide.

[1506] A "talk script" is a sentence of speech content corresponding to each slide, generated based on a narration prompt.

[1507] "Synthetic voice" refers to the generation of a talk script as voice data, and is a voice that imitates a human voice using synthetic voice technology.

[1508] "Subtitles" are texts that are generated based on synthesized speech and displayed on the screen, providing a visual complement to the narration content.

[1509] "Time code" is time information that indicates the timing of displaying and playing subtitles and narration.

[1510] An "avatar" is a computer-generated character used to give a presentation in place of a human.

[1511] "Video synthesis" is the process of combining individual slides, synthesized speech, subtitles, and avatars into a single video presentation.

[1512] "Providing" means distributing the generated presentation video to users in a form that allows them to view it online or download it.

[1513] MODE FOR CARRYING OUT THE INVENTION

[1514] This invention provides a system for efficiently generating presentation videos and reducing the user's workload. This system automates a series of processes: analyzing digital files for presentations, generating and integrating synthesized speech, subtitles, and avatars, and finally providing the presentation video.

[1515] Basic system configuration

[1516] The system mainly consists of the following components:

[1517] 1. User Interface

[1518] An interface for users to upload digital files for presentations and enter narration prompts.

[1519] It is typically implemented as a web-based application or a dedicated desktop application.

[1520] 2. Server Module

[1521] It is the main module that receives, analyzes and processes digital files.

[1522] Contains the following submodules:

[1523] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[1524] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[1525] Synthetic speech generation module: Generates synthetic speech based on the talk script.

[1526] Subtitle generation module: Generates subtitles based on synthesized speech.

[1527] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[1528] Program processing and data processing

[1529] Below, we will explain the details of this system's processing and the hardware and software used at each step.

[1530] 1. User request input

[1531] Users upload digital files (e.g., PDF, PowerPoint) for presentations to their devices through a dedicated web interface or desktop application, and simultaneously input prompts for narration.

[1532] Specifically, users upload files by dragging and dropping them into a form in their browser or using a file selection dialog, and enter narration prompts into text fields.

[1533] 2. Receiving and analyzing the file

[1534] The server receives the uploaded digital file and determines the file format using an appropriate library (e.g., Apache PDFBox, Apache POI).

[1535] Depending on the file format, the content of each slide is converted to text data. For example, if it is a PDF file, Apache PDFBox is used, and if it is a PowerPoint file, Apache POI is used.

[1536] 3. Parsing the prompt

[1537] The server uses a natural language processing library (e.g., spaCy, NLTK) to analyze the input narration prompts and generates a talk script for each slide based on the analysis results.

[1538] Specifically, the system analyzes the grammar and key keywords of the narration prompts and uses a script generation algorithm to generate a semantically consistent talk script.

[1539] 4. Generating Synthetic Speech

[1540] The server uses a synthetic speech engine (e.g., Google Text-to-Speech API, Amazon Polly) to convert the generated talk script into synthetic speech. It adjusts the tone and intonation as necessary to ensure quality.

[1541] 5. Subtitle Generation

[1542] The server generates subtitles based on the synthesized speech, extracts timestamps using speech waveform analysis, and generates subtitle data (e.g., SRT format).

[1543] Specifically, timestamps are extracted from the audio data, and the display timing is adjusted by generating and setting the subtitle time code.

[1544] 6. Avatar Integration

[1545] If the user wishes to use an avatar, the server uses an avatar generation API (e.g., Adobe Character Animator, VYOND) to generate avatar movements in accordance with the synthesized voice.

[1546] The avatar's movements and facial expressions are synchronized with the narration to create a natural presentation.

[1547] 7. Creating and providing presentation videos

[1548] The server uses a video editing tool (e.g., FFmpeg) to generate a presentation video that integrates slides, synthesized speech, subtitles, and avatars.

[1549] The final video file is uploaded to cloud storage, allowing users to download it or watch it online via their devices.

[1550] Specific examples

[1551] For example, when converting materials for in-house training into a presentation video, the following process is carried out.

[1552] 1. User request input: The user uploads a PDF file of training materials through a web interface and inputs a narration prompt: "This slide should explain the features of the new product."

[1553] 2. Receiving and parsing the file: The server receives the PDF file, parses the content, and converts it into text data.

[1554] 3. Parsing prompts: The server parses the narration prompts and generates a talk script.

[1555] 4. Generate synthetic voice: The server generates synthetic voice based on the talk script.

[1556] 5. Subtitle generation: The server generates subtitles from the synthesized speech and adjusts the display timing.

[1557] 6. Avatar integration: If the user wishes, the server will generate an avatar and integrate it into the video.

[1558] 7. Generating and providing presentation video: The server generates a presentation video that integrates all elements and provides it to the user.

[1559] In this way, the present invention can significantly reduce the user's workload and improve time efficiency, and also makes it possible to easily generate high-quality presentation videos.

[1560] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1561] Step 1: User Request Input

[1562] Input: The user inputs the digital file for the presentation (e.g., PDF, PowerPoint) and the narration prompt.

[1563] What it does: A user opens a dedicated web interface or desktop application, clicks the file upload button to upload a digital file, and simultaneously enters a narration prompt (e.g., "This slide should describe the features of our new product") into a text field.

[1564] Output: The digital files and narration prompts are sent to the server.

[1565] Step 2: Receiving and analyzing the file

[1566] Input: The server receives the digital file sent by the user.

[1567] What happens: The server first uses the appropriate library to determine the file format of the received file (e.g., Apache PDFBox for PDF, Apache POI for PowerPoint).

[1568] Output: The content of each slide is extracted as text data.

[1569] Step 3: Parse the prompt

[1570] Input: The server parses the narration prompt received from the user.

[1571] Specific operation: The server uses a natural language processing library (e.g., spaCy, NLTK) to parse the prompt and extract grammar and important keywords.

[1572] Output: A talk script corresponding to each slide is generated.

[1573] Step 4: Generate synthetic speech

[1574] Input: The server receives the generated talk script.

[1575] How it works: The server calls a synthetic speech engine (e.g., Google Text-to-Speech API, Amazon Polly) to convert the talk script into voice data. At this time, the voice quality and intonation are automatically set.

[1576] Output: Synthetic speech data is generated.

[1577] Step 5: Generate subtitles

[1578] Input: The server receives the generated synthesized voice data.

[1579] Specific operation: The server uses an audio waveform analysis tool to extract the timestamp of the audio data and generate subtitle data (e.g., SRT format).

[1580] Output: Subtitle files are generated.

[1581] Step 6: Integrating the Avatar

[1582] Input: Receives the user-selected avatar, synthesized voice data, and subtitle data.

[1583] Specific operation: The server uses an avatar generation API (e.g., Adobe Character Animator, VYOND) to generate avatar movements in sync with the synthesized voice, adjusting the timing and movement, and integrating the avatar data with slides, audio, and subtitles.

[1584] Output: Presentation data with avatars integrated is generated.

[1585] Step 7: Generate and serve presentation videos

[1586] Input: The server receives the consolidated presentation data.

[1587] What happens: The server uses a video editing tool (e.g. FFmpeg) to combine all elements (slides, synthesized speech, subtitles, avatars) and export them as a presentation video.

[1588] Output: The generated presentation video will be uploaded to cloud storage and a download link or streaming URL will be provided to users.

[1589] (Application example 1)

[1590] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1591] Conventional presentation video generation systems impose a heavy workload on educators when creating educational content, and have issues with the quality and adaptability of the videos created. In particular, for educational presentations, the coordination of slide content with narration and subtitles is crucial, and adjusting these elements satisfactorily can be time-consuming. Furthermore, systems lack the flexibility to adapt to different educational environments and learner needs. The present invention aims to solve these issues by providing a system that enables educators to easily and automatically generate high-quality educational presentation videos.

[1592] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1593] In this invention, the server includes means for receiving a digital presentation file input by a user, means for analyzing the received digital file and converting the content of each slide into text data, means for analyzing narration prompts input by the user and generating a talk script for each slide, means for generating synthetic speech based on the generated talk script, means for generating subtitles based on the generated synthetic speech, means for integrating the slides, the generated synthetic speech, and the generated subtitles to generate a presentation video, means for providing the generated presentation video to the user, means for generating a presentation video optimized as educational content based on content selected by the user, and means for providing an interface that is easy for educators to use and for automatically generating and adjusting content based on educator input. This enables educators to easily create high-quality educational presentation videos.

[1594] "User" refers to an individual or organization that uses this system to generate a presentation video.

[1595] A "digital presentation file" is an electronic file, such as a PDF or PowerPoint file, that contains the slides of a presentation.

[1596] "Text data" refers to the written information extracted from the slides in the digital file.

[1597] "Narration prompt" refers to text that the user enters for each slide and that serves as instructions for generating a talk script.

[1598] A "talk script" is a presentation text that is automatically generated based on narration prompts.

[1599] "Synthetic voice" is artificial voice data that is automatically generated based on a talk script.

[1600] "Subtitles" refers to text information that is displayed in response to synthesized speech.

[1601] A "presentation video" is a dynamic video file that combines slides, synthesized audio, and subtitles.

[1602] "Educational Content" refers to educational materials and videos created by educators for students and participants.

[1603] "Interface" refers to the user interface through which a user accesses the system and uploads files or enters prompts.

[1604] This invention provides a system that allows educators to easily create high-quality educational presentation videos. The system provides a series of processes that automatically generate educational content, including synthesized speech and subtitles, by simply uploading a digital file for the presentation and entering narration prompts.

[1605] Basic system configuration

[1606] The system mainly consists of the following components:

[1607] 1. User Interface

[1608] An interface through which users upload digital files for a presentation and enter narration prompts. This interface is typically implemented as a web-based application or a dedicated desktop application.

[1609] 2. Server Module

[1610] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[1611] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[1612] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[1613] Synthetic speech generation module: Generates synthetic speech based on the talk script. Software used includes Google Text-to-Speech (gTTS).

[1614] Subtitle generation module: Generates subtitles based on synthesized speech.

[1615] Video Integration Module: Generates a video presentation by integrating slides, synthesized speech, and subtitles, and optionally integrating avatars.

[1616] 3. Educational Module

[1617] Based on user input, it generates presentation videos that are optimized for educational content and automatically adjusts to meet the educator's requirements.

[1618] Program processing

[1619] The processing of the program will be explained in natural language below.

[1620] 1. User request input

[1621] Users upload digital files for presentations (e.g., PDF, PowerPoint) to the system via a web or desktop application, and also enter narration prompts through the user interface.

[1622] 2. Analyzing the file

[1623] The server receives the uploaded digital files, and a file analysis module converts the content of each slide into text data, making the presentation organized and easier to handle within the system.

[1624] 3. Parsing the prompt

[1625] The server uses a prompt analysis module to analyze the narration prompts input by the user and generate a talk script corresponding to each slide.

[1626] 4. Synthetic speech and subtitle generation

[1627] The server generates speech using a synthetic speech generation module based on the generated talk script, and creates corresponding subtitles using a subtitle generation module.

[1628] 5. Video Integration

[1629] The server uses a video integration module to integrate the slides, synthesized speech, and subtitles to generate a video presentation. If the user wishes, an avatar can also be used, which is also integrated into the video.

[1630] Specific examples

[1631] For example, if an educator is creating an educational presentation video on "Fundamental Concepts of Mathematics," they might do the following:

[1632] 1. Educators upload PowerPoint files of math teaching materials to the app.

[1633] 2. For each slide, include a prompt such as "Explain the difference between integers and natural numbers" or "On the next slide, provide an example of an integer."

[1634] 3. The server automatically generates narration and creates a presentation video.

[1635] Example prompt sentence:

[1636] "On this slide, explain the difference between integers and natural numbers."

[1637] "On the next slide, give an example of an integer."

[1638] "On the last slide, please summarize what you learned today."

[1639] The present invention allows educators to easily create high-quality educational presentation videos, thereby improving the quality of educational content.

[1640] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1641] Step 1:

[1642] A user uploads a digital file for a presentation to the system using a terminal, and simultaneously inputs a narration prompt corresponding to each slide through the interface, resulting in the digital file and the narration prompt as input data.

[1643] Input: Digital files for presentation, narration prompts

[1644] Output: Received digital files, narration prompts

[1645] Step 2:

[1646] The server analyzes the received digital files using a file analysis module, converting the content of each slide into text data and organizing it, thereby obtaining the text data for each slide.

[1647] Input: Received digital files

[1648] Output: Text data for each slide

[1649] Step 3:

[1650] The server uses a prompt analysis module to analyze the narration prompts entered by the user and generate a talk script corresponding to each slide. This analysis generates a talk script suitable for narration from the prompt text.

[1651] Input: Narration prompts, text data for each slide

[1652] Output: Talk script for each slide

[1653] Step 4:

[1654] The server uses a synthetic speech generation module to convert the generated talk script into synthetic speech, during which the quality of the synthetic speech is checked and adjusted if necessary.

[1655] Input: Talk script for each slide

[1656] Output: Synthesized voice data

[1657] Step 5:

[1658] The server generates subtitles using a subtitle generation module based on the synthesized speech, and the timing of the subtitles is set based on the time code of the synthesized speech.

[1659] Input: Synthetic voice data

[1660] Output: Subtitle data with timecode

[1661] Step 6:

[1662] The server uses a video integration module to integrate each slide, the synthesized speech, and the generated subtitles to generate a presentation video. If a user selects an avatar, the avatar is also integrated into the video in synchronization with the synthesized speech.

[1663] Input: Slide image data, synthesized voice data, subtitle data

[1664] Output: Presentation video data

[1665] Step 7:

[1666] The server provides the generated presentation video to the user, who can then download the video or watch it online using their device.

[1667] Input: Presentation video data

[1668] Output: Presentation video provided to the user

[1669] Step 8:

[1670] The server optimizes the generated presentation video as educational content, creating a video in a format that is easy for educators to use.

[1671] Input: Presentation video data

[1672] Output: Presentation video optimized for education

[1673] Step 9:

[1674] Users can use the video presentations optimized for education to achieve their educational goals by providing them to learners.

[1675] Input: Presentation video optimized for education

[1676] Output: Educational presentation use

[1677] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1678] MODE FOR CARRYING OUT THE INVENTION

[1679] This invention provides a system for efficiently generating presentation videos and providing high-quality videos that take user emotions into consideration. This system automates the entire process of analyzing digital files for presentations, generating synthesized speech and subtitles, and finally providing the presentation video, and further combines it with an emotion engine that recognizes user emotions.

[1680] Basic system configuration

[1681] The system mainly consists of the following components:

[1682] 1. User Interface

[1683] An interface through which users upload digital files for a presentation and enter narration prompts, typically implemented as a web-based application or a dedicated desktop application.

[1684] 2. Server Module

[1685] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[1686] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[1687] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[1688] Emotion Engine: Recognizes the user's emotions when entering narration prompts and takes them into account during analysis.

[1689] Synthetic speech generation module: Generates synthetic speech based on the talk script. Adjusts the tone and pace of the synthetic speech based on the user's emotions.

[1690] Subtitle generation module: Generates subtitles based on synthesized speech. Adjusts the expression of the subtitles based on the user's emotions.

[1691] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[1692] Program processing

[1693] The processing of the program will be explained in natural language below.

[1694] 1. User request input

[1695] Users upload digital presentation files (e.g., PDF, PowerPoint) to the system using their terminals, and also enter narration prompts through the user interface.

[1696] 2. Analyzing the file

[1697] The server receives the uploaded digital files and identifies the file format. A file analysis module then converts the content of each slide into text data, sorting out page numbers and header information. The extracted text data is then semantically analyzed using natural language processing (NLP) techniques to identify important elements.

[1698] 3. Prompt analysis and emotion recognition

[1699] The server's prompt analysis module analyzes the narration prompts entered by the user. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice and influences the content of the prompt. For example, if the user expresses "joy" as their emotion, a talk script with positive expressions will be generated.

[1700] 4. Synthetic speech and subtitle generation

[1701] The server's synthetic speech generation module converts the talk script into synthetic speech based on the results of the emotion engine. The tone and pace of the synthetic speech are adjusted based on the user's emotions. The subtitle generation module then generates subtitles based on the narration content, reflecting the user's emotions.

[1702] 5. Avatar Integration

[1703] If a user wishes to use an avatar, the server adds the avatar to the system and sets the avatar's lip sync and facial animation to match the synthesized voice.

[1704] 6. Creating and providing presentation videos

[1705] The server uses a video integration module to integrate slides, synthesized speech, subtitles, and avatars to generate a presentation video, which is then provided to users for download or online viewing on their devices.

[1706] Specific examples

[1707] For example, consider the creation of a marketing presentation. A user uploads marketing materials in PDF format and enters a narration prompt such as, "This slide should highlight the benefits of the product." As the user enters the narration prompt, an emotion engine recognizes the user's emotions, such as "excitement" or "anticipation." The server analyzes the file and generates a positive and emphatic talk script based on the emotion engine's results. A synthetic voice is then generated from the talk script, resulting in an expressive narration. Subtitles also reflect the user's emotions and use impactful expressions. If an avatar is added, the avatar will similarly behave to express excitement or anticipation. Finally, a video presentation integrating all elements is generated and served to the user.

[1708] In this way, the present invention can provide more effective and empathetic presentation videos that take into account the user's emotions, improving the quality and emotional appeal of presentations while also improving the efficiency of presentations.

[1709] The processing flow will be explained below.

[1710] Step 1:

[1711] The user uploads digital files for presentations to the system through a terminal, and inputs narration prompts through the user interface.

[1712] Step 2:

[1713] The server receives the uploaded digital file, determines the file format, and, in the case of PDF files, extracts the text and images page by page using a library such as Apache PDFBox.

[1714] Step 3:

[1715] The server's file analysis module organizes the text data by slide, extracting page numbers and header information, then uses natural language processing (NLP) techniques to semantically analyze the extracted text and identify key elements.

[1716] Step 4:

[1717] The server's prompt analysis module analyzes the narration prompts entered by the user. The emotion engine recognizes the user's emotions and reflects them in the prompt content. For example, if the user expresses "joy," a positive talk script is generated.

[1718] Step 5:

[1719] The server's synthetic speech generation module passes the talk script to a synthetic speech engine to generate the narration. The tone and pace of the speech are adjusted based on the user's emotions. Services such as Google Cloud Text-to-Speech may be used to generate the synthetic speech.

[1720] Step 6:

[1721] The server's subtitle generation module generates subtitles based on the content of the narration. It uses speech recognition technology to analyze the timing of the narration and produces text that reflects the user's emotions. For example, a cheerful expression that reflects the emotion of "joy" is used.

[1722] Step 7:

[1723] When a user requests to use an avatar, the server adds the avatar to the system. At this time, the server synchronizes the avatar's lip sync (mouth movements) and facial animation with the synthesized voice. The avatar also behaves in a way that expresses the user's emotions.

[1724] Step 8:

[1725] The server's video integration module integrates each slide, synthesized speech, subtitles, and avatars to generate a presentation video, which is then output with optimal image and sound quality using video editing software such as FFmpeg.

[1726] Step 9:

[1727] The server saves the generated presentation video and makes it accessible to users by uploading the video to cloud storage and providing users with a download link.

[1728] Step 10:

[1729] Users can download the presentation video or watch it online using the provided link on their devices. Users can check the video quality and make adjustments if necessary.

[1730] This is the specific processing flow of the program.

[1731] Example 2

[1732] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1733] The process of creating presentation videos manually is time-consuming and labor-intensive, making it inefficient. Furthermore, conventional systems struggle to generate videos that take user emotions into account, making it difficult to create high-quality presentation videos with emotional appeal. Furthermore, manually adjusting the intonation and timing of narration is tedious, making the creation of an integrated presentation video complex.

[1734] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a digital file for presentation input by a user, means for analyzing the received digital file and converting the content of each slide into text data, means for analyzing narration prompts input by the user and generating a talk script for each slide, means for recognizing the user's emotions and analyzing and generating narration prompts based on the emotions, means for generating synthetic speech based on the generated talk script, means for generating subtitles based on the generated synthetic speech, means for integrating the slides, the generated synthetic speech, and the generated subtitles to generate a presentation video, and means for providing the generated presentation video to the user. This enables automatic generation of high-quality presentation videos that take user emotions into consideration.

[1735] A "digital file for presentation" is a document for presentations saved in a format such as PDF or PowerPoint.

[1736] A "Presentation Video" is a video presentation generated by integrating digital file slides, narration audio, subtitles, and avatars.

[1737] A "narration prompt" is text that a user enters for each slide in a presentation to indicate what should be explained or what points should be discussed.

[1738] A "talk script" is a text of speech content corresponding to each slide of a presentation, which is generated based on file analysis and prompt analysis.

[1739] "Synthetic voice" is a mechanically generated voiced narration based on a talk script.

[1740] "Subtitles" are texts that are visually displayed within a presentation video and are used to display the contents of synthesized speech in text format.

[1741] An "avatar" is an image or animation of a character used in a presentation video that moves in sync with the synthesized voice.

[1742] An "emotion engine" is software that analyzes a user's facial expressions and voice to recognize their emotions and reflects the results in the presentation generation process.

[1743] MODE FOR CARRYING OUT THE INVENTION

[1744] System Overview

[1745] This invention provides a system for efficiently generating presentation videos and providing high-quality videos that take user emotions into consideration. This system automates the entire process of analyzing digital files for presentations, generating synthesized speech and subtitles, and finally providing the presentation video, and further combines it with an emotion engine that recognizes user emotions.

[1746] System Configuration

[1747] The system consists of the following main components:

[1748] 1. User Interface

[1749] An interface for users to upload digital files for presentations and enter narration prompts, implemented as a web-based application or a dedicated desktop application.

[1750] 2. Server Module

[1751] The main module that receives, analyzes and processes digital files. This module contains the following sub-modules:

[1752] File Analysis Module: Analyzes the contents of the uploaded file and extracts the text data of each slide.

[1753] Prompt analysis module: Analyzes the narration prompt entered by the user and generates a talk script.

[1754] Emotion Engine: Recognizes the user's emotions when entering narration prompts and analyzes them accordingly. For example, if the user expresses "joy" as their emotion, a talk script with positive expressions will be generated.

[1755] Synthetic speech generation module: Generates synthetic speech based on the talk script. Adjusts the tone and pace of the synthetic speech based on the user's emotions.

[1756] Subtitle generation module: Generates subtitles based on synthesized speech. Adjusts the expression of the subtitles based on the user's emotions.

[1757] Video Integration Module: Integrates slides, synthesized speech, subtitles, and avatars to generate presentation videos.

[1758] Operation explanation

[1759] 1. User request input

[1760] The user uploads digital files (e.g., PDF, PowerPoint) for the presentation through a user interface on the device, and also enters narration prompts through the user interface.

[1761] 2. Analyzing the file

[1762] The server receives the uploaded digital files and uses a file analysis module to determine the file format, which then converts the content of each slide into text data, which is then analyzed using natural language processing (NLP) techniques.

[1763] 3. Prompt analysis and emotion recognition

[1764] The server's prompt analysis module analyzes the narration prompts entered by the user. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice and reflects them in the prompt analysis. For example, if the user expresses "joy," a talk script with positive expressions is generated.

[1765] 4. Synthetic speech and subtitle generation

[1766] The server's synthetic speech generation module converts the talk script into synthetic speech based on the results of the emotion engine. The tone and pace are adjusted based on the user's emotions. The subtitle generation module then generates subtitles based on the narration content, reflecting the user's emotions.

[1767] 5. Avatar Integration

[1768] If a user wishes to use an avatar, the server adds the avatar to the system and sets the avatar's lip sync and facial animation to match the synthesized voice.

[1769] 6. Creating and providing presentation videos

[1770] The server uses a video integration module to integrate slides, synthesized speech, subtitles, and avatars to generate a presentation video, which is then provided to users for download or online viewing on their devices.

[1771] Specific examples

[1772] For example, consider a user creating a market analysis presentation. The user uploads the market analysis document in PDF format and enters a narration prompt such as, "In this slide, please emphasize the advantages of the new product." When the user enters the narration prompt, the emotion engine recognizes the user's "expectations" and generates a talk script based on them. The server analyzes the file and generates content that emphasizes important points based on the results of the emotion engine. Synthetic speech is then generated from the talk script, and subtitles also reflect the user's emotions. If an avatar is added, the avatar will also behave in a similar way, reflecting the emotion of expectation. Finally, a video presentation integrating all elements is generated and provided to the user.

[1773] Prompt Sentence Examples

[1774] 1. "Highlight the key points of your marketing strategy."

[1775] 2. "This slide should explain your efforts to improve customer satisfaction."

[1776] 3. "Please briefly summarize the technical features of your product."

[1777] In this way, the system of the present invention can take into account the user's emotions and provide more effective and empathetic presentation videos, improving the quality and emotional appeal of presentations while also improving the efficiency of presentations.

[1778] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1779] Step 1: User Request Input

[1780] ---

[1781] Users use terminals to access the system's user interface, upload digital files (e.g., PDF, PowerPoint) for presentations, and input narration prompts for the presentation content.

[1782] Input: Digital files for presentation, narration prompts

[1783] Output: Uploaded digital files, typed narration prompts

[1784] Specific behavior: The user interface provides buttons for uploading files and text boxes for entering narration prompts. Once the user enters the file and prompt, they are sent to the server.

[1785] Step 2: Receiving and analyzing the file

[1786] ---

[1787] The server receives the uploaded digital files and uses a file analysis module to convert the content of each slide into text data, identifying the file format and organizing page numbers and header information.

[1788] Input: Uploaded digital files

[1789] Output: Parsed text data

[1790] What it does: The server's file analysis module reads PDF or PowerPoint files, extracts the text within each slide, and uses natural language processing (NLP) techniques to identify important elements.

[1791] Step 3: Parsing the prompt and recognizing emotions

[1792] ---

[1793] The server's prompt analysis module analyzes the narration prompts entered by the user and generates a talk script. At this time, the emotion engine recognizes the user's emotions from their facial expressions and voice, and reflects those emotions in the prompt analysis.

[1794] Input: Narration prompts, user emotion information

[1795] Output: Talk script reflecting emotions

[1796] Specific operation: The emotion engine analyzes the user's facial expression and voice data acquired through the user interface, identifies the user's emotional state based on this data, and uses this information to generate a talk script appropriate for the narration prompt.

[1797] Step 4: Generate synthetic speech and subtitles

[1798] ---

[1799] The server uses a synthetic speech generation module to generate synthetic speech based on the talk script, adjusting the tone and pace of the speech based on the user's emotions. The subtitle generation module converts the content of the synthetic speech into text data and generates subtitles.

[1800] Input: Talk script, emotion information

[1801] Output: Synthesized speech, generated subtitles

[1802] Specific operation: The synthetic speech generation module receives the talk script as input data and dynamically adjusts the tone and pace of the speech based on the user's emotional information. The generated speech data is then converted into text and passed to the subtitle generation module, which finally creates subtitles for the presentation.

[1803] Step 5: Integrating the Avatar

[1804] ---

[1805] If a user wishes to use an avatar, the server adds the avatar to the system and sets the avatar's lip sync and facial animation to match the synthesized voice.

[1806] Input: Synthetic voice, user avatar preference

[1807] Output: Avatar with lip sync and animation added

[1808] Specific behavior: The server loads the avatar selected by the user from available avatar resources and sets the behavior based on the synthesized voice. Lip sync and facial animation are dynamically reflected based on emotional information, resulting in more natural behavior.

[1809] Step 6: Generate and serve presentation videos

[1810] ---

[1811] The server uses a video synthesis module to synthesize each slide, the generated synthetic speech, subtitles, and avatars into a single presentation video, which is then provided to users for download or online viewing.

[1812] Input: Slide data, synthesized speech, generated subtitles, avatar data

[1813] Output: Integrated presentation video

[1814] How it works: The video integration module combines the content of each slide, the generated audio, subtitles, and avatars, arranges them on a timeline, and generates a video file. The final presentation video is saved in a specified location for user access. For online viewing, a URL is provided to the user.

[1815] (Application example 2)

[1816] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1817] Conventional presentation video generation systems require users to input slide content individually, create narration, and generate synthesized speech, which is a complex process. Furthermore, because the system automatically generates the presentation without considering the user's emotions, it lacks appeal to viewers. Therefore, there is a need for a system that can efficiently generate presentation videos that respond to the user's emotions.

[1818] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving a digital file for presentation input by a user; means for analyzing the received digital file and converting the content of each slide into text data; means for analyzing narration prompts input by the user and generating a talk script for each slide; means for generating synthetic speech based on the generated talk script; means for generating subtitles based on the generated synthetic speech; means for recognizing a user's emotion and adjusting the tone and pace of the narration based on the recognized emotion; means for integrating each slide, the generated synthetic speech, the generated subtitles, and the recognized emotion information to generate a presentation video; and means for providing the generated presentation video to the user. This makes it possible to efficiently generate a highly appealing presentation video that is tailored to the user's emotion.

[1819] "Digital files for presentations" refer to slides and materials that users use for presentation purposes and are provided in digital formats such as PDF and PowerPoint.

[1820] "Narration prompt" refers to the content or wording of the narration entered by the user, including explanations and commentary for each slide.

[1821] A "talk script" is a spoken script generated by analyzing narration prompts, and is a narration instruction manual provided for each slide.

[1822] "Synthetic voice" is an artificial voice generated based on a talk script, and is reproduced to imitate human speech.

[1823] "Subtitles" are textual information that displays the contents of the synthesized speech, and serve to visually supplement the contents of the narration.

[1824] "Emotion information" is data of emotions recognized by analyzing the user's facial expressions, voice, etc., and indicates an emotional state such as joy, sadness, excitement, etc.

[1825] "Tone and pace adjustment" refers to changing the speaking style and speed of the synthesized voice based on emotional information, as a means of providing a more emotionally rich presentation.

[1826] "Video synthesis" is the process of combining slides, synthesized audio, and subtitles into a single video presentation that viewers can watch as a continuous story.

[1827] An "avatar" is a digital character that can be selected by the user to appear in the presentation video, and has lip-sync and facial animation.

[1828] This system analyzes a digital presentation file input by a user, generates a talk script based on narration prompts entered by the user, creates synthesized speech and subtitles, and provides a presentation video that integrates them. It is particularly notable for recognizing the user's emotions and adjusting the tone and pace of the narration based on those emotions.

[1829] Basic system configuration

[1830] 1. User Interface

[1831] Users use a device to upload digital files for presentations and enter narration prompts. The user interface may be implemented as a web-based application or a dedicated desktop application, typically a web application running in a browser.

[1832] 2. Server Module

[1833] The server includes the following sub-modules to analyze the received digital files and extract the content of each slide as text data:

[1834] File Analysis Module: Identifies the digital file format (PDF, PowerPoint, etc.) of the uploaded presentation and extracts the text data for each slide.

[1835] Prompt Analysis Module: Analyzes the narration prompts entered by the user and generates a talk script for each slide.

[1836] Emotion Recognition Engine: Recognizes emotions from the user's facial expressions and voice data. This is achieved by the user using the device's camera and microphone.

[1837] Synthetic voice generation module: Creates a synthetic voice based on the generated talk script, and adjusts the tone and pace according to the user's emotions.

[1838] Subtitle generation module: Generates subtitles in synchronization with the synthesized speech. The subtitles also include expressions based on the user's emotions.

[1839] Video Integration Module: Generates a presentation video by integrating each slide, synthesized speech, subtitles, and emotional information.

[1840] 3. Use of Avatars

[1841] A user-selected avatar can be integrated into the presentation video, lip-syncing to the generated synthetic speech and changing facial expressions based on emotions, enhancing the visual and emotional appeal.

[1842] Program processing explanation

[1843] The server uses the following hardware and software:

[1844] Hardware: Camera, microphone, user device (PC, smartphone, etc.)

[1845] Software: OpenCV, Natural Language Processing Library (NLP), Emotion Recognition Engine, Synthetic Speech Generation Library

[1846] Specific examples

[1847] For example, when creating a marketing presentation, a user uploads marketing materials in PDF format and enters a narration prompt such as, "This slide should highlight the benefits of the product." As the user enters the narration prompt, an emotion recognition engine recognizes the user's "joy." The server analyzes the file and generates a talk script that emphasizes a positive tone based on the emotion recognition results. Synthetic speech is then generated from this talk script, creating an expressive narration. Subtitles also reflect the user's emotions and use impactful expressions. If an avatar is added, the avatar will also behave in a similar way, expressing joy. Finally, a video presentation integrating all elements is generated and provided to the user.

[1848] Prompt Sentence Examples

[1849] "Analyze the text of marketing presentation slides, generate a narration that adjusts the tone and pace based on user emotion, add subtitles, and create a video of the presentation. The narration prompt is "Highlight the product benefits on this slide," and the user emotion is "Delight."

[1850] In this way, the present invention takes into account the user's emotions and can provide more effective and appealing presentation videos, improving the efficiency and quality of presentation creation and eliciting empathy from viewers.

[1851] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1852] Step 1:

[1853] User request input

[1854] Users use their terminals to upload digital files (e.g., PDF, PowerPoint) for presentations to the server and input narration prompts through a user interface.

[1855] Input: Digital files for presentation and narration prompts

[1856] Output: Received digital files and prompt data

[1857] Specific operation: The terminal displays a web form for uploading a file, from which the user selects a file and enters text for the narration prompt.

[1858] Step 2:

[1859] Analyzing files

[1860] The server's file analysis module analyzes the received digital files, extracting the content of each slide as text data and organizing page numbers and header information.

[1861] Input: Received digital presentation file

[1862] Output: Text data for each slide

[1863] Specific operation: The server determines the file format and extracts the text data using an appropriate library (e.g., PyMuPDF).

[1864] Step 3:

[1865] Parsing prompts

[1866] The server's prompt analysis module analyzes the narration prompts entered by the user and generates a talk script, while the emotion recognition engine recognizes the user's emotions and influences the content of the prompts.

[1867] Input: User-supplied narration prompts and user emotion data

[1868] Output: A talk script that reflects the user's emotions

[1869] Specific operation: The server uses an emotion recognition engine (e.g., OpenCV, dlib) to analyze the user's emotions and a natural language processing library (e.g., spaCy) to analyze the narration prompts.

[1870] Step 4:

[1871] Generate synthetic speech

[1872] The server's synthetic voice generation module generates synthetic voice based on the talk script and adjusts the tone and pace according to the user's emotions.

[1873] Input: A talk script that reflects the user's emotions

[1874] Output: Emotionally adjusted synthetic speech

[1875] Specific operation: The server generates an audio file using a synthetic speech generation library (e.g., Google Text-to-Speech API) and processes the audio (adjusts tone and pace) based on emotions.

[1876] Step 5:

[1877] Subtitle generation

[1878] A subtitle generation module of the server generates subtitles based on the synthesized speech, and includes expressions according to the user's emotions.

[1879] Input: Synthesized voice and talk script

[1880] Output: Emotionally sensitive subtitle data

[1881] Specific operation: The server uses a speech recognition library (e.g., Google Speech-to-Text API) to generate text from the speech and creates subtitle data in a style that corresponds to the user's emotions.

[1882] Step 6:

[1883] Presentation video generation

[1884] The video integration module of the server integrates each slide, synthesized speech, subtitles, and emotional information to generate a presentation video.

[1885] Input: Each slide, synthesized speech, subtitle data, emotional information

[1886] Output: Finished presentation video

[1887] Specific operation: The server uses a video editing library (e.g., FFmpeg) to integrate each element and generate a video file.

[1888] Step 7:

[1889] Providing generated presentation videos

[1890] The generated presentation video is provided to the user, who can download the video or watch it online using their device.

[1891] Input: Finished presentation video

[1892] Output: User-accessible video link of the presentation

[1893] Specific operation: The server stores the video file in storage and provides a download link or streaming URL to the user's device.

[1894] This processing flow enables users to efficiently generate emotionally-reflective presentation videos and deliver visually and emotionally appealing presentations.

[1895] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1896] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1897] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1898] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1899] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1900] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1901] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1902] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1903] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1904] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1905] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1906] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1907] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1908] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1909] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1910] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1911] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1912] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1913] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1914] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1915] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1916] The following is further disclosed regarding the above embodiment.

[1917] (Claim 1)

[1918] A program executed by a computer device,

[1919] means for receiving a digital presentation file input from a user;

[1920] means for analyzing the received digital files and converting the contents of each slide into text data;

[1921] A means for analyzing narration prompts input by a user and generating a talk script for each slide;

[1922] means for generating synthetic speech based on the generated talk script;

[1923] a means for generating subtitles based on the generated synthetic speech;

[1924] a means for integrating the slides, the generated synthetic voice, and the generated subtitles to generate a presentation video;

[1925] The system includes a means for providing the generated presentation video to a user.

[1926] (Claim 2)

[1927] 10. The system of claim 1, further comprising means for integrating a user-selected avatar into the presentation video.

[1928] (Claim 3)

[1929] 10. The system of claim 1, further comprising means for automatically adjusting intonation and timing of narration when generating said synthesized speech and subtitles.

[1930] "Example 1"

[1931] (Claim 1)

[1932] means for receiving a digital presentation file input from a user;

[1933] means for analyzing the received digital files and converting the contents of each slide into text data;

[1934] A means for analyzing narration prompts input by a user and generating a talk script for each slide;

[1935] means for generating synthetic speech based on the generated talk script;

[1936] a means for generating subtitles based on the generated synthetic speech;

[1937] a means for evaluating the quality of the generated synthesized speech and adjusting the tone and intonation as necessary; and

[1938] A means for setting the time code of the generated subtitles and adjusting the display timing;

[1939] means for integrating a user-selected avatar into the presentation video and synchronizing it with synthesized speech;

[1940] a means for integrating the slides, the generated synthetic voice, the generated subtitles, and the avatar to generate a presentation video;

[1941] The system includes a means for providing the generated presentation video to a user.

[1942] (Claim 2)

[1943] 10. The system of claim 1, further comprising means for integrating a user-selected avatar into the presentation video.

[1944] (Claim 3)

[1945] 10. The system of claim 1, further comprising means for automatically adjusting intonation and timing of narration when generating said synthesized speech and subtitles.

[1946] "Application Example 1"

[1947] (Claim 1)

[1948] means for receiving a digital presentation file input from a user;

[1949] means for analyzing the received digital files and converting the contents of each slide into text data;

[1950] A means for analyzing narration prompts input by a user and generating a talk script for each slide;

[1951] means for generating synthetic speech based on the generated talk script;

[1952] a means for generating subtitles based on the generated synthetic speech;

[1953] a means for integrating the slides, the generated synthetic voice, and the generated subtitles to generate a presentation video;

[1954] a means for providing the generated presentation video to a user;

[1955] means for generating a presentation video optimized as educational content based on the content selected by the user;

[1956] a means for providing an easy-to-use interface for educators and for automatically generating and adjusting content based on educator input;

[1957] A system including:

[1958] (Claim 2)

[1959] 10. The system of claim 1, further comprising means for integrating a user-selected avatar into the presentation video.

[1960] (Claim 3)

[1961] 10. The system of claim 1, further comprising means for automatically adjusting intonation and timing of narration when generating the synthesized speech and subtitles.

[1962] "Example 2: Combining Emotion Engines"

[1963] (Claim 1)

[1964] means for receiving a digital presentation file input from a user;

[1965] means for analyzing the received digital files and converting the contents of each slide into text data;

[1966] A means for analyzing narration prompts input by a user and generating a talk script for each slide;

[1967] means for recognizing a user's emotion and analyzing and generating narration prompts based on the emotion;

[1968] means for generating synthetic speech based on the generated talk script;

[1969] a means for generating subtitles based on the generated synthetic speech;

[1970] a means for integrating the slides, the generated synthetic voice, and the generated subtitles to generate a presentation video;

[1971] The system includes a means for providing the generated presentation video to a user.

[1972] (Claim 2)

[1973] 10. The system of claim 1, further comprising means for integrating a user-selected avatar into the presentation video.

[1974] (Claim 3)

[1975] 10. The system of claim 1, further comprising means for automatically adjusting intonation and timing of narration when generating said synthesized speech and subtitles.

[1976] "Application example 2 when combining emotion engines"

[1977] (Claim 1)

[1978] means for receiving a digital presentation file input from a user;

[1979] means for analyzing the received digital files and converting the contents of each slide into text data;

[1980] A means for analyzing narration prompts input by a user and generating a talk script for each slide;

[1981] means for generating synthetic speech based on the generated talk script;

[1982] a means for generating subtitles based on the generated synthetic speech;

[1983] means for recognizing a user's emotion and adjusting the tone and pace of the narration based on the recognized emotion;

[1984] a means for integrating the slides, the generated synthetic speech, the generated subtitles, and the recognized emotion information to generate a presentation video;

[1985] a means for providing the generated presentation video to a user;

[1986] A system including:

[1987] (Claim 2)

[1988] 10. The system of claim 1, further comprising means for integrating a user-selected avatar into the presentation video and adjusting lip-sync and facial animation of the avatar based on emotion.

[1989] (Claim 3)

[1990] 10. The system of claim 1, further comprising means for automatically adjusting intonation and timing of narration based on user emotion information when generating the synthetic speech and subtitles. [Explanation of symbols]

[1991] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A program executed by a computer device, means for receiving a digital presentation file input from a user; means for analyzing the received digital files and converting the contents of each slide into text data; A means for analyzing narration prompts input by a user and generating a talk script for each slide; means for generating synthetic speech based on the generated talk script; a means for generating subtitles based on the generated synthetic speech; a means for integrating the slides, the generated synthetic voice, and the generated subtitles to generate a presentation video; The system includes a means for providing the generated presentation video to a user.

2. The system of claim 1 , further comprising means for integrating a user-selected avatar into the presentation video.

3. The system of claim 1 , further comprising means for automatically adjusting intonation and timing of narration during the generation of the synthesized speech and subtitles.

Images